<a id="sphx-glr-auto-examples-pre-selection-plot-1-drop-correlated-py"></a>

<a id="drop-highly-correlated-assets"></a>

# Drop Highly Correlated Assets

This tutorial introduces the  [pre-selection transformers](https://skfolio.org/user_guide/pre_selection.html.md#pre-selection)
[`DropCorrelated`](https://skfolio.org/generated/skfolio.pre_selection.DropCorrelated.html.md#skfolio.pre_selection.DropCorrelated) to remove highly correlated assets before
the optimization.

Highly correlated assets tend to increase the instability of mean-variance optimization.

In this example, we will compare a mean-variance optimization with and without
pre-selection.

<a id="data"></a>

## Data

We load the FTSE 100 [dataset](https://skfolio.org/user_guide/datasets.html.md#datasets) composed of the daily prices of 64
assets from the FTSE 100 Index composition starting from 2000-01-04 up to 2023-05-31:

```Python
from plotly.io import show
from sklearn import set_config
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

from skfolio import Population, RatioMeasure
from skfolio.datasets import load_ftse100_dataset
from skfolio.model_selection import (
    CombinatorialPurgedCV,
    cross_val_predict,
    optimal_folds_number,
)
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.pre_selection import DropCorrelated, DropZeroVariance
from skfolio.preprocessing import prices_to_returns

prices = load_ftse100_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)
```

<a id="model"></a>

## Model

First, we create a maximum Sharpe Ratio model without pre-selection and fit it on the
training set:

```Python
model1 = MeanRisk(objective_function=ObjectiveFunction.MAXIMIZE_RATIO)
model1.fit(X_train)
model1.weights_
```

```none
array([5.72489249e-08, 6.92799374e-02, 2.99565070e-02, 5.15427369e-07,
       5.98248544e-08, 2.11954170e-07, 1.45472665e-07, 6.30531936e-08,
       1.38474971e-01, 5.39755883e-04, 1.19668650e-06, 2.04407315e-07,
       8.79495071e-03, 9.56173669e-08, 9.05345546e-08, 3.00727420e-07,
       8.30609346e-02, 3.48492428e-07, 1.18995205e-07, 1.41496349e-07,
       3.13544690e-02, 9.49328192e-08, 3.89233565e-02, 6.73529749e-08,
       1.08791263e-01, 1.03983577e-07, 1.81104294e-01, 1.71678032e-07,
       6.17026729e-08, 1.85876178e-07, 1.06482316e-07, 5.09757681e-08,
       1.97934916e-06, 4.57814908e-08, 8.41631508e-02, 6.52279086e-08,
       6.16740873e-03, 1.07868907e-07, 1.72699954e-07, 8.59571976e-08,
       1.12484994e-01, 1.77846765e-07, 7.68211300e-08, 8.46476491e-08,
       9.91391606e-08, 1.08820474e-07, 9.52131434e-08, 4.71020370e-07,
       9.27208522e-08, 1.40414870e-07, 3.82148954e-03, 8.28101461e-02,
       3.04499311e-03, 7.93434222e-08, 1.98631698e-07, 1.72168221e-02,
       1.14241628e-07, 1.24128912e-07, 2.22757658e-07, 3.33244837e-07,
       1.59411484e-07, 4.79955614e-07, 9.26676077e-08, 5.55225624e-07])
```

<a id="pipeline"></a>

## Pipeline

Then, we create a maximum Sharpe ratio model with pre-selection using `Pipeline` and
fit it on the training set:

```Python
set_config(transform_output="pandas")

model2 = Pipeline(
    [
        ("drop_zero_variance", DropZeroVariance(threshold=1e-6)),
        ("drop_correlated", DropCorrelated(threshold=0.5)),
        ("optimization", MeanRisk(objective_function=ObjectiveFunction.MAXIMIZE_RATIO)),
    ]
)
model2.fit(X_train)
model2.named_steps["optimization"].weights_
```

```none
array([8.18629046e-02, 2.99990921e-02, 1.16397541e-06, 3.33347825e-07,
       1.82548038e-01, 2.85482485e-03, 3.56265072e-06, 1.10513944e-02,
       2.32253197e-07, 2.05141268e-07, 7.34710302e-07, 8.21495217e-02,
       9.52097597e-07, 3.41747905e-07, 3.58028595e-02, 4.20420351e-02,
       1.56245104e-07, 2.35646825e-07, 1.85053358e-01, 3.80020780e-07,
       2.42398710e-07, 1.15032452e-03, 1.09539530e-07, 9.00503120e-02,
       2.41991003e-07, 3.83742727e-07, 1.23265556e-01, 3.91693342e-07,
       1.83251807e-07, 2.09337576e-07, 2.38594386e-07, 2.26123969e-07,
       1.11576161e-06, 2.18898633e-07, 8.33465412e-03, 8.57587244e-02,
       1.29547016e-02, 1.87531533e-07, 4.25937609e-07, 2.51052265e-02,
       2.73616225e-07, 3.03373275e-07, 5.56618079e-07, 3.50951178e-07,
       1.05787166e-06, 1.45723458e-06])
```

<a id="prediction"></a>

## Prediction

We predict both models on the test set:

```Python
ptf1 = model1.predict(X_test)
ptf1.name = "model1"
ptf2 = model2.predict(X_test)
ptf2.name = "model2"

print(ptf1.n_assets)
print(ptf2.n_assets)
```

```none
64
46
```

Each predicted object is a `MultiPeriodPortfolio`.
For improved analysis, we can add them to a `Population`:

```Python
population = Population([ptf1, ptf2])
```

Let’s plot the portfolios cumulative returns on the test set:

```Python
population.plot_cumulative_returns()
```

[plotly figure stripped from llms output]
<br />
<br />

<a id="combinatorial-purged-cross-validation"></a>

## Combinatorial Purged Cross-Validation

Only using one testing path (the historical path) may not be enough for comparing both
models. For a more robust analysis, we can use the
[`CombinatorialPurgedCV`](https://skfolio.org/generated/skfolio.model_selection.CombinatorialPurgedCV.html.md#skfolio.model_selection.CombinatorialPurgedCV) to create multiple testing
paths from different training folds combinations.

We choose `n_folds` and `n_test_folds` to obtain around 100 test paths and an average
training size of 800 days:

```Python
n_folds, n_test_folds = optimal_folds_number(
    n_observations=X_test.shape[0],
    target_n_test_paths=100,
    target_train_size=800,
)

cv = CombinatorialPurgedCV(n_folds=n_folds, n_test_folds=n_test_folds)
cv.summary(X_test)
```

```none
Number of Observations             1967
Total Number of Folds                10
Number of Test Folds                  6
Purge Size                            0
Embargo Size                          0
Average Training Size               786
Number of Test Paths                126
Number of Training Combinations     210
dtype: int64
```

```Python
pred_1 = cross_val_predict(
    model1,
    X_test,
    cv=cv,
    n_jobs=-1,
    portfolio_params=dict(annualization_factor=252, tag="model1"),
)

pred_2 = cross_val_predict(
    model2,
    X_test,
    cv=cv,
    n_jobs=-1,
    portfolio_params=dict(annualization_factor=252, tag="model2"),
)
```

The predicted object is a `Population` of `MultiPeriodPortfolio`. Each
`MultiPeriodPortfolio` represents one testing path of a rolling portfolio.
For improved analysis, we can merge the populations of each model:

```Python
population = pred_1 + pred_2
```

<a id="distribution"></a>

## Distribution

We plot the out-of-sample distribution of Sharpe ratio for both models:

```Python
fig = population.plot_distribution(
    measure_list=[RatioMeasure.SHARPE_RATIO], tag_list=["model1", "model2"], n_bins=40
)
show(fig)
```

<!doctype html>
[plotly figure stripped from llms output]

<br/>

Model 1:

```Python
print(
    "Average of Sharpe Ratio:"
    f" {pred_1.measures_mean(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO):0.2f}"
)
print(
    "Std of Sharpe Ratio:"
    f" {pred_1.measures_std(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO):0.2f}"
)
```

```none
Average of Sharpe Ratio: 0.46
Std of Sharpe Ratio: 0.20
```

Model 2:

```Python
print(
    "Average of Sharpe Ratio:"
    f" {pred_2.measures_mean(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO):0.2f}"
)
print(
    "Std of Sharpe Ratio:"
    f" {pred_2.measures_std(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO):0.2f}"
)
```

```none
Average of Sharpe Ratio: 0.51
Std of Sharpe Ratio: 0.21
```

**Total running time of the script:** (0 minutes 9.521 seconds)

<a id="sphx-glr-download-auto-examples-pre-selection-plot-1-drop-correlated-py"></a>