# Multiple Randomized Cross-Validation
This tutorial introduces [`MultipleRandomizedCV`](https://skfolio.org/generated/skfolio.model_selection.MultipleRandomizedCV.html.md#skfolio.model_selection.MultipleRandomizedCV),
which is based on the “Multiple Randomized Backtests” methodology of Palomar in [1](#id3).
This cross-validation strategy performs a resampling-based evaluation by repeatedly
sampling **distinct** asset subsets (without replacement) and **contiguous** time
windows, then applying an inner walk-forward split to each subsample, capturing both
temporal and cross-sectional variability in performance.
In this example, we build a portfolio model composed of a preselection of top
performers, followed by a Hierarchical Equal Risk Contribution optimization with
covariance shrinkage. We split the dataset into training and test sets, tune
hyperparameters on the training set, and then evaluate the final portfolio models on
the test set using [`MultipleRandomizedCV`](https://skfolio.org/generated/skfolio.model_selection.MultipleRandomizedCV.html.md#skfolio.model_selection.MultipleRandomizedCV).
## Data Loading
We load the FTSE 100 [dataset](https://skfolio.org/user_guide/datasets.html.md#datasets), which contains daily prices
of 64 assets from the FTSE 100 index, spanning 2000-01-04 to 2023-05-31.
```Python
import scipy.stats as stats
from plotly.io import show
from sklearn import set_config
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from skfolio import Population, RatioMeasure, RiskMeasure
from skfolio.datasets import load_ftse100_dataset
from skfolio.metrics import make_scorer
from skfolio.model_selection import MultipleRandomizedCV, WalkForward, cross_val_predict
from skfolio.moments import ShrunkCovariance
from skfolio.optimization import HierarchicalEqualRiskContribution
from skfolio.pre_selection import SelectKExtremes
from skfolio.preprocessing import prices_to_returns
from skfolio.prior import EmpiricalPrior
set_config(transform_output="pandas")
prices = load_ftse100_dataset()
returns = prices_to_returns(prices)
# Sequential train-test split: 67% training, 33% testing.
# `shuffle=False` preserves chronological order, crucial for time-series data.
X_train, X_test = train_test_split(returns, test_size=0.33, shuffle=False)
```
## Portfolio Construction
We build a pipeline that first selects the top-k assets by Sharpe ratio, then
allocates weights via Hierarchical Equal Risk Contribution using a shrunk
covariance estimator.
```Python
pre_selection = SelectKExtremes(k=10, measure=RatioMeasure.SHARPE_RATIO, highest=True)
optimization = HierarchicalEqualRiskContribution(
prior_estimator=EmpiricalPrior(
covariance_estimator=ShrunkCovariance(shrinkage=0.5)
),
risk_measure=RiskMeasure.VARIANCE,
)
model_bench = Pipeline(
[
("pre_selection", pre_selection),
("optimization", optimization),
]
)
```
## Rebalancing Strategy
We use [`WalkForward`](https://skfolio.org/generated/skfolio.model_selection.WalkForward.html.md#skfolio.model_selection.WalkForward) to define a monthly rebalancing
(20 trading days), training on the prior year (252 trading days):
```Python
walk_forward = WalkForward(test_size=20, train_size=252)
```
Note that [`WalkForward`](https://skfolio.org/generated/skfolio.model_selection.WalkForward.html.md#skfolio.model_selection.WalkForward) also supports specific
datetime frequencies. For examples, we could use
`walk_forward = WalkForward(test_size=1, train_size=12, freq="WOM-3FRI")` to
rebalance **monthly** on the **third Friday** (WOM-3FRI), training on the prior 12
months.
## Hyperparameter Tuning
Initially, the number of selected assets and the shrinkage parameter were
chosen randomly. We use `RandomizedSearchCV` to explore these parameters and
find the combination that maximizes the mean out-of-sample CVaR ratio.
```Python
random_search = RandomizedSearchCV(
estimator=model_bench,
cv=walk_forward,
n_jobs=-1,
param_distributions={
"pre_selection__k": stats.randint(10, 30),
"optimization__prior_estimator__covariance_estimator__shrinkage": stats.uniform(
0, 1
),
},
n_iter=30,
random_state=0,
scoring=make_scorer(RatioMeasure.CVAR_RATIO),
)
random_search.fit(X_train)
# Retrieve the best estimator from the search.
model_tuned = random_search.best_estimator_
model_tuned
```
[plotly figure stripped from llms output]
Display a summary of key performance metrics.
```Python
population.summary()
```
[plotly figure stripped from llms output]
We now compute and display the distribution of out-of-sample annualized Sharpe ratios:
```Python
population_mc.plot_distribution(
measure_list=[RatioMeasure.ANNUALIZED_SHARPE_RATIO],
tag_list=["Benchmark Model", "Tuned Model"],
)
```
[plotly figure stripped from llms output]
```Python
for pred in [pred_bench_mc, pred_tuned_mc]:
tag = pred[0].tag
mean_sr = pred.measures_mean(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO)
std_sr = pred.measures_std(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO)
print(f"{tag}\n{'=' * len(tag)}")
print(f"Average Sharpe Ratio: {mean_sr:0.2f}")
print(f"Sharpe Ratio Std Dev: {std_sr:0.2f}\n")
```
```none
Benchmark Model
===============
Average Sharpe Ratio: 0.36
Sharpe Ratio Std Dev: 0.38
Tuned Model
===========
Average Sharpe Ratio: 0.50
Sharpe Ratio Std Dev: 0.32
```
Let’s display the Box plot of the CVaR Ratio:
```Python
population_mc.boxplot_measure(
measure=RatioMeasure.CVAR_RATIO, tag_list=["Benchmark Model", "Tuned Model"]
)
```
[plotly figure stripped from llms output]
We plot the asset composition for the first two `MultiPeriodPortfolio`:
```Python
pred_tuned_mc[:2].plot_composition(display_sub_ptf_name=False)
```
[plotly figure stripped from llms output]
We plot the weights evolution over time for the first `MultiPeriodPortfolio`:
```Python
pred_tuned_mc[0].plot_weights_per_observation()
```
[plotly figure stripped from llms output]
## Conclusion
A single-path walk-forward analysis may understate the variability and uncertainty of
real-world performance. Multiple Randomized Cross-Validation, by contrast, applies
a resampling-based evaluation across asset subsets and time windows, yielding
performance estimates that are more robust and less prone to overfitting.
## References
* **[1]** “Portfolio Optimization: Theory and Application”, Chapter 8 Daniel P. Palomar (2025)
**Total running time of the script:** (5 minutes 50.772 seconds)