Note
Go to the end to download the full example code. or to run this example in your browser via JupyterLite or Binder
Schur Complementary Allocation#
This tutorial introduces the SchurComplementary
allocation.
Schur Complementary Allocation is a portfolio allocation method developed by Peter Cotton [1].
It uses Schur-complement-inspired augmentation of sub-covariance matrices, revealing a link between Hierarchical Risk Parity (HRP) and minimum-variance (MVO) portfolios.
By tuning the regularization factor gamma
, which governs how much off-diagonal
information is incorporated into the augmented covariance blocks, the method
smoothly interpolates from the heuristic divide-and-conquer allocation of HRP
(gamma = 0
) to the MVO solution (gamma -> 1
).
Note
A poorly conditioned covariance matrix can prevent convergence to the MVO solution
as gamma approaches one. Setting keep_monotonic=True
(the default) ensures that
the portfolio variance decreases monotonically with respect to gamma and remains
bounded by the variance of the HRP portfolio (variance(Schur) <= variance(HRP)
),
even in the presence of ill-conditioned covariance matrices. Additionally, you can
apply shrinkage or other conditioning techniques via the prior_estimator
parameter
to improve numerical stability and estimation accuracy.
Data Loading#
We load the S&P 500 dataset composed of the daily prices of 20 assets from the S&P 500 Index composition starting from 2020-01-02 up to 2022-12-28:
import numpy as np
import scipy.stats as stats
from plotly.io import show
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from skfolio import PerfMeasure, Population, RatioMeasure, RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.distance import KendallDistance, PearsonDistance
from skfolio.metrics import make_scorer
from skfolio.model_selection import MultipleRandomizedCV, WalkForward, cross_val_predict
from skfolio.moments import (
LedoitWolf,
)
from skfolio.optimization import (
HierarchicalRiskParity,
MeanRisk,
SchurComplementary,
)
from skfolio.preprocessing import prices_to_returns
from skfolio.prior import EmpiricalPrior
prices = load_sp500_dataset()
X = prices_to_returns(prices)
We select 10 assets from the 20 and split the data chronologically: 70% for training and 30% for testing.
# `shuffle=False` preserves chronological order, crucial for time-series data.
X_train, X_test = train_test_split(X.iloc[:, 10:], test_size=0.3, shuffle=False)
Schur Complementary Model#
We start by fitting a simple SchurComplementary
model
with gamma=0.5
:
model = SchurComplementary(gamma=0.5)
model.fit(X_train)
print(model.weights_)
[0.03687455 0.0613539 0.0659264 0.18201864 0.03743585 0.21311991
0.02811169 0.07392489 0.10507859 0.19615558]
Efficient Frontier Comparison#
Let’s take a closer look at how the Schur allocation behaves compared to other methods.
To do that, we’re going to fit a few different portfolio models on the training set:
Minimum-Variance (MVO)
Mean-Variance Efficient Frontier: 20 Markowitz portfolios spanning different risk levels
Hierarchical Risk Parity (HRP)
20 Schur portfolios with gamma values ranging from 0 to 1
We apply a Ledoit-Wolf shrinkage estimator to regularize the covariance matrix for every model.
Finally, we’ll evaluate all these portfolios on the test set to see how well they generalize.
prior = EmpiricalPrior(covariance_estimator=LedoitWolf())
population_train = Population([])
population_test = Population([])
# 20 Schur portfolios
for gamma in np.linspace(0.0, 1.0, 20):
schur = SchurComplementary(
gamma=gamma,
prior_estimator=prior,
portfolio_params={"name": f"Schur {gamma:0.2f}", "tag": "Schur"},
)
# Train
ptf = schur.fit_predict(X_train)
population_train.append(ptf)
# Test
ptf = schur.predict(X_test)
population_test.append(ptf)
# HRP portfolio
hrp = HierarchicalRiskParity(prior_estimator=prior, portfolio_params={"tag": "HRP"})
# Train
ptf = hrp.fit_predict(X_train)
population_train.append(ptf)
hrp_std = ptf.standard_deviation
# Test
ptf = hrp.predict(X_test)
population_test.append(ptf)
# 20 Markowitz (including MVO) portfolios
mean_variance = MeanRisk(
prior_estimator=prior,
efficient_frontier_size=20,
max_standard_deviation=hrp_std,
portfolio_params={"tag": "Markowitz"},
)
# Train
mv_population_train = mean_variance.fit_predict(X_train)
mv_population_train[0].tag = "MVO"
population_train += mv_population_train
# Test
mv_population_test = mean_variance.predict(X_test)
mv_population_test[0].tag = "MVO"
population_test += mv_population_test
Plot Mean-Variance Frontiers on training set
fig = population_train.plot_measures(
x=RiskMeasure.ANNUALIZED_STANDARD_DEVIATION,
y=PerfMeasure.ANNUALIZED_MEAN,
hover_measures=[RatioMeasure.ANNUALIZED_SHARPE_RATIO],
title="Training Set | Markowitz - HRP - Schur",
)
show(fig)
Plot Mean-Variance Frontiers on test set
population_test.plot_measures(
x=RiskMeasure.ANNUALIZED_STANDARD_DEVIATION,
y=PerfMeasure.ANNUALIZED_MEAN,
hover_measures=[RatioMeasure.ANNUALIZED_SHARPE_RATIO],
title="Test Set | Markowitz - HRP - Schur",
)
Plot portfolio compositions
population_train.filter(tags=["Schur", "MVO", "Markowitz"]).plot_composition()
Analysis#
When
gamma = 0
, the Schur portfolio is exactly equal to HRP.As
gamma
increases toward 1, it gradually approaches the MVO solution, without fully reaching it.On the training set, both Schur and HRP portfolios are Pareto dominated by the Markowitz portfolios, as expected, since those lie on the efficient frontier by construction.
On the test set, Schur portfolios dominate the Markowitz portfolios.
We observe a reversal in the relative frontiers (mean-variance dominance) between the training and test sets: Markowitz portfolios dominate in-sample, but their structure fails to hold out-of-sample versus Schur portfolios, which generalize more effectively.
Below, we’ll show how to model a more realistic train/test rebalancing strategy
and how to find the optimal gamma
parameter.
Rebalancing Strategy#
We use WalkForward
to define a quarterly rebalancing
(60 trading days), training on the prior three years (3*252 trading days):
walk_forward = WalkForward(test_size=60, train_size=252 * 3)
Note that WalkForward
also supports specific
datetime frequencies. For examples, we could use
walk_forward = WalkForward(test_size=3, train_size=36, freq="WOM-3FRI")
to
rebalance quarterly on the third Friday (WOM-3FRI), training on the prior 36
months.
Hyperparameter Tuning#
We’ll tune the Schur model’s gamma
and distance metric using
RandomizedSearchCV
, optimizing for out-of-sample mean-CDaR Ratio:
model = SchurComplementary(prior_estimator=prior)
random_search = RandomizedSearchCV(
estimator=model,
cv=walk_forward,
n_jobs=-1,
param_distributions={
"gamma": stats.uniform(0, 1),
"distance_estimator": [PearsonDistance(), KendallDistance()],
},
n_iter=10,
scoring=make_scorer(RatioMeasure.CDAR_RATIO),
random_state=0,
)
random_search.fit(X_train)
# Retrieve the best estimator from the search.
schur = random_search.best_estimator_
schur
In practice, it’s recommended to increase n_iter
to sample more parameter
combinations, then plot those samples to ensure adequate search-space coverage and
examine the convergence of training and test performance (see the
L1 and L2 Regularization tutorial).
Standard Walk-Forward Analysis#
We evaluate the MVO and tuned Schur models on the test set using standard walk-forward analysis:
mvo = MeanRisk(prior_estimator=prior)
pred_mvo = cross_val_predict(mvo, X_test, cv=walk_forward, n_jobs=-1)
pred_mvo.name = "MVO"
pred_schur = cross_val_predict(schur, X_test, cv=walk_forward, n_jobs=-1)
pred_schur.name = "Schur"
# Combine results for easier analysis.
population = Population([pred_schur, pred_mvo])
population.plot_cumulative_returns()
Let’s display a summary of key performance metrics:
summary = population.summary()
print(summary.loc[["Annualized Sharpe Ratio", "CDaR Ratio at 95%"]])
Schur MVO
Annualized Sharpe Ratio 0.99 0.72
CDaR Ratio at 95% 0.0052 0.0032
A single backtest path represents one possible trajectory of cumulative returns under the given rebalancing scheme and parameter set. While easy to compute, it may understate the variability and uncertainty of real-world performance compared to Monte Carlo-based methods.
Multiple Randomized Cross-Validation#
Using the MultipleRandomizedCV
methodology of
Palomar in [2], we perform Monte Carlo-style resampling by drawing 800 subsamples
of 10 distinct assets from the 20-asset universe and contiguous 5-year windows
(5 x 252 trading days). We then apply our walk-forward split to each subsample.
This approach captures both temporal and cross-sectional variability:
X_train, X_test = train_test_split(X, test_size=0.3, shuffle=False)
cv_mc = MultipleRandomizedCV(
walk_forward=walk_forward,
n_subsamples=800,
asset_subset_size=10,
window_size=5 * 252,
random_state=0,
)
# Generate cross-validated predictions for both models.
pred_mvo_mc = cross_val_predict(
mvo, X_test, cv=cv_mc, n_jobs=-1, portfolio_params={"tag": "MVO"}
)
pred_schur_mc = cross_val_predict(
schur, X_test, cv=cv_mc, n_jobs=-1, portfolio_params={"tag": "Schur"}
)
# Combine results for easier analysis.
population_mc = pred_mvo_mc + pred_schur_mc
Let’s plot the distribution of out-of-sample performance metrics (e.g., Sharpe ratio, CDaR ratio) across all Monte Carlo subsamples for both the Schur and MVO portfolios. This helps assess how robust each model is across different asset combinations and time periods:
population_mc.plot_distribution(
measure_list=[RatioMeasure.ANNUALIZED_SHARPE_RATIO], tag_list=["MVO", "Schur"]
)
population_mc.plot_distribution(
measure_list=[RatioMeasure.CDAR_RATIO], tag_list=["MVO", "Schur"]
)
for pred in [pred_mvo_mc, pred_schur_mc]:
tag = pred[0].tag
mean_sr = pred.measures_mean(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO)
std_sr = pred.measures_std(measure=RatioMeasure.ANNUALIZED_SHARPE_RATIO)
print(f"{tag}\n{'=' * len(tag)}")
print(f"Average Sharpe Ratio: {mean_sr:0.2f}")
print(f"Sharpe Ratio Std Dev: {std_sr:0.2f}\n")
MVO
===
Average Sharpe Ratio: 0.74
Sharpe Ratio Std Dev: 0.37
Schur
=====
Average Sharpe Ratio: 0.91
Sharpe Ratio Std Dev: 0.42
In this simple example, Schur portfolios tend to outperform MVO out-of-sample, exhibiting higher average Sharpe and CDaR ratios.
For a full tutorial on MultipleRandomizedCV
,
see L1 and L2 Regularization.
For additional cross-validation methods, such as
CombinatorialPurgedCV
from de Prado [3],
refer to the model selection section.
Conclusion#
This short example introduced the Schur Complementary Allocation method
and demonstrated how to use the skfolio
API to train, evaluate, tune, and compare
Schur portfolios with other allocation strategies.
References#
Total running time of the script: (4 minutes 4.733 seconds)