<a id="optimization"></a>

<a id="id1"></a>

# Optimization

The optimization module implements a set of methods intended for portfolio optimization.
They follow the same API as scikit-learn’s `estimator`: the `fit` method takes `X` as
the assets returns and stores the portfolio weights in its `weights_` attribute.

`X` can be any array-like structure (numpy array, pandas DataFrame, etc.)

All optimization inputs (expected returns, covariance, return scenarios) are expressed
in the periodicity of `X`: with daily returns, the optimizer works with daily moments
and scenarios rather than annualized ones. Parameters that share the unit of expected
returns, such as `transaction_costs` and `management_fees`, must be expressed in the
same periodicity. See [Periodicity Convention](https://skfolio.org/user_guide/data_preparation.html.md#periodicity-convention) for the
rationale and the cost conversion rules.

<a id="naive-allocation"></a>

## Naive Allocation

The naive module implements a set of naive allocations commonly used as benchmarks for
comparing different models:

> * [`EqualWeighted`](https://skfolio.org/generated/skfolio.optimization.EqualWeighted.html.md#skfolio.optimization.EqualWeighted)
> * [`InverseVolatility`](https://skfolio.org/generated/skfolio.optimization.InverseVolatility.html.md#skfolio.optimization.InverseVolatility)
> * [`Random`](https://skfolio.org/generated/skfolio.optimization.Random.html.md#skfolio.optimization.Random)

**Example:**

Naive inverse-volatility allocation:

```python
from sklearn.model_selection import train_test_split

from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import InverseVolatility
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = InverseVolatility()
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
```

<a id="mean-risk-optimization"></a>

## Mean-Risk Optimization

The [`MeanRisk`](https://skfolio.org/generated/skfolio.optimization.MeanRisk.html.md#skfolio.optimization.MeanRisk) estimator can solve the below 4 objective functions:

> * Minimize Risk:

> $$
> \begin{cases}
> \begin{aligned}
> &\min_{w} & & risk_{i}(w) \\
> &\text{s.t.} & & w^T\mu \ge min\_return \\
> & & & A w \ge b \\
> & & & risk_{j}(w) \le max\_risk_{j} \quad \forall \; j \ne i
> \end{aligned}
> \end{cases}

> $$

> * Maximize Expected Return:

> $$
> \begin{cases}
> \begin{aligned}
> &\max_{w} & & w^T\mu \\
> &\text{s.t.} & & risk_{i}(w) \le max\_risk_{i} \\
> & & & A w \ge b \\
> & & & risk_{j}(w) \le max\_risk_{j} \quad \forall \; j \ne i
> \end{aligned}
> \end{cases}

> $$

> * Maximize Utility:

> $$
> \begin{cases}
> \begin{aligned}
> &\max_{w} & & w^T\mu - \lambda \times risk_{i}(w)\\
> &\text{s.t.} & & risk_{i}(w) \le max\_risk_{i} \\
> & & & w^T\mu \ge min\_return \\
> & & & A w \ge b \\
> & & & risk_{j}(w) \le max\_risk_{j} \quad \forall \; j \ne i
> \end{aligned}
> \end{cases}

> $$

> * Maximize Ratio:

> $$
> \begin{cases}
> \begin{aligned}
> &\max_{w} & & \frac{w^T\mu - r_{f}}{risk_{i}(w)}\\
> &\text{s.t.} & & risk_{i}(w) \le max\_risk_{i} \\
> & & & w^T\mu \ge min\_return \\
> & & & A w \ge b \\
> & & & risk_{j}(w) \le max\_risk_{j} \quad \forall \; j \ne i
> \end{aligned}
> \end{cases}

> $$

With $risk_{i}$ a risk measure among:

> * Variance
> * Semi-Variance
> * Standard-Deviation
> * Semi-Deviation
> * Mean Absolute Deviation
> * First Lower Partial Moment
> * CVaR (Conditional Value at Risk)
> * EVaR (Entropic Value at Risk)
> * Worst Realization (worst return)
> * CDaR (Conditional Drawdown at Risk)
> * Maximum Drawdown
> * Average Drawdown
> * EDaR (Entropic Drawdown at Risk)
> * Ulcer Index
> * Gini Mean Difference

It supports the following parameters:

> * Weight Constraints
> * Budget Constraints
> * Group Constraints
> * Transaction Costs
> * Management Fees
> * L1 and L2 Regularization
> * Turnover Constraint
> * Tracking Error Constraint
> * Uncertainty Set on Expected Returns
> * Uncertainty Set on Covariance
> * Expected Return Constraints
> * Risk Measure Constraints
> * Custom Objective
> * Custom Constraints
> * Prior Estimator

**Example:**

Maximum Sharpe Ratio portfolio:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = MeanRisk(
    objective_function=ObjectiveFunction.MAXIMIZE_RATIO,
    risk_measure=RiskMeasure.VARIANCE,
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.sharpe_ratio)
```

<a id="prior-estimator"></a>

### Prior Estimator

Every portfolio optimization has a parameter named `prior_estimator`.
The [prior estimator](https://skfolio.org/user_guide/prior.html.md#prior) fits a [`ReturnDistribution`](https://skfolio.org/generated/skfolio.prior.ReturnDistribution.html.md#skfolio.prior.ReturnDistribution) containing
estimates of expected asset returns, covariance matrix, returns and Cholesky
decomposition of the covariance. It represents the investor’s prior beliefs about the
model used to estimate such distribution.

When the prior follows the native NaN-aware convention, compatible optimizers solve the
optimization problem on the investable subset and expand `weights_` back to the full
input universe. See [Missing Data and Changing Universes](https://skfolio.org/user_guide/data_representation.html.md#missing-data) for
details.

The available prior estimators are:

> * [`EmpiricalPrior`](https://skfolio.org/generated/skfolio.prior.EmpiricalPrior.html.md#skfolio.prior.EmpiricalPrior)
> * [`BlackLitterman`](https://skfolio.org/generated/skfolio.prior.BlackLitterman.html.md#skfolio.prior.BlackLitterman)
> * [`TimeSeriesFactorModel`](https://skfolio.org/generated/skfolio.prior.TimeSeriesFactorModel.html.md#skfolio.prior.TimeSeriesFactorModel)

**Example:**

Minimum Variance portfolio using a Factor Model:

```python
from sklearn.model_selection import train_test_split

from skfolio.datasets import load_factors_dataset, load_sp500_dataset
from skfolio.optimization import MeanRisk
from skfolio.preprocessing import prices_to_returns
from skfolio.prior import TimeSeriesFactorModel

prices = load_sp500_dataset()
factor_prices = load_factors_dataset()

X, factors = prices_to_returns(prices, factor_prices)
X_train, X_test, factors_train, factors_test = train_test_split(X, factors, test_size=0.33, shuffle=False)

model = MeanRisk(prior_estimator=TimeSeriesFactorModel())
model.fit(X_train, factors=factors_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
```

<a id="combining-prior-estimators"></a>

### Combining Prior Estimators

Prior estimators can be combined together, making it possible to design complex models:

**Example:**

This example is **purposely complex** to demonstrate how multiple estimators can be
combined.

The model below is a Maximum Sharpe Ratio optimization using a Factor Model for the
estimation of the **assets** expected returns and covariance matrix. A Black & Litterman
model is used for the estimation of the **factors** expected returns and covariance matrix,
incorporating the analysts’ views on the factors. Finally, the Black & Litterman prior
expected returns are estimated using an equal-weighted market equilibrium with a risk
aversion of 2 and a denoised prior covariance matrix:

```python
from sklearn.model_selection import train_test_split

from skfolio.datasets import load_factors_dataset, load_sp500_dataset
from skfolio.moments import DenoiseCovariance, EquilibriumMu
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.preprocessing import prices_to_returns
from skfolio.prior import BlackLitterman, EmpiricalPrior, TimeSeriesFactorModel

prices = load_sp500_dataset()
factor_prices = load_factors_dataset()

X, factors = prices_to_returns(prices, factor_prices)
X_train, X_test, factors_train, factors_test = train_test_split(X, factors, test_size=0.33, shuffle=False)

factor_views = ["MTUM - QUAL == 0.0003 ",
                "SIZE - USMV == 0.0004",
                "VLUE == 0.0006"]

model = MeanRisk(
    objective_function=ObjectiveFunction.MAXIMIZE_RATIO,
    prior_estimator=TimeSeriesFactorModel(
        factor_prior_estimator=BlackLitterman(
            prior_estimator=EmpiricalPrior(
                mu_estimator=EquilibriumMu(risk_aversion=2),
                covariance_estimator=DenoiseCovariance()
            ),
            views=factor_views)
    )
)

model.fit(X_train, factors=factors_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
```

<a id="custom-estimator"></a>

### Custom Estimator

It is very common to use a custom implementation for the moments estimators. For
example, you may want to use an in-house estimation for the covariance or a predictive
model for the expected returns.

Below is a simple example of how you would implement a custom covariance estimator.
For more complex cases and estimators, check the [API Reference](https://skfolio.org/api.html.md#api).

```python
import numpy as np

from skfolio.datasets import load_sp500_dataset
from skfolio.moments import BaseCovariance
from skfolio.optimization import MeanRisk
from skfolio.preprocessing import prices_to_returns
from skfolio.prior import EmpiricalPrior

prices = load_sp500_dataset()
X = prices_to_returns(prices)

class MyCustomCovariance(BaseCovariance):
    def __init__(self, my_param=0):
        super().__init__()
        self.my_param = my_param

    def fit(self, X, y=None):
        X = self._validate_data(X)
        # Your custom implementation goes here
        covariance = np.cov(X.T, ddof=self.my_param)
        self._set_covariance(covariance)
        return self

model = MeanRisk(
    prior_estimator=EmpiricalPrior(covariance_estimator=MyCustomCovariance(my_param=1)),
)
model.fit(X)
```

<a id="worst-case-optimization"></a>

### Worst-Case Optimization

With the `mu_uncertainty_set_estimator` parameter, the expected returns of the assets
are modeled with a [norm-ball uncertainty set](https://skfolio.org/user_guide/uncertainty_set.html.md#uncertainty-set-estimator). This
approach is known as worst-case optimization and falls under the class of robust
optimization. It mitigates the instability that arises from estimation errors of the
expected returns.

**Example:**

Worst-case maximum Mean/CDaR ratio (Conditional Drawdown at Risk) with an ellipsoidal
uncertainty set for the expected returns of the assets:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.preprocessing import prices_to_returns
from skfolio.uncertainty_set import BootstrapMuUncertaintySet

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = MeanRisk(
    objective_function=ObjectiveFunction.MAXIMIZE_RATIO,
    risk_measure=RiskMeasure.CDAR,
    mu_uncertainty_set_estimator=BootstrapMuUncertaintySet(confidence_level=0.9),
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
print(portfolio.cdar_ratio)
```

Covariance uncertainty is configured with `covariance_uncertainty_set_estimator`.
It is applied to the variance risk measure or a `max_variance` constraint. Generic
estimators use a lifted semidefinite formulation, while
[`OrthogonalCovarianceUncertaintySet`](https://skfolio.org/generated/skfolio.uncertainty_set.OrthogonalCovarianceUncertaintySet.html.md#skfolio.uncertainty_set.OrthogonalCovarianceUncertaintySet) uses a compact
representation in the factor model’s orthogonal space.

<a id="going-further"></a>

### Going Further

You can explore the remaining parameters (constraints, L1 and L2 regularization, costs,
turnover, tracking error, etc.) with the
[Mean-Risk examples](https://skfolio.org/auto_examples/mean_risk/index.html.md#mean-risk-examples) and the [`MeanRisk`](https://skfolio.org/generated/skfolio.optimization.MeanRisk.html.md#skfolio.optimization.MeanRisk) API.

<a id="risk-budgeting"></a>

## Risk Budgeting

The [`RiskBudgeting`](https://skfolio.org/generated/skfolio.optimization.RiskBudgeting.html.md#skfolio.optimization.RiskBudgeting) solves the below convex problem:

> $$
> \begin{cases}
> \begin{aligned}
> &\min_{w} & & risk_{i}(w) \\
> &\text{s.t.} & & b^T log(w) \ge c \\
> & & & w^T\mu \ge min\_return \\
> & & & A w \ge b \\
> & & & w \ge0
> \end{aligned}
> \end{cases}

> $$

with $b$ the risk budget vector and $c$ an auxiliary variable of the log
barrier.

And $risk_{i}$ a risk measure among:

> * Variance
> * Semi-Variance
> * Standard-Deviation
> * Semi-Deviation
> * Mean Absolute Deviation
> * First Lower Partial Moment
> * CVaR (Conditional Value at Risk)
> * EVaR (Entropic Value at Risk)
> * Worst Realization (worst return)
> * CDaR (Conditional Drawdown at Risk)
> * Maximum Drawdown
> * Average Drawdown
> * EDaR (Entropic Drawdown at Risk)
> * Ulcer Index
> * Gini Mean Difference
> * First Lower Partial Moment

It supports the following parameters:

> * Weight Constraints
> * Budget Constraints
> * Group Constrains
> * Transaction Costs
> * Management Fees
> * Expected Return Constraints
> * Custom Objective
> * Custom constraints
> * Prior Estimator

Limitations are imposed on certain constraints, such as long-only weights, to ensure the
problem remains convex.

**Example:**

CVaR (Conditional Value at Risk) Risk Parity portfolio:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import RiskBudgeting
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = RiskBudgeting(risk_measure=RiskMeasure.CVAR)
model.fit(X_train)
print(model.weights_)

portfolio_train = model.predict(X_train)
print(portfolio_train.annualized_sharpe_ratio)
print(portfolio_train.contribution(measure=RiskMeasure.CVAR))

portfolio_test = model.predict(X_test)
print(portfolio_test.annualized_sharpe_ratio)
print(portfolio_test.contribution(measure=RiskMeasure.CVAR))
```

<a id="maximum-diversification"></a>

## Maximum Diversification

The [`MaximumDiversification`](https://skfolio.org/generated/skfolio.optimization.MaximumDiversification.html.md#skfolio.optimization.MaximumDiversification) maximizes the diversification ratio, which is the
ratio of the weighted volatilities over the total volatility.

**Example:**

```python
from sklearn.model_selection import train_test_split

from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import MaximumDiversification
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = MaximumDiversification()
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.diversification)
```

<a id="distributionally-robust-cvar"></a>

## Distributionally Robust CVaR

The [`DistributionallyRobustCVaR`](https://skfolio.org/generated/skfolio.optimization.DistributionallyRobustCVaR.html.md#skfolio.optimization.DistributionallyRobustCVaR) constructs a Wasserstein ball in the space of
multivariate and non-discrete probability distributions centered at the uniform
distribution on the training samples and finds the allocation that minimizes the CVaR
of the worst-case distribution within this Wasserstein ball.
Esfahani and Kuhn proved that for piecewise linear objective functions,
which is the case of CVaR, the distributionally robust optimization problem
over a Wasserstein ball can be reformulated as finite convex programs.

A solver like `Mosek` that can handle a high number of constraints is preferred.

**Example:**

```python
from sklearn.model_selection import train_test_split

from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import DistributionallyRobustCVaR
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X = X["2020":]
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = DistributionallyRobustCVaR(wasserstein_ball_radius=0.01)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.cvar)
```

<a id="hierarchical-risk-parity"></a>

## Hierarchical Risk Parity

The [`HierarchicalRiskParity`](https://skfolio.org/generated/skfolio.optimization.HierarchicalRiskParity.html.md#skfolio.optimization.HierarchicalRiskParity) (HRP) is a portfolio optimization method developed
by Marcos Lopez de Prado.

This algorithm uses a distance matrix to compute hierarchical clusters using the
Hierarchical Tree Clustering algorithm then employs seriation to rearrange the assets
in the dendrogram, minimizing the distance between leaves.
in the dendrogram, minimizing the distance between leaves.

The final step is the recursive bisection where each cluster is split between two
sub-clusters by starting with the topmost cluster and traversing in a top-down
manner. For each sub-cluster, we compute the total cluster risk of an inverse-risk
allocation. A weighting factor is then computed from these two sub-cluster risks,
which is used to update the cluster weight.

#### NOTE
The original paper uses the variance as the risk measure and the single-linkage
method for the Hierarchical Tree Clustering algorithm. Here we generalize it to
multiple risk measures and linkage methods.
The default linkage method is set to the Ward
variance minimization algorithm, which is more stable and has better properties
than the single-linkage method.

It supports all [prior estimators](https://skfolio.org/user_guide/prior.html.md#prior) and [risk measures](https://skfolio.org/api.html.md#measures-ref)
as well as weight constraints.

It also supports all [distance estimators](https://skfolio.org/user_guide/distance.html.md#distance) through the
`distance_estimator` parameter. It fits a distance model for the
estimation of the codependence and the distance matrix used to compute the linkage
matrix:

> * [`PearsonDistance`](https://skfolio.org/generated/skfolio.distance.PearsonDistance.html.md#skfolio.distance.PearsonDistance)
> * [`KendallDistance`](https://skfolio.org/generated/skfolio.distance.KendallDistance.html.md#skfolio.distance.KendallDistance)
> * [`SpearmanDistance`](https://skfolio.org/generated/skfolio.distance.SpearmanDistance.html.md#skfolio.distance.SpearmanDistance)
> * [`CovarianceDistance`](https://skfolio.org/generated/skfolio.distance.CovarianceDistance.html.md#skfolio.distance.CovarianceDistance)
> * [`DistanceCorrelation`](https://skfolio.org/generated/skfolio.distance.DistanceCorrelation.html.md#skfolio.distance.DistanceCorrelation)
> * [`MutualInformation`](https://skfolio.org/generated/skfolio.distance.MutualInformation.html.md#skfolio.distance.MutualInformation)

**Example:**

Hierarchical Risk Parity with semi (downside) standard-deviation as the risk measure and
mutual information as the distance estimator:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.distance import MutualInformation
from skfolio.optimization import HierarchicalRiskParity
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = HierarchicalRiskParity(
    risk_measure=RiskMeasure.SEMI_DEVIATION, distance_estimator=MutualInformation()
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
print(portfolio.contribution(measure=RiskMeasure.SEMI_DEVIATION))
```

<a id="hierarchical-equal-risk-contribution"></a>

## Hierarchical Equal Risk Contribution

The [`HierarchicalEqualRiskContribution`](https://skfolio.org/generated/skfolio.optimization.HierarchicalEqualRiskContribution.html.md#skfolio.optimization.HierarchicalEqualRiskContribution) (HERC) is a portfolio optimization method
developed by Thomas Raffinot.

This algorithm uses a distance matrix to compute hierarchical clusters using the
Hierarchical Tree Clustering algorithm. It then computes, for each cluster, the total
cluster risk of an inverse-risk allocation.

The final step is the top-down recursive division of the dendrogram, where the assets
weights are updated using a naive risk parity within clusters.

It differs from the Hierarchical Risk Parity by exploiting the dendrogram shape
during the top-down recursive division instead of bisecting it.

#### NOTE
The default linkage method is set to the Ward
variance minimization algorithm, which is more stable and has better properties
than the single-linkage method.

It supports all [prior estimators](https://skfolio.org/user_guide/prior.html.md#prior) and [risk measures](https://skfolio.org/api.html.md#measures-ref)
as well as weight constraints.

It also supports all [distance estimators](https://skfolio.org/user_guide/distance.html.md#distance) through the
`distance_estimator` parameter. It fits a distance model for the
estimation of the codependence and the distance matrix used to compute the linkage
matrix:

> * [`PearsonDistance`](https://skfolio.org/generated/skfolio.distance.PearsonDistance.html.md#skfolio.distance.PearsonDistance)
> * [`KendallDistance`](https://skfolio.org/generated/skfolio.distance.KendallDistance.html.md#skfolio.distance.KendallDistance)
> * [`SpearmanDistance`](https://skfolio.org/generated/skfolio.distance.SpearmanDistance.html.md#skfolio.distance.SpearmanDistance)
> * [`CovarianceDistance`](https://skfolio.org/generated/skfolio.distance.CovarianceDistance.html.md#skfolio.distance.CovarianceDistance)
> * [`DistanceCorrelation`](https://skfolio.org/generated/skfolio.distance.DistanceCorrelation.html.md#skfolio.distance.DistanceCorrelation)
> * [`MutualInformation`](https://skfolio.org/generated/skfolio.distance.MutualInformation.html.md#skfolio.distance.MutualInformation)

**Example:**

Hierarchical Equal Risk Contribution with CVaR (Conditional Value at Risk) as the risk
measure and mutual information as the distance estimator:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.distance import MutualInformation
from skfolio.optimization import HierarchicalEqualRiskContribution
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = HierarchicalEqualRiskContribution(
    risk_measure=RiskMeasure.CVAR,
    distance_estimator = MutualInformation()
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
print(portfolio.contribution(measure=RiskMeasure.CVAR))
```

<a id="nested-clusters-optimization"></a>

## Nested Clusters Optimization

The [`NestedClustersOptimization`](https://skfolio.org/generated/skfolio.optimization.NestedClustersOptimization.html.md#skfolio.optimization.NestedClustersOptimization) (NCO) is a portfolio optimization method
developed by Marcos Lopez de Prado.

It uses a distance matrix to compute clusters using a clustering algorithm (
Hierarchical Tree Clustering, KMeans, etc.). For each cluster, the inner-cluster
weights are computed by fitting the inner-estimator on each cluster using the whole
training data. Then the outer-cluster weights are computed by training the
outer-estimator using out-of-sample estimates of the inner-estimators with
cross-validation. Finally, the final assets weights are the dot-product of the
inner-weights and outer-weights.

#### NOTE
The original paper uses KMeans as the clustering algorithm, minimum Variance for
the inner-estimator and equal-weighted for the outer-estimator. Here we generalize
it to all `sklearn` and `skfolio` clustering algorithms (Hierarchical Tree
Clustering, KMeans, etc.), all portfolio optimizations (Mean-Variance, HRP, etc.)
and risk measures (variance, CVaR, etc.).
To avoid data leakage at the outer-estimator, we use out-of-sample estimates to
fit the outer estimator.

It supports all [distance estimators](https://skfolio.org/user_guide/distance.html.md#distance)
and [clustering estimator](https://skfolio.org/user_guide/cluster.html.md#cluster) (both `skfolio` and `sklearn`)

**Example:**

Nested Clusters Optimization with KMeans as the clustering algorithm, Kendall Distance
as the distance estimator, Minimum Semi-Variance as the inner estimator, and CVaR Risk
Parity as the outer (meta) estimator trained on the out-of-sample estimates from the
KFold cross-validation and run with parallelization:

```python
from sklearn.cluster import KMeans
from sklearn.model_selection import KFold, train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.distance import KendallDistance
from skfolio.optimization import MeanRisk, NestedClustersOptimization, RiskBudgeting
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

model = NestedClustersOptimization(
    inner_estimator=MeanRisk(risk_measure=RiskMeasure.SEMI_VARIANCE),
    outer_estimator=RiskBudgeting(risk_measure=RiskMeasure.CVAR),
    distance_estimator=KendallDistance(),
    clustering_estimator=KMeans(n_init="auto"),
    cv=KFold(),
    n_jobs=-1,
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
print(portfolio.contribution(measure=RiskMeasure.CVAR))
```

The `cv` parameter can also be a combinatorial cross-validation, such as
`CombinatorialPurgedCV`, in which case each cluster’s
out-of-sample outputs are a collection of multiple paths instead of one single path.
The selected out-of-sample path among this collection of paths is chosen according to
the `quantile` and `quantile_measure` parameters.

<a id="stacking-optimization"></a>

## Stacking Optimization

[`StackingOptimization`](https://skfolio.org/generated/skfolio.optimization.StackingOptimization.html.md#skfolio.optimization.StackingOptimization) is an ensemble method that consists of stacking the outputs
of individual portfolio optimizations with a final portfolio optimization.

The final weights are the dot product of the individual optimizations’ weights and the final
optimization’s weights.

Stacking leverages the strengths of each individual portfolio optimization by
using their outputs as inputs to a final portfolio optimization.

To avoid data leakage, out-of-sample estimates are used to fit the outer
optimization.

**Example:**

Stacking Optimization with Minimum Semi-Variance and CVaR Risk Parity
stacked together using Minimum Variance as the final (meta) estimator.

```python
from sklearn.model_selection import KFold, train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import MeanRisk, RiskBudgeting, StackingOptimization
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

estimators = [
    ('model1', MeanRisk(risk_measure=RiskMeasure.SEMI_VARIANCE)),
    ('model2', RiskBudgeting(risk_measure=RiskMeasure.CVAR))
]

model = StackingOptimization(
    estimators=estimators,
    final_estimator=MeanRisk(),
    cv=KFold(),
    n_jobs=-1
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
```

The `cv` parameter can also be a combinatorial cross-validation, such as
`CombinatorialPurgedCV`, in which case each out-of-sample outputs are a
collection of multiple paths instead of one single path. The selected out-of-sample path
among this collection of paths is chosen according to the `quantile` and
`quantile_measure` parameters.

<a id="tracking-error-optimization"></a>

<a id="id2"></a>

## Tracking Error Optimization

Tracking error measures the deviation between a portfolio’s performance and a benchmark.
`skfolio` provides three approaches for tracking error optimization:

1. **Return-based tracking error constraint** (via `max_tracking_error`):
   Constrains the tracking error while optimizing another objective (e.g., minimize CVaR).
2. **Weight-based target** (via `target_weights`):
   Minimizes tracking error by finding weights that minimize deviation from a target
   portfolio allocation.
3. **Return-based target** (via [`BenchmarkTracker`](https://skfolio.org/generated/skfolio.optimization.BenchmarkTracker.html.md#skfolio.optimization.BenchmarkTracker)):
   Minimizes tracking error by optimizing on excess returns (portfolio returns
   minus benchmark returns).

**Example 1: Return-based tracking error constraint**

Minimize CVaR while constraining the tracking error to 0.30% vs a benchmark:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset, load_sp500_index
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()
spx_prices = load_sp500_index()

X, y = prices_to_returns(prices, spx_prices)
X_train, X_test, factors_train, factors_test = train_test_split(X, factors, test_size=0.33, shuffle=False)

model = MeanRisk(
    objective_function=ObjectiveFunction.MINIMIZE_RISK,
    risk_measure=RiskMeasure.CVAR,
    max_tracking_error=0.003,  # 0.30% tracking error constraint
)
model.fit(X_train, factors=factors_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.cvar)
```

**Example 2: Weight-based target**

Minimize tracking error vs an equal-weighted target portfolio:

```python
from sklearn.model_selection import train_test_split

import numpy as np
from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()

X = prices_to_returns(prices)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

# Define target portfolio (e.g., equal-weighted)
n_assets = X.shape[1]
target_weights = np.ones(n_assets) / n_assets

model = MeanRisk(
    objective_function=ObjectiveFunction.MINIMIZE_RISK,
    risk_measure=RiskMeasure.STANDARD_DEVIATION,
    target_weights=target_weights,
)
model.fit(X_train)
print(model.weights_)

portfolio = model.predict(X_test)
print(portfolio.annualized_sharpe_ratio)
```

**Example 3: Return-based target**

Minimize tracking error vs a benchmark’s returns:

```python
from sklearn.model_selection import train_test_split

from skfolio import RiskMeasure
from skfolio.datasets import load_sp500_dataset, load_sp500_index
from skfolio.optimization import BenchmarkTracker
from skfolio.preprocessing import prices_to_returns

prices = load_sp500_dataset()
benchmark_prices = load_sp500_index()

X, y = prices_to_returns(prices, benchmark_prices)
X_train, X_test, factors_train, factors_test = train_test_split(
    X, y["SP500"], test_size=0.33, shuffle=False
)

model = BenchmarkTracker(
    risk_measure=RiskMeasure.STANDARD_DEVIATION,
)
model.fit(X_train, factors=factors_train)
print(model.weights_)

portfolio = model.predict(X_test)
# Compare portfolio returns to benchmark
excess_returns = portfolio.returns - y_test.values
tracking_error = np.std(excess_returns, ddof=1)
print(f"Tracking Error: {tracking_error:0.2%}")
```

<a id="fallbacks"></a>

## Fallbacks

Optimization can sometimes fail during a given rebalancing. For example, a convex
mean-variance problem with strict risk or sector constraints may become infeasible on
specific dates.

All optimization estimators accept a `fallback` parameter that can be either a single
estimator or a list of estimators. When the primary optimization raises during `fit`,
the models in `fallback` are tried in order until one succeeds. The fitted weights and
core fitted attributes are copied back to the original estimator so you can keep a
single reference in your workflow. Fallbacks can also be set to the string
`previous_weights` to reuse the latest available allocation when the primary fit fails.

Each attempt is recorded in `fallback_chain_`, and the successful estimator is available
through `fallback_`.

This mechanism is critical in automated production, where optimization failures
shouldn’t interrupt pipelines and where you need reproducibility and auditability.
It can also be used to loosen optimization constraints gradually.

Example: The primary model is a minimum-variance optimization made intentionally
infeasible (the assets’ minimum weights are set to 10%, which exceeds the feasible
upper bound of 1/n_assets = 5%). As a fallback, we provide a feasible minimum-variance
model with a 2% minimum weight constraint:

```python
model = MeanRisk(
min_weights=0.1,  # intentionally infeasible
fallback=MeanRisk(min_weights=0.02),  # feasible fallback
)
model.fit(X_train)
print(model.weights_)

# Let's retrieve the fitted fallback that produced the final result:
print(model.fallback_)
# Let's display the sequence of attempts and their outcomes:
print(model.fallback_chain_)
# The fallback audit trail is also propagated to the predicted portfolio:
portfolio = model.predict(X_test)
assert portfolio.fallback_chain == model.fallback_chain_
```

When calling `predict`, the selected fallback and the full attempt log are propagated
to the resulting portfolio via `fallback_chain`.

For a step-by-step tutorial and more details, see
[Failure and Fallbacks](https://skfolio.org/auto_examples/mean_risk/plot_17_failure_and_fallbacks.html.md#sphx-glr-auto-examples-mean-risk-plot-17-failure-and-fallbacks-py).

<a id="failure-handling"></a>

## Failure Handling

In research, cross-validation and hyperparameter tuning (e.g., walk-forward, multiple
randomized cross-validation), it’s often useful to let all runs complete while keeping
a full record of failures instead of stopping on the first failed rebalancing.

The behavior on optimization failure is controlled by the `raise_on_failure`
parameter.

- If `raise_on_failure=True` (default): any error raised by the primary estimator is
  re-raised after fallbacks are exhausted. No `weights_` are set, and calling
  `predict` before a successful `fit` raises a `NotFittedError`.
- If `raise_on_failure=False`: errors are not raised. Instead, a warning is
  emitted, `weights_` is set to `None`, and `predict` returns a
  [`FailedPortfolio`](https://skfolio.org/generated/skfolio.portfolio.FailedPortfolio.html.md#skfolio.portfolio.FailedPortfolio) that carries diagnostics.

Diagnostics are exposed via:

- `error_`: the stringified error of the failed fit.
- `fallback_chain_`: a sequence of attempts with outcomes (`"success"` or the
  error message), starting from the primary estimator.

For online workflows based on `partial_fit`, the estimator first updates its stateful
components, such as the prior and moment estimators, then solves the next portfolio.
The `raise_on_failure` policy applies to solver failures at that rebalance. Errors raised
while updating stateful components are still raised because the estimator state may be
incomplete. The only fallback supported by `partial_fit` is
`fallback="previous_weights"`, which reuses the latest valid allocation. Estimator
fallbacks are reserved for regular `fit`, where each fallback can be fitted on the
complete training window.

Example: proceed without raising and retrieve failure diagnostics

```python
from skfolio import RiskMeasure
from skfolio.optimization import MeanRisk, ObjectiveFunction

# Configure an intentionally infeasible problem
model = MeanRisk(
    min_weights=1.0,
    raise_on_failure=False,  # do not raise; collect diagnostics instead
)

model.fit(X_train)  # does not raise; weights_ is None on failure
print(model.error_)          # stringified error message
print(model.fallback_chain_) # attempts and outcomes

ptf = model.predict(X_test)  # returns a FailedPortfolio sentinel
print(type(ptf).__name__)    # "FailedPortfolio"
print(ptf.optimization_error)
print(ptf.fallback_chain)
```

For a complete tutorial illustrating failure handling and fallbacks, see
[Failure and Fallbacks](https://skfolio.org/auto_examples/mean_risk/plot_17_failure_and_fallbacks.html.md#sphx-glr-auto-examples-mean-risk-plot-17-failure-and-fallbacks-py).