# Entropy Pooling
This tutorial introduces the [`EntropyPooling`](https://skfolio.org/generated/skfolio.prior.EntropyPooling.html.md#skfolio.prior.EntropyPooling) estimator.
## Introduction
Entropy Pooling, introduced by Attilio Meucci in 2008 as a generalization of the
Black-Litterman framework, is a nonparametric method for adjusting a baseline (“prior”)
probability distribution to incorporate user-defined views by finding the posterior
distribution closest to the prior while satisfying those views.
User-defined views can be **elicited** from domain experts or **derived** from
quantitative analyses.
Grounded in information theory, it updates the distribution in the least-informative
way by minimizing the Kullback-Leibler divergence (relative entropy) under the
specified view constraints.
Mathematically, the problem is formulated as:
$$
\begin{aligned}
\min_{\mathbf{q}} \quad & \sum_{i=1}^T q_i \log\left(\frac{q_i}{p_i}\right) \\
\text{subject to} \quad & \sum_{i=1}^T q_i = 1 \quad \text{(normalization constraint)} \\
& \mathbb{E}_q[f_j(X)] = v_j \quad(\text{or } \le v_j, \text{ or } \ge v_j), \quad j = 1,\dots,k, \text{(view constraints)} \\
& q_i \ge 0, \quad i = 1, \dots, T
\end{aligned}
$$
Where:
- $T$ is the number of observations (number of scenarios).
- $p_i$ is the prior probability of scenario $x_i$.
- $q_i$ is the posterior probability of scenario $x_i$.
- $X$ is the scenario matrix of shape (n_observations, n_assets).
- $f_j$ is the j th view function.
- $v_j$ is the target value imposed by the j th view.
- $k$ is the total number of views.
The `skfolio` implementation supports the following views:
: * Equalities
* Inequalities
* Ranking
* Linear combinations (e.g. relative views)
* Views on groups of assets
On the following measures:
: * Mean
* Variance
* Skew
* Kurtosis
* Correlation
* Value-at-Risk (VaR)
* Conditional Value-at-Risk (CVaR)
Entropy Pooling re-weights the sample probabilities of the prior distribution and is
therefore constrained by the support (completeness) of that distribution. For example,
if the historical distribution contains no returns below -10% for a given asset, we
cannot impose a CVaR view of 15%: no matter how we adjust the sample probabilities,
such tail data do not exist.
Therefore, to impose extreme views on a sparse historical distribution, one must
generate synthetic data. In that case, the EP posterior is only as reliable as the
synthetic scenarios. It is thus essential to use a generator capable of extrapolating
tail dependencies, such as [`VineCopula`](https://skfolio.org/generated/skfolio.distribution.VineCopula.html.md#skfolio.distribution.VineCopula), to model joint
extreme events accurately.
In general, for extreme stress tests, it is recommended to use conditional sampling from
[`VineCopula`](https://skfolio.org/generated/skfolio.distribution.VineCopula.html.md#skfolio.distribution.VineCopula) (see the previous tutorial
[Vine Copula & Stress Test](https://skfolio.org/auto_examples/synthetic_data/plot_2_vine_copula.html.md#sphx-glr-auto-examples-synthetic-data-plot-2-vine-copula-py)). However, when
conditional sampling does not provide sufficient granularity, one can combine Entropy
Pooling with Vine Copula, as demonstrated at the end of this tutorial.
In this tutorial, we will:
: 1. Apply Entropy Pooling to historical return data.
2. Construct portfolios based on the adjusted distribution.
3. Demonstrate factor-based and synthetic-data-enhanced Entropy Pooling.
4. Perform stress tests using Entropy Pooling.
## Data Loading and Preparation
We load the S&P 500 [dataset](https://skfolio.org/user_guide/datasets.html.md#datasets) and select seven stocks
(for demonstration purposes). We also load the factors dataset, composed of
daily prices for five ETFs representing common factors.
```Python
import numpy as np
import pandas as pd
from plotly.io import show
from skfolio import Population, RiskMeasure
from skfolio.datasets import load_factors_dataset, load_sp500_dataset
from skfolio.distribution import VineCopula
from skfolio.measures import (
correlation,
cvar,
kurtosis,
mean,
skew,
standard_deviation,
value_at_risk,
)
from skfolio.optimization import HierarchicalRiskParity, RiskBudgeting
from skfolio.preprocessing import prices_to_returns
from skfolio.prior import EntropyPooling, SyntheticData, TimeSeriesFactorModel
from skfolio.utils.figure import plot_kde_distributions
# Load stock price and factor data
prices = load_sp500_dataset()
prices = prices[["AMD", "BAC", "GE", "JNJ", "JPM", "LLY", "PG"]]
factor_prices = load_factors_dataset()
# Convert to daily returns
X, factors = prices_to_returns(prices, factor_prices)
print("Shapes:")
print(f"X: {X.shape}")
print(f"factors: {factors.shape}")
print(X.tail())
print(factors.tail())
```
```none
Shapes:
X: (2263, 7)
factors: (2263, 5)
AMD BAC GE ... JPM LLY PG
Date ...
2022-12-21 0.040430 0.015223 0.033001 ... 0.011248 0.023275 0.009170
2022-12-22 -0.056442 -0.008848 -0.014582 ... -0.011355 -0.007339 0.002308
2022-12-23 0.010335 0.002443 0.000235 ... 0.004749 0.007090 0.002825
2022-12-27 -0.019374 0.001875 0.012849 ... 0.003504 -0.008208 0.008713
2022-12-28 -0.011064 0.007360 -0.010502 ... 0.005463 0.000932 -0.012926
[5 rows x 7 columns]
MTUM QUAL SIZE USMV VLUE
Date
2022-12-21 0.014312 0.017884 0.014371 0.012005 0.013246
2022-12-22 -0.010977 -0.015411 -0.012070 -0.007315 -0.011989
2022-12-23 0.010897 0.005889 0.006287 0.005281 0.005844
2022-12-27 0.001770 -0.003138 -0.001320 0.001798 -0.000111
2022-12-28 -0.011778 -0.013325 -0.013914 -0.010489 -0.015238
```
### Summary Statistics
We create a helper function to compute key return statistics, optionally weighted by
sample probabilities:
```Python
def summary(X: pd.DataFrame, sample_weight: np.ndarray | None = None) -> pd.DataFrame:
return pd.DataFrame(
{
"Mean": mean(X, sample_weight=sample_weight),
"Volatility": standard_deviation(X, sample_weight=sample_weight),
"Skew": skew(X, sample_weight=sample_weight),
"Kurtosis": kurtosis(X, sample_weight=sample_weight),
"VaR at 90%": value_at_risk(X, beta=0.90, sample_weight=sample_weight),
"CVaR at 90%": cvar(X, beta=0.90, sample_weight=sample_weight),
}
)
print(f"Corr(BAC, JPM): {correlation(X[['BAC', 'JPM']])[0][1]:.2%}")
summary(X)
```
```none
Corr(BAC, JPM): 90.87%
```
[plotly figure stripped from llms output]
## Building a Portfolio based on Entropy Pooling
Now that we’ve shown how the Entropy Pooling estimator works in isolation, let’s
see how to implement a risk parity portfolio with CVaR-90% as the risk measure based
on EP:
```Python
bench = RiskBudgeting(risk_measure=RiskMeasure.CVAR, cvar_beta=0.9)
model = RiskBudgeting(
risk_measure=RiskMeasure.CVAR, cvar_beta=0.9, prior_estimator=entropy_pooling
)
bench.fit(X)
model.fit(X)
print(bench.weights_)
print(model.weights_)
```
```none
[0.07696064 0.10665992 0.11098196 0.19797973 0.11930806 0.16615742
0.22195227]
[0.0827406 0.0863148 0.07945442 0.18251824 0.06746977 0.22682055
0.27468163]
```
We notice that the weight on GE is lower in the portfolio based on EP versus the
benchmark, reflecting that GE’s tail risk was the most impacted by our views.
Note that instead of [`RiskBudgeting`](https://skfolio.org/generated/skfolio.optimization.RiskBudgeting.html.md#skfolio.optimization.RiskBudgeting), Entropy Pooling is
also compatible with the other [portfolio optimization](https://skfolio.org/user_guide/optimization.html.md#optimization) methods
such as [`MeanRisk`](https://skfolio.org/generated/skfolio.optimization.MeanRisk.html.md#skfolio.optimization.MeanRisk),
[`HierarchicalRiskParity`](https://skfolio.org/generated/skfolio.optimization.HierarchicalRiskParity.html.md#skfolio.optimization.HierarchicalRiskParity) etc.
### Comparing Risk Contributions
A CVaR risk-parity portfolio assigns weights so that each asset contributes the same
amount to the portfolio’s CVaR.
Therefore, as shown in the contribution graphs below:
* The benchmark has equal CVaR contributions under the **prior** distribution
* The EP portfolio has equal CVaR contributions under the **posterior** distribution
```Python
sample_weight = model.prior_estimator_.return_distribution_.sample_weight
portfolio_bench = bench.predict(X)
portfolio_bench.name = "Benchmark (Optimized on prior)"
portfolio_ep = model.predict(X)
portfolio_ep.name = "Optimized on EP posterior"
```
### Backtest using the Prior Distribution
```Python
population = Population([portfolio_bench, portfolio_ep])
population.plot_contribution(measure=RiskMeasure.CVAR)
```
[plotly figure stripped from llms output]
### Backtest using the EP Posterior Distribution
```Python
population.set_portfolio_params(sample_weight=sample_weight)
population.plot_contribution(measure=RiskMeasure.CVAR)
```
[plotly figure stripped from llms output]
## Factor Entropy Pooling
Instead of applying Entropy Pooling directly to asset returns, we can embed it
within a Factor Model.
This allows us to impose views on factor data such as the quality factor “QUAL”:
```Python
factor_entropy_pooling = EntropyPooling(mean_views=["QUAL == 0.0005"])
factor_model = TimeSeriesFactorModel(factor_prior_estimator=factor_entropy_pooling)
model = RiskBudgeting(risk_measure=RiskMeasure.CVAR, prior_estimator=factor_model)
model.fit(X, factors=factors)
print(model.weights_)
sample_weight = model.prior_estimator_.return_distribution_.sample_weight
summary(factors, sample_weight)
```
```none
[0.0858639 0.10101139 0.114691 0.20548907 0.11275435 0.17552912
0.20466117]
```
[plotly figure stripped from llms output]
## Conclusion
In this tutorial, we demonstrated how to leverage Entropy Pooling to integrate views
into every stage of portfolio management, from ex-ante optimization to ex-post
stress testing.
## References
> [1] “Fully Flexible Extreme Views”,
> : Journal of Risk, Meucci, Ardia & Keel (2011)
> [2] “Fully Flexible Views: Theory and Practice”,
> : Risk, Meucci (2013).
> [3] “Effective Number of Scenarios in Fully Flexible Probabilities”,
> : GARP Risk Professional, Meucci (2012)
> [4] “I-Divergence Geometry of Probability Distributions and Minimization
> : Problems”, The Annals of Probability, Csiszar (1975)
**Total running time of the script:** (0 minutes 9.361 seconds)