skfolio.moments.RegimeAdjustedEWCovariance#

class skfolio.moments.RegimeAdjustedEWCovariance(half_life=40, corr_half_life=None, hac_lags=None, regime_half_life=None, regime_target=PORTFOLIO, regime_method=FIRST_MOMENT, regime_portfolio_weights=None, regime_multiplier_clip=(0.7, 1.6), regime_min_observations=None, min_observations=None, assume_centered=True, nearest=True, higham=False, higham_max_iteration=100)[source]#

Exponentially weighted covariance estimator with regime adjustment via the Short-Term Volatility Update (STVU) [1].

This estimator computes an exponentially weighted covariance and applies a scalar multiplier $\phi_t$ to improve risk calibration when volatility regimes change more quickly than a plain EWMA can track.

This estimator also supports separate half life for variance and correlation. Lower half life for variance allows the model to adapt faster to volatility shifts, while higher half life for correlation enables more stable estimation of co-movements, which typically require more data for reliable inference and reduces estimation noise. This choice also aligns with empirical evidence that volatility tends to mean-revert faster than correlation. Using a lower (more responsive) decay factor for variance can capture this behavior.

Additionally, this estimator supports optional Newey-West HAC (Heteroskedasticity and Autocorrelation Consistent) correction via the hac_lags parameter. This adjusts for serial correlation in returns.

The STVU is configured by two parameters:

RegimeAdjustmentTarget: determines the statistic used to detect volatility regime changes (see the enum docstring for details and formulae).
RegimeAdjustmentMethod: determines how the raw statistic is transformed into the regime multiplier $\phi$ (see the enum docstring for details and formulae).

NaN handling:

The estimator handles missing data (NaN returns) caused by late listings, delistings, and holidays using EWMA updates together with active_mask. An asset with active_mask=True is treated as active at time $t$. If its return is finite, the EWMA is updated normally. If its return is NaN, the observation is treated as a holiday and covariance entries involving this asset are kept unchanged. An asset with active_mask=False is treated as inactive, for example during pre-listing or post-delisting periods, and covariance entries involving this asset are set to NaN.

Active with valid return: Normal EWMA update.
Active with NaN return (holiday): Freeze; covariance entries involving this asset are kept unchanged.
Inactive (active_mask=False): Covariance entries involving this asset are set to NaN.

When active_mask is not provided, trailing NaN returns are ambiguous: they could correspond either to holidays, in which case covariance is frozen, or to inactive periods, in which case covariance is set to NaN.

The min_observations parameter controls a warm-up period: an asset’s covariance entries remain NaN in the output until it has accumulated enough valid observations for a reliable estimate.

Late-listing bias correction:

The EWMA recursion is initialized at zero for every asset. This guarantees that the internal covariance state remains positive semi-definite at every step, but introduces a transient downward scale bias: after $n_i$ observations, the raw EWMA for asset $i$ is damped by a factor $(1 - \lambda^{n_i})$. At output time, a per-asset correction removes this bias:

\[\hat{\Sigma}_{ij} = \frac{S_{ij}}{\sqrt{(1 - \lambda^{n_i})(1 - \lambda^{n_j})}}\]

where $S$ is the raw internal EWMA. This is a congruence transform $D S D$ with $D = \text{diag}(1 / \sqrt{1 - \lambda^{n_i}})$, which preserves positive semi-definiteness while restoring the correct variance scale.

When corr_half_life is provided, the same bias correction is applied independently to the variance state (using $\lambda$) and the correlation state (using $\lambda_c$), then the covariance is reconstructed from the corrected components. The correlation bias correction uses pairwise co-observation counts rather than per-asset counts, so asynchronous late listings, holidays, and delistings are corrected at the pair level.

Estimation universe for STVU:

An optional estimation_mask defines the estimation universe used for the STVU regime multiplier without affecting pairwise covariance EWMA updates. The STVU is computed in a one-step-ahead manner: the return observed at time $t$ is standardized by the bias-corrected covariance estimate available at time $t-1$, and only assets that were already above min_observations before time $t$ contribute to the regime signal. This is important because the STVU statistic is sensitive to poorly-estimated assets. Noisy or illiquid assets with unreliable covariance estimates can inflate or deflate the distance, distorting the regime multiplier for the entire covariance matrix.

For standard exponentially weighted covariance without regime adjustment, see EWCovariance.

Parameters:

half_lifefloat, default=40

Half-life of the exponential weights for variance estimation, in number of observations.

When corr_half_life is None (default), this also controls the correlation estimation, resulting in standard EWMA covariance: $\Sigma_t = \lambda \Sigma_{t-1} + (1-\lambda) r_t r_t^\top$

When corr_half_life is provided, variance and correlation are updated with different half-lives to capture their different dynamics.

The half-life controls how quickly older observations lose their influence:

Larger half-life: More stable estimates, slower to adapt (robust to noise)
Smaller half-life: More responsive estimates, faster to adapt (sensitive to noise)

The decay factor $\lambda$ is computed as: $\lambda = 2^{-1/\text{half-life}}$

For example:

half-life = 40: $\lambda \approx 0.983$
half-life = 23: $\lambda \approx 0.970$
half-life = 11: $\lambda \approx 0.939$
half-life = 6: $\lambda \approx 0.891$

Note

For portfolio optimization, larger half-lives (>= 20) are generally preferred to avoid excessive turnover from estimation noise.

corr_half_lifefloat, optional

Half-life for correlation estimation, in number of observations.

If None (default), the same half_life is used for both variance and correlation, resulting in standard EWMA covariance.

If provided, enables separate half-lives: half_life governs variance and corr_half_life governs correlation. This is useful because volatility typically mean-reverts faster than correlation, so using a smaller (more responsive) half-life for variance can better capture regime changes.

hac_lagsint, optional

Number of lags for Newey-West HAC (Heteroskedasticity and Autocorrelation Consistent) correction. If None (default), no HAC correction is applied.

When enabled, the covariance update uses HAC-adjusted cross-products instead of simple outer products, accounting for autocorrelation in returns:

\[\text{hac_outer} = r_t r_t^T + \sum_{j=1}^{L} w_j (r_t r_{t-j}^T + r_{t-j} r_t^T)\]

where $w_j = 1 - j/(L+1)$ is the Bartlett kernel weight.

Typical values:

Daily equity data: 3-5 lags (weak autocorrelation from microstructure)
High-frequency data: 5-10 lags (stronger autocorrelation)
Monthly data: 1-2 lags

Must be a positive integer if specified.

regime_half_lifefloat, optional

Half-life for smoothing the volatility regime signal, in number of observations.

The regime signal is built from one-step-ahead standardized risk statistics and then transformed into the multiplier $\phi$ according to regime_target and regime_method. A shorter regime_half_life makes the multiplier react faster to abrupt changes in realized risk; a longer one produces a smoother, slower moving adjustment.

If None (default), it is automatically calibrated as: $\text{regime-half-life} = 0.5 \times \text{half-life}$

This makes the STVU more responsive (shorter half-life) than the covariance, allowing it to quickly rescale risk when realized volatility deviates from the slower EWMA estimate.

regime_targetRegimeAdjustmentTarget, default=RegimeAdjustmentTarget.PORTFOLIO

Target dimension used to calibrate the short-term volatility update:

PORTFOLIO: Portfolio variance $((w^T r)^2/(w^T \Sigma w))$
DIAGONAL: Individual volatilities $(\sum_i (r_i/\sigma_i)^2)$
MAHALANOBIS: Full covariance $(r^T \Sigma^{-1} r)$

regime_methodRegimeAdjustmentMethod, default=RegimeAdjustmentMethod.FIRST_MOMENT

Method used to transform the update statistic into the volatility multiplier $\phi$:

LOG: Robust to outliers (log compresses extremes)
FIRST_MOMENT: Calibrates the first moment of the standardized risk statistic
RMS: $\chi^2$ calibration (sensitive to extremes)

regime_portfolio_weightsarray-like of shape (n_assets,) or (n_portfolios, n_assets) or None, default=None

Portfolio weights used by the STVU PORTFOLIO target. Only used when regime_target=RegimeAdjustmentTarget.PORTFOLIO.

If None (default), uses inverse-volatility weights, which neutralizes asset volatility dispersion so high-volatility assets don’t dominate the calibration statistic. These weights are recomputed dynamically as variances evolve.

If a 1D array is provided, a single static portfolio is used. If a 2D array of shape (n_portfolios, n_assets) is provided, the STVU statistic is computed independently for each portfolio, transformed, and then averaged into a single regime signal. This calibrates the covariance along multiple traded directions without being affected by noise in uninvestable eigenvector directions (unlike MAHALANOBIS).

Weights are automatically normalized so each row sums to 1.

For equal-weight calibration, pass regime_portfolio_weights=np.ones(n_assets)/n_assets.

regime_multiplier_cliptuple[float, float] or None, default=(0.7, 1.6)

Clip $\phi$ to avoid extreme swings in the regime multiplier. Set to None to disable clipping. The multiplier is applied to the covariance as $\phi^2 \Sigma$. With the default bounds, the covariance scale remains between $0.7^2 = 0.49$ and $1.6^2 = 2.56$.

regime_min_observationsint, optional

Minimum number of one-step-ahead comparisons before enabling STVU. If insufficient data, STVU defaults to 1.0 (no adjustment).

If None (default), it is automatically set to int(regime_half_life), ensuring the STVU EWMA has seen roughly one half-life of data before being applied.

min_observationsint, optional

Minimum number of valid observations per asset before its covariance entries are considered reliable and exposed in the output covariance_. Until this threshold is reached, the asset’s covariance entries remain NaN.

This warm-up prevents noisy estimates from a few initial observations from being used by downstream optimizers.

The default (None) uses int(max(half_life, corr_half_life)) as the threshold when corr_half_life is set, or int(half_life) otherwise. This ensures both variance and correlation bias-correction factors have decayed to at most 50%. Set to 1 to disable warm-up entirely.

assume_centeredbool, default=True

If True (default), the EWMA update uses raw returns without demeaning. This is the standard convention for EWMA covariance estimation in finance. If False, returns are demeaned using an EWMA mean estimate before computing the covariance update, and location_ tracks the EWMA mean.

nearestbool, default=True

If this is set to True, the covariance is replaced by the nearest covariance matrix that is positive definite and with a Cholesky decomposition that can be computed. The variance is left unchanged. The default is True.

highambool, default=False

If this is set to True, the Higham (2002) algorithm is used to find the nearest PD covariance, otherwise the eigenvalues are clipped to a threshold above zeros (1e-13). The default is False and uses the clipping method as the Higham algorithm can be slow for large datasets.

higham_max_iterationint, default=100

Maximum number of iterations of the Higham (2002) algorithm. The default value is 100.

Attributes:

covariance_ndarray of shape (n_assets, n_assets): Estimated covariance matrix. Contains NaN for assets that are inactive or have not yet accumulated min_observations valid observations.
regime_multiplier_float: The volatility regime adjustment factor applied. Equal to 1.0 if insufficient data.
location_ndarray of shape (n_assets,): Estimated location (mean). If assume_centered=True, this is zeros. Otherwise, it tracks the EWMA mean of returns. Contains NaN for inactive assets.
n_features_in_int: Number of assets seen during fit.
feature_names_in_ndarray of shape (n_features_in_,): Names of features seen during fit. Defined only when X has feature names that are all strings.

Methods

`fit`(X[, y, active_mask, estimation_mask])	Fit the Regime-Adjusted Exponentially Weighted Covariance estimator.
`get_metadata_routing`()	Get metadata routing of this object.
`get_params`([deep])	Get parameters for this estimator.
`mahalanobis`(X_test)	Compute the squared Mahalanobis distance of observations.
`partial_fit`(X[, y, active_mask, estimation_mask])	Incrementally fit the Regime-Adjusted EW Covariance estimator.
`score`(X_test[, y])	Compute the mean log-likelihood of observations under the estimated model.
`set_fit_request`(*[, active_mask, ...])	Configure whether metadata should be requested to be passed to the `fit` method.
`set_params`(**params)	Set the parameters of this estimator.
`set_partial_fit_request`(*[, active_mask, ...])	Configure whether metadata should be requested to be passed to the `partial_fit` method.
`set_score_request`(*[, X_test])	Configure whether metadata should be requested to be passed to the `score` method.

Notes

The STVU compares predicted versus realized risk using a one-step-ahead standardized statistic $d^2_{t+1}$ computed from the covariance estimate at time $t$ and the return observed at time $t+1$. The exact statistic depends on regime_target:

PORTFOLIO calibrates covariance along one or more portfolio directions.
DIAGONAL calibrates the diagonal risk scale and ignores correlations.
MAHALANOBIS calibrates the full covariance structure.

Under correct calibration, the transformed statistic has unit scale in expectation. Persistent values above that level imply realized risk is higher than predicted, so $\phi > 1$ scales the covariance up. Persistent values below that level imply over-prediction, so $\phi < 1$ scales it down.

This approach is related to volatility updating in multivariate GARCH models, but implemented here as a multiplicative adjustment on top of an EWMA covariance estimator.

References

[1]

“The Elements of Quantitative Investing”, Wiley Finance, Giuseppe Paleologo (2025).

[2]

“Multivariate exponentially weighted moving covariance matrix”, Technometrics, Hawkins & Maboudou-Tchao (2008).

[3]

“Dynamic conditional correlation: A simple class of multivariate GARCH models”, Journal of Business & Economic Statistics, Engle (2002).

[4]

“Computing the nearest correlation matrix - A problem from finance”, IMA Journal of Numerical Analysis, Higham (2002)

[5]

“An Introduction to Multivariate Statistical Analysis”, Wiley, Anderson (2003).

Examples

>>> from skfolio.datasets import load_sp500_dataset
>>> from skfolio.moments import RegimeAdjustedEWCovariance, RegimeAdjustmentTarget, RegimeAdjustmentMethod
>>> from skfolio.preprocessing import prices_to_returns
>>> import numpy as np
>>>
>>> prices = load_sp500_dataset()
>>> X = prices_to_returns(prices)
>>>
>>> # Portfolio target with inverse-vol weights and FIRST_MOMENT method (default)
>>> model = RegimeAdjustedEWCovariance(half_life=23)
>>> model.fit(X)
>>> print(model.regime_multiplier_)
>>>
>>> # DIAGONAL target (individual asset volatilities)
>>> model2 = RegimeAdjustedEWCovariance(
...     regime_target=RegimeAdjustmentTarget.DIAGONAL,
...     regime_method=RegimeAdjustmentMethod.RMS,
... )
>>> model2.fit(X)
>>>
>>> # Mahalanobis target (full covariance structure)
>>> model_maha = RegimeAdjustedEWCovariance(
...     regime_target=RegimeAdjustmentTarget.MAHALANOBIS,
...     regime_method=RegimeAdjustmentMethod.FIRST_MOMENT,
... )
>>> model_maha.fit(X)
>>>
>>> # Portfolio target with equal weights
>>> n_assets = X.shape[1]
>>> model_equal = RegimeAdjustedEWCovariance(
...     regime_target=RegimeAdjustmentTarget.PORTFOLIO,
...     regime_portfolio_weights=np.ones(n_assets) / n_assets,
... )
>>> model_equal.fit(X)
>>>
>>> # With Newey-West HAC correction
>>> model_hac = RegimeAdjustedEWCovariance(
...     half_life=23,
...     hac_lags=5
... )
>>> model_hac.fit(X)

fit(X, y=None, *, active_mask=None, estimation_mask=None)[source]#

Fit the Regime-Adjusted Exponentially Weighted Covariance estimator.

Parameters:

Xarray-like of shape (n_observations, n_assets)

Price returns of the assets. May contain NaN for missing data (holidays, late listings, delistings).

yIgnored

Not used, present for API consistency by convention.

active_maskarray-like of shape (n_observations, n_assets), optional

Boolean mask indicating whether each asset is structurally active at each observation. Use this to distinguish between holidays (active_mask=True and NaN return: covariance is frozen) and inactive periods such as pre-listing or post-delisting (active_mask=False: covariance is set to NaN). If None (default), all pairs are assumed active.

estimation_maskarray-like of shape (n_observations, n_assets), optional

Boolean mask indicating which active assets should belong to the estimation universe for the STVU statistic computation on each day.

If None (default), all active assets with finite returns and finite covariance estimates are used.
If provided, only assets where the mask is True contribute to the regime multiplier calculation.

Pairwise covariance EWMA updates still use all active assets with valid observations; this mask only affects the STVU regime multiplier calculation.

This is important because the STVU statistic is sensitive to poorly-estimated assets. Noisy or illiquid assets with unreliable covariance estimates can inflate or deflate the distance, distorting the regime multiplier for the entire covariance matrix.

Use cases:

Focus on liquid assets to reduce noise in regime detection
Exclude recently-listed assets whose covariance is still poorly estimated
Match the estimation universe used in a factor model

Returns:

selfRegimeAdjustedEWCovariance: Fitted estimator.

get_metadata_routing()#

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns:

routingMetadataRequest: A MetadataRequest encapsulating routing information.

get_params(deep=True)#

Get parameters for this estimator.

Parameters:

deepbool, default=True: If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict: Parameter names mapped to their values.

mahalanobis(X_test)#

Compute the squared Mahalanobis distance of observations.

The squared Mahalanobis distance of an observation $r$ is defined as:

\[d^2 = (r - \mu)^T \Sigma^{-1} (r - \mu)\]

where $\Sigma$ is the estimated covariance matrix (self.covariance_) and $\mu$ is the estimated mean (self.location_ if available, otherwise zero).

This distance measure accounts for correlations between assets and is useful for:

Outlier detection in portfolio returns
Risk-adjusted distance calculations
Identifying unusual market regimes

Parameters:

X_testarray-like of shape (n_observations, n_assets) or (n_assets,): Observations for which to compute the squared Mahalanobis distance. Each row represents one observation. If 1D, treated as a single observation. Assets with non-finite fitted variance are excluded from inference. Inside the retained inference subspace, the observations must be finite.

Returns:

distancesndarray of shape (n_observations,) or float: Squared Mahalanobis distance for each observation. Returns a scalar if input is 1D.

Examples

>>> import numpy as np
>>> from skfolio.moments import EmpiricalCovariance
>>> X = np.random.randn(100, 3)
>>> model = EmpiricalCovariance()
>>> model.fit(X)
>>> distances = model.mahalanobis(X)
>>> # Distances follow approximately chi-squared distribution with n_assets DoF
>>> print(f"Mean distance: {distances.mean():.2f}, Expected: {3:.2f}")

partial_fit(X, y=None, *, active_mask=None, estimation_mask=None)[source]#

Incrementally fit the Regime-Adjusted EW Covariance estimator.

Parameters:

Xarray-like of shape (n_observations, n_assets): Price returns of the assets. May contain NaN for missing data (holidays, late listings, delistings).
yIgnored: Not used, present for API consistency by convention.
active_maskarray-like of shape (n_observations, n_assets), optional: Boolean mask indicating whether each asset is structurally active at each observation. Use this to distinguish between holidays (active_mask=True and NaN return: covariance is frozen) and inactive periods such as pre-listing or post-delisting (active_mask=False: covariance is set to NaN). If None (default), all pairs are assumed active.
estimation_maskarray-like of shape (n_observations, n_assets), optional: Boolean mask indicating which active assets should belong to the estimation universe for the STVU statistic computation on each day. See fit for details.

Returns:

selfRegimeAdjustedEWCovariance: Fitted estimator.

score(X_test, y=None)#

Compute the mean log-likelihood of observations under the estimated model.

Evaluates how well the fitted covariance matrix explains new observations, assuming a multivariate Gaussian distribution. This is useful for:

Model selection (comparing different covariance estimators)
Cross-validation of covariance estimation methods
Assessing goodness-of-fit

The log-likelihood for a single observation $r$ is:

\[\log p(r | \mu, \Sigma) = -\frac{1}{2} \left[ n \log(2\pi) + \log|\Sigma| + (r - \mu)^T \Sigma^{-1} (r - \mu) \right]\]

where $n$ is the number of assets, $\Sigma$ is the estimated covariance matrix (self.covariance_), and $\mu$ is the estimated mean (self.location_ if available, otherwise zero).

Parameters:

X_testarray-like of shape (n_observations, n_assets): Observations for which to compute the log-likelihood. Typically held-out test data not used during fitting. Assets with non-finite fitted variance are excluded from inference. This typically happens when the fitted covariance cannot be estimated for an asset, for example before listing, after delisting, or during a warmup period. After this asset-level filtering, each row of X_test is scored using the remaining available values only. This covers row-level missing values in X_test, such as market holidays or pre/post-listing.
yIgnored: Not used, present for scikit-learn API consistency.

Returns:

scorefloat: Mean log-likelihood of the observations. Higher values indicate better fit. The score is averaged over all observations.

Examples

>>> import numpy as np
>>> from skfolio.moments import EmpiricalCovariance, LedoitWolf
>>> X_train = np.random.randn(100, 5)
>>> X_test = np.random.randn(50, 5)
>>> emp = EmpiricalCovariance().fit(X_train)
>>> lw = LedoitWolf().fit(X_train)
>>> # Compare models on held-out data
>>> print(f"Empirical: {emp.score(X_test):.2f}")
>>> print(f"LedoitWolf: {lw.score(X_test):.2f}")

set_fit_request(*, active_mask='$UNCHANGED$', estimation_mask='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

active_maskstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for active_mask parameter in fit.
estimation_maskstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for estimation_mask parameter in fit.

Returns:

selfobject: The updated object.

set_params(**params)#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:

**paramsdict: Estimator parameters.

Returns:

selfestimator instance: Estimator instance.

set_partial_fit_request(*, active_mask='$UNCHANGED$', estimation_mask='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the partial_fit method.

The options for each parameter are:

True: metadata is requested, and passed to partial_fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to partial_fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

active_maskstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for active_mask parameter in partial_fit.
estimation_maskstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for estimation_mask parameter in partial_fit.

Returns:

selfobject: The updated object.

set_score_request(*, X_test='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the score method.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

X_teststr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED: Metadata routing for X_test parameter in score.

Returns:

selfobject: The updated object.