# Alpha Research and Factor-Neutral Portfolio
This tutorial shows how to research an alpha signal that forecasts the
idiosyncratic returns of the characteristics-based cross-sectional factor
model [`CharacteristicsFactorModel`](https://skfolio.org/generated/skfolio.prior.CharacteristicsFactorModel.html.md#skfolio.prior.CharacteristicsFactorModel), and how to trade it
in a factor-neutral long-short portfolio. The methodology is covered in the
[Alpha Estimators](https://skfolio.org/user_guide/factor_models.html.md#factor-model-alpha) and [Portfolio Construction](https://skfolio.org/user_guide/factor_models.html.md#factor-model-portfolio-construction) sections of the user guide.
We will:
* define an alpha signal that forecasts the factor model’s idiosyncratic returns
* evaluate its forecast quality with IC, portfolio and factor-correlation
diagnostics
* integrate the alpha estimator into the factor model
* optimize a factor-neutral portfolio that allocates to the orthogonal alpha
* jointly tune the optimizer, factor model and alpha estimator with online
search
* evaluate the strategy over a walk-forward test period
* run ex-ante and ex-post attribution of exposures, risk and performance,
verifying that the return comes from the orthogonal alpha rather than factor
premia
## Data
We reuse the synthetic characteristics panel from the [first tutorial](https://skfolio.org/auto_examples/factor_models/plot_characteristics_factor_model.html.md#sphx-glr-auto-examples-factor-models-plot-characteristics-factor-model-py).
It covers 500 assets over 1,500 trading days and includes late listings,
delistings, holidays and missing characteristics:
```Python
import numpy as np
from plotly.io import show
from skfolio.datasets import make_synthetic_characteristics
panel = make_synthetic_characteristics(
n_assets=500, n_observations=1500, random_state=0
)
```
## Model Definition
Next, we rebuild the 24-factor model with one global market factor, 10
industry factors and 13 style factors built from 29 descriptors. See the
[first tutorial](https://skfolio.org/auto_examples/factor_models/plot_characteristics_factor_model.html.md#sphx-glr-auto-examples-factor-models-plot-characteristics-factor-model-py)
for more details:
```Python
from skfolio.descriptor import (
AnalystDispersionToPrice,
AssetTurnover,
AssetsGrowthRate,
BookLeverage,
BookToPrice,
CapexToAssetsChangeInIntensity,
CashFlowToAssets,
CashFlowToPrice,
DebtToAssets,
DividendToPrice,
EWAmihudIlliquidity,
EWMarketBeta,
EWMomentum,
EWResidualVolatility,
EWShareTurnover,
EWVolatility,
EarningsChangeToPrice,
EarningsToPrice,
EbitdaToEnterpriseValue,
ForwardEarningsToPrice,
GrossMargin,
GrossProfitability,
IssuanceGrowthRate,
LogMarketCap,
MarketLeverage,
ReturnOnAssets,
ReturnOnEquity,
SalesGrowthRate,
SalesToPrice,
ShareholderYield,
ShortInterest,
)
from skfolio.factor_exposure import (
DerivedFactor,
FixedWeightedFactor,
GlobalFactor,
OneHotCategoricalFactors,
)
from skfolio.moments import EWMu, RegimeAdjustedEWCovariance
from skfolio.prior import CharacteristicsFactorModel, EmpiricalPrior
month = 21
quarter = 3 * month
half_year = 6 * month
year = 12 * month
global_factor = GlobalFactor(family="market")
industry_factors = OneHotCategoricalFactors(category="industry", family="industry")
beta_factor = FixedWeightedFactor(
descriptors=[("market_beta", EWMarketBeta(half_life=year))],
transform_by_group="industry",
)
momentum_factor = FixedWeightedFactor(
descriptors=[("momentum", EWMomentum(half_life=half_year, skip=month))],
transform_by_group="industry",
)
size_factor = FixedWeightedFactor(
descriptors=[("log_mcap", LogMarketCap())], transform_by_group="industry"
)
non_linear_size_factor = DerivedFactor(
source="size", func=lambda x: x**3, transform_by_group="industry"
)
value_factor = FixedWeightedFactor(
descriptors=[
("book_to_price", BookToPrice()),
("sales_to_price", SalesToPrice()),
("cash_flow_to_price", CashFlowToPrice()),
],
weights=[0.8, 0.1, 0.1],
transform_by_group="industry",
)
earnings_yield_factor = FixedWeightedFactor(
descriptors=[
("fwd_earnings_to_price", ForwardEarningsToPrice()),
("earnings_to_price", EarningsToPrice()),
("enterprise_multiple", EbitdaToEnterpriseValue()),
],
transform_by_group="industry",
)
growth_factor = FixedWeightedFactor(
descriptors=[
("earnings_change_to_price", EarningsChangeToPrice(lag=year)),
("sales_growth", SalesGrowthRate(lag=year)),
],
transform_by_group="industry",
)
profitability_factor = FixedWeightedFactor(
descriptors=[
("asset_turnover", AssetTurnover()),
("gross_profitability", GrossProfitability()),
("gross_margin", GrossMargin()),
("return_on_assets", ReturnOnAssets()),
("return_on_equity", ReturnOnEquity()),
("cash_flow_to_assets", CashFlowToAssets()),
],
transform_by_group="industry",
)
investment_factor = FixedWeightedFactor(
descriptors=[
("asset_growth", AssetsGrowthRate(lag=year)),
("issuance_growth", IssuanceGrowthRate(lag=year)),
("capex_growth", CapexToAssetsChangeInIntensity(lag=year)),
],
transform_by_group="industry",
)
dividend_yield_factor = FixedWeightedFactor(
descriptors=[
("dividend_to_price", DividendToPrice()),
("shareholder_yield", ShareholderYield()),
],
weights=[0.7, 0.3],
transform_by_group="industry",
)
leverage_factor = FixedWeightedFactor(
descriptors=[
("market_leverage", MarketLeverage()),
("debt_to_assets", DebtToAssets()),
("book_leverage", BookLeverage()),
],
transform_by_group="industry",
)
liquidity_factor = FixedWeightedFactor(
descriptors=[
("share_turnover", EWShareTurnover(half_life=quarter)),
("amihud_illiquidity", EWAmihudIlliquidity()),
],
transform_by_group="industry",
)
volatility_factor = FixedWeightedFactor(
descriptors=[
("vol", EWVolatility(half_life=quarter)),
(
"residual_vol",
EWResidualVolatility(half_life=quarter, beta_half_life=quarter),
),
],
transform_by_group="industry",
)
model = CharacteristicsFactorModel(
factors=[
("market", global_factor),
("industry", industry_factors),
("beta", beta_factor),
("momentum", momentum_factor),
("size", size_factor),
("non_linear_size", non_linear_size_factor),
("value", value_factor),
("earnings_yield", earnings_yield_factor),
("growth", growth_factor),
("profitability", profitability_factor),
("investment", investment_factor),
("dividend_yield", dividend_yield_factor),
("leverage", leverage_factor),
("liquidity", liquidity_factor),
("volatility", volatility_factor),
],
neutralize_against={
"non_linear_size": ["size"],
"volatility": ["beta"],
},
constrained_families=[("industry", None)],
exposure_lag=1,
inv_idio_variance_weight_shrinkage=0.5,
factor_prior_estimator=EmpiricalPrior(
covariance_estimator=RegimeAdjustedEWCovariance(
half_life=half_year, corr_half_life=year, regime_half_life=month
),
mu_estimator=EWMu(half_life=year),
),
n_jobs=-1,
)
```
## Alpha Research
We now build the alpha signal, a cross-sectional forecast of relative
idiosyncratic performance across assets at each date. The factor model
decomposes asset returns into systematic and idiosyncratic components [1](#id5), and
the signal targets the idiosyncratic returns. With raw returns as target,
the cross-sectional variation would also include each asset’s factor
exposures multiplied by the factor returns, so a signal correlated with the
exposures would pick up factor premia already captured by the factor model.
Targeting idiosyncratic returns removes this component and
keeps the forecast asset-specific (see [Alpha Estimators](https://skfolio.org/user_guide/factor_models.html.md#factor-model-alpha)).
We use the first three years for factor-model estimation and alpha
development and reserve the remaining three years for walk-forward
evaluation. After fitting the factor model, we use `enrich_asset_panel` to
add idiosyncratic returns, idiosyncratic variances, regression weights and
factor exposures to the training panel. We can then iterate on the alpha
estimator without refitting the factor model:
```Python
train_size = 3 * year
panel_train = panel[:train_size]
model.fit(characteristics=panel_train)
factor_model = model.factor_model_
panel_train_enriched = factor_model.enrich_asset_panel(panel_train)
```
### Alpha Signal
We build the signal from two characteristics: short interest and analyst
forecast dispersion. Empirical studies have associated high short interest
[2](#id6) and high analyst forecast dispersion [3](#id7) with lower subsequent
returns. We therefore combine the two descriptors with equal negative
weights, so assets with higher values receive lower alpha forecasts.
On real data, such a relationship would have to be discovered and tested.
Here the data is synthetic, so the relationship is built into the
generator: a persistent bearish component raises short interest and analyst
forecast dispersion and lowers future idiosyncratic returns. The
[`ShortInterest`](https://skfolio.org/generated/skfolio.descriptor.ShortInterest.html.md#skfolio.descriptor.ShortInterest) and
[`AnalystDispersionToPrice`](https://skfolio.org/generated/skfolio.descriptor.AnalystDispersionToPrice.html.md#skfolio.descriptor.AnalystDispersionToPrice) descriptors observe
this component with noise. Because the true signal is known by
construction, we can verify at the end of the tutorial that the workflow
recovers it from the observable characteristics.
We use [`FixedWeightedAlpha`](https://skfolio.org/generated/skfolio.alpha.FixedWeightedAlpha.html.md#skfolio.alpha.FixedWeightedAlpha) because the descriptor
directions and relative weights are specified in advance, which keeps the
focus on the alpha evaluation and portfolio-construction workflow. skfolio
also provides [`EWSharpeOptimalAlpha`](https://skfolio.org/generated/skfolio.alpha.EWSharpeOptimalAlpha.html.md#skfolio.alpha.EWSharpeOptimalAlpha) to estimate a
linear descriptor combination from historical idiosyncratic returns and
[`PredictorAlpha`](https://skfolio.org/generated/skfolio.alpha.PredictorAlpha.html.md#skfolio.alpha.PredictorAlpha) to use any ML predictor for
more flexible relationships:
```Python
from skfolio.alpha import FixedWeightedAlpha, alpha_forecast_evaluation
from skfolio.preprocessing import CSGaussianRankScaler
from skfolio.utils.stats import CSWeighting
holding_period = 10
alpha_estimator = FixedWeightedAlpha(
descriptors=[
("short_interest", ShortInterest()),
("analyst_dispersion", AnalystDispersionToPrice()),
],
weights=[-1.0, -1.0],
forecast_scale=7.5e-5,
scoring_transformer=CSGaussianRankScaler(),
n_jobs=-1,
)
```
[`CSGaussianRankScaler`](https://skfolio.org/generated/skfolio.preprocessing.CSGaussianRankScaler.html.md#skfolio.preprocessing.CSGaussianRankScaler) maps each descriptor and
the final composite to cross-sectional Gaussian rank scores. This places the
two descriptors on a comparable scale and limits the influence of extreme
values. [`FixedWeightedAlpha`](https://skfolio.org/generated/skfolio.alpha.FixedWeightedAlpha.html.md#skfolio.alpha.FixedWeightedAlpha) normalizes the weights by
their absolute sum, so `[-1.0, -1.0]` assigns an effective weight of
$-0.5$ to each descriptor.
`forecast_scale` converts one composite-score unit into expected
idiosyncratic return. Here, `7.5e-5` represents 0.75 basis points of expected
daily idiosyncratic return per score unit. The alpha forecast should be expressed in
expected-return units when it is combined with expected factor returns or used
in an optimization alongside return-denominated quantities such as transaction
costs, turnover constraints or return targets.
### Alpha Forecast Diagnostics
[`alpha_forecast_evaluation`](https://skfolio.org/generated/skfolio.alpha.alpha_forecast_evaluation.html.md#skfolio.alpha.alpha_forecast_evaluation) fits the estimator on the
enriched training panel and compares each historical forecast with the mean
idiosyncratic return over the next ten trading days. `signal_lag=1` pairs a
forecast observed at $t$ with returns beginning at $t+1$. The
default evaluation step equals the holding period, producing non-overlapping
target windows. `n_forward_periods=4` extends the decay analysis across four
consecutive ten-day windows.
We use regression weights for the Pearson IC, calibration and linear
factor-correlation diagnostics, while the Spearman IC evaluates
cross-sectional rank ordering and does not use them:
```Python
evaluation = alpha_forecast_evaluation(
alpha_estimator,
panel_train_enriched,
holding_period=holding_period,
signal_lag=1,
n_forward_periods=4,
cs_weighting=CSWeighting.REGRESSION,
)
evaluation.ic_summary()
```
[plotly figure stripped from llms output]
Both curves rise steadily with no prolonged flat or negative stretch,
consistent with the high hit rates of the IC summary.
Next, we check how quickly the signal decays. `plot_ic_decay` re-evaluates
each forecast over consecutive, disjoint ten-day windows:
```Python
evaluation.plot_ic_decay()
```
[plotly figure stripped from llms output]
The IC is strongest in the first window and weakens over the following ones.
The latent bearish component is highly persistent, so part of its predictive
power extends beyond the ten-day holding period.
Finally, we check whether the forecast overlaps with the risk factors.
`plot_factor_correlation` shows the contemporaneous cross-sectional
correlation between the raw alpha forecast and each factor exposure:
```Python
evaluation.plot_factor_correlation()
```
[plotly figure stripped from llms output]
All correlations are small, so the forecast is close to factor neutral.
Such small overlaps are not a concern for the portfolio below because the
factor model separates the forecast into spanned alpha and orthogonal alpha
and the optimization constraints keep the portfolio’s factor exposures near
zero. Unwanted tilts can also be removed at the alpha estimator level with
`neutralize_against`.
## Alpha Integration
After defining and evaluating the signal, we attach the alpha estimator to
the factor model:
```Python
model.set_params(alpha_estimator=alpha_estimator)
```
[plotly figure stripped from llms output]
Market and industry exposures are zero, while each style exposure remains
within its $\pm 0.05$ constraint.
We then inspect forecast volatility contributions:
```Python
predicted_attrib.plot_vol_contrib(top_n=15)
```
[plotly figure stripped from llms output]
The idiosyncratic component dominates the risk forecast, with only small
contributions from residual style exposures.
Next, we inspect expected return contributions:
```Python
predicted_attrib.plot_return_contrib(top_n=15)
```
[plotly figure stripped from llms output]
The idiosyncratic component dominates expected return. This is consistent with
the modeled expected return coming primarily from orthogonal alpha. In the previous
tutorial, the idiosyncratic
expected-return contribution was zero because the factor model had no alpha
estimator.
The same decomposition is available as a DataFrame:
```Python
predicted_attrib.summary_df()
```
[plotly figure stripped from llms output]
Next, we check the long, short, net and gross exposures through time. Net
exposure remains zero and gross exposure stays within the 300% cap:
```Python
mpp.plot_long_short_exposure()
```
[plotly figure stripped from llms output]
## Ex-Post Attribution
Now that we have the backtest, let’s check whether realized performance was
concentrated in idiosyncratic returns, as intended by the orthogonal alpha
forecast. `realized_attribution` decomposes the walk-forward portfolio using
realized factor returns, exposures and idiosyncratic returns. For this
descriptive ex-post analysis, we refit the factor model over the completed
sample. This fit occurs after the backtest and does not enter any portfolio
decision:
```Python
model.fit(characteristics=panel)
realized_factor_model = model.factor_model_
realized_attrib = mpp.realized_attribution(factor_model=realized_factor_model)
```
For each factor, we plot the mean realized exposure and its standard
deviation over the backtest:
```Python
realized_attrib.plot_exposure(top_n=15)
```
[plotly figure stripped from llms output]
Mean realized exposures remain close to zero. Their standard deviations
summarize time variation from rebalances, changing factor exposures and
within-period weight drift.
We then inspect realized return contributions:
```Python
fig = realized_attrib.plot_return_contrib(top_n=15)
# show(fig) is only used for the documentation sticker.
show(fig)
```
[plotly figure stripped from llms output]
The error bars show 95% confidence intervals on annualized mean return
contributions. The idiosyncratic return contribution is positive. Residual
factor exposures make a small aggregate contribution. The realized return decomposition is
consistent with the orthogonal alpha forecast.
The summary DataFrame adds `unattributed`, the difference between
observed portfolio returns and model-attributed returns. The portfolio
returns are net of transaction costs while the factor decomposition
explains gross returns, so the cost drag falls into this component, as
would management fees, slippage and cash:
```Python
realized_attrib.summary_df()
```
|
Volatility Contribution |
% of Total Variance |
Mean Return Contribution (95% CI) |
| Component |
|
|
|
| Systematic |
0.08% |
2.56% |
0.17% ± 0.28% |
| Idiosyncratic |
3.05% |
94.89% |
8.37% ± 0.28% |
| Unattributed |
0.08% |
2.55% |
-0.56% |
| Total |
3.21% |
100.00% |
7.99% |
Idiosyncratic returns contribute 8.06% annualized and 94.93% of realized
variance. Systematic factors contribute 0.24%, while the unattributed component
subtracts 0.66%. Total return over the attribution sample is 7.64% annualized.
## Conclusion
We recovered the synthetic idiosyncratic alpha from short interest and analyst
forecast dispersion, integrated it into the factor model and used the orthogonal
alpha in a factor-neutral portfolio.
#### SEE ALSO
The [Alpha Estimators](https://skfolio.org/user_guide/factor_models.html.md#factor-model-alpha) section of the
[Factor Models](https://skfolio.org/user_guide/factor_models.html.md#factor-models) user guide covers the learned
estimators [`EWSharpeOptimalAlpha`](https://skfolio.org/generated/skfolio.alpha.EWSharpeOptimalAlpha.html.md#skfolio.alpha.EWSharpeOptimalAlpha) and
[`PredictorAlpha`](https://skfolio.org/generated/skfolio.alpha.PredictorAlpha.html.md#skfolio.alpha.PredictorAlpha), and the [Portfolio
Construction](https://skfolio.org/user_guide/factor_models.html.md#factor-model-portfolio-construction) section covers
the optimizer conventions and orthogonal-space regularization.
## References
* **[1]** G. A. Paleologo, *The Elements of Quantitative Investing*, Wiley Finance (2025).
* **[2]** H. Desai, K. Ramesh, S. R. Thiagarajan, and B. V. Balachandran, “An Investigation of the Informational Role of Short Interest in the Nasdaq Market”, *The Journal of Finance*, vol. 57, no. 5, pp. 2263-2287 (2002). [doi:10.1111/0022-1082.00495](https://doi.org/10.1111/0022-1082.00495).
* **[3]** K. B. Diether, C. J. Malloy, and A. Scherbina, “Differences of Opinion and the Cross Section of Stock Returns”, *The Journal of Finance*, vol. 57, no. 5, pp. 2113-2141 (2002). [doi:10.1111/0022-1082.00490](https://doi.org/10.1111/0022-1082.00490).
* **[4]** R. C. Grinold and R. N. Kahn, *Active Portfolio Management: A Quantitative Approach for Producing Superior Returns and Controlling Risk*, McGraw-Hill (1999).
**Total running time of the script:** (0 minutes 54.393 seconds)