One of your first thoughts when looking at a stranger’s backtest is probably that it’s overfit, or that there is some look-ahead somewhere.

When you go a step further, you are probably constantly worried about overfitting your own backtests too!

In this article, we will introduce a framework that allows you to identify both! It’s a two-stage approach introduced in D. Nikolopoulos (2026). We will introduce the method, develop the two-stage approach, test the methodology empirically on a crypto dataset, look at some limitations, and also add some improvements of my own!

This article is free to read. You can support me immensely by sharing this article:

Share

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

How Backtests Lie

There are three main failure modes in backtests: Leakage, selection bias, and specification error. Let’s go over them.

Leakage

Leakage is when you are making decisions based on information not yet available at that time. This doesn’t have to be something extreme like using future price data when calculating momentum, or using the current day’s low as a stop loss when still mid-day; it can be more subtle things like normalizing based on the full-sample volatility.

We can describe this mathematically with the help of filtrations. If you are unfamiliar with this concept, please read the following article:

A signal x_t is valid if and only if it is measurable with respect to F_{t-1}:

F_{t-1} reflects everything that is knowable at time t-1, like all past prices p_1, …, p_{t-1}, all past return r_1, …, r_{t-1}, all past volumes, etc.

As time progresses, it only grows:

You never forget information; you only accumulate more.

The full-sample normalization is NOT measurable:

since mu_T and sigma_t depend on observations you haven’t seen yet at time t. The correct expanding window version restores measurability:

Selection Bias

Selection bias, or overfitting, is when you test many different strategies (or configurations of one strategy) and report the best one. Even on a dataset that is not predictable at all, like white noise, this will produce performance better than 0.

Formally, assume returns follow a martingale difference sequence under the null hypothesis:

For each candidate strategy k, we compute its in-sample mean return R, and measure its statistical significance using a HAC t-statistic (A t-statistic that accounts for autocorrelation and heteroskedasticity commonly observed in financial markets):

where V_HAC is the Newey-West variance estimator. For more on Newest-West and HAC t-statistics, read the following article:

Under H_0, Z_{IS,k} is standard normally distributed asymptotically; values above 1.96 should only occur 5% of the time by chance.

But when you search over K configurations and always report the best one:

This grows with K even under H_0:

where gamma (the Euler-Mascheroni constant) is roughly 0.5772 and

This is a result of Extreme Value Theory; you don’t need to memorize the formula.

At K=100, your expected best t-statistic is already around 2.79, well above the 1.96 significance threshold.

Specification Error

Specification error is when your model technically makes money, but you misinterpret why it makes money.

A classic example of this is exposure to factors like market, momentum, liquidity, etc.

Formally, assume your returns follow the following factor model:

where f_t is a known risk factor, beta_t is your exposure to it, and alpha_t is the true time-varying alpha not captured by your factor (You can, of course, extend this to multiple factors). Specification error here would be thinking alpha_t is bigger than it actually is.

The second form of specification error is fitting a transient regime. Your model found a pattern specific to a particular market period or regime that doesn’t reflect a structural inefficiency. It worked in your backtest window, but it won’t persist in live trading.

Our Pipeline

Before going into the stress-test framework, let’s first introduce the pipeline we will test this on. We will explore a moving-average crossover strategy on daily BTCUSD data from CoinAPI, optimized and tested walk-forward with an In-sample of 2 years and an out-of-sample of 6 months.

df = pd.read_csv("C:\\QuantData\\CoinAPI Survivorship Bias Free\\BTC_USD_1DAY_COMPOSITE.csv").set_index("date")["close"]
returns = df.pct_change().dropna()

# --- Signal ---
def ma_signal(returns, short, long):
    prices = (1 + returns).cumprod()
    short_ma = prices.rolling(short).mean()
    long_ma = prices.rolling(long).mean()
    signal = np.where(short_ma > long_ma, 1, -1)
    return pd.Series(signal, index=returns.index)

# --- HAC t-statistic ---
def hac_tstat(rets, max_lag=None):
    x = np.asarray(rets)  # ensures positional behavior
    T = len(x)
    
    if max_lag is None:
        max_lag = int(4 * (T / 100) ** (2/9))
    
    mu = x.mean()
    gamma0 = np.mean((x - mu) ** 2)
    
    V_hac = gamma0
    for l in range(1, max_lag + 1):
        w = 1 - l / (max_lag + 1)
        gamma_l = np.mean((x[l:] - mu) * (x[:-l] - mu))
        V_hac += 2 * w * gamma_l
    
    return mu / np.sqrt(V_hac / T)

# --- Walk-Forward ---
def walk_forward(returns, in_sample_days=365*2, out_sample_days=180):
    short_grid = range(5, 51, 5)
    long_grid = range(20, 201, 20)
    
    results = []
    starts = range(0, len(returns) - in_sample_days - out_sample_days, out_sample_days)
    
    for start in starts:
        r_IS = returns.iloc[start : start + in_sample_days]
        r_OOS = returns.iloc[start + in_sample_days : start + in_sample_days + out_sample_days]
        
        best_z, best_short, best_long = -np.inf, None, None
        for s in short_grid:
            for l in long_grid:
                if s >= l:
                    continue
                sig = ma_signal(r_IS, s, l).shift(1).dropna()
                strat_rets = (r_IS * sig).dropna()
                if len(strat_rets) < 30:
                    continue
                z = hac_tstat(strat_rets)
                if z > best_z:
                    best_z = z
                    best_short, best_long = s, l
        
        sig_OOS = ma_signal(r_OOS, best_short, best_long).shift(1).dropna()
        strat_rets_OOS = (r_OOS * sig_OOS).dropna()
        z_OOS = hac_tstat(strat_rets_OOS)
        
        results.append({
            "start": returns.index[start],
            "best_short": best_short,
            "best_long": best_long,
            "z_IS": best_z,
            "z_OOS": z_OOS,
            "rets_OOS": strat_rets_OOS
        })
    
    return pd.DataFrame(results)

results = walk_forward(returns)
print(results[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

Stage 1: Falsification

The core idea of Stage 1 is simple: If your pipeline finds alpha in data where alpha cannot exist by construction, your pipeline is broken.

We introduce 5 null environments that try to target different failure modes:

Environment A: White Noise

Returns are i.i.d. Gaussian with zero mean:

Then

Annualized volatility target sigma_ann = 0.20, giving

If your pipeline finds alpha here, it is purely selection bias.

Let’s test our moving-average crossover walk-forward pipeline with this:

def generate_env_a(returns, sigma_ann=0.20, seed=None):
    if seed is not None:
        np.random.seed(seed)
    sigma_daily = sigma_ann / np.sqrt(365)
    null_returns = pd.Series(
        np.random.normal(0, sigma_daily, len(returns)),
        index=returns.index
    )
    return null_returns

null_rets = generate_env_a(returns, seed=42)
null_results = walk_forward(null_rets)
print(null_results[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

The results are exactly what we expect: In-sample Z_IS^* values between 0.58 and 0.77 reflect modest inflation from grid search, while out-of-sample Z_OOS values hover around zero. There is no consistent OOS edge.

Environment B: Regime-Switching Volatility

This environment detects predictability arising from improper normalization rules that leak future volatility states, regime leakage, or scale-based trading rules that inadvertently exploit volatility regimes. Let s_t in {1,2} be a two-state Markov chain with transition matrix

Returns are generated as

with epsilon_t independent of F_{t-1}^(r). Therefore E[r_t|F_{t-1}^(r)] = 0.

The paper chooses the following values:

Low-regime annualized volatility sigma_ann = 0.10, high regime volatility multiplier m=3.0, regime presistence p_11 = p_22 = 0.98. Hence

Here’s the implementation:

def generate_env_b(returns, p11=0.98, p22=0.98, sigma_ann_low=0.10, sigma_ann_high=0.30, seed=None):
    if seed is not None:
        np.random.seed(seed)
    sigma_low = sigma_ann_low / np.sqrt(365)
    sigma_high = sigma_ann_high / np.sqrt(365)
    
    T = len(returns)
    states = np.zeros(T, dtype=int)
    states[0] = 0
    for t in range(1, T):
        if states[t-1] == 0:
            states[t] = 0 if np.random.rand() < p11 else 1
        else:
            states[t] = 1 if np.random.rand() < p22 else 0
    
    sigmas = np.where(states == 0, sigma_low, sigma_high)
    null_returns = pd.Series(
        np.random.normal(0, sigmas),
        index=returns.index
    )
    return null_returns

null_rets_b = generate_env_b(returns, seed=42)
results_b = walk_forward(null_rets_b)
print(results_b[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

Environment C: Friction Placebo (Bid-Ask Bounce)

Let u_t be the latent efficient return, the “true” return of the asset satisfying the martingale null (E[u_t | F_{t-1}^(u)] = 0):

The observed return is then contaminated by bid-ask bounce via an MA(1) process:

The negative theta creates negative first-order autocorrelation in observed returns, which are predictable, but NOT tradable! Short-term mean reversion strategies suffer especially hard from this bias.

Innovations are scaled to match a target annualised volatility sigma_ann. Let

so that

The paper picks sigma_ann = 0.20 and theta = -0.5.

If your pipeline profits from this, it has timing errors, indexing bugs, or execution misalignment. Your pipeline isn’t statistically broken, but economically.

Note: We are working with daily prices here, where the bid-ask bounce is typically no longer significant.

Here’s the implementation:

def generate_env_c(returns, theta=-0.5, sigma_ann=0.20, seed=None):
    if seed is not None:
        np.random.seed(seed)
    sigma_daily = sigma_ann / np.sqrt(365)
    sigma_eps = sigma_daily / np.sqrt(1 + theta**2)
    
    T = len(returns)
    u = np.random.normal(0, sigma_eps, T + 1)
    r = u[1:] + theta * u[:-1]
    
    null_returns = pd.Series(r, index=returns.index)
    return null_returnsnull_rets_c = generate_env_c(returns, seed=42)
results_c = walk_forward(null_rets_c)
print(results_c[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

null_rets_c = generate_env_c(returns, seed=42)
results_c = walk_forward(null_rets_c)
print(results_c[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

Environment D: Factor Null (One-Factor, Zero Alpha)

Returns are driven by a mean-zero risk factor with zero alpha:

with

and (f_t, e_t) independent of F_{t-1}^(r). Hence E[r_t |F_{t-1}^(r)] = 0.

The paper’s default calibration is beta = 1.0 and daily volatilities

This catches strategies mistaking beta for alpha. If your pipeline finds alpha here, it’s just harvesting systematic factor exposure rather than a genuine edge.

Here’s the implementation:

def generate_env_d(returns, beta=1.0, sigma_ann_f=0.20, sigma_ann_e=0.10, seed=None):
    if seed is not None:
        np.random.seed(seed)
    sigma_f = sigma_ann_f / np.sqrt(365)
    sigma_e = sigma_ann_e / np.sqrt(365)
    
    T = len(returns)
    f = np.random.normal(0, sigma_f, T)
    e = np.random.normal(0, sigma_e, T)
    r = beta * f + e
    
    null_returns = pd.Series(r, index=returns.index)
    return null_returns

null_rets_d = generate_env_d(returns, seed=42)
results_d = walk_forward(null_rets_d)
print(results_d[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

Environment E: Volatility Clustering Null (GARCH(1,1), Zero Mean)

This environment produces volatility clustering while preserving a zero conditional mean:

Conditional on F_{t-1}^(r), h_t is measurable and E[z_t] = 0, hence E[r_t | F_{t-1}^(r)] = 0.

The paper sets alpha = 0.10, beta = 0.85 (so alpha + beta = 0.95 < 1). They chose omega to match the target daily variance. Let

Then

When volatility is highly persistent (high alpha + beta), your return series has strong conditional heteroskedasticity. The HAC t-statistic assumes the variance estimator converges to the true long-run variance, but with finite OOS windows (6 months in our case) and persistent GARCH dynamics, the Newey-West estimator can underestimate the true variance, making your t-statistic look more significant than it really is. So we might report a significant Z_OOS^* not because we found alpha, but because the GARCH vol clustering causes the standard error to be underestimated, inflating the t-statistic artificially.

Here’s the implementation:

def generate_env_e(returns, alpha=0.10, beta=0.85, sigma_ann=0.20, seed=None):
    if seed is not None:
        np.random.seed(seed)
    sigma_daily = sigma_ann / np.sqrt(365)
    omega = (1 - alpha - beta) * sigma_daily**2
    
    T = len(returns)
    r = np.zeros(T)
    h = np.zeros(T)
    h[0] = sigma_daily**2
    
    z = np.random.normal(0, 1, T)
    for t in range(1, T):
        h[t] = omega + alpha * r[t-1]**2 + beta * h[t-1]
        r[t] = np.sqrt(h[t]) * z[t]
    
    null_returns = pd.Series(r, index=returns.index)
    return null_returns

null_rets_e = generate_env_e(returns, seed=42)
results_e = walk_forward(null_rets_e)
print(results_e[["start", "best_short", "best_long", "z_IS", "z_OOS"]])

Full Stage 1 Run

So far, we’ve only tested one realization per null-environment, which can look good or bad by chance. It’s much better to run on each null environment M (= 100) times and compute Z_OOS^* for each run. The pipeline is falsified for the environment e if the fraction of runs exceeding 1.96 is significantly higher than 5%:

Under the null, this should be ~5% by construction. If it’s materially higher, your pipeline is systematically finding significance in null data, which means it’s broken.

Under the null, each run has a 5% chance of exceeding 1.96. With M=100 runs, the number of rejections follows

The 95th percentile of this distribution gives you the threshold that is exceeded only 5% of the time by chance.

Note: This part differs from how the paper things!

Let’s run this for our pipeline:

from scipy.stats import binom

def run_null_environment(generate_fn, returns, M=100, **kwargs):
    z_IS_list = []
    z_OOS_list = []
    
    for m in range(M):
        null_rets = generate_fn(returns, seed=m, **kwargs)
        results = walk_forward(null_rets)
        all_rets_OOS = pd.concat(results["rets_OOS"].tolist())
        z_star_IS = results["z_IS"].abs().mean()
        z_star_OOS = abs(hac_tstat(all_rets_OOS))
        z_IS_list.append(z_star_IS)
        z_OOS_list.append(z_star_OOS)
    
    return {
        "z_IS_mean": np.mean(z_IS_list),
        "z_IS_std": np.std(z_IS_list),
        "z_OOS_mean": np.mean(z_OOS_list),
        "z_OOS_std": np.std(z_OOS_list),
        "z_OOS_99pct": np.quantile(z_OOS_list, 0.99),
        "rejection_rate": np.mean(np.array(z_OOS_list) > 1.96)
    }

rejection_rate_threshold = binom.ppf(0.95, n=100, p=0.05) / 100

envs = {
    "A: White Noise": (generate_env_a, {}),
    "B: Regime Vol": (generate_env_b, {}),
    "C: Bid-Ask Bounce": (generate_env_c, {}),
    "D: Factor Null": (generate_env_d, {}),
    "E: GARCH": (generate_env_e, {})
}

rows = []
for name, (fn, kwargs) in envs.items():
    print(f"Running {name}...")
    res = run_null_environment(fn, returns, M=100, **kwargs)
    rows.append({"Environment": name, **res})

summary = pd.DataFrame(rows).set_index("Environment")
summary["Falsified"] = summary["rejection_rate"] > rejection_rate_threshold
print(f"\nRejection rate threshold (5% significance): {rejection_rate_threshold:.3f}")
print(summary[["z_IS_mean", "z_IS_std", "z_OOS_mean", "z_OOS_std", "z_OOS_99pct", "rejection_rate", "Falsified"]])

As you can see, our pipeline only has a significantly high rejection rate for the Factor Null, which is to be expected with a strategy like ours and doesn’t mean we have a look-ahead or anything. We can safely proceed to stage 2.

Stage 2: Inflation Quantification

Now that we know that our pipeline itself is not broken, we wanna figure out how inflated the in-sample results are.

Effective Multiplicity K_eff

When you search over K configurations and pick the best one, not all K configurations are truly independent. Two MA crossover strategies with parameters (10, 50) and (10, 60) will produce highly correlated return streams. Treating them as independent searches overstates how much you’ve actually explored the strategy space.

K_eff measures the true dimensionality of your search, how many genuinely independent configurations you actually tried.

Here’s how to compute it:
For each candidate configuration k, collect its in-sample return stream

Stack all K return streams into a T_{IS} x K matrix and compute the K x K correlation matrix Sigma, where entry (i,j) is the correlation between strategy i and strategy j’s returns.

The effective multiplicity is the spectral participation ratio of Sigma:

where lambda_i are the eigenvalues of Sigma. Since Sigma is a correlation matrix, we have tr Sigma = K, so

If all strategies are identical, one eigenvalue equals K, the rest are 0, so K_eff = 1.
If, on the other hand, all strategies are independent, all eigenvalues equal 1, and K_eff = K.

When K is large relative to your in-sample period T_IS, the sample correlation matrix becomes numerically unreliable. You can improve the reliability of your correlation matrix using shrinkage techniques, which I discuss in the following article:

Your expected best in-sample t-statistic under the null is

So K_eff directly determines how inflated your in-sample results are expected to be.

Now, different optimization techniques will yield different K_eff for the same K. A grid search is likely gonna result in a larger K_eff than Bayesian optimization with optuna. Let’s put this to the test. To prevent numerical issues, since we are working with daily data and only have 1477 observations, I will use the full dataset as an IS:

This result is REALLY interesting. Because optuna causes all our candidates to cluster in one space, K_eff actually DECREASES as you increase K. For grid search, it increases as expected. It’s also bigger than Grid Search for small values of K because it’s still in its exploration phase. TPE hasn’t accumulated enough trials to identify the promising region yet, so it samples more diversely across the parameter space. This gives a higher K_eff than grid search, which randomly but uniformly samples. As K grows, Optuna transitions to exploitation; it concentrates trials around the best configs it found, collapsing K_eff.

Our K_eff are also really small; we effectively only test 1-2 unique configurations. This makes sense with our simple moving average crossover strategy, though; there is not much variation across strategies.

Backtest Inflation: Delta Z

We now have a way to measure how many truly independent searches we performed. But we still need to quantify how much our in-sample result was inflated by that search.

The absolute magnitude gap Delta Z measures the divergence between your optimized in-sample evidence and its walk-forward realization:

Z_OOS^* stays bound regardless of how much you searched. Meanwhile, Z_IS^* grows with K_eff, so Delta Z directly captures the inflation induced by your search.

The expected Delta Z under the null, when there is no real alpha, is:

where sqrt(2/pi), around 0.80, is the expected absolute value of a standard normal, which Z_OOS^* should be under the null.

Let’s use this to test our walk-forward pipeline using both optuna and a grid-search for K=50:

def walk_forward_grid(returns, K=50, in_sample_days=365*2, out_sample_days=180):
    short_grid = range(2, 201, 1)
    long_grid = range(10, 1001, 1)
    configs = [(s, l) for s in short_grid for l in long_grid if s < l]
    np.random.seed(42)
    configs = [configs[i] for i in np.random.choice(len(configs), min(K, len(configs)), replace=False)]
    
    results = []
    starts = range(0, len(returns) - in_sample_days - out_sample_days, out_sample_days)
    
    for start in starts:
        r_IS = returns.iloc[start : start + in_sample_days]
        r_OOS = returns.iloc[start + in_sample_days : start + in_sample_days + out_sample_days]
        
        best_z, best_short, best_long = -np.inf, None, None
        for s, l in configs:
            sig = ma_signal(r_IS, s, l).shift(1).dropna()
            strat_rets = (r_IS * sig).dropna()
            if len(strat_rets) < 30:
                continue
            z = hac_tstat(strat_rets)
            if z > best_z:
                best_z = z
                best_short, best_long = s, l
        
        sig_OOS = ma_signal(r_OOS, best_short, best_long).shift(1).dropna()
        strat_rets_OOS = (r_OOS * sig_OOS).dropna()
        z_OOS = hac_tstat(strat_rets_OOS)
        
        results.append({
            "start": returns.index[start],
            "best_short": best_short,
            "best_long": best_long,
            "z_IS": best_z,
            "z_OOS": z_OOS,
            "rets_OOS": strat_rets_OOS
        })
    
    return pd.DataFrame(results)

def walk_forward_optuna(returns, K=50, in_sample_days=365*2, out_sample_days=180):
    results = []
    starts = range(0, len(returns) - in_sample_days - out_sample_days, out_sample_days)
    
    for start in starts:
        r_IS = returns.iloc[start : start + in_sample_days]
        r_OOS = returns.iloc[start + in_sample_days : start + in_sample_days + out_sample_days]
        
        def objective(trial):
            s = trial.suggest_int("short", 2, 200)
            l = trial.suggest_int("long", 10, 1000)
            if s >= l:
                return -np.inf
            sig = ma_signal(r_IS, s, l).shift(1)
            strat_ret = (r_IS * sig).dropna()
            if len(strat_ret) < 30:
                return -np.inf
            return hac_tstat(strat_ret)
        
        study = optuna.create_study(direction="maximize",
                                    sampler=optuna.samplers.TPESampler(seed=42))
        optuna.logging.set_verbosity(optuna.logging.WARNING)
        study.optimize(objective, n_trials=K)
        
        best_short = study.best_params["short"]
        best_long = study.best_params["long"]
        best_z = study.best_value
        
        sig_OOS = ma_signal(r_OOS, best_short, best_long).shift(1).dropna()
        strat_rets_OOS = (r_OOS * sig_OOS).dropna()
        z_OOS = hac_tstat(strat_rets_OOS)
        
        results.append({
            "start": returns.index[start],
            "best_short": best_short,
            "best_long": best_long,
            "z_IS": best_z,
            "z_OOS": z_OOS,
            "rets_OOS": strat_rets_OOS
        })
    
    return pd.DataFrame(results)

results_grid = walk_forward_grid(returns, K=50)
results_optuna = walk_forward_optuna(returns, K=50)

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

for ax, (name, res) in zip(axes, [("Grid Search", results_grid), ("Optuna", results_optuna)]):
    delta_z = res["z_IS"] - res["z_OOS"]
    x = range(len(res))
    
    ax.bar(x, res["z_IS"], label="$Z^*_{IS}$", alpha=0.7)
    ax.bar(x, res["z_OOS"], label="$Z^*_{OOS}$", alpha=0.7)
    ax.axhline(0, color="black", linewidth=0.8)
    ax.set_xticks(list(x))
    ax.set_xticklabels(res["start"].tolist(), rotation=45, ha="right")
    ax.set_title(f"{name} — $\\Delta Z$ = {delta_z.mean():.2f}")
    ax.set_ylabel("$Z$")
    ax.legend()

plt.tight_layout()
plt.show()

Both produce pretty bad strategies OOS. Let’s compare for different values of K now:

def theoretical_z_IS(keff):
    a = np.sqrt(2 * np.log(2 * keff)) - (np.log(np.log(2 * keff)) + np.log(4 * np.pi)) / (2 * np.sqrt(2 * np.log(2 * keff)))
    b = 1 / np.sqrt(2 * np.log(2 * keff))
    gamma = 0.5772
    return a + gamma * b

all_grid_results = {}
all_optuna_results = {}
all_keff_grid = {}
all_keff_optuna = {}

for K in K_values:
    print(f"K={K}...")
    all_grid_results[K] = walk_forward_grid(returns, K=K)
    all_optuna_results[K] = walk_forward_optuna(returns, K=K)
    all_keff_grid[K] = compute_keff(get_grid_search_returns(returns.iloc[:365*2], K))
    all_keff_optuna[K] = compute_keff(get_optuna_returns(returns.iloc[:365*2], K))

fig, axes = plt.subplots(4, 2, figsize=(12, 16))

for fold_idx in range(4):
    ax_z = axes[fold_idx, 0]
    ax_dz = axes[fold_idx, 1]

    grid_z_IS = [all_grid_results[K]["z_IS"].iloc[fold_idx] for K in K_values]
    optuna_z_IS = [all_optuna_results[K]["z_IS"].iloc[fold_idx] for K in K_values]
    theoretical_grid = [theoretical_z_IS(all_keff_grid[K]) for K in K_values]
    theoretical_optuna = [theoretical_z_IS(all_keff_optuna[K]) for K in K_values]
    grid_delta_z = [all_grid_results[K]["z_IS"].iloc[fold_idx] - all_grid_results[K]["z_OOS"].iloc[fold_idx] for K in K_values]
    optuna_delta_z = [all_optuna_results[K]["z_IS"].iloc[fold_idx] - all_optuna_results[K]["z_OOS"].iloc[fold_idx] for K in K_values]

    ax_z.plot(K_values, grid_z_IS, marker="o", label="Grid Search")
    ax_z.plot(K_values, optuna_z_IS, marker="s", label="Optuna (TPE)")
    ax_z.plot(K_values, theoretical_grid, linestyle="--", color="blue", label="Theoretical (Grid)")
    ax_z.plot(K_values, theoretical_optuna, linestyle="--", color="orange", label="Theoretical (Optuna)")
    ax_z.set_xlabel("Nominal $K$")
    ax_z.set_ylabel("$Z^*_{IS}$")
    ax_z.set_title(f"Fold {fold_idx + 1} — $Z^*_{{IS}}$")
    ax_z.legend()

    ax_dz.plot(K_values, grid_delta_z, marker="o", label="Grid Search")
    ax_dz.plot(K_values, optuna_delta_z, marker="s", label="Optuna (TPE)")
    ax_dz.set_xlabel("Nominal $K$")
    ax_dz.set_ylabel("$\\Delta Z$")
    ax_dz.set_title(f"Fold {fold_idx + 1} — $\\Delta Z$")
    ax_dz.legend()

plt.tight_layout()
plt.show()

Your goal when designing a parameter optimization pipeline is to minimize Delta Z, while maximizing Z_OOS^*.

One decision rule for rejecting our pipeline could be:

Run M (= 100) white noise simulations, compute Delta Z for each, take the 99th percentile as your threshold, and if your observed Delta Z on real data exceeds this, falsify the pipeline.

Limitations

Sparse signals won’t be detected

If you have a strategy that rarely trades, like an event-based strategy, the walk-forward window simply won’t contain enough activations to reach significance even if the signal is genuine.

Passing doesn’t guarantee live performance

The audit is a stress test. A strategy can pass all 5 null environments, have a small Delta Z, and still lose money in live trading due to regime changes, liquidity constraints, or market impact. The audit only tells you your pipeline isn’t obviously broken.

Only for return prediction

The framework, as implemented, is designed for directional strategies with a well-defined trading lag. It doesn’t directly extend to portfolio optimization, classification tasks, volatility forecasting, or multi-asset strategies without adaptation.

Conclusion

Most quant researchers stress test their strategies manually and sporadically. This falsification framework can be permanently built into your research pipeline, running automatically every time you evaluate a new strategy or retune an existing one. Think of it like unit tests in software engineering: you don’t run them once before shipping, you run them on every commit.

Concretely, this means: every time your walk-forward backtest completes, Stage 1 and Stage 2 run automatically alongside it. Your research dashboard shows not just the equity curve and Sharpe ratio, but K_eff, Delta Z, and the Stage 1 rejection rates for all 5 environments. A strategy only advances to paper trading if it passes all steps, not because it looked good in-sample.

Join Quant Corner: https://discord.gg/X7TsxKNbXg

Everyone talks about alpha decay, but no one tells you how to actually quantify it. In this article, we’ll figure out how to do so and build a warning system that tells you when it’s time to pull the plug on a strategy.

We do this by implementing Minimum Regime Performance (MRP), a framework introduced by Alexander and Fabozzi (2026) that measures how a strategy holds up across structurally distinct market regimes. We will apply this method to a universe of crypto factors.

Beyond signal monitoring, MRP is also relevant in portfolio construction. Just as you would punish strategy tail risk like Value at Risk, you can also punish strategies that have a high tendency to decay.

The article will use the factors discussed in the following article:

VertoxQuant

The Myth of Factor-Free Crypto

People keep telling you to take what works in equities and apply it to crypto…

Read more

3 months ago · 9 likes · Vertox

Knowledge of regime switching models is NOT required.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

• 1 referral → 20% off

• 5 referrals → 1 month free

• 25 referrals → 6 months free

If you’ve gotten value from it, help me grow by recommending it. It costs you nothing!

You can find your personal referral link at the bottom of this email or right here:

Leaderboard

Thanks!

-Vertox

In this article, we will look at some useful model-agnostic methods (meaning that it doesn’t matter what model we apply it to) for figuring out why one specific prediction is what it is (Local Methods) and why predictions are what they are on average (Global Methods).

This is important for a few very important reasons.

1. Detecting problems in the model:
   Say you built a model that forecasts raw returns, and you see that the error has a lower error than some baseline model. A lot of people would quit here, but you dig deeper and find out that the only feature that matters to the model is volatility, and that the only thing that the model learned to do well is predict the magnitude of the return, not the sign. Suddenly, the model is completely useless for what it was supposed to do!

2. Regulatory & risk management requirements:

   Regulators increasingly require that automated trading and risk models be explainable. If your model flags a position as too risky or generates a trade signal, compliance needs to know why. A black box doesn’t pass that bar.

3. Detecting data leakage:

   If your model is suspiciously good, interpretability tells you what it’s actually learning. Example: a return-prediction model that learned to exploit a look-ahead bias in a feature, e.g., using end-of-day prices to predict intraday moves, shows up immediately as one feature with overwhelming importance.

4. Feature engineering feedback loop:

   Once you know which features are actually driving predictions, you can iterate intelligently; drop redundant factors, identify which alpha signals are being ignored, or understand why a signal that worked in research is being suppressed by the model in production.

5. Regime analysis:

   Local methods let you compare why the model made a prediction in a bull market vs. a crisis period. If the feature importances shift dramatically across regimes, that’s a red flag for out-of-sample robustness; your model may have overfit to one regime’s dynamics.

More interpretable models are often less accurate, and model-agnostic post-hoc methods let you have both: full model flexibility and explanations.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

1. Local Model-agnostic Methods

Local methods answer a specific question: why did the model predict this value for this observation? Rather than summarising model behaviour across the entire dataset, they explain a single prediction by attributing it to the input features of that one data point.

Example: Your model predicts a large positive return for a particular asset on a particular day. A global method might tell you that momentum is generally the most important feature in your model. But a local method tells you that for this specific prediction, the signal was driven almost entirely by a spike in order flow imbalance, while momentum actually pushed the prediction slightly lower. That distinction matters enormously when you're trying to understand a model's live behaviour, debug an unexpected signal, or explain a specific trade to a risk committee.

Let’s go over some of the most popular methods.

1.1 Ceteris Paribus Plots

The name sounds really complicated, but it’s actually the simplest method you could use. Ceteris Paribus is Latin for "all other things being equal". As the name implies, to analyze a certain feature, you simply change its value while keeping all other features unchanged and look at the resulting predictions.

Before looking more into this method, let us train a model that we will keep using as an example for all methods. We will do a 1-day-ahead prediction of SPY returns using some basic features like momentum, RSI, etc. Doesn’t matter if the model is garbage as long as we can run our analysis on it.

First we load our data:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import yfinance as yf

def load_spy():
    df = yf.download("SPY", start="2010-01-01", end="2026-03-27", auto_adjust=True)
    df.columns = df.columns.droplevel(1)

    df["ret"] = df["Close"].pct_change()
    df["target"] = df["ret"].shift(-1)  # next-day return

    # Momentum
    df["mom_5"]   = df["ret"].rolling(5).mean()
    df["mom_20"]  = df["ret"].rolling(20).mean()
    df["mom_60"]  = df["ret"].rolling(60).mean()

    # Volatility
    df["vol_5"]   = df["ret"].rolling(5).std()
    df["vol_20"]  = df["ret"].rolling(20).std()
    df["vol_ratio"] = df["vol_5"] / df["vol_20"]  # vol regime indicator

    # Mean reversion
    df["rsi_14"]  = _rsi(df["Close"], 14)
    df["zscore_20"] = (df["Close"] - df["Close"].rolling(20).mean()) / df["Close"].rolling(20).std()

    # Volume
    df["volume_z"] = (df["Volume"] - df["Volume"].rolling(20).mean()) / df["Volume"].rolling(20).std()

    # Price-based
    df["hl_range"] = (df["High"] - df["Low"]) / df["Close"]  # normalized daily range
    df["gap"]      = (df["Open"] - df["Close"].shift(1)) / df["Close"].shift(1)  # overnight gap

    # Autocorrelation signal
    df["ret_lag1"] = df["ret"].shift(1)
    df["ret_lag2"] = df["ret"].shift(2)

    features = [
        "mom_5", "mom_20", "mom_60",
        "vol_5", "vol_20", "vol_ratio",
        "rsi_14", "zscore_20",
        "volume_z", "hl_range", "gap",
        "ret_lag1", "ret_lag2"
    ]

    df = df[["target"] + features].dropna()
    return df

def _rsi(close, period=14):
    delta = close.diff()
    gain = delta.clip(lower=0).rolling(period).mean()
    loss = (-delta.clip(upper=0)).rolling(period).mean()
    rs = gain / loss
    return 100 - (100 / (1 + rs))

df = load_spy()
df

And now let’s train an XGBoost model to predict the next day’s return using our features (We won’t be doing any train-test splits or other important steps here, this is purely an exercise model!):

from xgboost import XGBRegressor
from sklearn.metrics import r2_score

FEATURES = [c for c in df.columns if c != "target"]
X = df[FEATURES]
y = df["target"]
 
model = XGBRegressor(n_estimators=300, max_depth=4, learning_rate=0.05,
                     subsample=0.8, colsample_bytree=0.8, random_state=42)
model.fit(X, y)

preds = model.predict(X)
df["forecast"] = preds
df[["target", "forecast"]].to_csv("forecasts.csv")

Price         target  forecast
Date                          
2010-03-31  0.006837  0.000646
2010-04-01  0.008149  0.000556
2010-04-05  0.002358 -0.000046
2010-04-06 -0.005712  0.000146
2010-04-07  0.003464  0.001303

Back to Ceteris Paribus.

Here is an implementation of it:

def cp_profile(model, X, obs_idx, feature, n_points=100):
    x_obs = X.iloc[[obs_idx]].copy()
    grid = np.linspace(X[feature].min(), X[feature].max(), n_points)
    preds = []
    for val in grid:
        x_obs[feature] = val
        preds.append(model.predict(x_obs)[0])
    return grid, np.array(preds)

Let’s try it on the 2026-03-03 prediction for the features "mom_20", "vol_20", "rsi_14", "zscore_20":

Price
target        0.007055
mom_5        -0.002033
mom_20       -0.001063
mom_60       -0.000015
vol_5         0.006754
vol_20        0.008314
vol_ratio     0.812388
rsi_14       39.089484
zscore_20    -1.367428
volume_z      1.027870
hl_range      0.019035
gap          -0.016492
ret_lag1      0.000569
ret_lag2     -0.004802
forecast      0.001172
Name: 2026-03-03 00:00:00, dtype: float64

obs_idx = X.index.get_loc("2026-03-03")
features_to_plot = ["mom_20", "vol_20", "rsi_14", "zscore_20"]

fig, axes = plt.subplots(len(features_to_plot), 1, figsize=(9, 14))
fig.suptitle("Ceteris Paribus Profiles — SPY XGBoost (2026-03-01)", fontsize=14, fontweight="bold")

for ax, feat in zip(axes, features_to_plot):
    grid, preds = cp_profile(model, X, obs_idx, feat)
    ax.plot(grid, preds, linewidth=1.8)
    ax.axvline(X.iloc[obs_idx][feat], color="red", linestyle="--", label="observed value")
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feat, fontsize=11)
    ax.set_ylabel("Predicted return", fontsize=10)
    ax.legend(fontsize=9)
    ax.grid(True, alpha=0.3)

plt.tight_layout()

If momentum were even more negative, we would have predicted an even more positive return (mean reversion effect). If volatility were higher, we would have flipped to strongly negative instead (flipping from mean reversion to momentum).

Many of the methods we will look at are creative combinations of multiple Ceteris Paribus plots that tell us more.

The BIG problem with this (and many other) methods is correlation among features.
Changing one feature while keeping the others unchanged can give us unrealistic feature combinations, like huge recent momentum while volatility is very low.

1.2 Individual Conditional Expectation (ICE)

An ICE plot is simply many Ceteris Paribus plots overlaid, one per observation.

Here is an implementation:

def ice_plot(model, X, feature, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42)
    grid = np.linspace(X[feature].min(), X[feature].max(), n_points)
    
    fig, ax = plt.subplots(figsize=(9, 4))
    for idx in range(len(sample)):
        _, preds = cp_profile(model, sample.reset_index(drop=True), idx, feature, n_points)
        ax.plot(grid, preds, color="steelblue", alpha=0.4, linewidth=0.8)
    
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"ICE Plot — {feature}")
    plt.tight_layout()

Let’s run it for “mom_20”:

ice_plot(model, X, "mom_20")

We can explore interactions with this method by using colors.

def ice_plot(model, X, feature, color_by=None, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42).reset_index(drop=True)
    grid = np.linspace(X[feature].quantile(0.01), X[feature].quantile(0.99), n_points)

    fig, ax = plt.subplots(figsize=(9, 4))
    for idx in range(len(sample)):
        _, preds = cp_profile(model, sample, idx, feature, n_points)
        if color_by is not None:
            median = sample[color_by].median()
            color = "#d62728" if sample[color_by].iloc[idx] > median else "#1f77b4"
        else:
            color = "steelblue"
        ax.plot(grid, preds, color=color, alpha=0.4, linewidth=0.8)

    if color_by is not None:
        ax.plot([], [], color="#d62728", label=f"{color_by} high")
        ax.plot([], [], color="#1f77b4", label=f"{color_by} low")
        ax.legend(fontsize=9)

    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"ICE Plot — {feature}" + (f" | colored by {color_by}" if color_by else ""))
    plt.tight_layout()

Let’s color in low/high volatility:

ice_plot(model, X, "mom_20", "vol_20")

Centered ICE Plot:

It can be hard to tell if the different curves are affected equally by the feature of interest because they have different levels.

A simple solution is to subtract each curve's value at some specific value of x. This removes the vertical spread caused by other features and lets you focus purely on the effect of varying the feature of interest.

def cp_profile(model, X, obs_idx, feature, grid):
    x_obs = X.iloc[[obs_idx]].copy()
    preds = [model.predict(x_obs.assign(**{feature: val}))[0] for val in grid]
    return np.array(preds)

def ice_plot(model, X, feature, color_by=None, centered=False, ref_value=None, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42).reset_index(drop=True)
    grid = np.linspace(X[feature].quantile(0.01), X[feature].quantile(0.99), n_points)
    if centered and ref_value is not None:
        grid = np.sort(np.append(grid, ref_value))  # ensure ref_value is exactly in grid

    fig, ax = plt.subplots(figsize=(9, 4))
    for idx in range(len(sample)):
        preds = cp_profile(model, sample, idx, feature, grid)
        if centered:
            ref = ref_value if ref_value is not None else grid[0]
            ref_pred = preds[np.searchsorted(grid, ref)]
            preds = preds - ref_pred
        color = "steelblue"
        if color_by is not None:
            color = "#d62728" if sample[color_by].iloc[idx] > sample[color_by].median() else "#1f77b4"
        ax.plot(grid, preds, color=color, alpha=0.3, linewidth=0.8)

    if color_by is not None:
        ax.plot([], [], color="#d62728", label=f"{color_by} high")
        ax.plot([], [], color="#1f77b4", label=f"{color_by} low")
        ax.legend(fontsize=9)

    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Change in predicted return" if centered else "Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"{'c-ICE' if centered else 'ICE'} Plot — {feature}" + (f" | colored by {color_by}" if color_by else ""))
    plt.tight_layout()

A natural reference point for momentum is 0:

ice_plot(model, X, "mom_20", "vol_20", centered=True, ref_value=0)

Can see a pretty clear pattern on the left! When volatility is high, momentum works. When volatility is low, mean reversion works.

1.3 Local interpretable model-agnostic explanations (LIME)

CP profiles and ICE show you how the prediction changes as you vary one feature. But they don’t give you a single number answering: how much did feature x contribute to this specific prediction? LIME does exactly that.

The basic idea is that our black box method f is too complex to interpret globally, so we interpret near a specific observation x (all our features at one specific date) by training a simpler model (for example linear) in a small neighborhood of x.

Here is how LIME does this exactly (using linear regression as an example for the interpretable model):

1. Perturbing the observation:
   Generate N synthetic samples by randomly perturbing the features around x. For tabular data, this is done by sampling each feature from its marginal distribution. For continuous data, this is done by sampling from a normal distribution with equal means and standard deviations to the training set (This again ignores correlations! A typical weakness of LIME).

2. Weighing by proximity:
   Each synthetic sample z is weighted by how close it is to x using an exponential kernel:

   In our case, d will be euclidian distance, but other measures of distance can be chosen too.

3. Fitting a weighted linear model:
   Run the black-box model on all synthetic samples to get labels, then fit a weighted linear regression:

4. Reading the explanation:
   Now we interpret the resulting model. In the case of linear regression, that’s easy, as we simply look at the coefficients.

Let’s implement this with ridge regression, which handles our correlated features better:

def lime_explain(model, X, obs_idx, n_samples=1000, sigma=1.0, ridge_alpha=1.0):
    x_obs = X.iloc[obs_idx].values
    means = X.mean().values
    stds  = X.std().values

    # 1. perturb
    Z = np.random.normal(means, stds, size=(n_samples, X.shape[1]))

    # 2. weights
    Z_std = (Z - x_obs) / stds
    distances = np.sqrt((Z_std ** 2).sum(axis=1))
    weights = np.exp(-distances ** 2 / sigma ** 2)

    # 3. black-box labels
    labels = model.predict(Z)

    # 4. fit weighted linear model
    lr = Ridge(alpha=ridge_alpha)
    lr.fit(Z, labels, sample_weight=weights)

    return dict(zip(X.columns, lr.coef_))

And run it again for our 2026-03-03 forecast:

obs_idx = X.index.get_loc("2026-03-03")
explanation = lime_explain(model, X, obs_idx)
print(explanation)

mom_5       -2.096855e-09
mom_20      -4.942316e-09
mom_60       4.846428e-09
vol_5        4.087705e-08
vol_20       1.238100e-08
vol_ratio    5.578009e-07
rsi_14      -1.130377e-05
zscore_20   -2.909305e-06
volume_z    -1.143078e-06
hl_range    -4.011315e-08
gap          2.016699e-08
ret_lag1     1.007680e-08
ret_lag2     5.326466e-08

Another challenge in using LIME is choosing the neighborhood of x. There is no clear solution to this, and results can generally be unstable as well.

1.4 SHAP

SHAP (SHapley Additive exPlanations) is a game-theoretic approach.

The question SHAP asks is: Given that the model made a prediction hat{y}, how do we fairly distribute the difference between this prediction and the average prediction E[y] among the features?

Think of the features as players in a cooperative game, where the "payout" is the model's prediction. The Shapley value of feature j is its average marginal contribution across all possible orderings (coalitions) of features:

where F is the full set of features, S is a subset excluding feature j, and \hat{f}_S is the model’s prediction using only features in S.

The Shapley values must satisfy the following four properties (and is the only attribution method that does so):

Efficiency:

so the contributions exactly decompose the prediction relative to the baseline.

Symmetry:

If two features j and k contribute equally to every coalition, i.e., for all

we have

then they must have the same Shapley value

Dummy:

If feature j contributes nothing to any coalition, i.e., for all

we have

then the Shapley value must be zero

Linearity:

If the model can be decomposed as f = f^A + f^B, then

Estimating Shapley values:

The number of possible coalitions increases exponentially as you increase the number of features in your model (2^|F|). You therefore typically want to approximate it via Monte-Carlo sampling. Instead of summing over all coalitions, sample random permutations pi of the features and estimate the marginal contribution of j in each:

where S^m is the set of features appearing before j in the m-th random permutation. This converges to the true Shapley value as M → infinity.

You want M to be reasonably big, so the variance of your estimated Shapley value is small. This means that for big models doing a SHAP analysis can take a long time, which is one major drawback of it.

For tree-based models (XGBoost, LightGBM, Random Forests, etc.), there is an exact polynomial-time algorithm that exploits the tree structure to compute exact Shapley values in O(TLD^2) time, where T is the number of trees, L is the number of leaves, and D is the maximum depth.

Let’s implement this with the shap library. There are a ton of useful tools!

import shap
import copy

explainer  = shap.TreeExplainer(model)
shap_values = explainer(X)

obs_idx = X.index.get_loc("2026-03-03")

# Transforming to basis-points for better readability
sv_bps = copy.deepcopy(shap_values)
sv_bps.values = shap_values.values * 10000
sv_bps.base_values = shap_values.base_values * 10000
sv_bps.data = shap_values.data

waterfall_plot:

Shows how each feature pushes the prediction up/down from the baseline E[f(.)] to the final prediction f(x).

shap.waterfall_plot(sv_bps[obs_idx])

As you can see, momentum pushed the prediction in one direction, but volume_z and RSI pushed it back in the other direction.

force_plot:

Same information as waterfall, but horizontal.

shap.initjs()
shap.force_plot(sv_bps[obs_idx])

decision_plot:

Also gives the same information by showing the cumulative sum of SHAP values as a path from the baseline E[f(.)] to the final prediction.

shap.decision_plot(explainer.expected_value, shap_values[obs_idx].values, X.iloc[obs_idx])

You can also pass multiple observations to compare them side by side:

shap.decision_plot(explainer.expected_value, shap_values.values[:50], X.iloc[:50])

We will get back to SHAP again when looking at global model-agnostic methods!

2. Global Model-agnostic Methods

Global methods answer a different question: not why did the model predict this value for this observation, but how does the model behave on average across the entire dataset? Rather than explaining a single prediction, they summarise the overall relationship between features and the target learned by the model.

Example: You’ve trained a return-forecasting model on SPY, and it performs well out of sample. A local method tells you why the model predicted a large positive return on one specific day. But a global method tells you that across all predictions, volatility is the dominant driver, momentum matters mainly in low-volatility regimes, and the overnight gap feature is essentially ignored by the model.

Let’s go over some of the most popular methods.

2.1 Partial Dependence Plot (PDP)

Remember Individual Conditional Expectation (ICE)? PDP is simply the average of all ICE curves. This also means the PDP inherits the main weakness of ICE: It assumes the feature is independent of the others.

Just like with ICE, a centered version also exists. Let’s implement it as a thick line inside our centered ICE plot:

def ice_plot(model, X, feature, color_by=None, centered=False, ref_value=None, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42).reset_index(drop=True)
    grid = np.linspace(X[feature].quantile(0.01), X[feature].quantile(0.99), n_points)
    if centered and ref_value is not None:
        grid = np.sort(np.append(grid, ref_value))

    fig, ax = plt.subplots(figsize=(9, 4))
    all_preds = []
    for idx in range(len(sample)):
        preds = cp_profile(model, sample, idx, feature, grid)
        if centered:
            ref = ref_value if ref_value is not None else grid[0]
            ref_pred = preds[np.searchsorted(grid, ref)]
            preds = preds - ref_pred
        all_preds.append(preds)
        color = "steelblue"
        if color_by is not None:
            color = "#d62728" if sample[color_by].iloc[idx] > sample[color_by].median() else "#1f77b4"
        ax.plot(grid, preds, color=color, alpha=0.3, linewidth=0.8)

    pdp = np.mean(all_preds, axis=0)
    ax.plot(grid, pdp, color="black", linewidth=2.5, label="PDP")

    if color_by is not None:
        ax.plot([], [], color="#d62728", label=f"{color_by} high")
        ax.plot([], [], color="#1f77b4", label=f"{color_by} low")

    ax.legend(fontsize=9)
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Change in predicted return" if centered else "Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"{'c-ICE' if centered else 'ICE'} Plot — {feature}" + (f" | colored by {color_by}" if color_by else ""))
    plt.tight_layout()

ice_plot(model, X, "mom_20", "vol_20", centered=True, ref_value=0)

2.2 Accumulated Local Effects (ALE)

PDP’s main weakness is that it sweeps feature x_j across its range while holding other features fixed at their observed values, generating unrealistic feature combinations when features are correlated. ALE fixes this by only looking at the effect of x_j locally, within small neighbourhoods where the feature combinations are realistics, and then accumulating these local effects to get a global picture.

Partition the range of x_j into K intervals [z_{k-1}, z_k]. For each interval, compute the local effect of x_j by averaging the finite difference of predictions over observations that actually fall in that interval:

where K(x) is the index of the inerval containing x, and n_k is the number of observations in interval k. x_{-j}^(i), z_k means that the j-th feature value in x^(i) got replaced with z_k (here i stays for the i-th observation).

Let’s implement this:

def ale_plot(model, X, feature, n_bins=20):
    quantiles = np.quantile(X[feature], np.linspace(0, 1, n_bins + 1))
    quantiles = np.unique(quantiles)

    ale = np.zeros(len(quantiles) - 1)
    bin_centers = np.zeros(len(quantiles) - 1)

    for k in range(len(quantiles) - 1):
        mask = (X[feature] >= quantiles[k]) & (X[feature] < quantiles[k + 1])
        if mask.sum() == 0:
            continue
        X_bin = X[mask].copy()
        preds_high = model.predict(X_bin.assign(**{feature: quantiles[k + 1]}))
        preds_low  = model.predict(X_bin.assign(**{feature: quantiles[k]}))
        ale[k] = (preds_high - preds_low).mean()
        bin_centers[k] = (quantiles[k] + quantiles[k + 1]) / 2

    ale = np.cumsum(ale)
    ale -= ale.mean()

    fig, ax = plt.subplots(figsize=(9, 4))
    ax.plot(bin_centers, ale, color="steelblue", linewidth=2)
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("ALE")
    ax.grid(True, alpha=0.3)
    plt.title(f"ALE Plot — {feature}")
    plt.tight_layout()

And run it for “mom_20”:

2.3 Friedman’s H statistic

So far, all methods look at the effect of one feature at a time. But in practice, features often interact; the effect of mom_20 on the prediction might be very different in high-volatility vs low-volatility regimes. Friedman’s H-statistic measures the strength of such interactions.

If two features x_j and x_k do not interact, then the point partial dependence can be decomposed as the sum of the individual partial dependences:

The H-statistic measures how much of the joint PDP’s variance is not explained by the sum of the individual PDPs:

H_{jk}^2 is in [0,1], where 0 means no interaction and 1 means the entire joint effect is due to interaction.

There is also a total interaction statistic for feature j measuring its interaction with all other features simultaneously:

Let’s implement both:

from sklearn.inspection import partial_dependence

def h_stat_pairwise(model, X, feat1, feat2, n_samples=500):
    X_s = X.sample(n_samples, random_state=42)
    
    pd12 = partial_dependence(model, X_s, [feat1, feat2], kind="average", grid_resolution=20)
    pd1  = partial_dependence(model, X_s, [feat1], kind="average", grid_resolution=20)
    pd2  = partial_dependence(model, X_s, [feat2], kind="average", grid_resolution=20)
    
    grid1 = pd12["grid_values"][0]
    grid2 = pd12["grid_values"][1]
    joint = pd12["average"][0]
    
    joint -= joint.mean()
    v1 = pd1["average"][0] - pd1["average"][0].mean()
    v2 = pd2["average"][0] - pd2["average"][0].mean()
    
    numerator, denominator = 0.0, 0.0
    for _, row in X_s.iterrows():
        f12 = joint[np.argmin(np.abs(grid1 - row[feat1])), np.argmin(np.abs(grid2 - row[feat2]))]
        f1  = v1[np.argmin(np.abs(pd1["grid_values"][0] - row[feat1]))]
        f2  = v2[np.argmin(np.abs(pd2["grid_values"][0] - row[feat2]))]
        numerator   += (f12 - f1 - f2) ** 2
        denominator += f12 ** 2
    
    return np.sqrt(numerator / denominator)

def h_stat_total(model, X, feature, n_samples=500):
    X_s = X.sample(n_samples, random_state=42).reset_index(drop=True)
    
    f_hats = model.predict(X_s)
    
    # PD_j: marginalise out all features except j (standard 1D PDP)
    pd_j = partial_dependence(model, X_s, [feature], kind="average", grid_resolution=20)
    grid_j = pd_j["grid_values"][0]
    v_j = pd_j["average"][0]

    # PD_{-j}: for each observation, fix x_{-j} and average over x_j
    pd_minus_j = np.zeros(len(X_s))
    for idx in range(len(X_s)):
        X_temp = X_s.copy()
        X_temp[feature] = X_s[feature]  # vary x_j over training dist
        X_temp.loc[:, [c for c in X_s.columns if c != feature]] = \
            X_s.iloc[idx][[c for c in X_s.columns if c != feature]].values
        pd_minus_j[idx] = model.predict(X_temp).mean()

    numerator, denominator = 0.0, 0.0
    for idx, (_, row) in enumerate(X_s.iterrows()):
        fj    = v_j[np.argmin(np.abs(grid_j - row[feature]))]
        f_hat = f_hats[idx]
        numerator   += (f_hat - fj - pd_minus_j[idx]) ** 2
        denominator += f_hat ** 2

    return np.sqrt(numerator / denominator)

from tqdm import tqdm
import seaborn as sns

features = X.columns.tolist()
h_matrix = np.zeros((len(features), len(features)))

pairs = [(i, j, f1, f2) for i, f1 in enumerate(features) for j, f2 in enumerate(features) if i < j]

for i, j, f1, f2 in tqdm(pairs, desc="Pairwise H"):
    h = h_stat_pairwise(model, X, f1, f2)
    h_matrix[i, j] = h
    h_matrix[j, i] = h

total_h = []
for f in tqdm(features, desc="Total H"):
    total_h.append(h_stat_total(model, X, f))

features_sorted, total_h_sorted = zip(*sorted(zip(features, total_h), key=lambda x: x[1]))

fig, axes = plt.subplots(1, 2, figsize=(16, 5))

sns.heatmap(h_matrix, xticklabels=features, yticklabels=features,
            annot=True, fmt=".2f", cmap="YlOrRd", ax=axes[0],
            linewidths=0.5, cbar_kws={"label": "H-statistic"}, vmin=0, vmax=1)
axes[0].set_title("Pairwise H-statistics", fontsize=13, fontweight="bold")

axes[1].barh(features_sorted, total_h_sorted, color="steelblue")
axes[1].axvline(0, color="black", linewidth=0.8)
axes[1].set_xlabel("H-statistic")
axes[1].set_title("Total H-statistics", fontsize=13, fontweight="bold")
axes[1].grid(True, alpha=0.3, axis="x")

plt.tight_layout()

This suffers from the typical drawback of all interpretable machine learning methods: It’s computationally expensive. It also only tells us the strength of the interactions, not what those interactions look like. A good workflow is to measure the interaction strengths with this method and then examine the strongest interactions in more detail using other methods (like specific SHAP plots that we will get to soon).

2.4 SHAP

We already covered how SHAP works so I will only showcase the plots that focus on global effect here.

beeswarm:

Shows the distribution of SHAP values for every feature across all observations.
Each dot is one observation, color indicates the feature value (red=high, blue=low).
Features are sorted by mean absolute SHAP value.

Can see here that very negative momentum causes very SHAP values (mean reversion).

mean absolute SHAP:

Mean absolute SHAP value per feature (global feature importance).
Simpler than beeswarm, good for reporting which features matter most on average.

shap.summary_plot(sv_bps, X, plot_type="bar")

Dependence Plots:

For each feature, it plots phi_j vs x_j across all observations.
Color is automatically chosen as the feature that interacts most with x_j, revealing how the effect of x_j changes depending on another feature's value.

for feat in X.columns:
    shap.dependence_plot(feat, sv_bps.values, X, interaction_index="auto")

Here is a notable ones:

When momentum is very negative and volume is very low, SHAP values become positive. The other ones didn’t have obvious interactions:

Headmap:

Shows phi_j for every feature j and every observation i simultaneously, features as rows and observations as columns. Each cell is colored by the SHAP value: red means the feature pushed the prediction up, blue means it pushed it down.

shap.plots.heatmap(sv_bps)

2.5 Permutation Feature Importance (PFI)

We already know about mean absolute SHAP as a general measure of how important a feature is in a model. PFI is another such singular value. If a feature x_j is important, then randomly shuffling its values across observations will destroy its relationship with the target and cause the model’s performance to degrade. If it’s unimportant, shuffling it changes nothing.

We first fit the model and compute a baseline performance metric L (e.g. MSE, R^2, IC) on the dataset. Then, for each feature j, we randomly permute the values of x_j across all observations, leaving all other features unchanged. We recompute the performance metric L^(j) on the permuted dataset, and the importance of feature j becomes

A large Delta L_j means the model relied heavily on x_j, and destroying it hurt performance a lot.

Since permutation is random, it’s standard to repeat this K times and average:

This is much faster to compute than SHAP and is directly tied to model performance, which is what we ultimately care about.

One BIG disadvantage is that if two features are highly correlated, then shuffling one leaves the other intact, and therefore both will receive a lot of permutation importance, even if they are super important to the model performance.

from sklearn.inspection import permutation_importance

result = permutation_importance(model, X, y, n_repeats=30, random_state=42, scoring="r2")

sorted_idx = result.importances_mean.argsort()

fig, ax = plt.subplots(figsize=(8, 5))
ax.barh(X.columns[sorted_idx], result.importances_mean[sorted_idx], 
        xerr=result.importances_std[sorted_idx], color="steelblue", capsize=3)
ax.axvline(0, color="black", linewidth=0.8)
ax.set_xlabel("Mean decrease in $R^2$")
ax.set_title("Permutation Feature Importance", fontsize=13, fontweight="bold")
ax.grid(True, alpha=0.3, axis="x")
plt.tight_layout()

2.6 Leave One Feature Out (LOFO) Importance

Instead of permuting a feature and evaluating with the same model, we completely drop a feature and retrain the model without it. LOFO Importance then measures the performance drop compared to the full model:

This is extremely useful for feature selection, but it is incredibly expensive and suffers from the same issue that highly correlated features get low LOFO importance. If you remove features one at a time, that problem becomes less relevant, as when one of them drops, the other increases in importance and won’t be dropped again.

Let’s implement this with R^2 as our performance metric:

from sklearn.metrics import r2_score

baseline = r2_score(y, model.predict(X))

loo_importance = {}
for feat in tqdm(features, desc="LOO"):
    X_loo = X.drop(columns=[feat])
    m = XGBRegressor(n_estimators=300, max_depth=4, learning_rate=0.05,
                     subsample=0.8, colsample_bytree=0.8, random_state=42)
    m.fit(X_loo, y)
    loo_importance[feat] = baseline - r2_score(y, m.predict(X_loo))

sorted_feats, sorted_vals = zip(*sorted(loo_importance.items(), key=lambda x: x[1]))

fig, ax = plt.subplots(figsize=(8, 5))
ax.barh(sorted_feats, sorted_vals, color="steelblue")
ax.axvline(0, color="black", linewidth=0.8)
ax.set_xlabel("Decrease in $R^2$")
ax.set_title("Leave-One-Out Feature Importance", fontsize=13, fontweight="bold")
ax.grid(True, alpha=0.3, axis="x")
plt.tight_layout()

3. Conclusion

I’ve never really written about ML before. If you enjoyed this article and would like to read more about ML, please let me know!

Join Quant Corner: https://discord.gg/X7TsxKNbXg

If you enjoyed this article, please share it!

Share

If you want to read more, consider subscribing!

This article provides a rigorous, hands-on walkthrough of portfolio optimization methods used in both research and industry. Rather than focusing on theory alone, we emphasize how these approaches behave in practice; what they optimize, which constraints matter, and the trade-offs they impose.

After briefly introducing the minimal notation, we move directly into the major families of portfolio construction techniques, including mean–variance, risk-based, and hierarchical approaches. The goal is not just to present these methods, but to understand when and why they work (or fail), especially in noisy and high-volatility environments.

Prerequisite (recommended):
This article assumes basic familiarity with Markowitz Mean–Variance Optimization (MVO). For full derivations of the efficient frontier, the tangency portfolio, and a detailed discussion of estimation error and its practical remedies (e.g., shrinkage, factor models, Black–Litterman, and robust optimization), see:

Notation and Setup

Consider n risky assets with (random) single-period returns r ∈ ℝⁿ. A portfolio is represented by weights w ∈ ℝⁿ. In the simplest setting, the portfolio return is

Let the expected returns be

and the covariance matrix be

Then the portfolio’s expected return and variance are

A standard budget constraint is

where the left 1 is the all-ones vector in ℝⁿ. Additional constraints may include long-only constraints w >= 0, leverage limits |w|_1 <= L, box constraints l <= w <= u, factor exposure constraints, and turnover constraints. Note: if w >= 0 and 1^⊤ w = 1, then |w|_1 = 1 is constant; l_1 leverage constraints/penalties matter primarily for long/short portfolios, active weights a = w - w_b, or turnover terms |wt - w_(t-1)|_1.

To set us up let’s first import the necessary libraries and set plot styles:

import os
import glob
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from scipy import linalg
from scipy import optimize
from scipy.optimize import linprog
from scipy.cluster.hierarchy import linkage, to_tree, dendrogram
from scipy.spatial.distance import squareform

# Plot style defaults.
plt.rcParams.update(
    {
        “figure.figsize”: (10, 6),
        “axes.grid”: True,
        “grid.alpha”: 0.3,
        “axes.titlesize”: 14,
        “axes.labelsize”: 12,
        “legend.fontsize”: 10,
    }
)

We don’t wanna be trading stablecoins, so we’ll exclude them:

STABLE_SYMBOLS = {
    “USDT”, “USDC”, “USDS”, “USDE”, “DAI”, “PYUSD”, “USD1”, “XAUT”, “USDF”,
    “PAXG”, “USDG”, “RLUSD”, “BFUSD”, “USDD”, “USDTB”, “USD0”, “FDUSD”, “A7A5”,
    “GHO”, “TUSD”, “CUSD”, “USDB”, “USR”, “KAU”, “EURC”, “CRVUSD”, “KAG”,
    “BUSD”, “USX”, “FRAX”, “USDC.N”, “YLDS”, “USDA”, “AUSD”, “DUSD”, “SATUSD”,
    “GUSD”, “EURS”, “DOLA”, “FRXUSD”, “PUSD”, “AIDAUSDC”, “USDZ”, “AVUSD”, “MNEE”,
    “USDR”, “PGOLD”, “CASH”, “EURCV”, “SUSDA”, “USDO”, “FEUSD”, “USDX”, “YZUSD”,
    “MIM”, “USDKG”, “REUSD”, “USDM”, “USD+”, “USDP”, “YUSD”, “XUSD”, “BOLD”,
    “HYUSD”, “LISUSD”, “USTC”, “USDN”, “LUSD”, “FXUSD”, “HBD”, “XTUSD”, “HONEY”,
    “EUSD”, “ZCHF”, “MIMATIC”, “USN”, “SUSD”, “EURE”, “BUCK”, “MUSD”, “EURA”,
    “GYD”, “USDH”, “MSUSD”, “UTY”, “WEMIX$”, “YU”, “HOLLAR”, “ALUSD”, “XSGD”,
    “EURR”,
}

Next, some useful helper function:

def _normalize_symbol(symbol: str) -> str:
    return “”.join(ch for ch in str(symbol).upper() if ch.isalnum())


def _base_symbol_from_ticker(ticker: str) -> str:
    t = str(ticker).upper()
    suffix = “_USD_1DAY_COMPOSITE”
    if t.endswith(suffix):
        return t[:-len(suffix)]
    return t.split(”_”)[0]


STABLE_SYMBOLS_NORM = {_normalize_symbol(s) for s in STABLE_SYMBOLS}

Whenever data isn’t available for a coin, we’ll be using synthetic data for demonstration purposes:

# Synthetic fallback data.
def synthetic_crypto_like_returns(
    n_assets: int = 12,
    n_obs: int = 1200,
    seed: int = 42,
    df_t: float = 5.0,
    regime_switch_prob: float = 0.02,
) -> pd.DataFrame:

    rng = np.random.default_rng(seed)
    dates = pd.date_range(”2020-01-01”, periods=n_obs, freq=”D”)

    # Build three correlation clusters.
    n_clusters = 3
    cluster_sizes = [n_assets // n_clusters] * n_clusters
    for i in range(n_assets % n_clusters):
        cluster_sizes[i] += 1
    clusters = np.concatenate([[k] * sz for k, sz in enumerate(cluster_sizes)])

    base_corr = np.eye(n_assets)
    for i in range(n_assets):
        for j in range(n_assets):
            if i == j:
                continue
            base_corr[i, j] = 0.55 if clusters[i] == clusters[j] else 0.15

    # Add symmetric jitter and keep the matrix positive definite.
    jitter = rng.normal(scale=0.03, size=(n_assets, n_assets))
    jitter = 0.5 * (jitter + jitter.T)
    corr = base_corr + jitter
    np.fill_diagonal(corr, 1.0)

    eig = np.linalg.eigvalsh(corr)
    if eig.min() <= 1e-6:
        corr += (abs(eig.min()) + 1e-3) * np.eye(n_assets)

    # Give each asset a different daily volatility.
    base_vol = rng.uniform(0.02, 0.08, size=n_assets)
    cov = np.outer(base_vol, base_vol) * corr
    L = np.linalg.cholesky(cov)

    # Switch between low- and high-volatility regimes.
    vol_mult = np.ones(n_obs)
    state = 0
    for t in range(1, n_obs):
        if rng.random() < regime_switch_prob:
            state = 1 - state
        vol_mult[t] = 1.0 if state == 0 else 2.0

    # Student-t shocks via normal / sqrt(chi2/df).
    Z = rng.standard_normal(size=(n_obs, n_assets))
    chi2 = rng.chisquare(df=df_t, size=n_obs) / df_t
    t_innov = Z / np.sqrt(chi2)[:, None]

    rets = (t_innov @ L.T) * vol_mult[:, None]
    cols = [f”Asset_{i:02d}“ for i in range(n_assets)]
    return pd.DataFrame(rets, index=dates, columns=cols)

The function for loading actual data (We are using CoinAPI):

# Loader for real data files.
def load_crypto_dataset_from_dir(
    data_dir: str = “/Users/admin/Documents/Projects/PortfolioOptimization/data”,
    max_assets: int = 100,
    min_obs: int = 365,
    coverage_threshold: float = 0.98,
    selection_metric: str = “mean_volume”,  # ‘mean_volume’ or ‘n_obs’
    verbose: bool = True,
):

    if not os.path.isdir(data_dir):
        raise FileNotFoundError(f”Directory not found: {data_dir}“)

    csv_files = sorted(glob.glob(os.path.join(data_dir, “*.csv”)))
    if len(csv_files) == 0:
        raise FileNotFoundError(f”No CSV files found in {data_dir}“)

    records = []
    price_series = {}
    excluded_stables = []

    for fp in csv_files:
        sym = os.path.splitext(os.path.basename(fp))[0].upper()
        base_sym = _base_symbol_from_ticker(sym)
        if _normalize_symbol(base_sym) in STABLE_SYMBOLS_NORM:
            excluded_stables.append(base_sym)
            continue
        try:
            df = pd.read_csv(fp)
        except Exception as e:
            if verbose:
                print(f”[WARN] Skipping {fp} (read error: {e})”)
            continue

        if “date” not in df.columns or “close” not in df.columns:
            if verbose:
                print(f”[WARN] Skipping {fp} (missing required columns ‘date’/’close’)”)
            continue

        keep_cols = [c for c in df.columns if c in (”date”, “close”, “volume_usd”)]
        df = df.loc[:, keep_cols].copy()

        df[”date”] = pd.to_datetime(df[”date”], errors=”coerce”)
        df[”close”] = pd.to_numeric(df[”close”], errors=”coerce”)
        df = df.dropna(subset=[”date”, “close”]).sort_values(”date”)
        df = df.drop_duplicates(subset=[”date”], keep=”last”)

        if df.shape[0] < min_obs:
            continue

        s = df.set_index(”date”)[”close”].astype(float)
        price_series[sym] = s

        mean_volume = float(df[”volume_usd”].astype(float).mean()) if “volume_usd” in df.columns else np.nan
        records.append(
            {
                “symbol”: sym,
                “n_obs”: int(df.shape[0]),
                “start”: df[”date”].min(),
                “end”: df[”date”].max(),
                “mean_volume_usd”: mean_volume,
                “file”: fp,
            }
        )

    if len(price_series) < 2:
        raise ValueError(f”Insufficient usable assets in {data_dir} after filtering (min_obs={min_obs}).”)

    info = pd.DataFrame(records).set_index(”symbol”)

    # Pick the sorting column.
    if selection_metric == “mean_volume” and info[”mean_volume_usd”].notna().any():
        sort_by = “mean_volume_usd”
    else:
        sort_by = “n_obs”

    info = info.sort_values(by=sort_by, ascending=False, na_position=”last”)
    selected = list(info.index[:max_assets])

    prices = pd.concat({sym: price_series[sym] for sym in selected}, axis=1).sort_index()
    rets = np.log(prices).diff()

    # Filter assets again after return coverage is computed.
    coverage = rets.notna().mean()
    keep_cols = coverage[coverage >= coverage_threshold].index.tolist()
    prices = prices[keep_cols]
    rets = rets[keep_cols]
    info = info.loc[keep_cols]

    before_rows = rets.shape[0]
    rets = rets.dropna(how=”any”)  # intersection of dates
    prices = prices.loc[rets.index]
    after_rows = rets.shape[0]

    if verbose:
        if excluded_stables:
            print(f”[DATA] Excluded {len(set(excluded_stables))} stable symbols before selection.”)
        print(f”[DATA] Loaded {len(keep_cols)} assets from {data_dir} (sorted by {sort_by}).”)
        print(f”[DATA] Date range: {rets.index.min().date()} → {rets.index.max().date()} ({after_rows} daily obs).”)
        print(
            “[DATA] Missing-data policy: drop any date with any missing return across selected assets “
            f”(kept {after_rows}/{before_rows} rows after alignment).”
        )

    if after_rows < 120:
        raise ValueError(”Not enough aligned observations after missing-data handling.”)

    return prices, rets, info

The following function computes returns from price data:

def get_crypto_returns(
    data_dir: str = “/Users/admin/Documents/Projects/PortfolioOptimization/data”,
    max_assets: int = 100,
    min_obs: int = 365,
    coverage_threshold: float = 0.98,
    seed_fallback: int = 42,
    verbose: bool = True,
):

    try:
        prices, rets, info = load_crypto_dataset_from_dir(
            data_dir=data_dir,
            max_assets=max_assets,
            min_obs=min_obs,
            coverage_threshold=coverage_threshold,
            verbose=verbose,
        )
        source = “real”
    except Exception as e:
        if verbose:
            print(f”[FALLBACK] Using synthetic data because real /Users/admin/Documents/Projects/PortfolioOptimization/data load failed: {e}“)
        prices = None
        rets = synthetic_crypto_like_returns(n_assets=max_assets, n_obs=1200, seed=seed_fallback)
        info = pd.DataFrame({”symbol”: rets.columns, “n_obs”: len(rets), “mean_volume_usd”: np.nan}).set_index(”symbol”)
        source = “synthetic”
    return prices, rets, info, source

Next, some helpful mathematical helper functions:

# Basic math helpers.
def regularize_cov(S: np.ndarray, eps: float = 1e-8) -> np.ndarray:
    “”“Symmetrize and add a small diagonal jitter for numerical stability.”“”
    S = np.asarray(S)
    S = 0.5 * (S + S.T)
    return S + eps * np.eye(S.shape[0])

def annualized_return(log_r: pd.Series, periods_per_year: int = 365) -> float:
    “”“Geometric annualized return from log returns.”“”
    years = len(log_r) / periods_per_year
    return float(np.exp(log_r.sum() / years) - 1.0) if years > 0 else np.nan

def annualized_vol(log_r: pd.Series, periods_per_year: int = 365) -> float:
    return float(log_r.std(ddof=1) * np.sqrt(periods_per_year))

def sharpe_ratio(log_r: pd.Series, rf_annual: float = 0.0, periods_per_year: int = 365) -> float:
    rf_log = np.log1p(rf_annual) / periods_per_year
    ex = log_r - rf_log
    vol = ex.std(ddof=1) * np.sqrt(periods_per_year)
    mean = ex.mean() * periods_per_year
    return float(mean / vol) if vol > 0 else np.nan

def max_drawdown(log_r: pd.Series) -> float:
    equity = np.exp(log_r.cumsum())
    peak = equity.cummax()
    dd = equity / peak - 1.0
    return float(dd.min())

def empirical_cvar(log_r: np.ndarray, alpha: float = 0.95) -> float:
    “”“
    Empirical CVaR of *losses* using simple returns:
      loss = - (exp(log_r) - 1)
    “”“
    simple = np.expm1(log_r)
    losses = -simple
    k = max(1, int(np.ceil((1 - alpha) * len(losses))))
    tail = np.sort(losses)[-k:]
    return float(tail.mean())

The following data loads our data and does some quick diagnostics:

# Load real data if available, otherwise use synthetic data.
crypto_prices, crypto_returns, asset_info, DATA_SOURCE = get_crypto_returns(
    data_dir=”/Users/admin/Documents/Projects/PortfolioOptimization/data”,
    max_assets=40,
    min_obs=365,
    coverage_threshold=0.98,
    seed_fallback=42,
    verbose=True,
)

print(f”\n[INFO] Data source: {DATA_SOURCE}“)
print(f”[INFO] Returns shape: {crypto_returns.shape[0]} dates × {crypto_returns.shape[1]} assets”)
print(f”[INFO] Assets: {list(crypto_returns.columns)}“)
print(”\n[INFO] Asset info (head):”)
print(asset_info.head())

# Quick diagnostics before the experiments.
pooled = crypto_returns.stack()
print(f”\n[DIAGNOSTIC] Pooled daily log-return mean: {pooled.mean():.6f}“)
print(f”[DIAGNOSTIC] Pooled daily log-return std : {pooled.std(ddof=1):.6f}“)
print(f”[DIAGNOSTIC] Pooled excess kurtosis      : {pooled.kurt():.2f}“)

# Plot pooled return histogram and correlation heatmap.
fig, ax = plt.subplots()
ax.hist(pooled.values, bins=80, density=True)
ax.set_title(”Pooled daily log returns (all assets)”)
ax.set_xlabel(”log return”)
ax.set_ylabel(”density”)
plt.tight_layout()
plt.show()

corr = crypto_returns.corr().values
fig, ax = plt.subplots()
im = ax.imshow(corr, aspect=”auto”)
ax.set_title(”Correlation matrix (aligned returns)”)
ax.set_xticks(range(crypto_returns.shape[1]))
ax.set_yticks(range(crypto_returns.shape[1]))
ax.set_xticklabels(crypto_returns.columns, rotation=90)
ax.set_yticklabels(crypto_returns.columns)
plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
plt.tight_layout()
plt.show()

[DATA] Excluded 17 stable symbols before selection.
[DATA] Loaded 18 assets from /Users/admin/Documents/Projects/PortfolioOptimization/data (sorted by mean_volume_usd).
[DATA] Date range: 2022-01-04 → 2026-01-16 (1472 daily obs).
[DATA] Missing-data policy: drop any date with any missing return across selected assets (kept 1472/1477 rows after alignment).

[INFO] Data source: real
[INFO] Returns shape: 1472 dates × 18 assets
[INFO] Assets: [’BTC_USD_1DAY_COMPOSITE’, ‘ETH_USD_1DAY_COMPOSITE’, ‘SOL_USD_1DAY_COMPOSITE’, ‘XRP_USD_1DAY_COMPOSITE’, ‘DOGE_USD_1DAY_COMPOSITE’, ‘ADA_USD_1DAY_COMPOSITE’, ‘FTM_USD_1DAY_COMPOSITE’, ‘RUNE_USD_1DAY_COMPOSITE’, ‘LINK_USD_1DAY_COMPOSITE’, ‘AVAX_USD_1DAY_COMPOSITE’, ‘SHIB_USD_1DAY_COMPOSITE’, ‘CFX_USD_1DAY_COMPOSITE’, ‘LTC_USD_1DAY_COMPOSITE’, ‘XLM_USD_1DAY_COMPOSITE’, ‘HBAR_USD_1DAY_COMPOSITE’, ‘CAKE_USD_1DAY_COMPOSITE’, ‘DYDX_USD_1DAY_COMPOSITE’, ‘XMR_USD_1DAY_COMPOSITE’]

[INFO] Asset info (head):
                         n_obs      start        end  mean_volume_usd  \
symbol                                                                  
BTC_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     1.168389e+09   
ETH_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     6.636419e+08   
SOL_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     2.891621e+08   
XRP_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     2.603798e+08   
DOGE_USD_1DAY_COMPOSITE   1477 2022-01-01 2026-01-16     1.068017e+08   

                                                                      file  
symbol                                                                      
BTC_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
ETH_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
SOL_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
XRP_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
DOGE_USD_1DAY_COMPOSITE  /Users/admin/Documents/Projects/PortfolioOptim...  

[DIAGNOSTIC] Pooled daily log-return mean: -0.000667
[DIAGNOSTIC] Pooled daily log-return std : 0.049747
[DIAGNOSTIC] Pooled excess kurtosis      : 13.75

1. Baseline: MVO Recap

We use MVO as the reference point for the rest of this notebook. For full derivations and numerical examples, see the following article:

1.1 Canonical formulations

(MVO-1) Minimum variance at target return:

(MVO-2) Penalized mean–variance utility:

Varying μ_p in (MVO-1) or γ in (MVO-2) traces the (constraint-dependent) efficient frontier: the set of portfolios that are not dominated in (σ, E[r]) space.

1.2 Two anchor portfolios

Global minimum variance (GMV) (unconstrained):

With a risk-free rate r_f and excess returns μ_e = μ - r_f 1, the tangency / maximum Sharpe portfolio (unconstrained) is:

We construct a simple synthetic market with a random covariance matrix and expected returns, allowing us to study the geometry of mean–variance optimization in a controlled setting.

Using closed-form solutions, we compute the global minimum variance (GMV) portfolio and the tangency portfolio (maximum Sharpe ratio). We then sweep across target returns to trace out the efficient frontier, which shows the best achievable return for each level of risk.

# Synthetic efficient frontier example.
rng = np.random.default_rng(0)

n = 6
A = rng.normal(size=(n, n))
Sigma = A @ A.T  # SPD covariance
mu = rng.normal(loc=0.0005, scale=0.0010, size=n)  # daily expected log returns

ones = np.ones(n)
Sigma = regularize_cov(Sigma, eps=1e-10)
Sigma_inv = linalg.inv(Sigma)

# GMV portfolio.
w_gmv = Sigma_inv @ ones / (ones @ Sigma_inv @ ones)
mu_gmv = float(w_gmv @ mu)
vol_gmv = float(np.sqrt(w_gmv @ Sigma @ w_gmv))

# Tangency portfolio (rf=0).
w_tan_unnorm = Sigma_inv @ mu
w_tan = w_tan_unnorm / (ones @ w_tan_unnorm)
mu_tan = float(w_tan @ mu)
vol_tan = float(np.sqrt(w_tan @ Sigma @ w_tan))

# Closed-form frontier with budget and target-return constraints.
A_ = mu @ Sigma_inv @ mu
B_ = mu @ Sigma_inv @ ones
C_ = ones @ Sigma_inv @ ones
D_ = A_ * C_ - B_**2

# Sweep target returns around GMV.
targets = np.linspace(mu_gmv - 1.5 * np.std(mu), mu_gmv + 1.5 * np.std(mu), 80)
frontier_vol = []
for mu_p in targets:
    # Frontier variance at target return mu_p.
    var = (C_ * mu_p**2 - 2.0 * B_ * mu_p + A_) / D_
    frontier_vol.append(np.sqrt(var))

frontier_vol = np.array(frontier_vol)

# Plot volatility vs expected return.
fig, ax = plt.subplots()
ax.plot(frontier_vol, targets, label=”Efficient frontier (unconstrained)”)
ax.scatter([vol_gmv], [mu_gmv], label=”GMV”, zorder=3)
ax.scatter([vol_tan], [mu_tan], label=”Tangency (rf=0)”, zorder=3)

ax.set_title(”Markowitz efficient frontier geometry (synthetic μ, Σ)”)
ax.set_xlabel(”Portfolio volatility  (σ)”)
ax.set_ylabel(”Expected portfolio return  (E[r])”)
ax.legend()
plt.tight_layout()
plt.show()

print(”[SYNTHETIC SUMMARY]”)
print(f”GMV:     E[r]={mu_gmv:.6f},  σ={vol_gmv:.6f},  sum(w)={w_gmv.sum():.3f}“)
print(f”Tangency E[r]={mu_tan:.6f},  σ={vol_tan:.6f},  sum(w)={w_tan.sum():.3f}“)
print(”\nGMV weights:”, np.round(w_gmv, 4))
print(”Tan weights:”, np.round(w_tan, 4))

[SYNTHETIC SUMMARY]
GMV:     E[r]=0.001123,  σ=0.344942,  sum(w)=1.000
Tangency E[r]=0.002090,  σ=0.470481,  sum(w)=1.000

GMV weights: [ 0.2473  0.3281 -0.0938 -0.0937 -0.0412  0.6533]
Tan weights: [ 0.1141  0.7469 -0.174  -0.1761 -0.512   1.0012]

1.3 Assumptions (why the baseline is clean, and why it breaks)

MVO is exact under normality or quadratic utility (more generally, elliptical return families) in a single-period setting with known (μ,Σ). In practice, (μ,Σ) are estimated, markets are non-stationary, and constraints/transaction costs are unavoidable. The rest of this notebook is largely about methods that either (i) reduce dependence on fragile estimates, (ii) change the risk objective, or (iii) impose structure to improve out-of-sample stability.

2. How We’ll Compare Portfolio Optimization Methods in Practice

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

Most models are trained to predict individual asset returns.

That seems reasonable. But for cross-sectional strategies, prediction accuracy is often the wrong objective.

Cross-sectional portfolios turn model outputs into trades by ranking assets at each rebalance date and going long the top names while shorting the bottom. Once trades are determined by rank, the absolute level of forecasts often becomes irrelevant: any monotone transformation of model scores produces the same portfolio.

In other words, getting the ordering right matters more than predicting returns precisely.

This creates a subtle but important mismatch. A model can achieve excellent pointwise accuracy (low RMSE) while producing a weak, or even negative, long–short spread. Conversely, a model with poor regression metrics may generate strong cross-sectional performance.

However, ranking is not always all that matters. When position sizing, risk budgeting, transaction cost models, or portfolio optimizers depend on forecast magnitudes and calibration, pointwise accuracy becomes important again.

In this article, we develop Learning to Rank (LTR) as a principled framework for cross-sectional strategies.

The argument proceeds in three steps:

1. We formalize the cross-sectional decision problem and show how the score-to-weight mapping determines which properties of model output actually affect PnL.

2. We demonstrate with numerical examples how low RMSE can coexist with negative long–short spreads, and vice versa.

3. We derive practical LTR objectives for statistical arbitrage, including label engineering, pair weighting, neutralization, and cost-aware evaluation.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

Real limit order books are discrete: all displayed prices lie on a grid with tick size > 0, and, especially in “large-tick” assets, the bid–ask spread is frequently pinned at one tick.
This changes the economics of liquidity provision (where rents come from, how they are competed away, and how adverse selection manifests) and it changes the mathematics of optimal control (from smooth optimization to discrete choice, hybrid/impulse control, and state augmentation for queue position).

This article provides a rigorous, pedagogical deep-dive that starts from the canonical continuous-time market-making framework and then explains, in detail, why and how tick size and queue priority alter optimal quoting.
We then survey the spectrum of modeling and control approaches used in practice: exact discrete-state dynamic programming, queue-aware Markov models, continuous relaxations with discrete corrections, and pragmatic heuristics used when the exact discrete control problem is computationally intractable.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

In this article, we’re gonna be covering more advanced regime detection and regime switching algorithms that go beyond HMM, which nobody talks about.

Read more

Mean–Variance Optimization (MVO) is a central framework for portfolio construction: choose weights that balance expected return against risk as measured by variance.

In its classical form, MVO is elegant, convex (under mild conditions), and analytically tractable. Yet practitioners quickly encounter a paradox: the mathematically “optimal” portfolio built from estimated inputs is often unstable, highly leveraged (explicitly or implicitly), and disappoints out-of-sample.

This is not a minor implementation detail; it is a structural consequence of combining a high-dimensional optimizer with noisy estimates of expected returns and covariances.

This article develops MVO from first principles and then explains, in a mathematically explicit way, why raw MVO tends to maximize estimation error.

Finally, it surveys the spectrum of practical fixes, organized around two levers: (i) improving or regularizing the inputs (expected returns and covariances), and (ii) constraining or regularizing the optimizer (the feasible set and the objective).

The unifying theme is that almost every successful “fix” works by injecting bias in exchange for a large reduction in variance of the resulting portfolio weights, thereby improving out-of-sample performance and implementability.

The Classical Framework

Notation

Consider N risky assets. Let the random vector of (excess) returns over a single period be

Define the (unknown) population mean and covariance

A portfolio is a weight vector interpreted as a fraction of capital invested in each asset.

The basic budget constraint is

where 1 denotes the all-ones vector in R^N. In other words, the weights should sum up to 1. Additional constraints like no-short, leverage limits, or sector bounds will be presented in the following chapters.

Under this setup, the portfolio return is linear in weights:

The expected portfolio return and variance are

Two foundational facts are worth stating explicitly because they explain why MVO becomes a quadratic program:

1. Expected return is linear in weights. This makes the “reward” side easy to compute but also extremely sensitive to errors in mu, because the optimizer can exploit tiny differences in mu via large weight changes.

2. Variance is quadratic in weights. The covariance matrix Sigma couples assets through correlations: diversification is precisely the exploitation of off-diagonal terms in Sigma. The quadratic form w^T Sigma w is convex in w if Sigma is positive semidefinite, and strictly convex if Sigma is positive definite, which ensures the uniqueness of the minimum-variance solution under typical linear constraints.

The Markowitz Problem in Matrix Form

Markowitz’s original formulation can be stated as: among all portfolios with a given expected return, choose the one with minimum variance. Fix a target expected return m. The constrained optimization is

If short-selling is disallowed, one adds w >= 0 componentwise. If leverage is limited, one might add |w|_1 <= L, where |.| denotes the L1 norm, and so on.

Why is the covariance matrix central here? Because for any two assets i and j,

The diagonal terms w_i^2 Sigma_{ii} represent contributions from each asset’s variance; the off-diagonal terms w_i w_j Sigma_{ij} represent interaction through co-movement. Diversification is not “holding many assets” per se; it is selecting weights so that positive and negative interactions among returns reduce overall variance.

A subtle but important point: variance is a second-moment object. It treats positive and negative deviations symmetrically and is fully described by Sigma. This makes MVO analytically convenient, but it also means the framework inherits all limitations of second-moment risk measures; non-normality, fat tails, and asymmetry are not captured unless the distribution is (approximately) elliptical. In many institutional contexts, however, variance remains a useful proxy because it aligns with tracking error, volatility targets, and risk budgeting infrastructure.

The penalized (risk-aversion) form and its equivalence

An equivalent way to pose the trade-off is to maximize a mean-variance utility function:

where gamma > 0 is the risk-aversion parameter. Larger gamma penalized variance more heavily, shifting the solution toward lower-risk portfolios.

The equivalence between (MVO-1) and (MVO-2) is practically important. The constrained form (MVO-1) traces the efficient frontier by varying m. The penalized form (MVO-2) traces it by varying gamma. In many production systems, gamma is tuned to meet a risk target or tracking error budget.

Solving the classical problem: Lagrangian and closed-form structure

To make mechanics concrete, consider (MVO-2) without additional constraints beyond the budget constraint. Form the Lagrangian

with Lagrange multiplier eta enforcing 1^T w = 1.

First-order optimality (assuming Sigma is positive definite) gives

hence

Imposing the budget constraint 1^T w = 1 yields

so

Substituting (1.2) into (1.1) gives the explicit optimizer.

This expression reveals a key structural fact that will matter later: the optimal weights are built from Sigma^{-1} mu and Sigma^{-1} 1. In other words, the inverse covariance matrix is the central operator transforming expected returns into weights. When Sigma^{-1} is unstable (ill-conditioned or poorly estimated), the entire solution becomes unstable.

For the return-target form (MVO-1), the Lagrangian with multiplies lambda, eta is

The first-order condition is

Define the classical scalars

Assuming Sigma is positive definite and mu is not collinear with 1 under Sigma^{-1}, one has Delta > 0. Solving for lambda, eta yields a closed-form frontier, and the efficient frontier in (sigma^2, m) space is a parabola:

The frontier’s curvature and location are completely determined by (A,B,C), i.e., by Sigma^{-1} and mu. This already hints at the practical challenge: every point on the frontier depends on inverting Sigma and multiplying by mu, precisely the operations most vulnerable to estimation noise.

Let’s implement this in Python and look at the resulting efficient frontier. We will also compare our closed-form solution of (MVO-1) to a numerical solution to verify that we did everything right.

As you can see, our closed-form solution and numerical solution are pretty much identical, and the (MVO-2) solutions lie on the efficient frontier traced by the (MVO-1) solution, so they are indeed equivalent.

Interpreting Sigma: Risk Geometry and Diversification

It is useful to interpret the quadratic form geometrically. If Sigma is positive definite, then the set of portfolios with equal variance sigma^2,

is an ellipsoid in weight space. The optimizer in (MVO-2) chooses the point on the budget hyperplane {1^T w = 1} that maximizes a linear functional w^T mu minus a quadratic penalty. The optimum balances moving “up” in the direction of mu while staying within low-risk ellipsoids determined by Sigma.

This picture is clean when mu and Sigma are known. The moment we replace them with estimates, the ellipsoids tilt and stretch unpredictably, and the direction of “up” becomes noisy. The optimizer, being deterministic, will still choose an extreme point, often an extreme point of the wrong geometry. That is the beginning of the “error maximization” story.

Here is an example using two assets:

Each ellipse here corresponds to portfolios with equal variance. As you can see, our optimal portfolio just barely touches the 0.03 variance ellipsoid, and any other portfolio on the line (that satisfies the budget constraint) results in a higher variance.

The “Error Maximization” Problem

Raw MVO is often described informally as “garbage in, garbage out.” That statement is true, but it understates the severity: MVO does not merely propagate input error; it can amplify it. In high dimensions, the amplification can be dramatic enough that the optimizer effectively learns the noise in the estimated inputs.

This section makes that mechanism explicit.

MVO is not an optimization problem; it is a statistical decision problem

In theory, (mu, Sigma) are population quantities. In practice, we never observe mu or Sigma. We observe a finite time series

and produce estimators hat{mu} and hat{Sigma}. The most common “plug-in” estimators are the sample mean and covariance

Then the raw MVO portfolio is

or its return-target equivalent.

Crucially, hat{w} is a function of the random sample; it is itself random. The “true” objective we actually care about is out-of-sample performance under the true distribution, e.g., maximizing

But plug-in MVO maximizes a different random objective,

The practical question is not “is hat{w} optimal for hat{U}?” (it is, by construction), but “how does U(hat{w}) compare to U(w^\star), where w^\star maximizes U?” That gap is the cost of estimation error and model uncertainty.

Why expected return estimation is the Achilles’ heel

Start with expected returns. For each asset i, the sample mean hat{mu}_i has a standard error on the order of

where

In many liquid asset classes, annualized volatilities might be 10% - 30% while annualized expected excess returns might be 2% - 8%. Translating to a monthly scale, the noise in the sample mean can be comparable to, or larger than, the signal. This is a fundamental signal-to-noise limitation, not an implementation defect.

Now multiply that limitation by dimensionality. MVO compares assets and tries to exploit differences in mu. When mu is noisy, the differences the optimizer sees are often dominated by noise. The optimizer is then rewarded (in-sample) for taking large positions in the assets that happened to have high realized returns in the estimation window, even if that was random luck.

Because the expected return term w^T mu is linear, any error in mu shifts the gradient of the objective directly. In contrast, the covariance term is quadratic and tends to act as a smoothness penalty. This asymmetry is why MVO is particularly fragile to errors in mu.

Here is a simple numerical simulation of how much noise can affect our estimate of mu:

And now the impact on our portfolio weights from MVO. We assume 20 assets with identical true means, so any variation in mu is pure noise.

Our gross exposure is through the roof! A typical MVO portfolio here is equivalent to 200% long and 200% short, 4x leverage. Our right tail on Gross Exposure is also huge, so the portfolio sometimes ends up being 6x levered. The largest single position is also 54% of the portfolio, even though we have 20 assets.

This shows just how much of an extreme impact estimation noise in mu has on MVO.

Why covariance estimation becomes dangerous when inverted

The second failure mode is subtler: even if covariance estimates are “more stable” than mean estimates, the optimizer requires hat{Sigma}^{-1}. Inversion is the mathematical operation that turns moderate estimation noise into potentially huge weight noise.

To see why, consider the eigen-decomposition of the true covariance matrix:

where Q is orthonormal and

with lambda_i > 0$ if Sigma is positive definite. Then

Small eigenvalues become large eigenvalues after inversion. In portfolio terms, eigenvectors associated with small variance directions are precisely the directions the optimizer finds attractive: they offer “cheap risk.” But in finite samples, the smallest eigenvalues of hat{Sigma} are often dominated by noise (especially when T is not much larger than N). When the optimizer leans on these noisy low-variance directions, it produces extreme, unstable weights.

This is not hypothetical. A basic dimensionality fact already creates a hard boundary: if T < N, the sample covariance hat{Sigma} is singular (rank at most T-1), so hat{Sigma}^{-1} does not exist. Even if T is only moderately larger than N, hat{Sigma} can be ill-conditioned, making numerical inversion unstable and conceptually unreliable.

Let’s consider three cases: T < N, T ≈ N, and T > N, and look at the impact that T has on the estimated eigenvalues:

You can clearly see one thing: For the estimates where T = 30, Sigma becomes singular, and the 30th and further smallest eigenvalues just become 0. For T = 80, we can still see that the smaller eigenvalues are systematically underestimated. The estimates become better as we increase T to 300.

Now, let’s look at estiamted eigenvalues of Sigma and Sigma^{-1} for T=300:

The smaller the estimated eigenvalues, the larger the estimated eigenvalues of the inverse of Sigma.

Note: The y-axis is logarithmic on both plots, so linear → exponential!

Sensitivity Analysis: How estimation errors translate into weight errors

A useful way to formalize “error maximization” is to compute how small perturbations in mu and Sigma affect the optimizer.

Consider the unconstrained (besides budget) mean–variance utility maximization (MVO-2). The population optimum satisfies

with eta^\star chosen to enforce 1^T w^\star = 1. The plug-in estimate is

Write estimation errors as

A first-order expansion (informally, a matrix Taylor approximation) uses

A first-order perturbation of the optimizer gives

The last term enforces the budget constraint (with delta eta the perturbation in the multiplier); dropping it isolates the two main channels.

Several qualitative conclusions drop out of (2.8):

1. Mean error passes through Sigma^{-1}. Even if delta mu is moderate, multiplying by Sigma^{-1} can produce large changes in w, especially in directions corresponding to small eigenvalues of Sigma (or hat{Sigma}).

2. Covariance error enters as a “sandwich” with Sigma^{-1}. The term Sigma^{-1} * delta Sigma, w^\star has Sigma^{-1} on the left; if w^\star already has large exposures along unstable directions, covariance errors further distort them.

3. Budget adjustment can amplify instability. The multiplier eta is computed using 1^T Sigma^{-1} mu and 1^T \Sigma^{-1} 1$. If either of these scalars is unstable due to \Sigma^{-1}, the adjustment needed to enforce 1^T w=1 can itself swing drastically.

The key is not that delta mu and delta Sigma exist—of course they do—but that MVO transforms them with Sigma^{-1}, a potentially high-gain operator.

Let’s see how a badly conditioned Sigma affects delta w. We’ll drop the budget constraint so eta doesn’t influence the analysis:

You can see clearly that as Sigma becomes ill-conditioned, the error in w grows.

Optimizer’s curse

Another lens is the optimizer’s curse, a general phenomenon in statistical decision-making: if you choose the argmax of a noisy objective, the achieved value is biased upward in-sample, and the chosen decision is biased toward noise.

Formally, because hat{w} maximizes hat{U}(w),

but what matters is U(hat{w}). The difference

is typically positive and can be large in high dimensions. Intuitively, among many portfolios, some will look exceptionally good in-sample purely due to noise in hat{mu} and hat{Sigma}. MVO systematically selects those portfolios and then “locks in” their noisy characteristics via extreme weights.

This selection effect is strongest when:

• The number of assets N is large relative to the sample size T

• Expected return estimates are weak (low signal-to-noise);

• Shorting/leverage is permitted (large feasible set);

• Constraints are loose (optimizer can chase small estimated edges);

• The covariance matrix has near-collinear assets (ill-conditioning).

Let’s demonstrate the optimizer’s curse via Monte-Carlo.

You can see that even in an environment with equal true in- and out-of-sample mu and Sigma, due to estimation error, we vastly overestimate our utility in the in-sample.

Practical Symptoms: Instability, extreme positions, turnover, and disappointment

When raw MVO meets real data, the mathematical mechanisms above manifest in operational ways:

• Extreme weights and implicit leverage. Even with the budget constraint 1^T w = 1, weights can be large positive and large negative (if shorting is allowed), producing large gross exposure |w|_1. Even with no-short constraints, solutions often sit on corners of the feasible region (many weights at bounds), because linear return objectives push to extremes.

• High sensitivity to small input changes. Updating the estimation window by one month can materially change hat{mu} and hat{Sigma}, leading to large changes in hat{w}. This is not merely “rebalancing”; it is model instability.

• High turnover and transaction cost drag. If weights change drastically, realized performance is dominated by trading costs and market impact, neither of which exists in the clean Markowitz formulation unless explicitly modeled.

• Out-of-sample underperformance relative to naive allocations. A simple equal-weight or risk-parity portfolio can outperform a naive MVO portfolio after costs, not because those heuristics are theoretically superior, but because they are robust to estimation error.

These observations motivate the central practical conclusion: raw plug-in MVO is a high-variance estimator of portfolio weights. In modern terms, it is an overfit model.

The Spectrum of Solutions

This concludes the theoretical foundation.

The remaining 40 pages implement and test 11 robust portfolio construction techniques.

Each technique includes: Mathematical derivation → Clean implementation → Parameter tuning → Comparative results

Plus: Full research notebook (1250+ lines) with production-ready code. If you've made it this far through the theory, the implementations are where it pays off.

If raw MVO is fragile because it optimizes a noisy objective with unstable operators, then fixes must do one (or both) of the following:

1. Fix the inputs: replace (hat{mu}, hat{Sigma}) with estimators that have lower estimation error, better conditioning, or an economically grounded structure.

2. Constrain or regularize the optimizer: modify the optimization problem so that it cannot translate small input errors into extreme weight changes.

These approaches are complementary. Many production-grade systems use both: structured/shrunk inputs and a constrained, regularized optimization.

Fixing the inputs

Covariance Shrinkage: Stabilizing \hat{Sigma} and its inverse

The sample covariance hat{Sigma} is unbiased under idealized assumptions, but it can have high variance in finite samples, especially for off-diagonal elements. Shrinkage addresses this by pulling the estimate toward a structured target F that is lower variance but potentially biased.

A canonical shrinkage estimator is

Common choices for F include:

• a diagonal matrix (imposing zero correlations),

• a constant-correlation model,

• a factor-based covariance (discussed next),

• an identity-scaled matrix (equal variance, no correlation).

Why does shrinkage help? Because it increases the smallest eigenvalues of hat{Sigma} (or, more precisely, it reduces eigenvalue dispersion), improving the condition number and stabilizing inversion. In terms of Sigma^{-1} mu, shrinkage reduces the optimizer’s ability to exploit noisy “cheap-risk” directions created by small sample eigenvalues.

A useful way to view this is that shrinkage imposes a prior belief: extreme correlation structures are unlikely unless supported by strong data. By nudging the estimate toward a simpler structure, you reduce estimation variance more than you increase bias, improving out-of-sample performance.

Oracle note: In the sweep below, the “best delta” is chosen by maximizing utility computed with the true (mu, Sigma) to isolate the mechanism. In practice, delta must be selected via an analytic estimator (e.g., Ledoit–Wolf/OAS) or via walk-forward / nested cross-validation. We also report a held-out test-sample utility proxy for evaluation, not for selection.

Let’s look at the effect of choosing different values of delta. We will let F be

which has the average estimated variance among all assets as its diagonal entries.

As you can see, the condition number of our estimated covariance matrix reduces pretty greatly as we increase delta.

Utility increases out of sample as we increase delta (This is what we REALLY care about!)

Small eigenvalues are inflated instead of dropping to extremely low values, which in turn causes the eigenvalues of Sigma^{-1} to not explode.

Factor Models

A deeper way to control covariance noise is to assume returns are driven by a small number of common factors. A standard linear factor model is

where

• f in R^K are factor returns with K << N

• B in R^{N x K} are factor loadings

• epsilon in R^N are idiosyncratic returns.

Assuming epsilon is uncorrelated with f and has diagonal covariance D, the covariance of r becomes

where Sigma_f = Cov(f) and D = Cov(epsilon) is typically diagonal.

The parameter-count reduction is dramatic. A full covariance matrix has N(N+1)/2 unique parameters. A factor model estimates:

• NK loadings (often constrained/regularized)

• K(K+1)/2 factor covariances

• N idiosyncratic variances.

When K << N, this is a much lower-dimensional estimation problem, leading to more stable covariance estimates and more stable inverses. Economically, factors capture persistent co-movement structure (industries, styles, macro exposures), while idiosyncratic risk captures asset-specific variance.

For optimization, (3.3) has another advantage: it makes risk decomposition interpretable and enables direct factor exposure constraints, a major practical tool for controlling unintended bets.

To test this method, we are gonna assume assets follow some “true” factor model, and then we will estimate this factor model to obtain an estimate for Sigma by plugging the estimates into (3.3):

As you can see, our median condition went from 4327 for the naive sample covariance to only 259.3 for the factor-informed covariance. Utility improved 100% of the time out of sample!

Time-varying covariance

Covariances evolve over time. Using very long histories can reduce sampling error but introduce model error if the distribution is non-stationary. Using short histories reduces model error but increases estimation noise.

A common compromise is an exponentially weighted covariance estimator. For a finite window, a normalized form is

This places more weight on recent observations, adapting to changing regimes, while keeping the weights normalized for finite T.

An equivalent way to write the estimator is the recursive form

which implicitly normalizes and becomes equivalent to the sum as T → infinity (since lambda^T → 0).

But note: exponential weighting alone does not solve ill-conditioning in high dimensions; it is often combined with shrinkage or factor models.

The practical point is that “better covariance” means balancing estimation variance against non-stationarity. MVO fails when either error dominates, and real markets usually give you both.

Expected Returns

If covariance estimation is difficult, expected return estimation is usually worse. Many practical systems, therefore, treat mu not as a directly estimated sample mean, but as a forecast that must be heavily regularized.

A basic shrinkage approach is

where mu_0 is a prior mean. Choices for mu_0 include:

• zero (implying no predictability in excess returns),

• a cross-sectional average (implying mean reversion across assets),

• factor-model-implied returns,

• equilibrium-implied returns from a benchmark portfolio.

The key is the logic: because hat{mu} is high-variance, we deliberately introduce bias by shrinking toward a stable prior. This reduces the variance of the resulting weights, which is often the dominant out-of-sample benefit.

A particularly influential equilibrium anchoring method is to infer implied expected returns pi from an observed “market” or benchmark portfolio w_m.

Before invoking the next equation, it helps to distinguish between two regimes:

• Risk-free asset (tangency portfolio). If a risk-free asset is available and w_m is the tangency portfolio, then pi are excess returns and reverse optimization yields pi = lambda Sigma w_m.

• Risky-only, fully invested. If we remain in the 1^T w = 1 setting, the KKT condition is mu = gamma Sigma w_m + eta 1. The intercept term is economically irrelevant under the budget constraint, so one can normalize it away or interpret eta as the baseline rate.

In a mean–variance world, if w_m is optimal for some representative investor with risk aversion lambda, then

This produces a return vector pi consistent with the covariance structure and observed holdings. Even if the exact assumptions are stylized, pi has a crucial practical advantage: it is anchored to a diversified, implementable portfolio, which prevents the optimizer from forming extreme views unsupported by the market’s aggregate positioning.

Black-Litterman as a canonical “fix-the-inputs” framework

Black–Litterman formalizes the idea that expected returns should be a blend of (i) equilibrium returns and (ii) investor views. While many variants exist, the classical setup can be summarized as follows.

Let pi be equilibrium implied returns (e.g., from (3.6)). Model the prior as

where tau > 0 scales uncertainty in the prior. Investor views are expressed as

where P in R^{M x N} selects linear combinations of returns, q in R^M are view returns, and view noise has covariance Omega in R^{M x M}$ (often diagonal).

The posterior mean under Gaussian assumptions is

Why does this help MVO?

• It shrinks noisy, unconstrained return estimates toward an equilibrium anchor.

• It expresses uncertainty explicitly via tau and Omega, preventing overconfidence.

• It ensures that when views are weak or absent, the optimizer defaults to a diversified baseline rather than a noisy corner solution.

Black–Litterman is not magic; it is a disciplined way of injecting structure and uncertainty into mu, which is exactly what raw MVO lacks.

Resampling and Bayesian Averaging: Stabilizing weights instead of moments

Another approach is to accept that inputs are noisy and average over that uncertainty. A simple version is bootstrap resampling:

1. Generate many synthetic datasets by resampling {r_t}.

2. For each dataset b, compute hat{mu}^(b), hat{Sigma}^(b) and solve MVO to get hat{w}^(b).

3. Average weights:

The intuition is that extreme positions caused by noise tend not to be stable across resamples; averaging dampens them. In decision-theoretic language, this approximates integrating over parameter uncertainty rather than conditioning on a single plug-in estimate.

This class of methods “fixes” MVO not by producing perfect estimates, but by reducing the variance of the decision w.

Let’s compare the weights generated by this method with those of MVO using just the sample estimates. We will also enforce long only to make the weights more comparable.

As you can see, half of the weights generated by the plug-in MVO are exactly zero, whereas with the bootstrap-averaged method, we have no zero weights. Those weights that are non-zero are also larger for the plug-in MVO. We are therefore better diversified using the bootstrap-averaged weights method.

Constraining and Regularizing the Optimizer

Even with improved inputs, the optimizer can still overreact. Constraining the optimization problem is therefore the second major lever. The guiding principle is simple: if you restrict the feasible set or penalize extreme solutions, the optimizer has fewer degrees of freedom to express estimation noise.

Box constraints, no-short constraints, and leverage limits

A basic set of implementability constraints is

along with the budget constraint 1^T w = 1. The no-short constraint is the special case (l_i = 0). A leverage constraint can be expressed as

When shorts are allowed, |w|_1 measures gross exposure; limiting it prevents the optimizer from creating offsetting long/short positions that are highly sensitive to small covariance or mean differences.

Why do these constraints work? They impose a cap on sensitivity. If the optimizer cannot push weights arbitrarily far, then errors in hat{mu} and hat{Sigma} cannot translate into arbitrarily extreme positions.

A common misconception is that constraints only reflect operational needs. In practice, they often serve a dual purpose: implementability and robustness.

Regularization Penalties: Making MVO behave like a well-posed estimation problem

Instead of hard constraints, one can add penalties to the objective. Consider augmenting (MVO-2) with an l2 (ridge) penalty:

Because |w|_2^2 = w^T I w, the risk term becomes

So ridge-regularized MVO is equivalent to replacing Sigma with

This is a profound and practical insight: a simple weight penalty is mathematically identical to eigenvalue inflation of the covariance matrix. It improves conditioning by pushing all eigenvalues away from zero, directly addressing the instability of Sigma^{-1}. In effect, it says: “do not trust low-variance directions too much; treat them as riskier than the raw estimate suggests.”

One can also use an l1 (lasso) penalty:

The l1 penalty encourages sparsity (many weights exactly zero) and discourages large gross exposure, especially when shorts are allowed. It is closely related to constraining |w|_1 as in (3.11). Sparse portfolios can be easier to implement and often exhibit lower turnover.

Note: if w >= 0 and 1^T w = 1, then |w|_1 = 1 is constant, so an l1 penalty on weights has no effect. l1 becomes meaningful when shorting is allowed (gross exposure varies), when applied to active weights a = w - w_b, or when penalizing turnover |w_t - w_{t-1}|_1.

Both l1 and l2 regularization can be understood as injecting a preference for “simple” portfolios, portfolios that do not require precise estimates to justify intricate long/short structures.

The following code demonstrates what happens if you adjust the ridge parameter lambda:

Higher lambda results in better conditioned Sigma.

Higher lambda results in lower L1 and L2 norm of w (The portfolios become “simpler”).

Turnover and Transaction Costs: Regularizing changes the weights

A portfolio is rarely optimized once. In practice, we solve a sequence of problems over time. Estimation noise then manifests as unstable changes in weights, i.e., turnover.

A common fix is to add trading costs or turnover penalties. Let w_{t-1} be current holdings and w_t the new decision. A quadratic turnover penalty gives

This makes the optimizer “reluctant” to move unless the gain in mean–variance utility is large enough to justify the change. Importantly, this is also a robustness device: it prevents the optimizer from chasing small, noisy changes in hat{mu} or hat{Sigma}.

Linear cost models use |w_t - w_{t-1}|_1 instead, which can better capture proportional costs and induce sparse trading (only a subset of names trade each period).

The deeper point is that multi-period implementation turns MVO into a control problem. Regularizing the control (weight changes) is often more impactful than regularizing the state (weights) alone.

Let’s compare the evolution of two portfolios: One via regular MVO and one with the L1 penalty for weight changes. Let’s also see how changing kappa affects our portfolio.

Above, we’ve used a kappa of 50. As you can see, turnover is very heavily reduced.

Here you can see that as you increase your turnover penalty kappa, your improvement vs an unpenalized portfolio becomes better and better, but returns are becoming diminishing with values >= 30.

In a real market that can change more abruptly, you may want to pick kappa slightly lower to be able to adapt more quickly.

Benchmark-relative optimization

Many institutional mandates are benchmark-relative. Let w_b be benchmark weights. Define active weights a = w - w_b. Then tracking error variance is

A common optimization is to maximize expected active return subject to tracking error:

where alpha represents expected alphas (expected excess returns relative to benchmark). This formulation has an important robustness advantage: it anchors the portfolio near a diversified benchmark, limiting the optimizer’s ability to create extreme positions driven by noise. In effect, the benchmark provides a strong prior on the weight vector.

Even if one is not formally benchmarked, this insight generalizes: anchoring optimization around a reference portfolio (market, equal weight, risk parity) often improves stability.

Robust Optimization: Optimizing against parameter uncertainty directly

A conceptually clean way to address estimation error is to incorporate uncertainty sets for mu and/or Sigma, then optimize a worst-case objective. This approach turns “garbage in, garbage out” into “uncertainty in, robustness out.”

One standard robust mean model assumes mu lies in an ellipsoid around hat{mu}:

Consider the robust counterpart of maximizing expected return minus risk, taking the worst-case mean:

The inner minimization has a closed form:

So the robust problem becomes

This is illuminating. The robust adjustment subtracts a term proportional to portfolio volatility, effectively reducing the attractiveness of portfolios whose expected return advantage might be explained by estimation error. The optimizer is forced to find portfolios whose expected return is high relative to uncertainty.

Robust covariance uncertainty sets lead to related regularization effects, often inflating risk in uncertain directions; again echoing the intuition behind shrinkage and ridge penalties. Robust optimization provides a principled bridge between statistical uncertainty and optimization regularization.

Let’s compare this to standard MVO:

Our Robust MVO weights are a lot more “clean” compared to the standard MVO weights.

Conclusion

It is tempting to treat the practical modifications above as an ad hoc toolbox: shrinkage here, constraints there, turnover penalties elsewhere. A more coherent view is that these methods all solve the same underlying problem:

The plug-in MVO portfolio is a high-variance estimator of the optimal weights.

Reducing that variance requires injecting structure, equivalently, adding bias.

• Shrinkage of Sigma reduces the variance of covariance estimates (and stabilizes inversion) at the cost of bias toward the target structure.

• Factor models impose a low-rank structure that may be misspecified but dramatically reduces estimation noise.

• Shrinkage and Bayesian methods for mu explicitly acknowledge that sample means are too noisy to trust fully.

• Constraints and penalties reduce the effective degrees of freedom of the portfolio, preventing the optimizer from encoding noise into intricate weight patterns.

• Turnover penalties reduce the variance of changes in weights, which is often the real operational pain point.

• Robust optimization is an explicit uncertainty-aware form of regularization.

From this perspective, the “right” portfolio is not the one that is optimal for the estimated parameters; it is the one that is optimal for the decision problem under uncertainty, including estimation error, non-stationarity, and implementability constraints.

Quant Corner: https://discord.gg/X7TsxKNbXg

Code: https://gist.github.com/vertoxquant/e417b8afd2abe0e15ae0ff486813842a

At first glance, that contradiction seems justified. Crypto markets look nothing like equities: trading is continuous, liquidity is fragmented across venues, listings and delistings are frequent, and returns are dominated by a long tail of illiquid tokens.

Let’s answer once and for all if equity factors exist in crypto and if you can profit from them.

By the end of this article, you’ll have a framework for answering that question yourself. You’ll see how to test new factors and alphas in a way that is robust to listings and delistings, a changing universe, and the realities of trading crypto markets. The full research notebook used in the analysis is available for download at the end of the article for paid subscribers.

Read more

In the previous article, we built all of the theory behind HMMs for microstructural data. This article is where it finally gets real, and we fit those models to actual market data efficiently.

We are going to take something that is completely infeasible in practice, on the order of 10^430 computations. To put that in perspective, if you took every atom in our universe, put a new universe inside each atom, and repeated that process five times, you would reach a number around that size. Then we collapse it down to roughly 9000 computations that you can actually run.

By the end of this article, you will be able to take real market data, fit an HMM to it, and identify which market regime you are most likely in

Read more

• Small communities that are invite only just tend to die down eventually.

• Big servers don’t really offer much value or do anything fun.

That’s exactly what I want to change with Quant Corner!

The goal is simple: build a community that actually does things.
Think tournaments, live Q&As, useful resource giveaways, and open discussions where your feedback genuinely shapes what we build next.

Quant Corner is also the best place to engage directly with me, share ideas, suggest events, and help steer the direction of the community.

We’re currently at ~440 members, and once we hit 800, I’ll be hosting something big.

If this sounds like something you’d want to be part of, join us here:
https://discord.gg/X7TsxKNbXg

Many financial features exhibit regime behavior: low vs. high volatility, choppy vs. trending markets, liquid vs. illiquid conditions.

These regimes are not directly observable. Instead, we only see their manifestations in market data such as prices, returns, volumes, and spreads.

As a result, microstructure features may exhibit different statistical behaviour depending on the underlying market state, raising the question of how they should be analysed in practice.

Read more

They dump every book and paper the author has heard of (but never read). You’re left with a 50-item reading list and no idea where to start.

This roadmap is different. Every resource here has been tested. Every month builds on the last. And it contains just what you actually need to become a quant researcher.

Day 1: Python

As a quantitative researcher, 99% of your time will be spent with Python. You don’t need to be proficient in the language, but you should be able to code up any ideas that come to mind.

If you have no coding experience whatsoever, you can honestly just get started with any “Python for Beginners” YouTube video like this:

By itself, Python isn’t all that powerful for research. The reason it’s still the Nr 1 choice for most researchers is the libraries.

You will see the following 3 libraries in pretty much every project:

• numpy (Fast library for working with arrays and matrices)

• pandas (Data manipulation and analysis, Dataframe data structure)

• matplotlib (Plotting)

Other than that, you still have many other libraries that make your life easier, like sklearn, scipy, polars, statsmodels, and cvxpy.

Your main goal when doing research should be to leverage libraries as much as you can to get results as quickly as possible and to move on to the next problem.

Don’t spend too much time in this stage. You can spend a day coding up a few small applications to get a basic understanding of the language, but other than that, you’ll learn the language by doing real quant projects.

Week 1 and 2: Calculus and Linear Algebra

You don’t need to be a PhD-level math genius to be doing quant (unless you want to be doing complex derivatives pricing, but we can learn that later).

A solid understanding on first semester college level Calculus and Linear Algebra will get you really far. There are 2 ways to learn this:

1. Go to university.

2. Read the following 2 articles that summarize everything in an intuitive way:

Universities heavily focus on proofs, as that’s what you need to discover (or invent?) new math. As a quant, I’ve never used the proof-writing skills that I learned in university before. Proofs can absolutely help with intuitive understanding of math, so if you want to gain a deeper understanding, try proving some things!

The book “How to Solve It” by George Polya teaches you a mental framework for how to approach proofs.

Just like with coding, you learn math by using it. That can either mean doing quant projects that involve a certain type of math or doing practice problems.

One of the best resources out there that I can recommend for both learning math and solving practice problems is Paul’s Online Notes:

https://tutorial.math.lamar.edu/

Week 3 and 4: Statistics

With the basic mathematical prerequisites down, you can move on to stuff you will apply more directly when doing research: Statistics!

There are 2 main branches of statistics:

Descriptive Statistics

This is all about summarizing data. Mean, Median, Variance, Volatility, Skewness, Correlation, Histograms, Boxplots, etc.

Those are all things you hear on a daily basis.

Inferential Statistics

This is all about inferring things. You are trying to predict the behavior of a non-observed set of information or generalize about a larger population using data from a sample. Things like linear regression, time series analysis, decision trees, and statistical tests are all part of inferential statistics.

The Organic Chemistry Tutor and StatQuest both have fantastic playlists on statistics:

https://www.youtube.com/playlist?list=PL0o_zxa4K1BVsziIRdfv4Hl4UIqDZhXWV

https://www.youtube.com/playlist?list=PLblh5JKOoLUK0FLuzwntyYI10UQFUhsY9

And again: Don’t just watch those videos blindly and try to remember everything, but actually try to apply what you’ve learned.
Just found out about histograms, boxplots, and other visualization techniques? Try to visualize some real financial data to build intuition!

Month 2: Machine Learning

Machine learning isn’t all about huge deep neural networks and LLMs. If I were to define it, I’d say it is “all about algorithms that learn patterns from data in order to make predictions or decisions, without being explicitly programmed for each task.”

You can again categorize types of machine learning models.

Supervised learning

You work with labeled data.

By far the most important models of this type are regression models:

• Linear Regression

• Logistic Regression

• Polynomial Regression

• Ridge Regression

• Lasso Regression

• Quantile Regression

Those are the machine learning models you’ll be working with most of the time, so you should get familiar with them.

https://www.geeksforgeeks.org/machine-learning/types-of-regression-techniques/

Another very powerful type of model is tree-based models, like:

• Decision Trees

• Random Forests

• Gradient Boosted Trees (XGBoost, LightGBM)

You often see Gradient Boosted Trees winning Kaggle competitions.

Unsupervised Learning

No labels; you discover structure.

Examples:

• Clustering (Grouping similar data)

• Dimensionality Reduction (Finding structure in data)

• Outlier Detection

• Density Estimation

Reinforcement Learning

Learning via interaction and feedback.

This is pretty niche, and typically other types of models do better anyway, so I wouldn’t recommend you to learn this if you are starting out.

StatQuest has a video on pretty much every topic I mentioned here, and more:

Most importantly: If your features and targets are garbage, no amount of model tuning will help you. Keep this in mind when working on projects!

Month 3: Optimization

Your chef comes up to you and tells you he wants a portfolio with 10% expected yearly return and volatility at most 8%. How do you do that? The answer: Optimization!

Optimization is an absolutely huge field, including:

• Convex Optimization

• Linear Programming

• Semidefinite Programming

My favorite introductory book, which I would read myself if I had to start over again, is “Optimization Methods in Finance - Second Edition”.

This will also be your first time dealing with finance specifically (This goes to show how much of quant isn’t about finance, but about math and science.)

For a deep dive into Convex Optimization (which you don’t need at that level!), read “Convex Optimization” by Boyd and Vandenberghe.

Month 4: Portfolio Optimization and Management

You will already learn some portfolio optimization in the book we mentioned in the last section, but what I really like about “Robust Portfolio Optimization and Management (Frank J. Fabozzi Series)” is that it tells you about all the practically important techniques you need to know to create robust portfolios.

The truth is that if you just apply the portfolio optimization techniques you’ve learned about so far blindly to your sample returns and covariance, your portfolio weights will mostly be determined by noise.

Month 5: Numerical Methods

Most problems can’t be solved analytically, or it’s very cumbersome to do so. This is where numerical methods, which are incredibly powerful, come in!

Root Finding

The goal of root-finding algorithms is to figure out where a function is equal to 0. If you apply this to the derivative of a function, you can find its maxima and minima.

This is the only playlist you need:

https://www.youtube.com/playlist?list=PLb0Tx2oJWuYIpNE23qYHGQD42TIR3ThNz

Gradient Descent

The goal here is to find minimums / maximums of functions. There are many algorithms that are derived from basic gradient descent, so you should learn about it first:

Integration

Those are used to approximate integrals. Sometimes integrals are difficult or even impossible to solve analytically:

Differentiation

If we can integrate numerically, then we can also differentiate numerically:

Interpolation and Approximation

This is all about curve fitting and smoothing. The theory here can go really deep. This video explains the most important techniques:

Linear Algebra

Whenever you want to solve a linear system (Ax=b), compute eigenvalues and eigenvectors, or perform a matrix factorization, you need numerically stable algorithms. The following playlist covers all of those topics:

Those two playlists can be used as a lookup for anything about numerical methods:

https://www.youtube.com/playlist?list=PLDea8VeK4MUTppAXQzHBNz3KiyEd9SQms

https://www.youtube.com/playlist?list=PLkZjai-2Jcxn35XnijUtqqEg0Wi5Sn8ab

Month 6: Derivatives and Pricing

You can’t talk about derivatives and pricing without mentioning the two most iconic books:

• “Options, Futures, and Other Derivatives” by John Hull

• “Option Volatility and Pricing” by Natenberg.

The former is about all derivatives, while the latter covers options specifically.

You shouldn’t read them front-to-end, though, but look up what you want to become more familiar with.

QuantStart is a good start (pun not intended) to get familiar with the basics:

https://www.quantstart.com/articles/derivatives-pricing-i-pricing-under-the-black-scholes-model/

https://www.quantstart.com/articles/derivatives-pricing-ii-volatility-is-rough/

https://www.quantstart.com/articles/derivatives-pricing-iii-models-driven-by-levy-processes/

Months 7 and 8: Risk Management

I’ve written a huge article on risk management basics:

If you care specifically about tail-risk, I recommend you learn about Extreme Value Theory. The following book is a perfect introduction:

“An Introduction to Statistical Modeling of Extreme Values” by Stuart Coles.

Volatility forecasting is naturally also very important for risk management:

There are many new topics you need to learn here, especially Extreme Value Theory, which is more challenging! Expect to spend some more time on this area. But trust me, it pays off!

Conclusion

Quant is a huge topic, and you can learn about it for the rest of your life.
This article tells you what to learn step by step to start doing real, meaningful research.

There are, of course, many more topics not covered in this crash course, like market microstructure, time series analysis, data engineering, execution algorithms, and everything high frequency. With the things you learned using the crash course, you are more than ready to dive into those topics yourself, though!

And again, because it’s so important: Don’t just learn by reading resources and watching videos. Apply what you learn to real projects to build REAL understanding and intuition!

Btw: I have written 77+ articles on everything quant on this site!

VertoxQuant

Quantitative Research

You build a clean, elegant beta-neutral portfolio.

Longs and shorts, carefully balanced, as all things should be.
Your regression says market beta is 0.02. Basically flat.
You go to sleep relaxed.

Then the market sells off 3%.
Your “beta-neutral” book is down 1.4%.

You can’t believe that happened.
You start digging:

• Did correlations spike?

• Was it exposure to another factor?

• Is it all a conspiracy?

No.
The problem was already there before the sell-off even happened.
Your beta estimates were lying to you!

SOL doesn’t actually have a beta of 3.1; this just screams that it was estimated from noisy data.

But there’s a fix. A technique far more powerful than clamping beta estimates. One that shrinks your betas towards reality before they blow up your portfolio.

Why “Beta-neutral” Portfolios Aren’t Actually Neutral

Sample betas are just estimates. They’re noisy, unstable, and prone to exaggeration. The more extreme a beta looks, the more likely it’s lying.

The more aggressively you hedge with extreme betas, the more estimation error creeps in, and the less reliable your beta-neutral portfolio becomes.

Negative betas are especially dangerous. Imagine Asset A has a measured beta of 1.5 while Asset B has a measured beta of -1.5. You go long both, hoping to be beta-neutral. A few hours pass, and suddenly Asset B’s beta flips to 1.0, and your portfolio is suddenly long the market.

One thing many people do is clamp betas, which is not the right approach.
If you clamp betas at 2.0, then all your betas beyond that level just clump together at 2.0. There is suddenly no difference between a 2.1 and a 3.9 beta asset. Even within your capped group of assets, some positions are overweighted, some underweighted.

Statistically, clamping is crude because it introduces bias without reducing variance intelligently.

The Bias-Variance Tradeoff

In statistics and machine learning, every prediction faces a fundamental tradeoff: Bias vs Variance.

Bias is a systematic error: On average, your estimate is off.
Variance is the variability of your estimates across different samples.

Our goal now is to introduce some bias in order to reduce variance.
But why would we want to be wrong?!
Think of it like walking across a tightrope in a windstorm. You could try to react to every gust of wind, but if you misjudge once, you fall.
Or you can take a slightly biased, controlled path; You lean a little into the wind, and you make it across safely.

But instead of walking across a tightrope, you build a beta-neutral portfolio. And instead of falling, you lose the house.

Honey, I Shrunk the Betas

So how do we stop our “beta-neutral” portfolios from blowing up?

The answer is simple: shrink your betas.

Instead of trusting every extreme sample beta, we pull them toward a sensible center; usually 1 for market beta, or the average beta across your universe. If you want to be fancy, you could even group your assets by sector and shrink the betas within each sector to the average one.

Mild betas barely move, while extreme betas get pulled towards the center much more strongly.

In practice, the simplest form is linear shrinkage:

where

• \hat{beta} = raw sample beta

• beta_target = The center every beta gets pulled towards

• w = How strong to pull.

w controls your bias-variance tradeoff. A smaller w leads to a lower variance, in exchange for higher bias. The longer the window is that you estimate betas on, the larger you typically want your w to be.

For those who like to go deeper, there are more advanced ways to shrink betas:

• Bayesian shrinkage: Incorporate prior knowledge about beta distributions.

• James-Stein: Learn the target from the cross-sectional distribution of all assets.

Conclusion

Raw beta estimates are noisy. Extreme betas are overconfident.
Trying to neutralize the market using raw beta estimates is a disaster waiting to happen.

Shrinkage is a smarter approach that pulls extreme estimates towards a reasonable center and keeps the portfolio’s relative structure intact.

It’s counterintuitive, but true: Sometimes being a little wrong on purpose makes you much more right when it matters.

So next time your regression spits out a 3.1 beta, don’t panic, don’t clamp it. Just shrink it.

If you wish for more practical Quant insights like this one, consider supporting us!
What you get:

• Access to over 50 premium articles.

• 3 new premium articles per month.

• Access to all the project code in the premium articles.

• Access to the premium section of the Discord server.

Here you can have a taste of the premium articles:

Getting access to high-frequency data in crypto is easier than in any other market. Even if you don’t want to spend a couple of hundred or thousand dollars on historical data, you can set up data collection yourself pretty cost-efficiently and reliably.

We show how to use this high-frequency data to get better forecasts of volatility, compare different volatility forecasting models, and show how to properly diagnose a volatility forecasting model.

Table of Contents

1. Realised Variance from High-Frequency Returns — Our Forecast Target

2. Range-based Variance Estimators — Sometimes, Less is More

3. Sampling Frequency — The Problem with High Frequency Data

4. Rolling Averages and EWMA — Baseline Variance Forecasts

5. Tuning EWMA via QLIKE — A Proper Scoring Rule for Variance

6. GARCH Family — Conditional Heteroskedasticity Models

7. Stochastic Volatility and Kalman Filters — How to Actually Use a Kalman Filter

8. Volatility Forecast Diagnostics — What Model is The Best?

9. Final Remarks — Conclusion, Code, and Discord

Read more

A: “Kelly is too dangerous, use half-kelly instead!”
B: “No use 0.75x Kelly, 0.5x is too low!”
C: “Actually, you should use this formula for continuous Kelly!”

In this article, we will delve into the actual math of Kelly, rather than relying on word-of-mouth.

Thanks to the amazing support of our premium readers, we are able to release an article for free from time to time.
If you wish to unlock over 50 premium articles like this one, as well as 3 articles per month, consider supporting us as well!

Table of Contents

1. The Kelly Objective — What Are We Even Optimising?

2. Classic Kelly — Model and Closed-Form Solution

3. Fractional Kelly — What’s the point?

4. A Bounded-Below Model — The Impact of Tail Risk

5. A Quadratic Approximation — The Dangerous Formula You Find Online

6. Continuous-Time Kelly (GBM) — Moving On From Discrete Models

7. Multi-Asset Continuous-Time Kelly — When We Trade Multiple Assets

8. Estimation Error: Bayesian Approach — Updating Your Beliefs of Odds

9. Dynamic Sizing — Don’t Use Constant Kelly!

10. Final Remarks — Final Remarks and Discord

The Kelly Objective — What Are We Even Optimising?

We start with some notation.
Let:

• r = per-period risk-free simple return

• X_{t+1} = per-period excess simple return of the risky asset

• f = dollar exposure to the risky asset, as a fraction of current wealth (f=1 means fully invested; f>1 means levered; f<0 means short)

Then discrete-time wealth evolves as:

The Kelly objective that we want to maximize is the per-period expected log growth:

Working with the log of wealth has a couple of nice properties:

• log(0) = -inf, going bankrupt is unacceptable.

• Maximizing E[log(1+R)] maximizes the long-run growth rate of wealth.

• Log utility represents risk aversion: Going from $1000 → $2000 is much nicer than going from $10.000 → $11.000.

g(f) is an expected value, and we know that log(0) is -inf, so for g(f) to be finite we have the necessary condition:

This also explains why you can’t use an unbound normal distribution to model your returns, since even if the probability of getting a return that wipes out your portfolio is astronomically small, the expected value (g(f)) will be -inf.

Classic Kelly — Model and Closed-Form Solution

Let’s start with a super simple binary model:

• Win probability: p

• Lose probability: q = 1-p

• Odds: risk 1, win b

By betting a fraction f of wealth, our new wealth will be multiplied by:

• Win: 1 + bf

• Lose: 1 - f

The Kelly objective g(f) will now look as follows:

We can find a closed-form solution for f that maximizes g by taking the derivative and setting it to 0:

This gives us the traditional formula for the Kelly fraction:

Let’s code this up!

First, some imports and configs that we will use in this project:

Now the code to compute the Kelly fraction and the expected log growth:

With p = 0.55 and b = 1.0, we obtain a Kelly fraction of 0.1, meaning we should bet 10% of our entire wealth on this every time.

We can verify that this is correct using a plot:

Fractional Kelly — What’s the point?

There is one biiiig problem with the Kelly fraction: it just barely avoids ruin.
At the Kelly fraction f, our probability of ruin is 0 almost surely. Any increase in f above Kelly gives a positive probability of eventual ruin.

If you overestimated your probability of winning, then even Kelly will give you a leverage that gives you a non-zero probability of eventually going to 0.

So the intuitive idea of just using a fraction of Kelly is actually pretty sound, although the execution isn’t the best (we will get to this later).

Let’s compare what happens at Half Kelly, Full Kelly, and Double Kelly.
The following function will simulate the binary game with probability of winning p, a payout b, and leverage f:

To better compare the 3, I will use the same random numbers in all 3 simulations:

A Bounded-Below Model — The Impact of Tail Risk

We usually only have samples of excess return x_i available.
We can then approximate the Kelly objective g via:

where n is the number of samples available.

We maximize over a feasible interval where 1+r+fx_i > 0 for all samples.
To compute g from samples and to compute the feasible interval, we can use the following functions:

To illustrate how much of an impact tail risk has on Kelly, consider the following scenario:

• With probability p_good: Sample normal returns, clipped at a crash floor.

• With probability 1-p_good: Sample a fixed crash x_crash (a point mass at the floor)

We can sample those returns with the following function:

We can maximise the log growth rate (Kelly objective) on our feasible interval using golden section search:

I talked about the golden section search in more detail in the following article:

We will further use the following two functions to summarize information about log wealth:

Let’s sample from our bounded fat-tailed distribution now and find the optimal fraction to bet:

As you can see, we shouldn’t be betting at all.
Let’s see what happens if we increase the mean of the normal distribution from 0.1% to 1%:

Much better, but we are also pretty close to the leverage that will cause us to reach ruin with probability 1 almost surely.

A Quadratic Approximation — The Dangerous Formula You Find Online

Recall the formula for the Kelly objective:

The Taylor expansion of log(1+y) around y = 0 is:

The Second-order approximation is therefore:

Substitute y = r+fX, then:

Taking expectation on both sides, we finally obtain (since r and f are constants):

Now we simplify r - r^2/2 using the second-order approximation of log(1+r) and factor out the (1+r) term to finally obtain:

A very naive formula I’ve seen online now further assumes r ≈ 0 and E[X^2] ≈ sigma^2.
With this, we get the final formula:

Where mu = E[X].

To find the f that maximises this, we take the derivative and set it to 0:

This has the following solution that you often see online:

We’ve done so many approximations and assumptions here. This formula becomes extremely unreliable in real markets.

Here is what this approximation would have given us on our fat-tailed samples:

Continuous-Time Kelly (GBM) — Moving On From Discrete Models

One really nice property of continuous-time models is that we avoid the discrete-time “one-step wipeout” issue because the wealth process stays strictly positive under diffusion dynamics.

Let’s model a risky asset with excess drift mu via Geometric Brownian Motion:

Let’s say we have wealth W and maintain constant exposure f to the risky asset.
Our wealth changes from two sources:

• Cash portion (1-f)*W earns r.

• Risky asset portion f*W follows the asset’s dynamics.

Our wealth return dynamics are therefore:

What we care about are the dynamics of log wealth, however.
We can apply Itô’s lemma to obtain:

The drift component of log wealth is:

To maximize the drift of log wealth, we again take the derivative of this with respect to f and set it equal to zero (you know the drill) to obtain:

That’s the same formula we obtained in our second-order approximation!

Let’s code it up and confirm our formula is correct:

Multi-Asset Continuous-Time Kelly — When We Trade Multiple Assets

Why limit ourselves to one asset when we can trade multiple assets at once?

Let X be the vector of excess returns in continuous time with drift mu and covariance Sigma. We have the multivariate Gaussian Process:

Doing the same process as in the previous section, we arrive at the drift of our log wealth:

with weights w.

The Kelly solution is:

Let’s compute and verify this:

There is one major issue with the Kelly solution (besides the fact that markets don’t follow Brownian motion): Sigma^-1 is extremely unstable. Covariance matrices are nearly singular for correlated assets, making small changes to Sigma can give you a vastly different Sigma^-1.

Our goal is to reduce the variance by introducing some bias. A simple way to do this is using Ridge Regularisation:

I talk in more detail about regularisation in the following article:

What you need to know here is: The more we increase lambda, the more our weights will shrink towards zero (more bias), but the smaller the variance will be.

Here is the impact of different values of lambda on our weights:

Estimation Error: Bayesian Approach — Updating Your Beliefs of Odds

Imagine you are walking around in Las Vegas, and someone in an alley calls you to play a game. Because you have no survival instincts, you decide to give it a try. He suggests throwing a coin. If you win, your payout is b=1. If he wins, his payout is 1.

You believe the nice gentleman is using a fair coin, but you are sceptical. Your prior belief on the win probability is a Beta(a0, b0) distribution with a0 = 2 and b0 = 2. This is a symmetric distribution centred at p=0.5:

You guys decide to play 100 games, and he wins 70 times… You no longer believe the coin is fair and update your beliefs about the probability of him winning. Your new posterior distribution looks as follows:

You still decide to play another 300 games, and this time he only wins 162 times.
Your new posterior distribution now looks as follows:

The process we just did is called Bayesian inference, where we compute the posterior probability according to Bayes’ theorem:

where:

• H is a hypothesis whose probability you are trying to estimate.

• P(H) is your prior probability for the hypothesis.

• E is the evidence you collect by playing.

• P(H|E) is the posterior probability. The probability of H with new evidence E.

Our prior is a Beta(a0, b0) distribution. The density is:

We observe n=100 trials with k=70 wins. The likelihood is binomial:

According to Bayes’ Theorem, we have:

Substituting:

This is the kernel of a Beta distribution:

So after our first round of 100 games, our posterior distribution is Beta(72,32).

Note: The mean of a Beta(a,b) distribution is a/(a+b).

We are gonna test 4 betting strategies against the mysterious alley guy now. For all 4 strategies we start by playing 100 test games to get an idea of the true probability of heads and tails.

1. We estimate p as the fraction of games won. We then bet using Kelly: f = 2p-1.

2. Same as strategy 1, but we bet with half-Kelly.

3. We use our Bayesian inference strategy and bet with Kelly using the mean of our posterior distribution as the probability.

4. We use Bayesian inference like in strategy 3, but use the 10th percentile of our posterior distribution instead of the mean.

Here is all of that coded up:

Half Kelly has the highest mean log wealth growth, and no strategy ever went bankrupt.

If we increase our warmup games to 1000 and then play 10000 games in total (across 20000 experiments), those numbers change:

The more optimal strategies start to outperform!

Here are the distributions of the final log wealths:

Note: The true probability of us winning was 55% (Thank you, mysterious coin guy!).

Dynamic Sizing — Don’t Use Constant Kelly!

An even better strategy would be to update your posterior distribution in the Bayesian inference strategy after every single game (or every few games).
There is a small problem with this, though: what if the true probability of winning changed over time? The more games we played, the slower Bayesian inference updates.

There are a few major ways to solve this issue:

1. Use a sliding window and only use the last N observations for your inference.

2. Weight recent data more heavily.

3. Monitor for regime changes and reset when detected.

Those are left as an exercise to the reader.

Imagine the shady alley man actually has 2 coins now that he switches between at random, and instead of flipping a coin, he samples from a normal distribution. Further, with a small probability, we get a crash.

We can simulate this with the following code:

We are gonna use a rolling window to bet with Kelly and hopefully adapt to the changing regimes:

Let’s test out 3 different strategies now:

1. A crystal ball that knows what regime we are in and knows all the true probabilities to bet optimally using Kelly.

2. Our rolling Kelly.

3. Play a couple of warm-up games and then play with fixed Kelly.

Final Remarks — Final Remarks and Discord

Thanks to the amazing support of our premium readers, we are able to release an article for free from time to time.
If you wish to unlock over 50 premium articles like this one, as well as 3 articles per month, consider supporting us as well!

Discord: https://discord.gg/X7TsxKNbXg

What You Get:

• Full access to 54 premium articles, all significantly longer and more detailed than the 21 free articles.

• 3 new premium deep-dives every month covering high-frequency trading, market making, options, funding-arbitrage, and more.

• Access to the Premium section of the Discord server.

• Entry into exclusive giveaways (we recently gave away a brand-new copy of The Elements of Quantitative Investing).

What subscribers are saying:

• ”Hey thanks for sharing on twitter, subbed to your substack! You content has been extremely valuable and appreciate your work much.”

• ”Alot of your writing directly helps with my job!”

• ”It’s clear, concise and to the point.”

Now only $29.90/month (previously $39.90/month):

https://www.vertoxquant.com/subscribe

At first glance, a trade executing on Binance, an earthquake rupturing along a fault line, and a cat video going viral seem to have nothing in common. Yet they all have one important characteristic: They cluster in time!

When an earthquake occurs, the aftershocks that follow can last for days or weeks. When a tweet gains traction, retweets create more traction, causing even more retweets! And when a large trade executes in a financial market, it doesn’t occur in isolation; You have copy and momentum traders that flock in. Algorithmic order execution creates lots of smaller trades that happen in bursts, and so on.

In this article, we will be exploring models that are able to capture those phenomena and use them to model order-flow, a very important metric to keep an eye on in high-frequency trading!

Full Code available at the End!

Table of Contents

1. Point Process — The Mathematics of Random Events

2. Poisson Process — Why the Textbook Model Fails

3. Poisson-GLM and Binned Intensities — Smarter Order-Flow Model

4. Hawkes Process — When Past Trades Trigger Future Trades

5. Multivariate Hawkes Process — How Buys Excite Sells (and Vice Versa)

6. Extensions and Conclusion — Extensions, Final Remarks and Discord

Point Process — The Mathematics of Random Events

At its core, a point process is a random collection of points in some space, in our case, time. A point process in space could be where a raindrop hits the ground, for example. When modelling Order-Flow, each event happens at a specific time, and the pattern of these occurrences is random rather than deterministic.

Formally, a point process N on a space S is a random countable collection of points. For any measurable set A ⊆ S, the quantity N(A) represents the (random) number of points falling within A.

Counting Process Representation

When working with point processes in time (S = [0, ∞)), we often use the counting process representation. Define N(t) as the number of events that have occurred in the interval [0, t]:

where T_1, T_2, … are random arrival times of events with 0 < T_1 < T_2 < ….
The function N(t) is non-decreasing, right-continuous, and takes integer values, jumping by 1 at each arrival time.

Here is one example of a realisation of a point process and the corresponding counting process representation:

Key Characteristics

There are several quantities that characterise the behaviour of a point process. Consider, for example, the following 2 point process realisations:

Right off the bat, you can tell that the second one seems to be more intense, whatever that means.

The so-called intensity function described the instantaneous rate at which events occur:

This captures the expected number of events per unit time in an infinitesimal interval after t. For many point processes, we can write the expected number of points in an interval [a,b] as:

The conditional intensity provides even more information by incorporating the history of the process up to time t:

with respect to a filtration F_t. You can imagine F_t as the history of events preceding time t. If you need to catch up on filtrations, I recommend that you check out the following article:

I explain Filtrations intuitively there.

Read more

This is fine if you are working with BTC and ETH, but what do you do if you want to bet on a coin like SOL, XRP, or DOGE that doesn’t have a liquid options market?

We will be covering the answer to that in this article!

Table of Contents

1. Project Structure

2. Pricing Model

3. Data Gathering

4. SSVI Model

5. Cross-Asset Statistics

6. Baseline Proxy Model

7. Behaviour-Aware Proxy Model

8. Conclusion

Project Structure

The project structure looks as follows:

project/
│
├── notebook.ipynb
│
└── src/
    ├── analysis/
    │   └── factors.py
    │
    ├── data/
    │   └── loader.py
    │
    ├── models/
    │   └── proxy.py
    │
    └── numerics/
        ├── black.py
        └── ssvi.py

factors.py:

Contains all the functions necessary to compute betas of returns and factors of a volatility surface.

loader.py:

Handles data gathering from Deribit using CCXT.

proxy.py:

Contains the proxy model that allows us to map the IV surface of a liquid coin like BTC or ETH, to an altcoin.

black.py:

Contains the pricing model and functions for computing Greeks and implied volatilities.

ssvi.py:

Contains all the functions for representing and fitting volatility surfaces using SSVI as well as checking for no-arbitrage conditions.

notebook.ipynb

Wraps everything together in a structured way, following this article.

Pricing Model

Read more