The moment you care about speed, like when quoting live, or when calibrating a pricing model where you need to reprice options thousands of times, Monte Carlo quickly becomes unusable.

This article presents three Fourier-based methods that solve this problem, reducing pricing from seconds to microseconds. We’ll use the Heston model as our running example throughout, and achieve a speedup of 30,000x against Monte Carlo!

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why Monte Carlo breaks down for live pricing and model calibration, and what makes it fundamentally unsuitable for production options trading.

• How the Fourier Transform and FFT work and why they matter for options pricing.

• What the characteristic function is and how it enables us to price options even when the risk-neutral density of the model has no known closed-form analytical solution.

• How Carr-Madan transforms option pricing into an FFT problem.

• How Lewis reformulates the same problem via contour integration, eliminating the dampening parameter that makes Carr-Madan brittle.

• How the COS method takes a completely different approach, approximating the density via a cosine series, and why it converges exponentially fast.

• Python implementations for all methods, together with speed comparisons.

This article is free to read, so I would appreciate it if you shared it with someone who would also like it!

The Characteristic Function

To price options using Fourier methods, we need one thing from our model: the characteristic function of log S_T where S_T is the price at expiry.

For a random variable X, the characteristic function is defined as:

For log S_T we write:

where Q is the risk-neutral measure.

Just like the probability density function, it completely characterises the distribution of X. But while the density q(x) often has no closed-form expression, the characteristic function often does and is much easier to find.

In fact, the characteristic function and the density are a Fourier transform pair. Given the characteristic function, you can recover the density via the inverse Fourier transform:

This is the core idea behind all Fourier pricing methods. You don’t need to know the distribution of log S_T explicitly; you just need the characteristic function.

Why does the characteristic function always exist?

For any model where log S_T is a well-defined random variable, the characteristic function is guaranteed to exist for all real u, since

and thus the expectation is always finite.

Pricing in Fourier Space

In theory, you could now go ahead and compute the distribution from the characteristic function numerically, and then integrate against the payoff to get the fair value of the option. In practice, Fourier pricing methods skip this step entirely and price directly in Fourier space, which is both faster and more numerically stable.

Two Examples

For Black-Scholes, log S_T is Gaussian, so the characteristic function is that of a Gaussian random variable, with appropriate mean and standard deviation plugged in:

For Heston, log S_T has no nice closed-form density expression. But the characteristic function is still available in closed form:

where:

with

These are derived by applying Itô’s lemma to log S_t, exploiting the affine structure of the Heston dynamics, and solving the resulting Riccati ODEs in closed form.

How to properly derive this will be the subject of future articles.

Fourier Transform and FFT

Before deriving any pricing formulas, we need to understand the Fourier Transform and the Fast Fourier Transform algorithm.

The Fourier Transform

For a real square integrable function f(x), the Fourier transform is:

This function lives in frequency space; It tells you how much of each frequency xi is present in f. You recover f from hat{f} via the inverse Fourier transform:

Two real functions f and g that are both square integrable have a well-defined inner product. Parseval’s theorem says this inner product can also be computed in Fourier space:

This will be used later on in the Lewis method.

The Discrete Fourier Transform

In practice, you work with a finite grid of N points rather than a continuous function. The Discrete Fourier Transform (DFT) of a sequence x_0, x_1, …, x_{N-1} is:

Each output X_k is the inner product of the input sequence against a complex exponential at frequency k. Computing all N outputs naively requires evaluating N terms for each of the N outputs, a total of N^2 multiplications.

The Fast Fourier Transform

For N a power of 2, the Fast Fourier Transform, introduced by Cooley and Tukey, reduces the cost from O(N^2) to O(N log N) by exploiting the fact that the DFT of a sequence of length N can be written as the sum of two DFTs of length N/2.
Applying this decomposition recursively, splitting each half into its own two halves, reduces the total number of multiplications to O(N log N). For N = 4096, this is roughly a 340x speedup over the naive approach!

More formally, the DFT of a sequence of length N can be written as the sum of the DFT over the even-indexed elements and the DFT over the odd-indexed elements:

where E_k and O_k are the DFTs of the even and odd subsequences.

Carr-Madan

The Carr-Madan method (1999) is one of the most popular option pricing methods using the Fast Fourier Transform.

Let s = log S_T and k = log K. The risk-neutral density of s is q_T(s), with characteristic function

The call price as a function of log strike is:

We want to Fourier transform C_T(k). The problem is that C_T(k) is not square-integrable, since as k→ -infinity, C_T(k) → S_0 rather than decaying to zero. So its Fourier Transform doesn’t exist in the classical sense.

The fix is to multiply by an exponential dampening factor. Define the modified call price:

For suitable alpha, c_T(k) is square integrable, since the e^{alpha k) term decays fast enough as k → -infty to kill the blow-up. Therefore, this has a well-defined Fourier transform:

Our goal now is to write an analytical expression of Psi_T in terms of phi_T, and then obtain call prices numerically using the inverse transform.

Deriving Psi_T

Substitute the definitions of c_T(k) and C_T(k):

The region of integration is

which is equivalent to:

The integrand is non-negative, so we can apply Tonelli’s theorem and swap the order of integration:

Expand the inner integrand:

For the first term:

For the second term:

Substituting back:

Factor out

and recognise that since

we have:

So:

After simplifying, we finally get:

This is expressed entirely in terms of phi_T, the characteristic function of log S_T under our model of choice, which we said earlier we often have a closed-form solution for, so this whole thing is closed-form!

Recovery Formula

Since Psi_T is the Fourier transform of the dampened call price, standard Fourier inversion gives:

Dividing both sides by e^{alpha k}:

Since C_T(k) is real, we have

so the integrand over (-infty, 0) is the complex conjugate of the integrand over (0,infty), and their sum is twice the real part:

Choice of alpha

Two conditions constrain alpha. First, for Psi_T(0) to be finite, we need

Another condition is alpha not being equal to 0, to avoid a singularity in the denominator at v = 0. For Black-Scholes and Heston, all positive moments exist, so alpha is unconstrained. Carr and Madan suggest using one quarter of the upper bound implied by the moment condition, and find alpha = 1.5 works well in practice.

Truncation bound

The recovery integral runs from 0 to infinity. We need to verify it converges and can be safely truncated at some finite value a. For that, let’s first look at the numerator of |Psi_T(v)|.

It’s easy to see that

which is independent of v.

The denominator of Psi_T(v) grows like v^4 for large v, so:

for some constant A. Therefore

and the tail integral from any truncation point a satisfies:

Thus, the truncation error is bounded by:

To achieve a target error epsilon, choose:

FFT discretization

Discretize the recovery integral with N points at spacing eta, setting

Applying the trapezoid rule:

The effective upper limit of integration is a = N*eta. We want to evaluate this at a grid of N log-strikes

which gives us log strike levels ranging from -b to b, centred around ATM by setting

Substituting v_j and k_u:

For the sum to match the DFT structure

we need

This is the grid coupling constraint; Fine integration grid (eta small) forces coarse strike spacing (lambda large) and vice versa. The sum becomes:

which is exactly the DFT of the sequence

A single FFT call evaluates this for all N strikes simultaneously.

So you build the input vector

once and then call the FFT algorithm. Then multiply the u-th output by

to get the final call prices for each strike.

Simpson’s rule

The trapezoid rule has errors O(eta^2). To get accurate integration with larger eta, which, via the grid coupling constraint, means finer strike spacing, we replace the uniform weight eta with Simpson’s rule weights:

where delta_{j-1} is the Kronecker delta, equal to 1 only at j=1. The error improves from O(eta^2) to O(eta^4), and the FFT structure is preserved since replacing eta with w_j just multiplies each term by a scalar before applying the FFT.

The final formula is:

The weakness of Carr-Madan

The grid coupling constraint is the fundamental limitation; you cannot independently choose integration accuracy and strike grid density. Additionally, alpha requires model-specific tuning. Both issues are addressed by the methods that follow.

Python Implementation

We will be working with Heston in this article. Let’s first implement Monte Carlo pricing using Euler-Maruyama as a ground truth and for speed comparison:

import numpy as np
from scipy.stats import norm
import time
import matplotlib.pyplot as plt
from scipy.integrate import quad

def heston_mc(S0, K, r, q, T, kappa, theta, eta, rho, v0,
              n_steps=200, n_paths=1_000_000, seed=42):
    rng = np.random.default_rng(seed)
    dt  = T / n_steps
    S   = np.full(n_paths, float(S0))
    v   = np.full(n_paths, float(v0))

    for _ in range(n_steps):
        Z1    = rng.standard_normal(n_paths)
        Z2    = rho * Z1 + np.sqrt(1 - rho**2) * rng.standard_normal(n_paths)
        v_pos = np.maximum(v, 0)
        S    *= np.exp((r - q - 0.5 * v_pos) * dt + np.sqrt(v_pos * dt) * Z1)
        v    += kappa * (theta - v_pos) * dt + eta * np.sqrt(v_pos * dt) * Z2

    return np.exp(-r * T) * np.mean(np.maximum(S - K, 0))

Next, we implement the Heston Characteristic Function:

def heston_cf(omega, S0, r, q, T, kappa, theta, eta, rho, v0):
    x  = np.log(S0)
    xi = kappa - 1j * rho * eta * omega
    d  = np.sqrt(xi**2 + eta**2 * (omega**2 + 1j * omega))
    g  = (xi - d) / (xi + d)

    C = ((r - q) * 1j * omega * T
         + (kappa * theta / eta**2) * (
             (xi - d) * T - 2 * np.log((1 - g * np.exp(-d * T)) / (1 - g))
         ))
    D = ((xi - d) / eta**2) * (1 - np.exp(-d * T)) / (1 - g * np.exp(-d * T))

    return np.exp(C + D * v0 + 1j * omega * x)

and finally, the FFT implementation of Carr-Madan using Simpson weights:

def carr_madan_fft(S0, r, q, T, kappa, theta, eta, rho, v0,
                   alpha=1.5, N=4096, eta_grid=0.25):
    lam = 2 * np.pi / (N * eta_grid)
    b   = 0.5 * N * lam

    j   = np.arange(1, N + 1)
    v_j = eta_grid * (j - 1)

    cf  = lambda omega: heston_cf(omega, S0, r, q, T, kappa, theta, eta, rho, v0)

    psi = (np.exp(-r * T) * cf(v_j - (alpha + 1) * 1j)
           / (alpha**2 + alpha - v_j**2 + 1j * (2 * alpha + 1) * v_j))

    w   = (eta_grid / 3) * (3 + (-1)**j - (j == 1).astype(float))
    k_u = -b + lam * (np.arange(1, N + 1) - 1)

    x       = np.fft.fft(np.exp(1j * b * v_j) * psi * w)
    calls   = (np.exp(-alpha * k_u) / np.pi) * np.real(x)
    strikes = np.exp(k_u)

    return strikes, calls

Let’s compare MC to Carr-Mada:

# ── parameters ───────────────────────────────────────────────────────────────

params = dict(S0=100, r=0.05, q=0, T=1.0,
              kappa=2.0, theta=0.04, eta=0.3, rho=-0.7, v0=0.04)

strikes_to_price = np.arange(60, 141, 5, dtype=float)

# ── MC ground truth ───────────────────────────────────────────────────────────

t0 = time.perf_counter()
mc_prices = np.array([heston_mc(**params, K=K) for K in strikes_to_price])
mc_time   = time.perf_counter() - t0

# ── Carr-Madan ────────────────────────────────────────────────────────────────

t0 = time.perf_counter()
cm_strikes, cm_calls = carr_madan_fft(**params)
cm_time = time.perf_counter() - t0
cm_prices = np.interp(strikes_to_price, cm_strikes, cm_calls)

# ── print results ─────────────────────────────────────────────────────────────

print(f"{'Strike':>8} {'MC':>10} {'Carr-Madan':>12} {'Error':>10}")
print("-" * 44)
for K, mc, cm in zip(strikes_to_price, mc_prices, cm_prices):
    print(f"{K:>8.0f} {mc:>10.4f} {cm:>12.4f} {abs(mc-cm):>10.4f}")

print(f"\nMC time:          {mc_time:.2f}s  ({len(strikes_to_price)} strikes, 100k paths each)")
print(f"Carr-Madan time:  {cm_time*1000:.2f}ms (all strikes simultaneously)")

# ── plot ──────────────────────────────────────────────────────────────────────

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(strikes_to_price, mc_prices, 'o', label='Monte Carlo', color='steelblue')
ax1.plot(strikes_to_price, cm_prices, '-', label='Carr-Madan', color='tomato', linewidth=2)
ax1.set_xlabel('Strike')
ax1.set_ylabel('Call Price')
ax1.set_title('Heston Call Prices')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.bar(['Monte Carlo\n(100k paths, 17 strikes)', 'Carr-Madan FFT\n(all strikes)'],
        [mc_time, cm_time],
        color=['steelblue', 'tomato'])
ax2.set_ylabel('Time (seconds)')
ax2.set_title('Computation Time')
ax2.grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

  Strike         MC   Carr-Madan      Error
--------------------------------------------
      60    43.0926      43.0487     0.0439
      65    38.4447      38.4019     0.0428
      70    33.8703      33.8281     0.0423
      75    29.4033      29.3614     0.0420
      80    25.0861      25.0447     0.0415
      85    20.9701      20.9298     0.0403
      90    17.1172      17.0758     0.0415
      95    13.5838      13.5438     0.0401
     100    10.4286      10.3950     0.0336
     105     7.7081       7.6798     0.0283
     110     5.4500       5.4306     0.0194
     115     3.6676       3.6572     0.0104
     120     2.3451       2.3334     0.0117
     125     1.4146       1.4061     0.0085
     130     0.8051       0.8002     0.0049
     135     0.4343       0.4309     0.0034
     140     0.2238       0.2206     0.0031

MC time:          6.46s  (17 strikes, 100k paths each)
Carr-Madan time:  1.05ms (all strikes simultaneously)

The results speak for themselves! Even at 100k paths, which isn’t all that much, and gives somewhat inaccurate results, running MC takes around 6.5 seconds, while Carr-Madan only takes a millisecond. But we can do even better!

Lewis

Lewis (2001) reformulates option pricing as a complex contour integral. The result is a cleaner formula than Carr-Mada; no dampening parameter alpha to tune, and the strip condition is easily satisfied for essentially all models used in practice.

The core idea

The option price is a discounted expectation under Q (We are restating this since the paper has slightly different notation):

where x = log S_T, w(x) is the payoff function, and q_T(x) is the risk-neutral density of log S_T. This is an inner product of w and q_T. Parseval’s theorem says inner products are preserved under Fourier transformation:

Since q_T(x) is real, we have

So:

The problem

That formula looks clean, but there’s a catch. For a call, the payoff is

which grows like e^x as x → infty. Its ordinary Fourier transform diverges!

Carr-Madan’s fix was to multiply the payoff by a dampening factor before transforming. Lewis takes a cleaner route: allow the transform variable to be complex.

Generalized Fourier Transform

Let z = u+iv be a complex number. Define the generalized Fourier transform of the payoff:

Now e^{izx} = e^{iux-vx}, and the factor e^{-vx} provides the decay that kills the growth of w(x), as long as v is large enough. For the call:

This does not converge unless Im(z) > 1. With Im(z) > 1 we have:

We call this area the strip of regularity for the call payoff:

You can refer to the following table from the paper for other options:

The characteristic function has its own regularity strip. For the characteristic function of X_T to exist, we need

The set of valid complex v with this property forms the strip S_X.

Define the reflected strip

This is where phi_T(-z) is regular, which is what appears in the pricing formula.

Shifting the contour

By Cauchy’s theorem, the integration contour may be shifted to any horizontal line inside the common strip of regularity:

The Lewis Formula

The option price finally becomes:

where v = Im(z) in S_V and Y = ln S_0 + (r-q)T is the log forward price, and phi denotes the characteristic function of the centred log-return X_T = log S_T - Y.

Call Pricing Formula

Substituting the call payoff transform hat{w} from above:

where b is some model-dependent upper bound below which the characteristic function stays regular. Lewi’s preferred contour is v=1/2, obtained by shifting the contour from (1,b) down to (0,1) via the residue theorem and picking up the pole at z=i. That gives

where k = ln(S_0/K) + (r-q)T is the log-moneyness. The contour v=1/2 sits symmetrically between the poles at z=0 and z=i, and requires only

satisfies by practically every standard model, and is a much weaker requirement than

that Carr-Madan needs for alpha > 0.

Python Implementation

Lewis uses the characteristic function of X_T:

def heston_cf_xt(omega, S0, r, q, T, kappa, theta, eta, rho, v0):
    """CF of X_T = log(S_T/S_0) - (r-q)T under Heston."""
    xi = kappa - 1j * rho * eta * omega
    d  = np.sqrt(xi**2 + eta**2 * (omega**2 + 1j * omega))
    g  = (xi - d) / (xi + d)

    C = (kappa * theta / eta**2) * (
            (xi - d) * T - 2 * np.log((1 - g * np.exp(-d * T)) / (1 - g))
        )
    D = ((xi - d) / eta**2) * (1 - np.exp(-d * T)) / (1 - g * np.exp(-d * T))

    # no (r-q) drift term since X_T is already centered
    return np.exp(C + D * v0 - 0.5 * 1j * omega * eta**2 * 0)

Now since Lewis doesn’t use FFT by design, we have to compute strikes one by one using the contour integral, or we are smart and vectorize:

def lewis_calls_vectorized(S0, strikes, r, q, T, kappa, theta, eta, rho, v0,
                           N=4096, eta_grid=0.25):
    u_j = eta_grid * np.arange(0, N)
    
    # Simpson weights
    w = np.ones(N) * 2
    w[1::2] = 4
    w[0] = w[-1] = 1
    w *= eta_grid / 3

    phi = heston_cf_xt(u_j - 0.5j, S0, r, q, T, kappa, theta, eta, rho, v0)
    phi /= (u_j**2 + 0.25)

    # k for each strike: shape (n_strikes,)
    k = np.log(S0 / strikes) + (r - q) * T

    # outer product: shape (n_strikes, N)
    phase = np.exp(1j * np.outer(k, u_j))

    I = np.real(phase * phi[np.newaxis, :]) @ w / np.pi

    return S0 * np.exp(-q * T) - np.sqrt(S0 * strikes) * np.exp(-(r + q) * T / 2) * I

And the comparison:

t0 = time.perf_counter()
lw_prices = lewis_calls_vectorized(strikes=strikes_to_price, **params)
lw_time = time.perf_counter() - t0

print(f"{'Strike':>8} {'MC':>10} {'Carr-Madan':>12} {'Lewis':>10} "
      f"{'CM Err':>10} {'LW Err':>10}")
print("-" * 64)
for K, mc, cm, lw in zip(strikes_to_price, mc_prices, cm_prices, lw_prices):
    print(f"{K:>8.0f} {mc:>10.4f} {cm:>12.4f} {lw:>10.4f} "
          f"{abs(mc-cm):>10.4f} {abs(mc-lw):>10.4f}")

print(f"\nMC time:          {mc_time:.2f}s")
print(f"Carr-Madan time:  {cm_time*1000:.2f}ms")
print(f"Lewis time:       {lw_time*1000:.2f}ms")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(strikes_to_price, mc_prices, 'o', label='Monte Carlo', color='steelblue')
ax1.plot(strikes_to_price, cm_prices, '-', label='Carr-Madan', color='tomato', linewidth=2)
ax1.plot(strikes_to_price, lw_prices, '--', label='Lewis', color='seagreen', linewidth=2)
ax1.set_xlabel('Strike')
ax1.set_ylabel('Call Price')
ax1.set_title('Heston Call Prices')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.bar(['Carr-Madan\nFFT', 'Lewis\nVectorized'],
        [cm_time, lw_time],
        color=['tomato', 'seagreen'])
ax2.set_ylabel('Time (seconds)')
ax2.set_title('Computation Time (excl. Monte Carlo)')
ax2.grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

  Strike         MC   Carr-Madan      Lewis     CM Err     LW Err
----------------------------------------------------------------
      60    43.0926      43.0487    43.1459     0.0439     0.0533
      65    38.4447      38.4019    38.5020     0.0428     0.0573
      70    33.8703      33.8281    33.9311     0.0423     0.0608
      75    29.4033      29.3614    29.4673     0.0420     0.0640
      80    25.0861      25.0447    25.1536     0.0415     0.0674
      85    20.9701      20.9298    21.0417     0.0403     0.0716
      90    17.1172      17.0758    17.1902     0.0415     0.0730
      95    13.5838      13.5438    13.6612     0.0401     0.0774
     100    10.4286      10.3950    10.5150     0.0336     0.0864
     105     7.7081       7.6798     7.8025     0.0283     0.0944
     110     5.4500       5.4306     5.5570     0.0194     0.1070
     115     3.6676       3.6572     3.7858     0.0104     0.1182
     120     2.3451       2.3334     2.4652     0.0117     0.1201
     125     1.4146       1.4061     1.5412     0.0085     0.1266
     130     0.8051       0.8002     0.9381     0.0049     0.1329
     135     0.4343       0.4309     0.5717     0.0034     0.1375
     140     0.2238       0.2206     0.3646     0.0031     0.1409

MC time:          8.09s
Carr-Madan time:  1.18ms
Lewis time:       2.49ms

As you can see, when using the same N and eta as with Carr-Madan, Lewis starts diverging pretty quickly and is still over twice as slow. If we increase N and reduce eta, we are quickly multiple times slower than Carr-Madan when pricing many strikes at once. On the other hand, we get rid of the dampening parameter alpha, which can cause real problems.

Let’s finally look at our method of choice, which performs better than both Lewis and Carr-Madan.

The COS Method

Both Carr-Madan and Lewis price options by evaluating a Fourier integral numerically. The integrand is oscillatory, which is expensive to integrate accurately with equally spaced quadrature.

The COS method (2008) takes a different route. Instead of integrating the characteristic function directly, it expands the risk-neutral density in a cosine series and reads the series coefficients directly off the characteristic function, with those coefficients often being available analytically.

Fourier-Cosine Series Expansion

Starting from risk-neutral pricing (once again, slightly different notation!):

where x=ln(S_0/K), y = ln(S_T/K), v(y,T) is the payoff, and f(y|x) is the risk-neutral transition density of the log-return.

We truncate to a finite interval [a,b]:

Now expand f(y|x) in a cosine series on [a,b]:

where the ‘ in the sum means the k=0 term is weighted by 1/2, and the cosine coefficients are:

Substituting into the pricing integral and swapping sum and integral (which we can do thanks to Fubini):

where:

Note that the V_k are the cosine series coefficients of v(y,T) in y. Thus we have transformed the product of two real functions, f(y|x) and v(y,T), to that of their Fourier-cosine series coefficients.

Coefficients from the characteristic function

The density f(y|x) is unknown, that’s the whole reason we use Fourier methods in the first place. Comparing the definition of A_k(x) with the characteristic function phi_T(w), one finds that:

where phi_1 is the characteristic function computed over the truncated range [a,b]. Since [a,b] was chosen to capture essentially all the density mass, phi_1 is approx phi and so:

The COS Formula

Finally, replacing A_k by F_k in the formula for V(x, t_0), we obtain:

This is the COS formula.

Payoff coefficients for calls and puts

For a call, v(y,T) = K(e^y-1)^+, the integration domain reduces to [0,b] since the payoff is zero for y < 0. The coefficients are:

where chi and psi have closed forms:

For a put the integration domain is [a,0], and

In practice, you can price puts or calls, and recover the other via put-call parity

Truncation Range

The interval [a,b] must contain enough of the density without being so wide that the cosine approximation needs excessive terms to resolve the tails. Fang & Oosterlee propose using cumulants of ln(S_T/K):

where c_1, c_2, c_4 are the first, second and fourth cumulants of ln(S_T/K). Including c_4 matters for short maturities and fat-tailed processes where the density is sharply peaked.

You can additionally also include a c_6 term as well for extremely short maturities, but the sixth cumulant can be difficult to derive for many models. You can also go the other direction and only include the c_1 and c_2 terms.

Benefits of the COS method

For Carr-Madan and Lewis, accuracy is limited by how well a trapezoidal or Simpson sum approximates a slowly decaying oscillatory integral. The COS method avoids this entirely. For smooth densities like GBM, Heston, and many Lévy processes at moderate maturities, the density is C^{infty} on [a,b] and the cosine coefficients decay exponentially, yielding exponential convergence.

The computational complexity is O(N) for a single strike. For a vector of strikes, the F_k are the same across all strikes, and the V_k differ only through the x-dependent phase; pricing M strikes simultaneously costs O(MN) via matrix-vector multiplication.

Python Implementation

We need the first and second cumulants of log(S_T/K) under Heston:

def heston_cumulants(r, q, T, kappa, theta, eta, rho, v0):
    """First and second cumulants of log(S_T/K) under Heston."""
    c1 = (r - q) * T + (1 - np.exp(-kappa * T)) * (theta - v0) / (2 * kappa) - 0.5 * theta * T

    c2 = (1 / (8 * kappa**3)) * (
        eta * T * kappa * np.exp(-kappa * T) * (v0 - theta) * (8 * kappa * rho - 4 * eta)
        + kappa * rho * eta * (1 - np.exp(-kappa * T)) * (16 * theta - 8 * v0)
        + 2 * theta * kappa * T * (-4 * kappa * rho * eta + eta**2 + 4 * kappa**2)
        + eta**2 * ((theta - 2 * v0) * np.exp(-2 * kappa * T)
                    + theta * (6 * np.exp(-kappa * T) - 7) + 2 * v0)
        + 8 * kappa**2 * (v0 - theta) * (1 - np.exp(-kappa * T))
    )

    return c1, c2

Fast vectorized implementation of the COS method:

def cos_calls(S0, strikes, r, q, T, kappa, theta, eta, rho, v0, N=128, L=12):
    c1, c2 = heston_cumulants(r, q, T, kappa, theta, eta, rho, v0)
    a = c1 - L * np.sqrt(abs(c2))
    b = c1 + L * np.sqrt(abs(c2))

    k     = np.arange(N)
    omega = k * np.pi / (b - a)

    def varphi(omega):
        xi = kappa - 1j * rho * eta * omega
        d  = np.sqrt(xi**2 + eta**2 * (omega**2 + 1j * omega))
        g  = (xi - d) / (xi + d)
        C  = ((r - q) * 1j * omega * T
              + (kappa * theta / eta**2) * (
                  (xi - d) * T - 2 * np.log((1 - g * np.exp(-d * T)) / (1 - g))
              ))
        D  = ((xi - d) / eta**2) * (1 - np.exp(-d * T)) / (1 - g * np.exp(-d * T))
        return np.exp(C + D * v0)

    def chi(c, d):
        kpi = np.where(k != 0, omega, 1.0)
        val = (np.cos(kpi * (d - a)) * np.exp(d)
             - np.cos(kpi * (c - a)) * np.exp(c)
             + kpi * np.sin(kpi * (d - a)) * np.exp(d)
             - kpi * np.sin(kpi * (c - a)) * np.exp(c)) / (1 + kpi**2)
        val[0] = np.exp(d) - np.exp(c)
        return val

    def psi(c, d):
        val      = np.empty(N)
        val[0]   = d - c
        val[1:]  = (b - a) / (k[1:] * np.pi) * (
                       np.sin(omega[1:] * (d - a)) - np.sin(omega[1:] * (c - a))
                   )
        return val

    # precompute strike-independent quantities
    basis    = varphi(omega) * np.exp(-1j * omega * a)  # vphi * e^{-i*omega*a}
    A        = np.real(basis)
    B        = np.imag(basis)
    Uk       = (2 / (b - a)) * (-chi(a, 0) + psi(a, 0))

    # cache constants
    disc     = np.exp(-r * T)
    fwd_S0   = S0 * np.exp(-q * T)
    disc_K   = np.exp(-r * T)

    # vectorize over strikes
    x        = np.log(S0 / strikes)                    # shape (M,)
    theta_mx = np.outer(x, omega)                      # shape (M, N)
    F        = A * np.cos(theta_mx) - B * np.sin(theta_mx)  # shape (M, N)
    F[:, 0] *= 0.5

    puts  = disc * strikes * (F @ Uk)
    calls = puts + fwd_S0 - strikes * disc_K
    return calls

And last but not least, the comparison code:

t0 = time.perf_counter()
cos_prices = cos_calls(strikes=strikes_to_price, **params)
cos_time = time.perf_counter() - t0

print(f"{'Strike':>8} {'MC':>10} {'Carr-Madan':>12} {'Lewis':>10} {'COS':>10} "
      f"{'CM Err':>10} {'LW Err':>10} {'COS Err':>10}")
print("-" * 84)
for K, mc, cm, lw, cos in zip(strikes_to_price, mc_prices, cm_prices, lw_prices, cos_prices):
    print(f"{K:>8.0f} {mc:>10.4f} {cm:>12.4f} {lw:>10.4f} {cos:>10.4f} "
          f"{abs(mc-cm):>10.4f} {abs(mc-lw):>10.4f} {abs(mc-cos):>10.4f}")

print(f"\nMC time:          {mc_time:.2f}s")
print(f"Carr-Madan time:  {cm_time*1000:.2f}ms")
print(f"Lewis time:       {lw_time*1000:.2f}ms")
print(f"COS time:         {cos_time*1000:.2f}ms")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(strikes_to_price, mc_prices, 'o', label='Monte Carlo', color='steelblue')
ax1.plot(strikes_to_price, cm_prices, '-', label='Carr-Madan', color='tomato', linewidth=2)
ax1.plot(strikes_to_price, lw_prices, '--', label='Lewis', color='seagreen', linewidth=2)
ax1.plot(strikes_to_price, cos_prices, ':', label='COS', color='purple', linewidth=2)
ax1.set_xlabel('Strike')
ax1.set_ylabel('Call Price')
ax1.set_title('Heston Call Prices')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.bar(['Carr-Madan\nFFT', 'Lewis\nVectorized', 'COS'],
        [cm_time, lw_time, cos_time],
        color=['tomato', 'seagreen', 'purple'])
ax2.set_ylabel('Time (seconds)')
ax2.set_title('Computation Time (excl. Monte Carlo)')
ax2.grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

  Strike         MC   Carr-Madan      Lewis        COS     CM Err     LW Err    COS Err
------------------------------------------------------------------------------------
      60    43.0926      43.0487    43.1459    43.0487     0.0439     0.0533     0.0439
      65    38.4447      38.4019    38.5020    38.4019     0.0428     0.0573     0.0428
      70    33.8703      33.8281    33.9311    33.8280     0.0423     0.0608     0.0424
      75    29.4033      29.3614    29.4673    29.3612     0.0420     0.0640     0.0421
      80    25.0861      25.0447    25.1536    25.0446     0.0415     0.0674     0.0416
      85    20.9701      20.9298    21.0417    20.9297     0.0403     0.0716     0.0403
      90    17.1172      17.0758    17.1902    17.0753     0.0415     0.0730     0.0419
      95    13.5838      13.5438    13.6612    13.5434     0.0401     0.0774     0.0404
     100    10.4286      10.3950    10.5150    10.3942     0.0336     0.0864     0.0344
     105     7.7081       7.6798     7.8025     7.6788     0.0283     0.0944     0.0293
     110     5.4500       5.4306     5.5570     5.4303     0.0194     0.1070     0.0197
     115     3.6676       3.6572     3.7858     3.6562     0.0104     0.1182     0.0114
     120     2.3451       2.3334     2.4652     2.3326     0.0117     0.1201     0.0125
     125     1.4146       1.4061     1.5412     1.4057     0.0085     0.1266     0.0089
     130     0.8051       0.8002     0.9381     0.7996     0.0049     0.1329     0.0055
     135     0.4343       0.4309     0.5717     0.4303     0.0034     0.1375     0.0039
     140     0.2238       0.2206     0.3646     0.2203     0.0031     0.1409     0.0035

MC time:          8.09s
Carr-Madan time:  1.18ms
Lewis time:       2.49ms
COS time:         0.34ms

Awesome! We are more than 3 times faster than Carr-Madan, while getting rid of annoying parameters and gaining exponential convergence!

The method naturally vectorizes into dense linear algebra operations, meaning you could still make this way way faster if you optimized it properly.

Conclusion

Apologies for the delay since the last article! This one took a little longer than I expected since I had to verify all the math, and turn 3 papers into one article instead of the usual one paper.

I hope you enjoyed this rather math-heavy article anyway! I plan on covering more options content soon.

Other Articles You Would Enjoy

Quant Corner

Join Quant Corner, a community that actually does things. Tournaments, live Q&As, open discussions, and the best place to shape what gets built next.

Join here: https://discord.gg/X7TsxKNbXg

The problem is straightforward: When you post a limit order, you know your price and your size, but you don’t know where you actually sit in the queue and how much volume is ahead of you before you get filled. An order at the front of a 10 BTC queue has a completely different fill probability than one at the back of it.

There is one particular blog post by Rigtop (2013) that proposed splitting observed volume changes proportionally between ahead and behind. It works as a heuristic, but it has a problem: it treats cancellations as events where some volume gets cancelled ahead of you and some behind, but in reality, each cancellation is either fully ahead of you or fully behind you. This means it also only gives you a point estimate of where in the queue you are, with no measure of uncertainty.

We do something more powerful. We maintain a distribution over all possible queue positions using a particle filter, which gives us a whole distribution of where we likely are in the queue.

We derive the framework from scratch, implement it live on Bybit, and show the particle distribution evolving live in real time.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why market-by-price data makes queue position a latent variable, and why the naive proportional heuristic isn’t enough.

• How to model queue position as a distribution rather than a point estimate, and why that matters for execution and market making.

• How to derive a particle filter for queue position from scratch, handling trades, cancellations, and fills correctly.

• How to implement it live with Bybit’s WebSocket feed in Python.

• What the queue position distribution actually looks like in real life, and how it evolves as the market moves.

Read more

The models we used to forecast returns were themself not “online” though. They were trained on a warm-up set and then kept frozen. The first improvement that comes to mind is simply retraining them periodically, but it turns out there is a more powerful method that lets you learn every single timestep and not miss out on crucial shifts.

That’s what we’ll be implementing in this article.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why rolling window retraining is fundamentally limited and what assumptions it silently makes about your data.

• How AROWR works: a second-order Bayesian online regression algorithm that maintains a full posterior over model weights and updates every single timestep.

• How ARCOR extends AROWR with a principled covariance reset mechanism that detects when the market has changed and restores the model's ability to adapt.

• How to implement AROWR and ARCOR from scratch in Python and apply them to any regression problem that updates in real time.

• How we applied ARCOR to BTC beta estimation across 355 crypto assets over 4 years, significantly improving performance over rolling OLS, especially during LUNA, FTX, and similar situations.

Read more

Maybe one is a linear autoregressive model that performs well in trending markets. Another is a mean-reversion model, and a third is a more complex machine learning model that captures nonlinear patterns. Each has its strengths and weaknesses, and most importantly, you don’t know in advance which one will be best tomorrow.

The naive approach is to pick the best model in-sample and deploy it. Another naive approach is equal weighting. But those two approaches ignore everything you learn as new data arrives.

What if there exists a principled way to combine your models that:

• Allocates a lot of weight to models that currently perform well and less weight to models that perform poorly.

• Requires no retraining, no hyperparameter tuning, no rebalancing decisions.

• Comes with a mathematical guarantee that you perform nearly as well as the best model.

In this article, we present an algorithm that is able to achieve all of the above, demonstrate its behavior, and show how this model shines when there are different regimes where different models perform best.

I write about quantitative trading the way it’s actually practiced:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn:

• The theoretical framework of online learning and how it applies to forecast combination.

• What regret is, why minimizing it is the right objective, and how it differs from standard in-sample loss minimization.

• Why square loss has a special property called 2-mixability that enables a provably optimal combination algorithm.

• How the aggregating forecaster works, why it is minimax optimal, and how to implement it from scratch in Python.

• How to extend the algorithm with polynomial discounting to handle non-stationary markets with regime changes.

• Why the optimal discount rate depends on regime persistence, and how to tune it empirically.

Read more

What is the point of such a measure?

Let’s say you are testing a moving-average crossover strategy, and you test 20 different combinations of moving-average lookbacks. Even for different parameter settings, the strategy returns will be correlated, so you didn’t really test 20 different strategies.

Let’s now dive into ways of measuring this and why it’s incredibly useful.

I write about quantitative trading the way it’s actually practiced:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

What you’ll learn:

• Why the raw number of tested strategies dramatically overstates overfitting risk.

• Why dependence geometry determines multiple-testing complexity.

• How expected chi-squared maxima define a continuous effective number of tests.

• How to compute K_eff numerically.

The 5 Axioms

If I think of a measure of how many strategies were effectively tested, I think of a few intuitive axioms that such a measure should follow:

• 1 <= K_eff <= K: You can’t test fewer than one strategy, and you can’t test more strategies than you’ve actually tested.

• K_eff = 1 iff all strategies are perfectly correlated/anticorrelated: If all your strategies are perfectly correlated/anticorrelated, then you’ve effectively only tested a single strategy.

• K_eff = K iff all strategies are uncorrelated: If all your strategies are perfectly uncorrelated (or independent, which is the same in the Gaussian world), then you can’t get more information; you really tested as many strategies as you got.

• K_eff is non-decreasing in K: If we test more strategies, our effective number of strategies that we tested can’t decrease; it should only increase, or stay constant if the new strategy is perfectly correlated/anticorrelated to a previously tested strategy.

• The expectancy of the maximum of the absolute Z-scores of your K strategies grows asymptotically like the square root of twice the logarithm of K_eff: Okay, this one may not be so intuitive… Let me explain! Let Z_i be the Z-score of strategy i. We assume the Z’s to be standard Gaussian random variables with covariance matrix Sigma. A standard result in extreme value theory is that:

  We want K_eff to be defined so that your correlated portfolio of K strategies looks exactly like K_eff independent strategies in terms of expected best absolute Z-score.

Why Popular Measures Fail

I’m not the first one to come up with a notion of “effective number of strategies tested”, so let’s look at some of the existing definitions and where they ultimately break down.

Spectral Participation Rate

Spectral Participation Rate is the measure that was used in the following article:

It’s defined as follows:

where λ_1, …, λ_K are the eigenvalues of Sigma, which is the correlation matrix of our Z-scores.

By construction, K_eff = K under independence and decreases toward 1 as configurations become collinear. And there you have your problem, K_eff decreases toward 1 as configurations become collinear. Let’s say you have strategies A and B, which are completely independent, so K_eff = 2. Now imagine you were to test strategy B over and over again. You would want K_eff to stay 2, but something else happens.

If you keep adding copies of strategy B, your correlation matrix becomes:

The eigenvalues of Sigma are:

So:

And as K goes to infinity, K_eff converges towards 1!

While this is an extreme example, it happens in reality with adaptive search as well. In Bayesian optimization, you explore parameter areas that previously gave you good results more thoroughly, which in turn gives you a highly correlated strategy and causes K_eff to decrease.

Axiom 4 is clearly violated, and thus, this is not usable for us.

Effective Rank

Effective rank is defined in Roy & Vetterli (2007), and measures effective dimensionality by providing a real-valued extension to the rank of a matrix.

Consider the singular value decomposition (SVD) of Sigma:

where U and V are unitary matrices, and D is a diagonal matrix containing the singular values

Let the singular value distribution be

where sigma is the vector of singular values and ||.||_1 the l1-norm.

The effective rank of the matrix Sigma is defined as

where H is the Shannon entropy given by

We will use this effective rank as our K_eff.

Consider again the scenario of independent strategies A and B, and testing strategy B over and over again. Since Sigma is SPD, the eigenvalues and singular values are the same:

The l1 norm is:

And the singular value distribution is:

With this, erank(Sigma), or K_eff, becomes

This suffers from the same problem as the spectral participation rate! The problem is that those two measures only look at the shape of the spectrum, not its scale. Adding more copies of B drowns out the contribution of A in the normalized distribution, pulling K_eff towards 1.

Bailey & López de Prado

In the paper "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality", Bailey and López de Prado present another method of measuring the effective number of strategies tested.

They define

where the equal-weighted average pairwise correlation is

Here is an example where this measure also breaks down:

Let A and B be two perfectly anti-correlated strategies. Then

and the average pairwise correlation is -1.

We get

With this definition, our effective number of strategies tested is 5, even though we only tested 2 strategies. It also violates a bunch of other axioms.

A Definition That Works

Here is what I propose. Let M(x) be the expected maximum of x i.i.d. squared standard normal random variables:

Equivalently, since squaring a standard normal random variable yields a chi-squared random variable, this is the expected maximum of x i.i.d. chi-squared random variables. Using the PDF and CDF, we write the expected maximum as:

M is strictly increasing and continuous, so it has a well-defined inverse

We then define the effective number of tested strategies as:

where Z_1, …, Z_K are the actual strategy Z-scores with correlation matrix Sigma. In other words, K_eff is the number of independent strategies that would produce the same expected best squared Z-score as your actual correlated portfolio of K strategies.

Now, let’s prove that this definition actually follows the 5 axioms.

Axiom 1

Since M is strictly increasing, it suffices to show that

Lower bound:

Upper bound:

By Šidák’s inequality for centered Gaussian vectors,

Since each Z_i is standard normal, we have

and hence

where st means stochastically dominated. And because x → x^2 is increasing,

Taking expectations gives

Axiom 2

If all strategies are perfectly correlated or anticorrelated, then for every i,j,

Since each Z_i is standard Gaussian, this implies there exists a single standard normal random variable X and signs epsilon_i, either -1 or 1, such that

Therefore,

so

Taking expectations,

Applying the inverse of M,

Now the other direction. Assume K_eff = 1. Since M(1) = 1,

Since each Z_i is standard Gaussian, we have E[Z_i^2] = 1.

Since

the equality

can occur only if

Thus

For jointly Gaussian variables, this implies

and therefore

Axiom 3

If all strategies are uncorrelated, we have

which implies that

Therefore,

Applying the inverse of M, we get

Now the other direction. Assume K_eff = K. Since M is strictly increasing,

Now let

From the proof of Axiom 1, we know that

i.e.

Using the tail-integral formula,

The integrand is nonnegative everywhere, and the left-hand side is zero since E[X] = E[Y]. Hence

Thus, X and Y have the same distribution. Equivalently,

By the equality case of Šidák’s inequality, this implies that Z_1, …, Z_K are independent, which for Gaussian random variables is equivalent to being uncorrelated.

Axiom 4:

We have

Taking expectation

and applying the inverse of M, we get

Axiom 5:

Let

Then

A standard result from Gaussian extreme-value theory states that

Moreover,

is uniformly integrable. Hence, convergence in probability implies convergence of expectations:

Hence

Substituting k = K_eff yields

Since

we obtain

Since Gaussian maxima concentrate sharply, the first and second moments are asymptotically equivalent up to square-root scaling. Therefore,

Simulation

Below is the code used for computing K_eff:

import numpy as np
from scipy.stats import chi2
from scipy.optimize import brentq


def M(x: float, n_quad: int = 4000) -> float:
    """
    Compute M(x) = E[max of x i.i.d. chi^2_1 variables] via numerical
    quadrature, working in log space to avoid underflow.

    Parameters
    ----------
    x       : effective count (any real >= 1)
    n_quad  : number of quadrature points

    Returns
    -------
    float : E[max_i Z_i^2] for x i.i.d. N(0,1) variables
    """
    # Adaptive upper limit: tail of chi^2_1 maximum decays beyond this
    hi = chi2.ppf(1 - 1e-10 / x, df=1) if x > 1 else chi2.ppf(1 - 1e-8, df=1)
    t  = np.linspace(0, hi, n_quad + 1)[1:]  # avoid t=0 where pdf diverges

    log_F   = chi2.logcdf(t, df=1)
    log_f   = chi2.logpdf(t, df=1)
    log_ker = np.log(t) + np.log(x) + log_f + (x - 1) * log_F

    # Zero out terms that underflow (negligible contribution)
    mask       = log_ker > -700
    integrand  = np.zeros_like(t)
    integrand[mask] = np.exp(log_ker[mask])

    return float(np.trapezoid(integrand, t))


def M_inv(val: float, tol: float = 1e-8) -> float:
    """
    Compute M^{-1}(val) via Brent's method.

    Parameters
    ----------
    val : target value, must be >= M(1) = 1
    tol : tolerance for root finding

    Returns
    -------
    float : x such that M(x) = val
    """
    if val <= 1.0:
        return 1.0

    # Bracket: expand upper bound until M(hi) > val
    hi = 2.0
    while M(hi) < val:
        hi *= 2.0

    return brentq(lambda x: M(x) - val, 1.0, hi, xtol=tol)


def K_eff(Sigma: np.ndarray, n_sim: int = 200_000, seed: int = 42) -> float:
    """
    Compute K_eff for a portfolio of K strategies with correlation matrix Sigma.

    Uses Monte Carlo to estimate E[max_i Z_i^2], then inverts M analytically.

    Parameters
    ----------
    Sigma : (K, K) correlation matrix
    n_sim : number of Monte Carlo samples
    seed  : random seed

    Returns
    -------
    float : K_eff in [1, K]
    """
    K   = Sigma.shape[0]
    rng = np.random.default_rng(seed)
    L   = np.linalg.cholesky(Sigma)
    Z   = (L @ rng.standard_normal((K, n_sim)))  # (K, n_sim)
    E_max_Z2 = float(np.mean(np.max(Z**2, axis=0)))
    return M_inv(E_max_Z2)

And now let’s simulate how K_eff evolves as K grows for different families of strategies:

import matplotlib.pyplot as plt


# ============================================================
# Helper: build equicorrelation matrix
# ============================================================

def equicorr_matrix(K, rho):
    """
    Build K x K equicorrelation matrix with off-diagonal rho.
    """

    Sigma = np.full((K, K), rho, dtype=float)
    np.fill_diagonal(Sigma, 1.0)

    return Sigma


# ============================================================
# Sequential simulation
# ============================================================

def sequential_keff(mode, K_max=100):

    K_vals    = []
    Keff_vals = []

    # Persistent storage for sequential random strategies
    random_vectors = []

    rng = np.random.default_rng(123)

    for K in range(1, K_max + 1):

        # ----------------------------------------------------
        # Independent
        # ----------------------------------------------------
        if mode == "independent":

            Sigma = np.eye(K, dtype=float)

        # ----------------------------------------------------
        # Perfect dependence
        # ----------------------------------------------------
        elif mode == "dependent":

            Sigma = np.ones((K, K), dtype=float)

        # ----------------------------------------------------
        # rho = 0.5
        # ----------------------------------------------------
        elif mode == "rho_0.5":

            Sigma = equicorr_matrix(K, 0.5)

        # ----------------------------------------------------
        # Maximally negative equicorrelation
        # ----------------------------------------------------
        elif mode == "negative_boundary":

            if K == 1:
                rho = 0.0
            else:
                # PSD boundary:
                # rho >= -1/(K-1)
                rho = -0.9 / (K - 1)

            Sigma = equicorr_matrix(K, rho)

        # ----------------------------------------------------
        # Sequential random correlated strategies
        # ----------------------------------------------------
        elif mode == "random":

            # Add ONE new latent-factor strategy
            v = rng.standard_normal(5)

            # Normalize
            v /= np.linalg.norm(v)

            random_vectors.append(v)

            X = np.stack(random_vectors)

            # Correlation matrix
            Sigma = X @ X.T

        else:
            raise ValueError(f"Unknown mode: {mode}")

        # ----------------------------------------------------
        # Numerical stabilization
        # ----------------------------------------------------

        Sigma = Sigma.astype(float)

        Sigma += 1e-10 * np.eye(K)

        # ----------------------------------------------------
        # Compute K_eff
        # ----------------------------------------------------

        keff = K_eff(Sigma)

        K_vals.append(K)
        Keff_vals.append(keff)

        print(
            f"{mode:20s} | "
            f"K = {K:3d} | "
            f"K_eff = {keff:8.3f}"
        )

    return np.array(K_vals), np.array(Keff_vals)


# ============================================================
# Run all simulations
# ============================================================

modes = [
    "independent",
    "dependent",
    "rho_0.5",
    "negative_boundary",
    "random"
]

results = {}

for mode in modes:

    K, Keff = sequential_keff(mode, K_max=100)

    results[mode] = (K, Keff)


# ============================================================
# Plot
# ============================================================

plt.figure(figsize=(12, 8))

for mode in modes:

    K, Keff = results[mode]

    plt.plot(K, Keff, linewidth=2, label=mode)

# Reference line
plt.plot(
    np.arange(1, 101),
    np.arange(1, 101),
    linestyle="--",
    linewidth=2,
    label="K_eff = K"
)

plt.xlabel("Raw Number of Tested Strategies (K)")
plt.ylabel("Effective Number of Strategies (K_eff)")

plt.title(
    "Sequential Growth of K_eff Under Different Correlation Structures"
)

plt.legend()
plt.grid(True)

plt.show()

Conclusion

K_eff is not just theoretical; it has immediate practical applications.

The most direct one is improving the Deflated Sharpe Ratio presented by Bailey and De Prado. More broadly, K_eff can be used anywhere you need to correct for multiple testing across correlated strategies.

It can also directly guide you in determining when to stop searching while tuning hyperparameters.

I will explore those and many other things K_eff is capable of, and the math behind K_eff in more detail in future articles, which I then plan to all combine into a research paper!

Join Quant Corner: https://discord.gg/X7TsxKNbXg

Quick reminder that I’m now open for consulting and short-term engagements. If your team needs help with anything in the quant space, like strategy research, portfolio construction, execution analysis, signal development, or just a second brain on a hard problem, I'm available. You can DM me on Discord or email me at vertoxquant@gmail.com.

I have some news to share. As of today, I'm going fully independent and dedicating all of my time to VertoxQuant.

For the past few years, I've been splitting my hours between my work as a quant researcher and the newsletter. That chapter is now closed, which means every hour I would have spent on industry work is now going straight into research, writing, and building things here.

In practice, that means more articles, deeper dives, and more time to explore the corners of quant finance I actually find interesting, not just the stuff that's immediately commercially useful.

For those newer here: my background is in quantitative research and systematic trading, with experience across strategy research, portfolio construction, execution analysis, and signal development. I also hold a degree in mathematics, though I tend to approach problems very practically while maintaining the right level of mathematical rigor. VertoxQuant started as a place to share ideas and research publicly, and it has grown into a community of people who care deeply about quantitative finance.

I'm also now open for consulting and short-term engagements. If your team needs help with anything in the quant space, like strategy research, portfolio construction, execution analysis, signal development, or just a second brain on a hard problem, I'm available. You can DM me on Discord or email me at vertoxquant@gmail.com.

I’m incredibly excited for what comes next.

-VertoxQuant

Rather than trying to fix a method, you can also go down the route of using specially designed portfolio optimization techniques that don’t suffer from instability as much. Hierarchical Risk Parity (HRP) is one of the most popular such techniques, replacing the covariance inverse with a recursive clustering procedure that is more robust by construction. But even HRP has its limitations; It’s designed to be long-only, and it takes no signal input whatsoever. It’s a pure risk allocation method. Any alpha you’ve worked so hard to generate is completely ignored by it.

This matters enormously for stat-arb and long-short strategies, where both the signal and the risk structure are crucial. What you want is a portfolio construction method that uses market structure to shape position sizes while preserving signal direction. That is exactly what Topological Risk Parity (TRP), introduced by Nayar, Ainasse, and Kulkarni (2026), is designed to do.

In this article, we implement TRP from scratch in Python on a crypto universe. We also implement the Semi-Supervised variant of TRP, which allows you to impose an economic prior on the hierarchy, like forcing BTC to be the market root and major L1s as the second layer. We then apply Random Matrix Theory (RMT) to further improve TRP across every performance metric!

What you’ll learn:

• What Topological Risk Parity is and why it outperforms HRP for long-short strategies

• How to build a Minimum Spanning Tree from a correlation matrix and what it reveals about market structure

• How Random Matrix Theory separates genuine economic signals from statistical noise in correlation matrices

• How to denoise a correlation matrix using Ledoit-Wolf shrinkage and hard thresholding

• How to implement TRP from scratch in Python, including the Semi-Supervised variant with economic anchoring

I write about quantitative trading the way it’s actually practiced:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Mapping the Market with Correlation Networks

Read more

One of your first thoughts when looking at a stranger’s backtest is probably that it’s overfit, or that there is some look-ahead somewhere.

When you go a step further, you are probably constantly worried about overfitting your own backtests too!

In this article, we will introduce a framework that allows you to identify both! It’s a two-stage approach introduced in D. Nikolopoulos (2026). We will introduce the method, develop the two-stage approach, test the methodology empirically on a crypto dataset, look at some limitations, and also add some improvements of my own!

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

How Backtests Lie

Read more

Everyone talks about alpha decay, but no one tells you how to actually quantify it. In this article, we’ll figure out how to do so and build a warning system that tells you when it’s time to pull the plug on a strategy.

We do this by implementing Minimum Regime Performance (MRP), a framework introduced by Alexander and Fabozzi (2026) that measures how a strategy holds up across structurally distinct market regimes. We will apply this method to a universe of crypto factors.

Beyond signal monitoring, MRP is also relevant in portfolio construction. Just as you would punish strategy tail risk like Value at Risk, you can also punish strategies that have a high tendency to decay.

The article will use the factors discussed in the following article:

VertoxQuant

The Myth of Factor-Free Crypto

People keep telling you to take what works in equities and apply it to crypto…

Read more

5 months ago · 9 likes · Vertox

Knowledge of regime switching models is NOT required.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

• 1 referral → 20% off

• 5 referrals → 1 month free

• 25 referrals → 6 months free

If you’ve gotten value from it, help me grow by recommending it. It costs you nothing!

You can find your personal referral link at the bottom of this email or right here:

Leaderboard

Thanks!

-Vertox

In this article, we will look at some useful model-agnostic methods (meaning that it doesn’t matter what model we apply it to) for figuring out why one specific prediction is what it is (Local Methods) and why predictions are what they are on average (Global Methods).

This is important for a few very important reasons.

1. Detecting problems in the model:
   Say you built a model that forecasts raw returns, and you see that the error has a lower error than some baseline model. A lot of people would quit here, but you dig deeper and find out that the only feature that matters to the model is volatility, and that the only thing that the model learned to do well is predict the magnitude of the return, not the sign. Suddenly, the model is completely useless for what it was supposed to do!

2. Regulatory & risk management requirements:

   Regulators increasingly require that automated trading and risk models be explainable. If your model flags a position as too risky or generates a trade signal, compliance needs to know why. A black box doesn’t pass that bar.

3. Detecting data leakage:

   If your model is suspiciously good, interpretability tells you what it’s actually learning. Example: a return-prediction model that learned to exploit a look-ahead bias in a feature, e.g., using end-of-day prices to predict intraday moves, shows up immediately as one feature with overwhelming importance.

4. Feature engineering feedback loop:

   Once you know which features are actually driving predictions, you can iterate intelligently; drop redundant factors, identify which alpha signals are being ignored, or understand why a signal that worked in research is being suppressed by the model in production.

5. Regime analysis:

   Local methods let you compare why the model made a prediction in a bull market vs. a crisis period. If the feature importances shift dramatically across regimes, that’s a red flag for out-of-sample robustness; your model may have overfit to one regime’s dynamics.

More interpretable models are often less accurate, and model-agnostic post-hoc methods let you have both: full model flexibility and explanations.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

1. Local Model-agnostic Methods

Local methods answer a specific question: why did the model predict this value for this observation? Rather than summarising model behaviour across the entire dataset, they explain a single prediction by attributing it to the input features of that one data point.

Example: Your model predicts a large positive return for a particular asset on a particular day. A global method might tell you that momentum is generally the most important feature in your model. But a local method tells you that for this specific prediction, the signal was driven almost entirely by a spike in order flow imbalance, while momentum actually pushed the prediction slightly lower. That distinction matters enormously when you're trying to understand a model's live behaviour, debug an unexpected signal, or explain a specific trade to a risk committee.

Let’s go over some of the most popular methods.

1.1 Ceteris Paribus Plots

The name sounds really complicated, but it’s actually the simplest method you could use. Ceteris Paribus is Latin for "all other things being equal". As the name implies, to analyze a certain feature, you simply change its value while keeping all other features unchanged and look at the resulting predictions.

Before looking more into this method, let us train a model that we will keep using as an example for all methods. We will do a 1-day-ahead prediction of SPY returns using some basic features like momentum, RSI, etc. Doesn’t matter if the model is garbage as long as we can run our analysis on it.

First we load our data:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import yfinance as yf

def load_spy():
    df = yf.download("SPY", start="2010-01-01", end="2026-03-27", auto_adjust=True)
    df.columns = df.columns.droplevel(1)

    df["ret"] = df["Close"].pct_change()
    df["target"] = df["ret"].shift(-1)  # next-day return

    # Momentum
    df["mom_5"]   = df["ret"].rolling(5).mean()
    df["mom_20"]  = df["ret"].rolling(20).mean()
    df["mom_60"]  = df["ret"].rolling(60).mean()

    # Volatility
    df["vol_5"]   = df["ret"].rolling(5).std()
    df["vol_20"]  = df["ret"].rolling(20).std()
    df["vol_ratio"] = df["vol_5"] / df["vol_20"]  # vol regime indicator

    # Mean reversion
    df["rsi_14"]  = _rsi(df["Close"], 14)
    df["zscore_20"] = (df["Close"] - df["Close"].rolling(20).mean()) / df["Close"].rolling(20).std()

    # Volume
    df["volume_z"] = (df["Volume"] - df["Volume"].rolling(20).mean()) / df["Volume"].rolling(20).std()

    # Price-based
    df["hl_range"] = (df["High"] - df["Low"]) / df["Close"]  # normalized daily range
    df["gap"]      = (df["Open"] - df["Close"].shift(1)) / df["Close"].shift(1)  # overnight gap

    # Autocorrelation signal
    df["ret_lag1"] = df["ret"].shift(1)
    df["ret_lag2"] = df["ret"].shift(2)

    features = [
        "mom_5", "mom_20", "mom_60",
        "vol_5", "vol_20", "vol_ratio",
        "rsi_14", "zscore_20",
        "volume_z", "hl_range", "gap",
        "ret_lag1", "ret_lag2"
    ]

    df = df[["target"] + features].dropna()
    return df

def _rsi(close, period=14):
    delta = close.diff()
    gain = delta.clip(lower=0).rolling(period).mean()
    loss = (-delta.clip(upper=0)).rolling(period).mean()
    rs = gain / loss
    return 100 - (100 / (1 + rs))

df = load_spy()
df

And now let’s train an XGBoost model to predict the next day’s return using our features (We won’t be doing any train-test splits or other important steps here, this is purely an exercise model!):

from xgboost import XGBRegressor
from sklearn.metrics import r2_score

FEATURES = [c for c in df.columns if c != "target"]
X = df[FEATURES]
y = df["target"]
 
model = XGBRegressor(n_estimators=300, max_depth=4, learning_rate=0.05,
                     subsample=0.8, colsample_bytree=0.8, random_state=42)
model.fit(X, y)

preds = model.predict(X)
df["forecast"] = preds
df[["target", "forecast"]].to_csv("forecasts.csv")

Price         target  forecast
Date                          
2010-03-31  0.006837  0.000646
2010-04-01  0.008149  0.000556
2010-04-05  0.002358 -0.000046
2010-04-06 -0.005712  0.000146
2010-04-07  0.003464  0.001303

Back to Ceteris Paribus.

Here is an implementation of it:

def cp_profile(model, X, obs_idx, feature, n_points=100):
    x_obs = X.iloc[[obs_idx]].copy()
    grid = np.linspace(X[feature].min(), X[feature].max(), n_points)
    preds = []
    for val in grid:
        x_obs[feature] = val
        preds.append(model.predict(x_obs)[0])
    return grid, np.array(preds)

Let’s try it on the 2026-03-03 prediction for the features "mom_20", "vol_20", "rsi_14", "zscore_20":

Price
target        0.007055
mom_5        -0.002033
mom_20       -0.001063
mom_60       -0.000015
vol_5         0.006754
vol_20        0.008314
vol_ratio     0.812388
rsi_14       39.089484
zscore_20    -1.367428
volume_z      1.027870
hl_range      0.019035
gap          -0.016492
ret_lag1      0.000569
ret_lag2     -0.004802
forecast      0.001172
Name: 2026-03-03 00:00:00, dtype: float64

obs_idx = X.index.get_loc("2026-03-03")
features_to_plot = ["mom_20", "vol_20", "rsi_14", "zscore_20"]

fig, axes = plt.subplots(len(features_to_plot), 1, figsize=(9, 14))
fig.suptitle("Ceteris Paribus Profiles — SPY XGBoost (2026-03-01)", fontsize=14, fontweight="bold")

for ax, feat in zip(axes, features_to_plot):
    grid, preds = cp_profile(model, X, obs_idx, feat)
    ax.plot(grid, preds, linewidth=1.8)
    ax.axvline(X.iloc[obs_idx][feat], color="red", linestyle="--", label="observed value")
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feat, fontsize=11)
    ax.set_ylabel("Predicted return", fontsize=10)
    ax.legend(fontsize=9)
    ax.grid(True, alpha=0.3)

plt.tight_layout()

If momentum were even more negative, we would have predicted an even more positive return (mean reversion effect). If volatility were higher, we would have flipped to strongly negative instead (flipping from mean reversion to momentum).

Many of the methods we will look at are creative combinations of multiple Ceteris Paribus plots that tell us more.

The BIG problem with this (and many other) methods is correlation among features.
Changing one feature while keeping the others unchanged can give us unrealistic feature combinations, like huge recent momentum while volatility is very low.

1.2 Individual Conditional Expectation (ICE)

An ICE plot is simply many Ceteris Paribus plots overlaid, one per observation.

Here is an implementation:

def ice_plot(model, X, feature, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42)
    grid = np.linspace(X[feature].min(), X[feature].max(), n_points)
    
    fig, ax = plt.subplots(figsize=(9, 4))
    for idx in range(len(sample)):
        _, preds = cp_profile(model, sample.reset_index(drop=True), idx, feature, n_points)
        ax.plot(grid, preds, color="steelblue", alpha=0.4, linewidth=0.8)
    
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"ICE Plot — {feature}")
    plt.tight_layout()

Let’s run it for “mom_20”:

ice_plot(model, X, "mom_20")

We can explore interactions with this method by using colors.

def ice_plot(model, X, feature, color_by=None, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42).reset_index(drop=True)
    grid = np.linspace(X[feature].quantile(0.01), X[feature].quantile(0.99), n_points)

    fig, ax = plt.subplots(figsize=(9, 4))
    for idx in range(len(sample)):
        _, preds = cp_profile(model, sample, idx, feature, n_points)
        if color_by is not None:
            median = sample[color_by].median()
            color = "#d62728" if sample[color_by].iloc[idx] > median else "#1f77b4"
        else:
            color = "steelblue"
        ax.plot(grid, preds, color=color, alpha=0.4, linewidth=0.8)

    if color_by is not None:
        ax.plot([], [], color="#d62728", label=f"{color_by} high")
        ax.plot([], [], color="#1f77b4", label=f"{color_by} low")
        ax.legend(fontsize=9)

    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"ICE Plot — {feature}" + (f" | colored by {color_by}" if color_by else ""))
    plt.tight_layout()

Let’s color in low/high volatility:

ice_plot(model, X, "mom_20", "vol_20")

Centered ICE Plot:

It can be hard to tell if the different curves are affected equally by the feature of interest because they have different levels.

A simple solution is to subtract each curve's value at some specific value of x. This removes the vertical spread caused by other features and lets you focus purely on the effect of varying the feature of interest.

def cp_profile(model, X, obs_idx, feature, grid):
    x_obs = X.iloc[[obs_idx]].copy()
    preds = [model.predict(x_obs.assign(**{feature: val}))[0] for val in grid]
    return np.array(preds)

def ice_plot(model, X, feature, color_by=None, centered=False, ref_value=None, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42).reset_index(drop=True)
    grid = np.linspace(X[feature].quantile(0.01), X[feature].quantile(0.99), n_points)
    if centered and ref_value is not None:
        grid = np.sort(np.append(grid, ref_value))  # ensure ref_value is exactly in grid

    fig, ax = plt.subplots(figsize=(9, 4))
    for idx in range(len(sample)):
        preds = cp_profile(model, sample, idx, feature, grid)
        if centered:
            ref = ref_value if ref_value is not None else grid[0]
            ref_pred = preds[np.searchsorted(grid, ref)]
            preds = preds - ref_pred
        color = "steelblue"
        if color_by is not None:
            color = "#d62728" if sample[color_by].iloc[idx] > sample[color_by].median() else "#1f77b4"
        ax.plot(grid, preds, color=color, alpha=0.3, linewidth=0.8)

    if color_by is not None:
        ax.plot([], [], color="#d62728", label=f"{color_by} high")
        ax.plot([], [], color="#1f77b4", label=f"{color_by} low")
        ax.legend(fontsize=9)

    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Change in predicted return" if centered else "Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"{'c-ICE' if centered else 'ICE'} Plot — {feature}" + (f" | colored by {color_by}" if color_by else ""))
    plt.tight_layout()

A natural reference point for momentum is 0:

ice_plot(model, X, "mom_20", "vol_20", centered=True, ref_value=0)

Can see a pretty clear pattern on the left! When volatility is high, momentum works. When volatility is low, mean reversion works.

1.3 Local interpretable model-agnostic explanations (LIME)

CP profiles and ICE show you how the prediction changes as you vary one feature. But they don’t give you a single number answering: how much did feature x contribute to this specific prediction? LIME does exactly that.

The basic idea is that our black box method f is too complex to interpret globally, so we interpret near a specific observation x (all our features at one specific date) by training a simpler model (for example linear) in a small neighborhood of x.

Here is how LIME does this exactly (using linear regression as an example for the interpretable model):

1. Perturbing the observation:
   Generate N synthetic samples by randomly perturbing the features around x. For tabular data, this is done by sampling each feature from its marginal distribution. For continuous data, this is done by sampling from a normal distribution with equal means and standard deviations to the training set (This again ignores correlations! A typical weakness of LIME).

2. Weighing by proximity:
   Each synthetic sample z is weighted by how close it is to x using an exponential kernel:

   In our case, d will be euclidian distance, but other measures of distance can be chosen too.

3. Fitting a weighted linear model:
   Run the black-box model on all synthetic samples to get labels, then fit a weighted linear regression:

4. Reading the explanation:
   Now we interpret the resulting model. In the case of linear regression, that’s easy, as we simply look at the coefficients.

Let’s implement this with ridge regression, which handles our correlated features better:

def lime_explain(model, X, obs_idx, n_samples=1000, sigma=1.0, ridge_alpha=1.0):
    x_obs = X.iloc[obs_idx].values
    means = X.mean().values
    stds  = X.std().values

    # 1. perturb
    Z = np.random.normal(means, stds, size=(n_samples, X.shape[1]))

    # 2. weights
    Z_std = (Z - x_obs) / stds
    distances = np.sqrt((Z_std ** 2).sum(axis=1))
    weights = np.exp(-distances ** 2 / sigma ** 2)

    # 3. black-box labels
    labels = model.predict(Z)

    # 4. fit weighted linear model
    lr = Ridge(alpha=ridge_alpha)
    lr.fit(Z, labels, sample_weight=weights)

    return dict(zip(X.columns, lr.coef_))

And run it again for our 2026-03-03 forecast:

obs_idx = X.index.get_loc("2026-03-03")
explanation = lime_explain(model, X, obs_idx)
print(explanation)

mom_5       -2.096855e-09
mom_20      -4.942316e-09
mom_60       4.846428e-09
vol_5        4.087705e-08
vol_20       1.238100e-08
vol_ratio    5.578009e-07
rsi_14      -1.130377e-05
zscore_20   -2.909305e-06
volume_z    -1.143078e-06
hl_range    -4.011315e-08
gap          2.016699e-08
ret_lag1     1.007680e-08
ret_lag2     5.326466e-08

Another challenge in using LIME is choosing the neighborhood of x. There is no clear solution to this, and results can generally be unstable as well.

1.4 SHAP

SHAP (SHapley Additive exPlanations) is a game-theoretic approach.

The question SHAP asks is: Given that the model made a prediction hat{y}, how do we fairly distribute the difference between this prediction and the average prediction E[y] among the features?

Think of the features as players in a cooperative game, where the "payout" is the model's prediction. The Shapley value of feature j is its average marginal contribution across all possible orderings (coalitions) of features:

where F is the full set of features, S is a subset excluding feature j, and \hat{f}_S is the model’s prediction using only features in S.

The Shapley values must satisfy the following four properties (and is the only attribution method that does so):

Efficiency:

so the contributions exactly decompose the prediction relative to the baseline.

Symmetry:

If two features j and k contribute equally to every coalition, i.e., for all

we have

then they must have the same Shapley value

Dummy:

If feature j contributes nothing to any coalition, i.e., for all

we have

then the Shapley value must be zero

Linearity:

If the model can be decomposed as f = f^A + f^B, then

Estimating Shapley values:

The number of possible coalitions increases exponentially as you increase the number of features in your model (2^|F|). You therefore typically want to approximate it via Monte-Carlo sampling. Instead of summing over all coalitions, sample random permutations pi of the features and estimate the marginal contribution of j in each:

where S^m is the set of features appearing before j in the m-th random permutation. This converges to the true Shapley value as M → infinity.

You want M to be reasonably big, so the variance of your estimated Shapley value is small. This means that for big models doing a SHAP analysis can take a long time, which is one major drawback of it.

For tree-based models (XGBoost, LightGBM, Random Forests, etc.), there is an exact polynomial-time algorithm that exploits the tree structure to compute exact Shapley values in O(TLD^2) time, where T is the number of trees, L is the number of leaves, and D is the maximum depth.

Let’s implement this with the shap library. There are a ton of useful tools!

import shap
import copy

explainer  = shap.TreeExplainer(model)
shap_values = explainer(X)

obs_idx = X.index.get_loc("2026-03-03")

# Transforming to basis-points for better readability
sv_bps = copy.deepcopy(shap_values)
sv_bps.values = shap_values.values * 10000
sv_bps.base_values = shap_values.base_values * 10000
sv_bps.data = shap_values.data

waterfall_plot:

Shows how each feature pushes the prediction up/down from the baseline E[f(.)] to the final prediction f(x).

shap.waterfall_plot(sv_bps[obs_idx])

As you can see, momentum pushed the prediction in one direction, but volume_z and RSI pushed it back in the other direction.

force_plot:

Same information as waterfall, but horizontal.

shap.initjs()
shap.force_plot(sv_bps[obs_idx])

decision_plot:

Also gives the same information by showing the cumulative sum of SHAP values as a path from the baseline E[f(.)] to the final prediction.

shap.decision_plot(explainer.expected_value, shap_values[obs_idx].values, X.iloc[obs_idx])

You can also pass multiple observations to compare them side by side:

shap.decision_plot(explainer.expected_value, shap_values.values[:50], X.iloc[:50])

We will get back to SHAP again when looking at global model-agnostic methods!

2. Global Model-agnostic Methods

Global methods answer a different question: not why did the model predict this value for this observation, but how does the model behave on average across the entire dataset? Rather than explaining a single prediction, they summarise the overall relationship between features and the target learned by the model.

Example: You’ve trained a return-forecasting model on SPY, and it performs well out of sample. A local method tells you why the model predicted a large positive return on one specific day. But a global method tells you that across all predictions, volatility is the dominant driver, momentum matters mainly in low-volatility regimes, and the overnight gap feature is essentially ignored by the model.

Let’s go over some of the most popular methods.

2.1 Partial Dependence Plot (PDP)

Remember Individual Conditional Expectation (ICE)? PDP is simply the average of all ICE curves. This also means the PDP inherits the main weakness of ICE: It assumes the feature is independent of the others.

Just like with ICE, a centered version also exists. Let’s implement it as a thick line inside our centered ICE plot:

def ice_plot(model, X, feature, color_by=None, centered=False, ref_value=None, n_obs=100, n_points=100):
    sample = X.sample(n_obs, random_state=42).reset_index(drop=True)
    grid = np.linspace(X[feature].quantile(0.01), X[feature].quantile(0.99), n_points)
    if centered and ref_value is not None:
        grid = np.sort(np.append(grid, ref_value))

    fig, ax = plt.subplots(figsize=(9, 4))
    all_preds = []
    for idx in range(len(sample)):
        preds = cp_profile(model, sample, idx, feature, grid)
        if centered:
            ref = ref_value if ref_value is not None else grid[0]
            ref_pred = preds[np.searchsorted(grid, ref)]
            preds = preds - ref_pred
        all_preds.append(preds)
        color = "steelblue"
        if color_by is not None:
            color = "#d62728" if sample[color_by].iloc[idx] > sample[color_by].median() else "#1f77b4"
        ax.plot(grid, preds, color=color, alpha=0.3, linewidth=0.8)

    pdp = np.mean(all_preds, axis=0)
    ax.plot(grid, pdp, color="black", linewidth=2.5, label="PDP")

    if color_by is not None:
        ax.plot([], [], color="#d62728", label=f"{color_by} high")
        ax.plot([], [], color="#1f77b4", label=f"{color_by} low")

    ax.legend(fontsize=9)
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("Change in predicted return" if centered else "Predicted return")
    ax.grid(True, alpha=0.3)
    plt.title(f"{'c-ICE' if centered else 'ICE'} Plot — {feature}" + (f" | colored by {color_by}" if color_by else ""))
    plt.tight_layout()

ice_plot(model, X, "mom_20", "vol_20", centered=True, ref_value=0)

2.2 Accumulated Local Effects (ALE)

PDP’s main weakness is that it sweeps feature x_j across its range while holding other features fixed at their observed values, generating unrealistic feature combinations when features are correlated. ALE fixes this by only looking at the effect of x_j locally, within small neighbourhoods where the feature combinations are realistics, and then accumulating these local effects to get a global picture.

Partition the range of x_j into K intervals [z_{k-1}, z_k]. For each interval, compute the local effect of x_j by averaging the finite difference of predictions over observations that actually fall in that interval:

where K(x) is the index of the inerval containing x, and n_k is the number of observations in interval k. x_{-j}^(i), z_k means that the j-th feature value in x^(i) got replaced with z_k (here i stays for the i-th observation).

Let’s implement this:

def ale_plot(model, X, feature, n_bins=20):
    quantiles = np.quantile(X[feature], np.linspace(0, 1, n_bins + 1))
    quantiles = np.unique(quantiles)

    ale = np.zeros(len(quantiles) - 1)
    bin_centers = np.zeros(len(quantiles) - 1)

    for k in range(len(quantiles) - 1):
        mask = (X[feature] >= quantiles[k]) & (X[feature] < quantiles[k + 1])
        if mask.sum() == 0:
            continue
        X_bin = X[mask].copy()
        preds_high = model.predict(X_bin.assign(**{feature: quantiles[k + 1]}))
        preds_low  = model.predict(X_bin.assign(**{feature: quantiles[k]}))
        ale[k] = (preds_high - preds_low).mean()
        bin_centers[k] = (quantiles[k] + quantiles[k + 1]) / 2

    ale = np.cumsum(ale)
    ale -= ale.mean()

    fig, ax = plt.subplots(figsize=(9, 4))
    ax.plot(bin_centers, ale, color="steelblue", linewidth=2)
    ax.axhline(0, color="black", linewidth=0.8, linestyle="--")
    ax.set_xlabel(feature)
    ax.set_ylabel("ALE")
    ax.grid(True, alpha=0.3)
    plt.title(f"ALE Plot — {feature}")
    plt.tight_layout()

And run it for “mom_20”:

2.3 Friedman’s H statistic

So far, all methods look at the effect of one feature at a time. But in practice, features often interact; the effect of mom_20 on the prediction might be very different in high-volatility vs low-volatility regimes. Friedman’s H-statistic measures the strength of such interactions.

If two features x_j and x_k do not interact, then the point partial dependence can be decomposed as the sum of the individual partial dependences:

The H-statistic measures how much of the joint PDP’s variance is not explained by the sum of the individual PDPs:

H_{jk}^2 is in [0,1], where 0 means no interaction and 1 means the entire joint effect is due to interaction.

There is also a total interaction statistic for feature j measuring its interaction with all other features simultaneously:

Let’s implement both:

from sklearn.inspection import partial_dependence

def h_stat_pairwise(model, X, feat1, feat2, n_samples=500):
    X_s = X.sample(n_samples, random_state=42)
    
    pd12 = partial_dependence(model, X_s, [feat1, feat2], kind="average", grid_resolution=20)
    pd1  = partial_dependence(model, X_s, [feat1], kind="average", grid_resolution=20)
    pd2  = partial_dependence(model, X_s, [feat2], kind="average", grid_resolution=20)
    
    grid1 = pd12["grid_values"][0]
    grid2 = pd12["grid_values"][1]
    joint = pd12["average"][0]
    
    joint -= joint.mean()
    v1 = pd1["average"][0] - pd1["average"][0].mean()
    v2 = pd2["average"][0] - pd2["average"][0].mean()
    
    numerator, denominator = 0.0, 0.0
    for _, row in X_s.iterrows():
        f12 = joint[np.argmin(np.abs(grid1 - row[feat1])), np.argmin(np.abs(grid2 - row[feat2]))]
        f1  = v1[np.argmin(np.abs(pd1["grid_values"][0] - row[feat1]))]
        f2  = v2[np.argmin(np.abs(pd2["grid_values"][0] - row[feat2]))]
        numerator   += (f12 - f1 - f2) ** 2
        denominator += f12 ** 2
    
    return np.sqrt(numerator / denominator)

def h_stat_total(model, X, feature, n_samples=500):
    X_s = X.sample(n_samples, random_state=42).reset_index(drop=True)
    
    f_hats = model.predict(X_s)
    
    # PD_j: marginalise out all features except j (standard 1D PDP)
    pd_j = partial_dependence(model, X_s, [feature], kind="average", grid_resolution=20)
    grid_j = pd_j["grid_values"][0]
    v_j = pd_j["average"][0]

    # PD_{-j}: for each observation, fix x_{-j} and average over x_j
    pd_minus_j = np.zeros(len(X_s))
    for idx in range(len(X_s)):
        X_temp = X_s.copy()
        X_temp[feature] = X_s[feature]  # vary x_j over training dist
        X_temp.loc[:, [c for c in X_s.columns if c != feature]] = \
            X_s.iloc[idx][[c for c in X_s.columns if c != feature]].values
        pd_minus_j[idx] = model.predict(X_temp).mean()

    numerator, denominator = 0.0, 0.0
    for idx, (_, row) in enumerate(X_s.iterrows()):
        fj    = v_j[np.argmin(np.abs(grid_j - row[feature]))]
        f_hat = f_hats[idx]
        numerator   += (f_hat - fj - pd_minus_j[idx]) ** 2
        denominator += f_hat ** 2

    return np.sqrt(numerator / denominator)

from tqdm import tqdm
import seaborn as sns

features = X.columns.tolist()
h_matrix = np.zeros((len(features), len(features)))

pairs = [(i, j, f1, f2) for i, f1 in enumerate(features) for j, f2 in enumerate(features) if i < j]

for i, j, f1, f2 in tqdm(pairs, desc="Pairwise H"):
    h = h_stat_pairwise(model, X, f1, f2)
    h_matrix[i, j] = h
    h_matrix[j, i] = h

total_h = []
for f in tqdm(features, desc="Total H"):
    total_h.append(h_stat_total(model, X, f))

features_sorted, total_h_sorted = zip(*sorted(zip(features, total_h), key=lambda x: x[1]))

fig, axes = plt.subplots(1, 2, figsize=(16, 5))

sns.heatmap(h_matrix, xticklabels=features, yticklabels=features,
            annot=True, fmt=".2f", cmap="YlOrRd", ax=axes[0],
            linewidths=0.5, cbar_kws={"label": "H-statistic"}, vmin=0, vmax=1)
axes[0].set_title("Pairwise H-statistics", fontsize=13, fontweight="bold")

axes[1].barh(features_sorted, total_h_sorted, color="steelblue")
axes[1].axvline(0, color="black", linewidth=0.8)
axes[1].set_xlabel("H-statistic")
axes[1].set_title("Total H-statistics", fontsize=13, fontweight="bold")
axes[1].grid(True, alpha=0.3, axis="x")

plt.tight_layout()

This suffers from the typical drawback of all interpretable machine learning methods: It’s computationally expensive. It also only tells us the strength of the interactions, not what those interactions look like. A good workflow is to measure the interaction strengths with this method and then examine the strongest interactions in more detail using other methods (like specific SHAP plots that we will get to soon).

2.4 SHAP

We already covered how SHAP works so I will only showcase the plots that focus on global effect here.

beeswarm:

Shows the distribution of SHAP values for every feature across all observations.
Each dot is one observation, color indicates the feature value (red=high, blue=low).
Features are sorted by mean absolute SHAP value.

Can see here that very negative momentum causes very SHAP values (mean reversion).

mean absolute SHAP:

Mean absolute SHAP value per feature (global feature importance).
Simpler than beeswarm, good for reporting which features matter most on average.

shap.summary_plot(sv_bps, X, plot_type="bar")

Dependence Plots:

For each feature, it plots phi_j vs x_j across all observations.
Color is automatically chosen as the feature that interacts most with x_j, revealing how the effect of x_j changes depending on another feature's value.

for feat in X.columns:
    shap.dependence_plot(feat, sv_bps.values, X, interaction_index="auto")

Here is a notable ones:

When momentum is very negative and volume is very low, SHAP values become positive. The other ones didn’t have obvious interactions:

Headmap:

Shows phi_j for every feature j and every observation i simultaneously, features as rows and observations as columns. Each cell is colored by the SHAP value: red means the feature pushed the prediction up, blue means it pushed it down.

shap.plots.heatmap(sv_bps)

2.5 Permutation Feature Importance (PFI)

We already know about mean absolute SHAP as a general measure of how important a feature is in a model. PFI is another such singular value. If a feature x_j is important, then randomly shuffling its values across observations will destroy its relationship with the target and cause the model’s performance to degrade. If it’s unimportant, shuffling it changes nothing.

We first fit the model and compute a baseline performance metric L (e.g. MSE, R^2, IC) on the dataset. Then, for each feature j, we randomly permute the values of x_j across all observations, leaving all other features unchanged. We recompute the performance metric L^(j) on the permuted dataset, and the importance of feature j becomes

A large Delta L_j means the model relied heavily on x_j, and destroying it hurt performance a lot.

Since permutation is random, it’s standard to repeat this K times and average:

This is much faster to compute than SHAP and is directly tied to model performance, which is what we ultimately care about.

One BIG disadvantage is that if two features are highly correlated, then shuffling one leaves the other intact, and therefore both will receive a lot of permutation importance, even if they are super important to the model performance.

from sklearn.inspection import permutation_importance

result = permutation_importance(model, X, y, n_repeats=30, random_state=42, scoring="r2")

sorted_idx = result.importances_mean.argsort()

fig, ax = plt.subplots(figsize=(8, 5))
ax.barh(X.columns[sorted_idx], result.importances_mean[sorted_idx], 
        xerr=result.importances_std[sorted_idx], color="steelblue", capsize=3)
ax.axvline(0, color="black", linewidth=0.8)
ax.set_xlabel("Mean decrease in $R^2$")
ax.set_title("Permutation Feature Importance", fontsize=13, fontweight="bold")
ax.grid(True, alpha=0.3, axis="x")
plt.tight_layout()

2.6 Leave One Feature Out (LOFO) Importance

Instead of permuting a feature and evaluating with the same model, we completely drop a feature and retrain the model without it. LOFO Importance then measures the performance drop compared to the full model:

This is extremely useful for feature selection, but it is incredibly expensive and suffers from the same issue that highly correlated features get low LOFO importance. If you remove features one at a time, that problem becomes less relevant, as when one of them drops, the other increases in importance and won’t be dropped again.

Let’s implement this with R^2 as our performance metric:

from sklearn.metrics import r2_score

baseline = r2_score(y, model.predict(X))

loo_importance = {}
for feat in tqdm(features, desc="LOO"):
    X_loo = X.drop(columns=[feat])
    m = XGBRegressor(n_estimators=300, max_depth=4, learning_rate=0.05,
                     subsample=0.8, colsample_bytree=0.8, random_state=42)
    m.fit(X_loo, y)
    loo_importance[feat] = baseline - r2_score(y, m.predict(X_loo))

sorted_feats, sorted_vals = zip(*sorted(loo_importance.items(), key=lambda x: x[1]))

fig, ax = plt.subplots(figsize=(8, 5))
ax.barh(sorted_feats, sorted_vals, color="steelblue")
ax.axvline(0, color="black", linewidth=0.8)
ax.set_xlabel("Decrease in $R^2$")
ax.set_title("Leave-One-Out Feature Importance", fontsize=13, fontweight="bold")
ax.grid(True, alpha=0.3, axis="x")
plt.tight_layout()

3. Conclusion

I’ve never really written about ML before. If you enjoyed this article and would like to read more about ML, please let me know!

Join Quant Corner: https://discord.gg/X7TsxKNbXg

If you enjoyed this article, please share it!

Share

If you want to read more, consider subscribing!

This article provides a rigorous, hands-on walkthrough of portfolio optimization methods used in both research and industry. Rather than focusing on theory alone, we emphasize how these approaches behave in practice; what they optimize, which constraints matter, and the trade-offs they impose.

After briefly introducing the minimal notation, we move directly into the major families of portfolio construction techniques, including mean–variance, risk-based, and hierarchical approaches. The goal is not just to present these methods, but to understand when and why they work (or fail), especially in noisy and high-volatility environments.

Prerequisite (recommended):
This article assumes basic familiarity with Markowitz Mean–Variance Optimization (MVO). For full derivations of the efficient frontier, the tangency portfolio, and a detailed discussion of estimation error and its practical remedies (e.g., shrinkage, factor models, Black–Litterman, and robust optimization), see:

Notation and Setup

Consider n risky assets with (random) single-period returns r ∈ ℝⁿ. A portfolio is represented by weights w ∈ ℝⁿ. In the simplest setting, the portfolio return is

Let the expected returns be

and the covariance matrix be

Then the portfolio’s expected return and variance are

A standard budget constraint is

where the left 1 is the all-ones vector in ℝⁿ. Additional constraints may include long-only constraints w >= 0, leverage limits |w|_1 <= L, box constraints l <= w <= u, factor exposure constraints, and turnover constraints. Note: if w >= 0 and 1^⊤ w = 1, then |w|_1 = 1 is constant; l_1 leverage constraints/penalties matter primarily for long/short portfolios, active weights a = w - w_b, or turnover terms |wt - w_(t-1)|_1.

To set us up let’s first import the necessary libraries and set plot styles:

import os
import glob
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from scipy import linalg
from scipy import optimize
from scipy.optimize import linprog
from scipy.cluster.hierarchy import linkage, to_tree, dendrogram
from scipy.spatial.distance import squareform

# Plot style defaults.
plt.rcParams.update(
    {
        “figure.figsize”: (10, 6),
        “axes.grid”: True,
        “grid.alpha”: 0.3,
        “axes.titlesize”: 14,
        “axes.labelsize”: 12,
        “legend.fontsize”: 10,
    }
)

We don’t wanna be trading stablecoins, so we’ll exclude them:

STABLE_SYMBOLS = {
    “USDT”, “USDC”, “USDS”, “USDE”, “DAI”, “PYUSD”, “USD1”, “XAUT”, “USDF”,
    “PAXG”, “USDG”, “RLUSD”, “BFUSD”, “USDD”, “USDTB”, “USD0”, “FDUSD”, “A7A5”,
    “GHO”, “TUSD”, “CUSD”, “USDB”, “USR”, “KAU”, “EURC”, “CRVUSD”, “KAG”,
    “BUSD”, “USX”, “FRAX”, “USDC.N”, “YLDS”, “USDA”, “AUSD”, “DUSD”, “SATUSD”,
    “GUSD”, “EURS”, “DOLA”, “FRXUSD”, “PUSD”, “AIDAUSDC”, “USDZ”, “AVUSD”, “MNEE”,
    “USDR”, “PGOLD”, “CASH”, “EURCV”, “SUSDA”, “USDO”, “FEUSD”, “USDX”, “YZUSD”,
    “MIM”, “USDKG”, “REUSD”, “USDM”, “USD+”, “USDP”, “YUSD”, “XUSD”, “BOLD”,
    “HYUSD”, “LISUSD”, “USTC”, “USDN”, “LUSD”, “FXUSD”, “HBD”, “XTUSD”, “HONEY”,
    “EUSD”, “ZCHF”, “MIMATIC”, “USN”, “SUSD”, “EURE”, “BUCK”, “MUSD”, “EURA”,
    “GYD”, “USDH”, “MSUSD”, “UTY”, “WEMIX$”, “YU”, “HOLLAR”, “ALUSD”, “XSGD”,
    “EURR”,
}

Next, some useful helper function:

def _normalize_symbol(symbol: str) -> str:
    return “”.join(ch for ch in str(symbol).upper() if ch.isalnum())


def _base_symbol_from_ticker(ticker: str) -> str:
    t = str(ticker).upper()
    suffix = “_USD_1DAY_COMPOSITE”
    if t.endswith(suffix):
        return t[:-len(suffix)]
    return t.split(”_”)[0]


STABLE_SYMBOLS_NORM = {_normalize_symbol(s) for s in STABLE_SYMBOLS}

Whenever data isn’t available for a coin, we’ll be using synthetic data for demonstration purposes:

# Synthetic fallback data.
def synthetic_crypto_like_returns(
    n_assets: int = 12,
    n_obs: int = 1200,
    seed: int = 42,
    df_t: float = 5.0,
    regime_switch_prob: float = 0.02,
) -> pd.DataFrame:

    rng = np.random.default_rng(seed)
    dates = pd.date_range(”2020-01-01”, periods=n_obs, freq=”D”)

    # Build three correlation clusters.
    n_clusters = 3
    cluster_sizes = [n_assets // n_clusters] * n_clusters
    for i in range(n_assets % n_clusters):
        cluster_sizes[i] += 1
    clusters = np.concatenate([[k] * sz for k, sz in enumerate(cluster_sizes)])

    base_corr = np.eye(n_assets)
    for i in range(n_assets):
        for j in range(n_assets):
            if i == j:
                continue
            base_corr[i, j] = 0.55 if clusters[i] == clusters[j] else 0.15

    # Add symmetric jitter and keep the matrix positive definite.
    jitter = rng.normal(scale=0.03, size=(n_assets, n_assets))
    jitter = 0.5 * (jitter + jitter.T)
    corr = base_corr + jitter
    np.fill_diagonal(corr, 1.0)

    eig = np.linalg.eigvalsh(corr)
    if eig.min() <= 1e-6:
        corr += (abs(eig.min()) + 1e-3) * np.eye(n_assets)

    # Give each asset a different daily volatility.
    base_vol = rng.uniform(0.02, 0.08, size=n_assets)
    cov = np.outer(base_vol, base_vol) * corr
    L = np.linalg.cholesky(cov)

    # Switch between low- and high-volatility regimes.
    vol_mult = np.ones(n_obs)
    state = 0
    for t in range(1, n_obs):
        if rng.random() < regime_switch_prob:
            state = 1 - state
        vol_mult[t] = 1.0 if state == 0 else 2.0

    # Student-t shocks via normal / sqrt(chi2/df).
    Z = rng.standard_normal(size=(n_obs, n_assets))
    chi2 = rng.chisquare(df=df_t, size=n_obs) / df_t
    t_innov = Z / np.sqrt(chi2)[:, None]

    rets = (t_innov @ L.T) * vol_mult[:, None]
    cols = [f”Asset_{i:02d}“ for i in range(n_assets)]
    return pd.DataFrame(rets, index=dates, columns=cols)

The function for loading actual data (We are using CoinAPI):

# Loader for real data files.
def load_crypto_dataset_from_dir(
    data_dir: str = “/Users/admin/Documents/Projects/PortfolioOptimization/data”,
    max_assets: int = 100,
    min_obs: int = 365,
    coverage_threshold: float = 0.98,
    selection_metric: str = “mean_volume”,  # ‘mean_volume’ or ‘n_obs’
    verbose: bool = True,
):

    if not os.path.isdir(data_dir):
        raise FileNotFoundError(f”Directory not found: {data_dir}“)

    csv_files = sorted(glob.glob(os.path.join(data_dir, “*.csv”)))
    if len(csv_files) == 0:
        raise FileNotFoundError(f”No CSV files found in {data_dir}“)

    records = []
    price_series = {}
    excluded_stables = []

    for fp in csv_files:
        sym = os.path.splitext(os.path.basename(fp))[0].upper()
        base_sym = _base_symbol_from_ticker(sym)
        if _normalize_symbol(base_sym) in STABLE_SYMBOLS_NORM:
            excluded_stables.append(base_sym)
            continue
        try:
            df = pd.read_csv(fp)
        except Exception as e:
            if verbose:
                print(f”[WARN] Skipping {fp} (read error: {e})”)
            continue

        if “date” not in df.columns or “close” not in df.columns:
            if verbose:
                print(f”[WARN] Skipping {fp} (missing required columns ‘date’/’close’)”)
            continue

        keep_cols = [c for c in df.columns if c in (”date”, “close”, “volume_usd”)]
        df = df.loc[:, keep_cols].copy()

        df[”date”] = pd.to_datetime(df[”date”], errors=”coerce”)
        df[”close”] = pd.to_numeric(df[”close”], errors=”coerce”)
        df = df.dropna(subset=[”date”, “close”]).sort_values(”date”)
        df = df.drop_duplicates(subset=[”date”], keep=”last”)

        if df.shape[0] < min_obs:
            continue

        s = df.set_index(”date”)[”close”].astype(float)
        price_series[sym] = s

        mean_volume = float(df[”volume_usd”].astype(float).mean()) if “volume_usd” in df.columns else np.nan
        records.append(
            {
                “symbol”: sym,
                “n_obs”: int(df.shape[0]),
                “start”: df[”date”].min(),
                “end”: df[”date”].max(),
                “mean_volume_usd”: mean_volume,
                “file”: fp,
            }
        )

    if len(price_series) < 2:
        raise ValueError(f”Insufficient usable assets in {data_dir} after filtering (min_obs={min_obs}).”)

    info = pd.DataFrame(records).set_index(”symbol”)

    # Pick the sorting column.
    if selection_metric == “mean_volume” and info[”mean_volume_usd”].notna().any():
        sort_by = “mean_volume_usd”
    else:
        sort_by = “n_obs”

    info = info.sort_values(by=sort_by, ascending=False, na_position=”last”)
    selected = list(info.index[:max_assets])

    prices = pd.concat({sym: price_series[sym] for sym in selected}, axis=1).sort_index()
    rets = np.log(prices).diff()

    # Filter assets again after return coverage is computed.
    coverage = rets.notna().mean()
    keep_cols = coverage[coverage >= coverage_threshold].index.tolist()
    prices = prices[keep_cols]
    rets = rets[keep_cols]
    info = info.loc[keep_cols]

    before_rows = rets.shape[0]
    rets = rets.dropna(how=”any”)  # intersection of dates
    prices = prices.loc[rets.index]
    after_rows = rets.shape[0]

    if verbose:
        if excluded_stables:
            print(f”[DATA] Excluded {len(set(excluded_stables))} stable symbols before selection.”)
        print(f”[DATA] Loaded {len(keep_cols)} assets from {data_dir} (sorted by {sort_by}).”)
        print(f”[DATA] Date range: {rets.index.min().date()} → {rets.index.max().date()} ({after_rows} daily obs).”)
        print(
            “[DATA] Missing-data policy: drop any date with any missing return across selected assets “
            f”(kept {after_rows}/{before_rows} rows after alignment).”
        )

    if after_rows < 120:
        raise ValueError(”Not enough aligned observations after missing-data handling.”)

    return prices, rets, info

The following function computes returns from price data:

def get_crypto_returns(
    data_dir: str = “/Users/admin/Documents/Projects/PortfolioOptimization/data”,
    max_assets: int = 100,
    min_obs: int = 365,
    coverage_threshold: float = 0.98,
    seed_fallback: int = 42,
    verbose: bool = True,
):

    try:
        prices, rets, info = load_crypto_dataset_from_dir(
            data_dir=data_dir,
            max_assets=max_assets,
            min_obs=min_obs,
            coverage_threshold=coverage_threshold,
            verbose=verbose,
        )
        source = “real”
    except Exception as e:
        if verbose:
            print(f”[FALLBACK] Using synthetic data because real /Users/admin/Documents/Projects/PortfolioOptimization/data load failed: {e}“)
        prices = None
        rets = synthetic_crypto_like_returns(n_assets=max_assets, n_obs=1200, seed=seed_fallback)
        info = pd.DataFrame({”symbol”: rets.columns, “n_obs”: len(rets), “mean_volume_usd”: np.nan}).set_index(”symbol”)
        source = “synthetic”
    return prices, rets, info, source

Next, some helpful mathematical helper functions:

# Basic math helpers.
def regularize_cov(S: np.ndarray, eps: float = 1e-8) -> np.ndarray:
    “”“Symmetrize and add a small diagonal jitter for numerical stability.”“”
    S = np.asarray(S)
    S = 0.5 * (S + S.T)
    return S + eps * np.eye(S.shape[0])

def annualized_return(log_r: pd.Series, periods_per_year: int = 365) -> float:
    “”“Geometric annualized return from log returns.”“”
    years = len(log_r) / periods_per_year
    return float(np.exp(log_r.sum() / years) - 1.0) if years > 0 else np.nan

def annualized_vol(log_r: pd.Series, periods_per_year: int = 365) -> float:
    return float(log_r.std(ddof=1) * np.sqrt(periods_per_year))

def sharpe_ratio(log_r: pd.Series, rf_annual: float = 0.0, periods_per_year: int = 365) -> float:
    rf_log = np.log1p(rf_annual) / periods_per_year
    ex = log_r - rf_log
    vol = ex.std(ddof=1) * np.sqrt(periods_per_year)
    mean = ex.mean() * periods_per_year
    return float(mean / vol) if vol > 0 else np.nan

def max_drawdown(log_r: pd.Series) -> float:
    equity = np.exp(log_r.cumsum())
    peak = equity.cummax()
    dd = equity / peak - 1.0
    return float(dd.min())

def empirical_cvar(log_r: np.ndarray, alpha: float = 0.95) -> float:
    “”“
    Empirical CVaR of *losses* using simple returns:
      loss = - (exp(log_r) - 1)
    “”“
    simple = np.expm1(log_r)
    losses = -simple
    k = max(1, int(np.ceil((1 - alpha) * len(losses))))
    tail = np.sort(losses)[-k:]
    return float(tail.mean())

The following data loads our data and does some quick diagnostics:

# Load real data if available, otherwise use synthetic data.
crypto_prices, crypto_returns, asset_info, DATA_SOURCE = get_crypto_returns(
    data_dir=”/Users/admin/Documents/Projects/PortfolioOptimization/data”,
    max_assets=40,
    min_obs=365,
    coverage_threshold=0.98,
    seed_fallback=42,
    verbose=True,
)

print(f”\n[INFO] Data source: {DATA_SOURCE}“)
print(f”[INFO] Returns shape: {crypto_returns.shape[0]} dates × {crypto_returns.shape[1]} assets”)
print(f”[INFO] Assets: {list(crypto_returns.columns)}“)
print(”\n[INFO] Asset info (head):”)
print(asset_info.head())

# Quick diagnostics before the experiments.
pooled = crypto_returns.stack()
print(f”\n[DIAGNOSTIC] Pooled daily log-return mean: {pooled.mean():.6f}“)
print(f”[DIAGNOSTIC] Pooled daily log-return std : {pooled.std(ddof=1):.6f}“)
print(f”[DIAGNOSTIC] Pooled excess kurtosis      : {pooled.kurt():.2f}“)

# Plot pooled return histogram and correlation heatmap.
fig, ax = plt.subplots()
ax.hist(pooled.values, bins=80, density=True)
ax.set_title(”Pooled daily log returns (all assets)”)
ax.set_xlabel(”log return”)
ax.set_ylabel(”density”)
plt.tight_layout()
plt.show()

corr = crypto_returns.corr().values
fig, ax = plt.subplots()
im = ax.imshow(corr, aspect=”auto”)
ax.set_title(”Correlation matrix (aligned returns)”)
ax.set_xticks(range(crypto_returns.shape[1]))
ax.set_yticks(range(crypto_returns.shape[1]))
ax.set_xticklabels(crypto_returns.columns, rotation=90)
ax.set_yticklabels(crypto_returns.columns)
plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
plt.tight_layout()
plt.show()

[DATA] Excluded 17 stable symbols before selection.
[DATA] Loaded 18 assets from /Users/admin/Documents/Projects/PortfolioOptimization/data (sorted by mean_volume_usd).
[DATA] Date range: 2022-01-04 → 2026-01-16 (1472 daily obs).
[DATA] Missing-data policy: drop any date with any missing return across selected assets (kept 1472/1477 rows after alignment).

[INFO] Data source: real
[INFO] Returns shape: 1472 dates × 18 assets
[INFO] Assets: [’BTC_USD_1DAY_COMPOSITE’, ‘ETH_USD_1DAY_COMPOSITE’, ‘SOL_USD_1DAY_COMPOSITE’, ‘XRP_USD_1DAY_COMPOSITE’, ‘DOGE_USD_1DAY_COMPOSITE’, ‘ADA_USD_1DAY_COMPOSITE’, ‘FTM_USD_1DAY_COMPOSITE’, ‘RUNE_USD_1DAY_COMPOSITE’, ‘LINK_USD_1DAY_COMPOSITE’, ‘AVAX_USD_1DAY_COMPOSITE’, ‘SHIB_USD_1DAY_COMPOSITE’, ‘CFX_USD_1DAY_COMPOSITE’, ‘LTC_USD_1DAY_COMPOSITE’, ‘XLM_USD_1DAY_COMPOSITE’, ‘HBAR_USD_1DAY_COMPOSITE’, ‘CAKE_USD_1DAY_COMPOSITE’, ‘DYDX_USD_1DAY_COMPOSITE’, ‘XMR_USD_1DAY_COMPOSITE’]

[INFO] Asset info (head):
                         n_obs      start        end  mean_volume_usd  \
symbol                                                                  
BTC_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     1.168389e+09   
ETH_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     6.636419e+08   
SOL_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     2.891621e+08   
XRP_USD_1DAY_COMPOSITE    1477 2022-01-01 2026-01-16     2.603798e+08   
DOGE_USD_1DAY_COMPOSITE   1477 2022-01-01 2026-01-16     1.068017e+08   

                                                                      file  
symbol                                                                      
BTC_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
ETH_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
SOL_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
XRP_USD_1DAY_COMPOSITE   /Users/admin/Documents/Projects/PortfolioOptim...  
DOGE_USD_1DAY_COMPOSITE  /Users/admin/Documents/Projects/PortfolioOptim...  

[DIAGNOSTIC] Pooled daily log-return mean: -0.000667
[DIAGNOSTIC] Pooled daily log-return std : 0.049747
[DIAGNOSTIC] Pooled excess kurtosis      : 13.75

1. Baseline: MVO Recap

We use MVO as the reference point for the rest of this notebook. For full derivations and numerical examples, see the following article:

1.1 Canonical formulations

(MVO-1) Minimum variance at target return:

(MVO-2) Penalized mean–variance utility:

Varying μ_p in (MVO-1) or γ in (MVO-2) traces the (constraint-dependent) efficient frontier: the set of portfolios that are not dominated in (σ, E[r]) space.

1.2 Two anchor portfolios

Global minimum variance (GMV) (unconstrained):

With a risk-free rate r_f and excess returns μ_e = μ - r_f 1, the tangency / maximum Sharpe portfolio (unconstrained) is:

We construct a simple synthetic market with a random covariance matrix and expected returns, allowing us to study the geometry of mean–variance optimization in a controlled setting.

Using closed-form solutions, we compute the global minimum variance (GMV) portfolio and the tangency portfolio (maximum Sharpe ratio). We then sweep across target returns to trace out the efficient frontier, which shows the best achievable return for each level of risk.

# Synthetic efficient frontier example.
rng = np.random.default_rng(0)

n = 6
A = rng.normal(size=(n, n))
Sigma = A @ A.T  # SPD covariance
mu = rng.normal(loc=0.0005, scale=0.0010, size=n)  # daily expected log returns

ones = np.ones(n)
Sigma = regularize_cov(Sigma, eps=1e-10)
Sigma_inv = linalg.inv(Sigma)

# GMV portfolio.
w_gmv = Sigma_inv @ ones / (ones @ Sigma_inv @ ones)
mu_gmv = float(w_gmv @ mu)
vol_gmv = float(np.sqrt(w_gmv @ Sigma @ w_gmv))

# Tangency portfolio (rf=0).
w_tan_unnorm = Sigma_inv @ mu
w_tan = w_tan_unnorm / (ones @ w_tan_unnorm)
mu_tan = float(w_tan @ mu)
vol_tan = float(np.sqrt(w_tan @ Sigma @ w_tan))

# Closed-form frontier with budget and target-return constraints.
A_ = mu @ Sigma_inv @ mu
B_ = mu @ Sigma_inv @ ones
C_ = ones @ Sigma_inv @ ones
D_ = A_ * C_ - B_**2

# Sweep target returns around GMV.
targets = np.linspace(mu_gmv - 1.5 * np.std(mu), mu_gmv + 1.5 * np.std(mu), 80)
frontier_vol = []
for mu_p in targets:
    # Frontier variance at target return mu_p.
    var = (C_ * mu_p**2 - 2.0 * B_ * mu_p + A_) / D_
    frontier_vol.append(np.sqrt(var))

frontier_vol = np.array(frontier_vol)

# Plot volatility vs expected return.
fig, ax = plt.subplots()
ax.plot(frontier_vol, targets, label=”Efficient frontier (unconstrained)”)
ax.scatter([vol_gmv], [mu_gmv], label=”GMV”, zorder=3)
ax.scatter([vol_tan], [mu_tan], label=”Tangency (rf=0)”, zorder=3)

ax.set_title(”Markowitz efficient frontier geometry (synthetic μ, Σ)”)
ax.set_xlabel(”Portfolio volatility  (σ)”)
ax.set_ylabel(”Expected portfolio return  (E[r])”)
ax.legend()
plt.tight_layout()
plt.show()

print(”[SYNTHETIC SUMMARY]”)
print(f”GMV:     E[r]={mu_gmv:.6f},  σ={vol_gmv:.6f},  sum(w)={w_gmv.sum():.3f}“)
print(f”Tangency E[r]={mu_tan:.6f},  σ={vol_tan:.6f},  sum(w)={w_tan.sum():.3f}“)
print(”\nGMV weights:”, np.round(w_gmv, 4))
print(”Tan weights:”, np.round(w_tan, 4))

[SYNTHETIC SUMMARY]
GMV:     E[r]=0.001123,  σ=0.344942,  sum(w)=1.000
Tangency E[r]=0.002090,  σ=0.470481,  sum(w)=1.000

GMV weights: [ 0.2473  0.3281 -0.0938 -0.0937 -0.0412  0.6533]
Tan weights: [ 0.1141  0.7469 -0.174  -0.1761 -0.512   1.0012]

1.3 Assumptions (why the baseline is clean, and why it breaks)

MVO is exact under normality or quadratic utility (more generally, elliptical return families) in a single-period setting with known (μ,Σ). In practice, (μ,Σ) are estimated, markets are non-stationary, and constraints/transaction costs are unavoidable. The rest of this notebook is largely about methods that either (i) reduce dependence on fragile estimates, (ii) change the risk objective, or (iii) impose structure to improve out-of-sample stability.

2. How We’ll Compare Portfolio Optimization Methods in Practice

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

Most models are trained to predict individual asset returns.

That seems reasonable. But for cross-sectional strategies, prediction accuracy is often the wrong objective.

Cross-sectional portfolios turn model outputs into trades by ranking assets at each rebalance date and going long the top names while shorting the bottom. Once trades are determined by rank, the absolute level of forecasts often becomes irrelevant: any monotone transformation of model scores produces the same portfolio.

In other words, getting the ordering right matters more than predicting returns precisely.

This creates a subtle but important mismatch. A model can achieve excellent pointwise accuracy (low RMSE) while producing a weak, or even negative, long–short spread. Conversely, a model with poor regression metrics may generate strong cross-sectional performance.

However, ranking is not always all that matters. When position sizing, risk budgeting, transaction cost models, or portfolio optimizers depend on forecast magnitudes and calibration, pointwise accuracy becomes important again.

In this article, we develop Learning to Rank (LTR) as a principled framework for cross-sectional strategies.

The argument proceeds in three steps:

1. We formalize the cross-sectional decision problem and show how the score-to-weight mapping determines which properties of model output actually affect PnL.

2. We demonstrate with numerical examples how low RMSE can coexist with negative long–short spreads, and vice versa.

3. We derive practical LTR objectives for statistical arbitrage, including label engineering, pair weighting, neutralization, and cost-aware evaluation.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

Real limit order books are discrete: all displayed prices lie on a grid with tick size > 0, and, especially in “large-tick” assets, the bid–ask spread is frequently pinned at one tick.
This changes the economics of liquidity provision (where rents come from, how they are competed away, and how adverse selection manifests) and it changes the mathematics of optimal control (from smooth optimization to discrete choice, hybrid/impulse control, and state augmentation for queue position).

This article provides a rigorous, pedagogical deep-dive that starts from the canonical continuous-time market-making framework and then explains, in detail, why and how tick size and queue priority alter optimal quoting.
We then survey the spectrum of modeling and control approaches used in practice: exact discrete-state dynamic programming, queue-aware Markov models, continuous relaxations with discrete corrections, and pragmatic heuristics used when the exact discrete control problem is computationally intractable.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

In this article, we’re gonna be covering more advanced regime detection and regime switching algorithms that go beyond HMM, which nobody talks about.

Read more

Mean–Variance Optimization (MVO) is a central framework for portfolio construction: choose weights that balance expected return against risk as measured by variance.

In its classical form, MVO is elegant, convex (under mild conditions), and analytically tractable. Yet practitioners quickly encounter a paradox: the mathematically “optimal” portfolio built from estimated inputs is often unstable, highly leveraged (explicitly or implicitly), and disappoints out-of-sample.

This is not a minor implementation detail; it is a structural consequence of combining a high-dimensional optimizer with noisy estimates of expected returns and covariances.

This article develops MVO from first principles and then explains, in a mathematically explicit way, why raw MVO tends to maximize estimation error.

Finally, it surveys the spectrum of practical fixes, organized around two levers: (i) improving or regularizing the inputs (expected returns and covariances), and (ii) constraining or regularizing the optimizer (the feasible set and the objective).

The unifying theme is that almost every successful “fix” works by injecting bias in exchange for a large reduction in variance of the resulting portfolio weights, thereby improving out-of-sample performance and implementability.

The Classical Framework

Notation

Consider N risky assets. Let the random vector of (excess) returns over a single period be

Define the (unknown) population mean and covariance

A portfolio is a weight vector interpreted as a fraction of capital invested in each asset.

The basic budget constraint is

where 1 denotes the all-ones vector in R^N. In other words, the weights should sum up to 1. Additional constraints like no-short, leverage limits, or sector bounds will be presented in the following chapters.

Under this setup, the portfolio return is linear in weights:

The expected portfolio return and variance are

Two foundational facts are worth stating explicitly because they explain why MVO becomes a quadratic program:

1. Expected return is linear in weights. This makes the “reward” side easy to compute but also extremely sensitive to errors in mu, because the optimizer can exploit tiny differences in mu via large weight changes.

2. Variance is quadratic in weights. The covariance matrix Sigma couples assets through correlations: diversification is precisely the exploitation of off-diagonal terms in Sigma. The quadratic form w^T Sigma w is convex in w if Sigma is positive semidefinite, and strictly convex if Sigma is positive definite, which ensures the uniqueness of the minimum-variance solution under typical linear constraints.

The Markowitz Problem in Matrix Form

Markowitz’s original formulation can be stated as: among all portfolios with a given expected return, choose the one with minimum variance. Fix a target expected return m. The constrained optimization is

If short-selling is disallowed, one adds w >= 0 componentwise. If leverage is limited, one might add |w|_1 <= L, where |.| denotes the L1 norm, and so on.

Why is the covariance matrix central here? Because for any two assets i and j,

The diagonal terms w_i^2 Sigma_{ii} represent contributions from each asset’s variance; the off-diagonal terms w_i w_j Sigma_{ij} represent interaction through co-movement. Diversification is not “holding many assets” per se; it is selecting weights so that positive and negative interactions among returns reduce overall variance.

A subtle but important point: variance is a second-moment object. It treats positive and negative deviations symmetrically and is fully described by Sigma. This makes MVO analytically convenient, but it also means the framework inherits all limitations of second-moment risk measures; non-normality, fat tails, and asymmetry are not captured unless the distribution is (approximately) elliptical. In many institutional contexts, however, variance remains a useful proxy because it aligns with tracking error, volatility targets, and risk budgeting infrastructure.

The penalized (risk-aversion) form and its equivalence

An equivalent way to pose the trade-off is to maximize a mean-variance utility function:

where gamma > 0 is the risk-aversion parameter. Larger gamma penalized variance more heavily, shifting the solution toward lower-risk portfolios.

The equivalence between (MVO-1) and (MVO-2) is practically important. The constrained form (MVO-1) traces the efficient frontier by varying m. The penalized form (MVO-2) traces it by varying gamma. In many production systems, gamma is tuned to meet a risk target or tracking error budget.

Solving the classical problem: Lagrangian and closed-form structure

To make mechanics concrete, consider (MVO-2) without additional constraints beyond the budget constraint. Form the Lagrangian

with Lagrange multiplier eta enforcing 1^T w = 1.

First-order optimality (assuming Sigma is positive definite) gives

hence

Imposing the budget constraint 1^T w = 1 yields

so

Substituting (1.2) into (1.1) gives the explicit optimizer.

This expression reveals a key structural fact that will matter later: the optimal weights are built from Sigma^{-1} mu and Sigma^{-1} 1. In other words, the inverse covariance matrix is the central operator transforming expected returns into weights. When Sigma^{-1} is unstable (ill-conditioned or poorly estimated), the entire solution becomes unstable.

For the return-target form (MVO-1), the Lagrangian with multiplies lambda, eta is

The first-order condition is

Define the classical scalars

Assuming Sigma is positive definite and mu is not collinear with 1 under Sigma^{-1}, one has Delta > 0. Solving for lambda, eta yields a closed-form frontier, and the efficient frontier in (sigma^2, m) space is a parabola:

The frontier’s curvature and location are completely determined by (A,B,C), i.e., by Sigma^{-1} and mu. This already hints at the practical challenge: every point on the frontier depends on inverting Sigma and multiplying by mu, precisely the operations most vulnerable to estimation noise.

Let’s implement this in Python and look at the resulting efficient frontier. We will also compare our closed-form solution of (MVO-1) to a numerical solution to verify that we did everything right.

As you can see, our closed-form solution and numerical solution are pretty much identical, and the (MVO-2) solutions lie on the efficient frontier traced by the (MVO-1) solution, so they are indeed equivalent.

Interpreting Sigma: Risk Geometry and Diversification

It is useful to interpret the quadratic form geometrically. If Sigma is positive definite, then the set of portfolios with equal variance sigma^2,

is an ellipsoid in weight space. The optimizer in (MVO-2) chooses the point on the budget hyperplane {1^T w = 1} that maximizes a linear functional w^T mu minus a quadratic penalty. The optimum balances moving “up” in the direction of mu while staying within low-risk ellipsoids determined by Sigma.

This picture is clean when mu and Sigma are known. The moment we replace them with estimates, the ellipsoids tilt and stretch unpredictably, and the direction of “up” becomes noisy. The optimizer, being deterministic, will still choose an extreme point, often an extreme point of the wrong geometry. That is the beginning of the “error maximization” story.

Here is an example using two assets:

Each ellipse here corresponds to portfolios with equal variance. As you can see, our optimal portfolio just barely touches the 0.03 variance ellipsoid, and any other portfolio on the line (that satisfies the budget constraint) results in a higher variance.

The “Error Maximization” Problem

Raw MVO is often described informally as “garbage in, garbage out.” That statement is true, but it understates the severity: MVO does not merely propagate input error; it can amplify it. In high dimensions, the amplification can be dramatic enough that the optimizer effectively learns the noise in the estimated inputs.

This section makes that mechanism explicit.

MVO is not an optimization problem; it is a statistical decision problem

In theory, (mu, Sigma) are population quantities. In practice, we never observe mu or Sigma. We observe a finite time series

and produce estimators hat{mu} and hat{Sigma}. The most common “plug-in” estimators are the sample mean and covariance

Then the raw MVO portfolio is

or its return-target equivalent.

Crucially, hat{w} is a function of the random sample; it is itself random. The “true” objective we actually care about is out-of-sample performance under the true distribution, e.g., maximizing

But plug-in MVO maximizes a different random objective,

The practical question is not “is hat{w} optimal for hat{U}?” (it is, by construction), but “how does U(hat{w}) compare to U(w^\star), where w^\star maximizes U?” That gap is the cost of estimation error and model uncertainty.

Why expected return estimation is the Achilles’ heel

Start with expected returns. For each asset i, the sample mean hat{mu}_i has a standard error on the order of

where

In many liquid asset classes, annualized volatilities might be 10% - 30% while annualized expected excess returns might be 2% - 8%. Translating to a monthly scale, the noise in the sample mean can be comparable to, or larger than, the signal. This is a fundamental signal-to-noise limitation, not an implementation defect.

Now multiply that limitation by dimensionality. MVO compares assets and tries to exploit differences in mu. When mu is noisy, the differences the optimizer sees are often dominated by noise. The optimizer is then rewarded (in-sample) for taking large positions in the assets that happened to have high realized returns in the estimation window, even if that was random luck.

Because the expected return term w^T mu is linear, any error in mu shifts the gradient of the objective directly. In contrast, the covariance term is quadratic and tends to act as a smoothness penalty. This asymmetry is why MVO is particularly fragile to errors in mu.

Here is a simple numerical simulation of how much noise can affect our estimate of mu:

And now the impact on our portfolio weights from MVO. We assume 20 assets with identical true means, so any variation in mu is pure noise.

Our gross exposure is through the roof! A typical MVO portfolio here is equivalent to 200% long and 200% short, 4x leverage. Our right tail on Gross Exposure is also huge, so the portfolio sometimes ends up being 6x levered. The largest single position is also 54% of the portfolio, even though we have 20 assets.

This shows just how much of an extreme impact estimation noise in mu has on MVO.

Why covariance estimation becomes dangerous when inverted

The second failure mode is subtler: even if covariance estimates are “more stable” than mean estimates, the optimizer requires hat{Sigma}^{-1}. Inversion is the mathematical operation that turns moderate estimation noise into potentially huge weight noise.

To see why, consider the eigen-decomposition of the true covariance matrix:

where Q is orthonormal and

with lambda_i > 0$ if Sigma is positive definite. Then

Small eigenvalues become large eigenvalues after inversion. In portfolio terms, eigenvectors associated with small variance directions are precisely the directions the optimizer finds attractive: they offer “cheap risk.” But in finite samples, the smallest eigenvalues of hat{Sigma} are often dominated by noise (especially when T is not much larger than N). When the optimizer leans on these noisy low-variance directions, it produces extreme, unstable weights.

This is not hypothetical. A basic dimensionality fact already creates a hard boundary: if T < N, the sample covariance hat{Sigma} is singular (rank at most T-1), so hat{Sigma}^{-1} does not exist. Even if T is only moderately larger than N, hat{Sigma} can be ill-conditioned, making numerical inversion unstable and conceptually unreliable.

Let’s consider three cases: T < N, T ≈ N, and T > N, and look at the impact that T has on the estimated eigenvalues:

You can clearly see one thing: For the estimates where T = 30, Sigma becomes singular, and the 30th and further smallest eigenvalues just become 0. For T = 80, we can still see that the smaller eigenvalues are systematically underestimated. The estimates become better as we increase T to 300.

Now, let’s look at estiamted eigenvalues of Sigma and Sigma^{-1} for T=300:

The smaller the estimated eigenvalues, the larger the estimated eigenvalues of the inverse of Sigma.

Note: The y-axis is logarithmic on both plots, so linear → exponential!

Sensitivity Analysis: How estimation errors translate into weight errors

A useful way to formalize “error maximization” is to compute how small perturbations in mu and Sigma affect the optimizer.

Consider the unconstrained (besides budget) mean–variance utility maximization (MVO-2). The population optimum satisfies

with eta^\star chosen to enforce 1^T w^\star = 1. The plug-in estimate is

Write estimation errors as

A first-order expansion (informally, a matrix Taylor approximation) uses

A first-order perturbation of the optimizer gives

The last term enforces the budget constraint (with delta eta the perturbation in the multiplier); dropping it isolates the two main channels.

Several qualitative conclusions drop out of (2.8):

1. Mean error passes through Sigma^{-1}. Even if delta mu is moderate, multiplying by Sigma^{-1} can produce large changes in w, especially in directions corresponding to small eigenvalues of Sigma (or hat{Sigma}).

2. Covariance error enters as a “sandwich” with Sigma^{-1}. The term Sigma^{-1} * delta Sigma, w^\star has Sigma^{-1} on the left; if w^\star already has large exposures along unstable directions, covariance errors further distort them.

3. Budget adjustment can amplify instability. The multiplier eta is computed using 1^T Sigma^{-1} mu and 1^T \Sigma^{-1} 1$. If either of these scalars is unstable due to \Sigma^{-1}, the adjustment needed to enforce 1^T w=1 can itself swing drastically.

The key is not that delta mu and delta Sigma exist—of course they do—but that MVO transforms them with Sigma^{-1}, a potentially high-gain operator.

Let’s see how a badly conditioned Sigma affects delta w. We’ll drop the budget constraint so eta doesn’t influence the analysis:

You can see clearly that as Sigma becomes ill-conditioned, the error in w grows.

Optimizer’s curse

Another lens is the optimizer’s curse, a general phenomenon in statistical decision-making: if you choose the argmax of a noisy objective, the achieved value is biased upward in-sample, and the chosen decision is biased toward noise.

Formally, because hat{w} maximizes hat{U}(w),

but what matters is U(hat{w}). The difference

is typically positive and can be large in high dimensions. Intuitively, among many portfolios, some will look exceptionally good in-sample purely due to noise in hat{mu} and hat{Sigma}. MVO systematically selects those portfolios and then “locks in” their noisy characteristics via extreme weights.

This selection effect is strongest when:

• The number of assets N is large relative to the sample size T

• Expected return estimates are weak (low signal-to-noise);

• Shorting/leverage is permitted (large feasible set);

• Constraints are loose (optimizer can chase small estimated edges);

• The covariance matrix has near-collinear assets (ill-conditioning).

Let’s demonstrate the optimizer’s curse via Monte-Carlo.

You can see that even in an environment with equal true in- and out-of-sample mu and Sigma, due to estimation error, we vastly overestimate our utility in the in-sample.

Practical Symptoms: Instability, extreme positions, turnover, and disappointment

When raw MVO meets real data, the mathematical mechanisms above manifest in operational ways:

• Extreme weights and implicit leverage. Even with the budget constraint 1^T w = 1, weights can be large positive and large negative (if shorting is allowed), producing large gross exposure |w|_1. Even with no-short constraints, solutions often sit on corners of the feasible region (many weights at bounds), because linear return objectives push to extremes.

• High sensitivity to small input changes. Updating the estimation window by one month can materially change hat{mu} and hat{Sigma}, leading to large changes in hat{w}. This is not merely “rebalancing”; it is model instability.

• High turnover and transaction cost drag. If weights change drastically, realized performance is dominated by trading costs and market impact, neither of which exists in the clean Markowitz formulation unless explicitly modeled.

• Out-of-sample underperformance relative to naive allocations. A simple equal-weight or risk-parity portfolio can outperform a naive MVO portfolio after costs, not because those heuristics are theoretically superior, but because they are robust to estimation error.

These observations motivate the central practical conclusion: raw plug-in MVO is a high-variance estimator of portfolio weights. In modern terms, it is an overfit model.

The Spectrum of Solutions

This concludes the theoretical foundation.

The remaining 40 pages implement and test 11 robust portfolio construction techniques.

Each technique includes: Mathematical derivation → Clean implementation → Parameter tuning → Comparative results

Plus: Full research notebook (1250+ lines) with production-ready code. If you've made it this far through the theory, the implementations are where it pays off.

If raw MVO is fragile because it optimizes a noisy objective with unstable operators, then fixes must do one (or both) of the following:

1. Fix the inputs: replace (hat{mu}, hat{Sigma}) with estimators that have lower estimation error, better conditioning, or an economically grounded structure.

2. Constrain or regularize the optimizer: modify the optimization problem so that it cannot translate small input errors into extreme weight changes.

These approaches are complementary. Many production-grade systems use both: structured/shrunk inputs and a constrained, regularized optimization.

Fixing the inputs

Covariance Shrinkage: Stabilizing \hat{Sigma} and its inverse

The sample covariance hat{Sigma} is unbiased under idealized assumptions, but it can have high variance in finite samples, especially for off-diagonal elements. Shrinkage addresses this by pulling the estimate toward a structured target F that is lower variance but potentially biased.

A canonical shrinkage estimator is

Common choices for F include:

• a diagonal matrix (imposing zero correlations),

• a constant-correlation model,

• a factor-based covariance (discussed next),

• an identity-scaled matrix (equal variance, no correlation).

Why does shrinkage help? Because it increases the smallest eigenvalues of hat{Sigma} (or, more precisely, it reduces eigenvalue dispersion), improving the condition number and stabilizing inversion. In terms of Sigma^{-1} mu, shrinkage reduces the optimizer’s ability to exploit noisy “cheap-risk” directions created by small sample eigenvalues.

A useful way to view this is that shrinkage imposes a prior belief: extreme correlation structures are unlikely unless supported by strong data. By nudging the estimate toward a simpler structure, you reduce estimation variance more than you increase bias, improving out-of-sample performance.

Oracle note: In the sweep below, the “best delta” is chosen by maximizing utility computed with the true (mu, Sigma) to isolate the mechanism. In practice, delta must be selected via an analytic estimator (e.g., Ledoit–Wolf/OAS) or via walk-forward / nested cross-validation. We also report a held-out test-sample utility proxy for evaluation, not for selection.

Let’s look at the effect of choosing different values of delta. We will let F be

which has the average estimated variance among all assets as its diagonal entries.

As you can see, the condition number of our estimated covariance matrix reduces pretty greatly as we increase delta.

Utility increases out of sample as we increase delta (This is what we REALLY care about!)

Small eigenvalues are inflated instead of dropping to extremely low values, which in turn causes the eigenvalues of Sigma^{-1} to not explode.

Factor Models

A deeper way to control covariance noise is to assume returns are driven by a small number of common factors. A standard linear factor model is

where

• f in R^K are factor returns with K << N

• B in R^{N x K} are factor loadings

• epsilon in R^N are idiosyncratic returns.

Assuming epsilon is uncorrelated with f and has diagonal covariance D, the covariance of r becomes

where Sigma_f = Cov(f) and D = Cov(epsilon) is typically diagonal.

The parameter-count reduction is dramatic. A full covariance matrix has N(N+1)/2 unique parameters. A factor model estimates:

• NK loadings (often constrained/regularized)

• K(K+1)/2 factor covariances

• N idiosyncratic variances.

When K << N, this is a much lower-dimensional estimation problem, leading to more stable covariance estimates and more stable inverses. Economically, factors capture persistent co-movement structure (industries, styles, macro exposures), while idiosyncratic risk captures asset-specific variance.

For optimization, (3.3) has another advantage: it makes risk decomposition interpretable and enables direct factor exposure constraints, a major practical tool for controlling unintended bets.

To test this method, we are gonna assume assets follow some “true” factor model, and then we will estimate this factor model to obtain an estimate for Sigma by plugging the estimates into (3.3):

As you can see, our median condition went from 4327 for the naive sample covariance to only 259.3 for the factor-informed covariance. Utility improved 100% of the time out of sample!

Time-varying covariance

Covariances evolve over time. Using very long histories can reduce sampling error but introduce model error if the distribution is non-stationary. Using short histories reduces model error but increases estimation noise.

A common compromise is an exponentially weighted covariance estimator. For a finite window, a normalized form is

This places more weight on recent observations, adapting to changing regimes, while keeping the weights normalized for finite T.

An equivalent way to write the estimator is the recursive form

which implicitly normalizes and becomes equivalent to the sum as T → infinity (since lambda^T → 0).

But note: exponential weighting alone does not solve ill-conditioning in high dimensions; it is often combined with shrinkage or factor models.

The practical point is that “better covariance” means balancing estimation variance against non-stationarity. MVO fails when either error dominates, and real markets usually give you both.

Expected Returns

If covariance estimation is difficult, expected return estimation is usually worse. Many practical systems, therefore, treat mu not as a directly estimated sample mean, but as a forecast that must be heavily regularized.

A basic shrinkage approach is

where mu_0 is a prior mean. Choices for mu_0 include:

• zero (implying no predictability in excess returns),

• a cross-sectional average (implying mean reversion across assets),

• factor-model-implied returns,

• equilibrium-implied returns from a benchmark portfolio.

The key is the logic: because hat{mu} is high-variance, we deliberately introduce bias by shrinking toward a stable prior. This reduces the variance of the resulting weights, which is often the dominant out-of-sample benefit.

A particularly influential equilibrium anchoring method is to infer implied expected returns pi from an observed “market” or benchmark portfolio w_m.

Before invoking the next equation, it helps to distinguish between two regimes:

• Risk-free asset (tangency portfolio). If a risk-free asset is available and w_m is the tangency portfolio, then pi are excess returns and reverse optimization yields pi = lambda Sigma w_m.

• Risky-only, fully invested. If we remain in the 1^T w = 1 setting, the KKT condition is mu = gamma Sigma w_m + eta 1. The intercept term is economically irrelevant under the budget constraint, so one can normalize it away or interpret eta as the baseline rate.

In a mean–variance world, if w_m is optimal for some representative investor with risk aversion lambda, then

This produces a return vector pi consistent with the covariance structure and observed holdings. Even if the exact assumptions are stylized, pi has a crucial practical advantage: it is anchored to a diversified, implementable portfolio, which prevents the optimizer from forming extreme views unsupported by the market’s aggregate positioning.

Black-Litterman as a canonical “fix-the-inputs” framework

Black–Litterman formalizes the idea that expected returns should be a blend of (i) equilibrium returns and (ii) investor views. While many variants exist, the classical setup can be summarized as follows.

Let pi be equilibrium implied returns (e.g., from (3.6)). Model the prior as

where tau > 0 scales uncertainty in the prior. Investor views are expressed as

where P in R^{M x N} selects linear combinations of returns, q in R^M are view returns, and view noise has covariance Omega in R^{M x M}$ (often diagonal).

The posterior mean under Gaussian assumptions is

Why does this help MVO?

• It shrinks noisy, unconstrained return estimates toward an equilibrium anchor.

• It expresses uncertainty explicitly via tau and Omega, preventing overconfidence.

• It ensures that when views are weak or absent, the optimizer defaults to a diversified baseline rather than a noisy corner solution.

Black–Litterman is not magic; it is a disciplined way of injecting structure and uncertainty into mu, which is exactly what raw MVO lacks.

Resampling and Bayesian Averaging: Stabilizing weights instead of moments

Another approach is to accept that inputs are noisy and average over that uncertainty. A simple version is bootstrap resampling:

1. Generate many synthetic datasets by resampling {r_t}.

2. For each dataset b, compute hat{mu}^(b), hat{Sigma}^(b) and solve MVO to get hat{w}^(b).

3. Average weights:

The intuition is that extreme positions caused by noise tend not to be stable across resamples; averaging dampens them. In decision-theoretic language, this approximates integrating over parameter uncertainty rather than conditioning on a single plug-in estimate.

This class of methods “fixes” MVO not by producing perfect estimates, but by reducing the variance of the decision w.

Let’s compare the weights generated by this method with those of MVO using just the sample estimates. We will also enforce long only to make the weights more comparable.

As you can see, half of the weights generated by the plug-in MVO are exactly zero, whereas with the bootstrap-averaged method, we have no zero weights. Those weights that are non-zero are also larger for the plug-in MVO. We are therefore better diversified using the bootstrap-averaged weights method.

Constraining and Regularizing the Optimizer

Even with improved inputs, the optimizer can still overreact. Constraining the optimization problem is therefore the second major lever. The guiding principle is simple: if you restrict the feasible set or penalize extreme solutions, the optimizer has fewer degrees of freedom to express estimation noise.

Box constraints, no-short constraints, and leverage limits

A basic set of implementability constraints is

along with the budget constraint 1^T w = 1. The no-short constraint is the special case (l_i = 0). A leverage constraint can be expressed as

When shorts are allowed, |w|_1 measures gross exposure; limiting it prevents the optimizer from creating offsetting long/short positions that are highly sensitive to small covariance or mean differences.

Why do these constraints work? They impose a cap on sensitivity. If the optimizer cannot push weights arbitrarily far, then errors in hat{mu} and hat{Sigma} cannot translate into arbitrarily extreme positions.

A common misconception is that constraints only reflect operational needs. In practice, they often serve a dual purpose: implementability and robustness.

Regularization Penalties: Making MVO behave like a well-posed estimation problem

Instead of hard constraints, one can add penalties to the objective. Consider augmenting (MVO-2) with an l2 (ridge) penalty:

Because |w|_2^2 = w^T I w, the risk term becomes

So ridge-regularized MVO is equivalent to replacing Sigma with

This is a profound and practical insight: a simple weight penalty is mathematically identical to eigenvalue inflation of the covariance matrix. It improves conditioning by pushing all eigenvalues away from zero, directly addressing the instability of Sigma^{-1}. In effect, it says: “do not trust low-variance directions too much; treat them as riskier than the raw estimate suggests.”

One can also use an l1 (lasso) penalty:

The l1 penalty encourages sparsity (many weights exactly zero) and discourages large gross exposure, especially when shorts are allowed. It is closely related to constraining |w|_1 as in (3.11). Sparse portfolios can be easier to implement and often exhibit lower turnover.

Note: if w >= 0 and 1^T w = 1, then |w|_1 = 1 is constant, so an l1 penalty on weights has no effect. l1 becomes meaningful when shorting is allowed (gross exposure varies), when applied to active weights a = w - w_b, or when penalizing turnover |w_t - w_{t-1}|_1.

Both l1 and l2 regularization can be understood as injecting a preference for “simple” portfolios, portfolios that do not require precise estimates to justify intricate long/short structures.

The following code demonstrates what happens if you adjust the ridge parameter lambda:

Higher lambda results in better conditioned Sigma.

Higher lambda results in lower L1 and L2 norm of w (The portfolios become “simpler”).

Turnover and Transaction Costs: Regularizing changes the weights

A portfolio is rarely optimized once. In practice, we solve a sequence of problems over time. Estimation noise then manifests as unstable changes in weights, i.e., turnover.

A common fix is to add trading costs or turnover penalties. Let w_{t-1} be current holdings and w_t the new decision. A quadratic turnover penalty gives

This makes the optimizer “reluctant” to move unless the gain in mean–variance utility is large enough to justify the change. Importantly, this is also a robustness device: it prevents the optimizer from chasing small, noisy changes in hat{mu} or hat{Sigma}.

Linear cost models use |w_t - w_{t-1}|_1 instead, which can better capture proportional costs and induce sparse trading (only a subset of names trade each period).

The deeper point is that multi-period implementation turns MVO into a control problem. Regularizing the control (weight changes) is often more impactful than regularizing the state (weights) alone.

Let’s compare the evolution of two portfolios: One via regular MVO and one with the L1 penalty for weight changes. Let’s also see how changing kappa affects our portfolio.

Above, we’ve used a kappa of 50. As you can see, turnover is very heavily reduced.

Here you can see that as you increase your turnover penalty kappa, your improvement vs an unpenalized portfolio becomes better and better, but returns are becoming diminishing with values >= 30.

In a real market that can change more abruptly, you may want to pick kappa slightly lower to be able to adapt more quickly.

Benchmark-relative optimization

Many institutional mandates are benchmark-relative. Let w_b be benchmark weights. Define active weights a = w - w_b. Then tracking error variance is

A common optimization is to maximize expected active return subject to tracking error:

where alpha represents expected alphas (expected excess returns relative to benchmark). This formulation has an important robustness advantage: it anchors the portfolio near a diversified benchmark, limiting the optimizer’s ability to create extreme positions driven by noise. In effect, the benchmark provides a strong prior on the weight vector.

Even if one is not formally benchmarked, this insight generalizes: anchoring optimization around a reference portfolio (market, equal weight, risk parity) often improves stability.

Robust Optimization: Optimizing against parameter uncertainty directly

A conceptually clean way to address estimation error is to incorporate uncertainty sets for mu and/or Sigma, then optimize a worst-case objective. This approach turns “garbage in, garbage out” into “uncertainty in, robustness out.”

One standard robust mean model assumes mu lies in an ellipsoid around hat{mu}:

Consider the robust counterpart of maximizing expected return minus risk, taking the worst-case mean:

The inner minimization has a closed form:

So the robust problem becomes

This is illuminating. The robust adjustment subtracts a term proportional to portfolio volatility, effectively reducing the attractiveness of portfolios whose expected return advantage might be explained by estimation error. The optimizer is forced to find portfolios whose expected return is high relative to uncertainty.

Robust covariance uncertainty sets lead to related regularization effects, often inflating risk in uncertain directions; again echoing the intuition behind shrinkage and ridge penalties. Robust optimization provides a principled bridge between statistical uncertainty and optimization regularization.

Let’s compare this to standard MVO:

Our Robust MVO weights are a lot more “clean” compared to the standard MVO weights.

Conclusion

It is tempting to treat the practical modifications above as an ad hoc toolbox: shrinkage here, constraints there, turnover penalties elsewhere. A more coherent view is that these methods all solve the same underlying problem:

The plug-in MVO portfolio is a high-variance estimator of the optimal weights.

Reducing that variance requires injecting structure, equivalently, adding bias.

• Shrinkage of Sigma reduces the variance of covariance estimates (and stabilizes inversion) at the cost of bias toward the target structure.

• Factor models impose a low-rank structure that may be misspecified but dramatically reduces estimation noise.

• Shrinkage and Bayesian methods for mu explicitly acknowledge that sample means are too noisy to trust fully.

• Constraints and penalties reduce the effective degrees of freedom of the portfolio, preventing the optimizer from encoding noise into intricate weight patterns.

• Turnover penalties reduce the variance of changes in weights, which is often the real operational pain point.

• Robust optimization is an explicit uncertainty-aware form of regularization.

From this perspective, the “right” portfolio is not the one that is optimal for the estimated parameters; it is the one that is optimal for the decision problem under uncertainty, including estimation error, non-stationarity, and implementability constraints.

Quant Corner: https://discord.gg/X7TsxKNbXg

Code: https://gist.github.com/vertoxquant/e417b8afd2abe0e15ae0ff486813842a

At first glance, that contradiction seems justified. Crypto markets look nothing like equities: trading is continuous, liquidity is fragmented across venues, listings and delistings are frequent, and returns are dominated by a long tail of illiquid tokens.

Let’s answer once and for all if equity factors exist in crypto and if you can profit from them.

By the end of this article, you’ll have a framework for answering that question yourself. You’ll see how to test new factors and alphas in a way that is robust to listings and delistings, a changing universe, and the realities of trading crypto markets. The full research notebook used in the analysis is available for download at the end of the article for paid subscribers.

Read more

In the previous article, we built all of the theory behind HMMs for microstructural data. This article is where it finally gets real, and we fit those models to actual market data efficiently.

We are going to take something that is completely infeasible in practice, on the order of 10^430 computations. To put that in perspective, if you took every atom in our universe, put a new universe inside each atom, and repeated that process five times, you would reach a number around that size. Then we collapse it down to roughly 9000 computations that you can actually run.

By the end of this article, you will be able to take real market data, fit an HMM to it, and identify which market regime you are most likely in

Read more

• Small communities that are invite only just tend to die down eventually.

• Big servers don’t really offer much value or do anything fun.

That’s exactly what I want to change with Quant Corner!

The goal is simple: build a community that actually does things.
Think tournaments, live Q&As, useful resource giveaways, and open discussions where your feedback genuinely shapes what we build next.

Quant Corner is also the best place to engage directly with me, share ideas, suggest events, and help steer the direction of the community.

We’re currently at ~440 members, and once we hit 800, I’ll be hosting something big.

If this sounds like something you’d want to be part of, join us here:
https://discord.gg/X7TsxKNbXg

Many financial features exhibit regime behavior: low vs. high volatility, choppy vs. trending markets, liquid vs. illiquid conditions.

These regimes are not directly observable. Instead, we only see their manifestations in market data such as prices, returns, volumes, and spreads.

As a result, microstructure features may exhibit different statistical behaviour depending on the underlying market state, raising the question of how they should be analysed in practice.

Read more