This matters quite a lot, as the previous models didn’t fit empirically observed patterns well, like sharply steepening ATM skew for short-dated options. But as you know from my articles, just having a model fit well and calling it a day is not enough for us. We want to know WHY a certain model works well or doesn’t from an economic perspective. This reasoning is what allows us to develop truly robust models.

So in this article we will:

• measure how much BTC volatility actually moves across different timescales and let the data tell us what kind of process it is,

• ask what that process does to an option book, derive the short-dated skew it implies, and check the prediction against Deribit,

• ask where the process comes from in the first place, build the mechanism up from three facts about how markets trade, and estimate it from Binance data.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

EigenScore is the first rated contest platform for quants: timed rounds, Elo ratings, and problems across probability, pricing, optimization, and forecasting.

Solve your first problem

Read more

You can use it to practice your quant skills, prep for interviews, or prove your skills against other quants in the rated contests.

The first such rated contest is today in a few hours. Join before it’s too late:

https://eigenscore.com/contests/round-1

The problems will range from easy to hard with 3 types of problems:

• Standard: Ground-truth problems. You either solve them or not.

• Prediction: Better out-of-sample forecast performance gets more points.

• Optimization: Better final minimum/maximum out-of-sample gets more points.

Problems come from all kinds of areas: probability theory, combinatorics, portfolio optimization, graphs, and much more.

So anyone can join, no matter your level of experience.

Join here: https://eigenscore.com/contests/round-1

eigenscore.com

In quant, everyone claims to be good, and there's no arena to settle it. Competitive programming solved that twenty years ago with rated contests. I built the same thing for us.

What it is

EigenScore is a competitive platform for quants: a problem archive and rated contests with a persistent rating (Elo).

You get a problem archive of 50 problems at launch, across probability, combinatorics, martingales, stochastic calculus, option pricing, portfolio optimization, time series, games, graphs, and numerical methods. You write Python in the browser, hit Submit, and a judge tells you within seconds whether you were right.

But “right” means different things in quant, so there are three kinds of problems:

• Standard: Those have an exact answer, like a probability or an option price, and the judge checks it against the reference within tolerance.

• Prediction: You’re given data and are asked for an estimate, like the next values of a process, a covariance matrix, or tail quantiles. There’s no correct output; you’re scored on how close your estimate gets to the best one that was theoretically possible from the data.

• Optimization: You’re asked to make a decision. Accept or reject an offer, allocate a portfolio, choose a stopping rule, and get scored by how well your decisions perform relative to the optimal policy.

Then there are rated contests: two hours, five problems, everyone at once. When the round settles, every participant’s rating updates. You start as a Newbie, and then climb your ranks through Pupil, Apprentice, Specialist, Expert, Master, Grandmaster, all the way to Legend.

What went into it

Every problem is original. Written, solved, and tested by me, with reference solutions, hidden test sets, and tolerance checks.

Every submission runs inside its own hardware-isolated virtual machine that boots in milliseconds, runs your code with strict time and memory limits, and is destroyed afterwards. I built the judge, the worker fleet, the queuing, the scoring for all the problems, and the settlement engine.

And then everything else: accounts, contest scheduling, reminder emails, leaderboards, profiles with rating graphs, an announcement system, and the entire legal setup.

What I need from you

It’s day one. The archive is small and will grow every week. Things will break. If you hit a wrong verdict, an unclear statement, or anything that feels off, use the contact form or tell me directly. I read everything and most fixes ship within days.

If you know someone who’d enjoy this, like a colleague or a student, send them the link. A rating is only worth something when there are people to be rated against.

I’ll see you on Monday, 8 September, 20:00-22:00 CEST when the first rated contest goes live!

-Vertox

The entire purpose of a risk system is to quantify potential losses that have yet to occur. We care about losses so extreme they have never (or very rarely) occurred. And that is precisely what makes tail-risk estimation so difficult: the number you are trying to estimate is the one with the least data.

Let’s put this in perspective. Let’s say you have hourly BTC returns from August 2017 to May 2026 (what we will use in this article), which gives us n = 76,908 observations. That sounds like plenty of data, and it is for your typical moves, since even at the 99% level you still have around 769 observations. But a desk does not blow up at a 99% event; let’s push this to a 1-in-10,000 event, and suddenly you only have around seven to eight data points you can use for estimation! And even worse, that’s assuming you use the ENTIRE dataset for estimation, which you probably don’t want to do since data from 2017 won’t be as relevant to you when trading in a 2026 environment.

And what if you care about a 1-in-100,000 event? You realistically have at most one data point to estimate that! Worry not, I will tell you how you can still quite reliably estimate those!

The parametric approach: a distribution for everything

The first instinct is to fit a distribution to the data and read the quantile off the fitted form. Because we are trying to estimate tail risk, let’s use a fat-tailed distribution like a Student-t distribution:

so L itself has density

The ν degrees of freedom govern the tail: as |t| → infty the density decays like |t|^(-v+1), a polynomial rather than an exponential, so the survival function is genuinely power-law,

Writing

for the standard-t quantile, we get a clean closed form for VaR and CVaR:

where VaR and CVaR are defined as

The Student-t distributions appear to have done everything right. So where is the failure? Not in the shape of the tail, but in the data the fit uses to find it. We have one parameter, the degrees of freedom v, that governs both the body and tail of the distribution. We care about the v that is accurate for the tail, but when fitting the Student-t distribution, most of our samples will be in the body. Maximum likelihood will prioritize finding a v that fits the body well rather than the tails. Another problem is that the Student-t distribution is symmetric, while real loss distributions are asymmetric: the left tail is heavier than the right tail, and a single parameter can’t capture both.

Historical simulation: no distribution at all

The opposite instinct is to assume nothing. Sort the realized losses, take the empirical quantile, and compute the average loss beyond that quantile. Boom, you have VaR and CVaR. Many desks, in fact, run exactly this on a rolling basis. And true, it never lies about the shape of what has already happened, and here is exactly the limitation of this method: It can’t tell you about tail events that have never happened before (or have not happened in your rolling window).

What if I told you there is a principled way to model just the tail of a distribution, and, under mild assumptions, the tails of almost every distribution approach the same special one.

What you’ll learn

• Why the loss big enough to blow up a book is the one you have the least data to measure, and why two standard answers both break in the tail.

• How extreme value theory models the tail on its own terms, assuming nothing about the rest of the distribution, and lets you put a number on a loss larger than anything that ever happened in history.

• Why a single VaR is only half the answer, and what expected shortfall tells you about how bad losses actually get once that line is crossed.

• Why a tail fitted once over all your data is wrong on any given day, and how conditional EVT builds a tail that adapts to the market.

• How to prove the final model performs better than the baseline models.

• A full implementation!

Chapter 2 - The Limit Laws of Extremes

Read more

Suppose you want to predict tomorrow’s return using today’s information. We collect some features (X_t) and fit a regression model:

This seems reasonable at first. If our model predicts a return of 0.5%, we can use that prediction to decide whether to trade, how much to trade, or which assets to prefer.

But there is a fundamental problem with this approach:

It condenses all information into a single number.

Distributions can look vastly different while having the same mean.
Here are 6 distributions, all with a mean of 0.5%:

Consider only the tight normal and high-vol normal. We would always wish for the tight normal, since we are way more confident about our predicted return.

Now you could go and layer on top something like conformal prediction (which we discussed in our previous article) to get a sense of uncertainty, but even that doesn’t tell you the whole distribution.

And here is another idea: If you are able to predict the entire distribution, you can back out numbers like:

• The probability of the trade being profitable.

• The probability of a loss of a certain size.

• Any arbitrary quantity of the distribution.

This is what Distributional Regression gives you.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why predicting the conditional mean throws away most of the information contained in a return distribution.

• How distributional regression turns a conventional regression problem into one of predicting the entire conditional probability distribution.

• How link functions map unconstrained predictors to valid distributional parameters, allowing us to model quantities such as scale or tail parameters while respecting their constraints.

• How NGBoost combines gradient boosting with probabilistic prediction to learn these distributional parameters.

• How distributional predictions can give you quantities such as volatility, quantiles, tail probabilities, and expected shortfall from a single model.

Read more

Now no need to be sceptical about the “neural network” part! We also break up all the common misconceptions people have about them in the article.

With that said, our forecast there is just a single point, and we have no notion of how certain the neural network is about that forecast. And that second number drives a lot of decisions in the real world: How wide to quote, how much size to put on, etc.

What we want is a range we believe realized volatility to land in with a certain probability! A naive way to get something like that is to assume a distribution: Fit the model, look at the residuals, assume they’re Gaussian, and then read off the bands you get. This fails immediately for volatility, whose residuals are skewed and heavy-tailed. Plus we’ve now bolted a distributional assumption onto a model we picked deliberately because it was flexible and assumption-light.

What we’d like is a distribution-free, finite-sample guarantee that the true value lands inside the interval with the probability we asked for, and being able to use any model. That is exactly what conformal prediction gives us.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why a point forecast is only half of a volatility model, and what a proper prediction interval gives you that a single number can’t.

• How conformal prediction wraps any model in a distribution-free coverage guarantee that holds in finite samples, without assuming anything about the shape of the errors.

• Why the naive symmetric, constant-width interval is wrong for volatility, and how conformalized quantile regression (CQR) fixes both.

• The assumptions the coverage guarantee rests on and why time series badly violate it.

• How Adaptive Conformal Inference (ACI) restored a coverage guarantee under arbitrary distribution shift, and what it loses.

• A full implementation on top of our neural-network-based volatility forecaster.

Split Conformal Prediction

Start with the model you’ve already trained and call it f^, which in our case is the neural-network-based volatility forecaster. Conformal prediction will never actually look inside the model and treat it as a complete black box. Instead, we set aside a chunk of data the model never saw during training, called the calibration set, which consists of n points (x_1, y_1), …, (x_n, y_n).

For each calibration point, we compute a nonconformity score:

For a target coverage of 1 - alpha (say 90%, so alpha = 0.1), we take an empirical quantile of the scores with a small finite-sample correction:

Finally, for any new point x_{n+1}, we form the interval by padding the point forecast with the above quantile:

For a new point drawn from the same distribution, the interval covers the truth with at least the probability you asked for:

This holds for ANY model f^, no matter how bad. A really bad model will simply give you very wide intervals. Note that this holds in finite samples, and without any assumptions on the distribution of the data!

Why it works, and the one assumption hiding inside

Okay, maybe it’s not all sunshine and rainbows. The guarantee that the interval covers the truth of the new point with at least the probability you asked for rests on a single condition: the calibration scores and the new point’s score must be exchangeable, meaning that their joint distribution remains unchanged by any reordering. Exchangeability is the slightly weaker cousin of “i.i.d.”, since it doesn’t require independence, only that the ordering carries no information.

Our data is not that. A volatility series is ordered in time, and the ordering carries an enormous amount of information. Two more problems with this method for our volatility series are symmetry and constant-width. We apply the same padding up and down at a given point, which can even make the interval go into the negative for low-volatility periods (Which isn’t necessarily wrong, but we’d much rather have more accurate intervals). We also apply the same padding across different points in time, which is broken by heteroskedasticity in volatility.

Conformalized Quantile Regression

Instead of producing a point forecast and padding it symmetrically, we will have the model predict its own lower and upper bounds directly. We give the network 2 output neurons that give us the quantiles

its own estimate of where the lower and upper ends of the outcome sit at that particular x. This already gives us good intervals, and conformal prediction will then come in and make the whole thing valid.

Predicting Quantiles

To get the low and high quantiles at levels alpha/2 and 1-alpha/2 (for 90% coverage, the 5th and 95th percentiles), we train the network with the pinball loss rather than a squared error. For a target quantile level tau, the pinball loss on the prediction q^and truth y is

This is an asymmetric loss that penalizes being below the truth and above the truth at different rates, and the rate is set by tau. Minimising it drives q^ toward the true conditional tau-quantile of y. In our neural network, we do this once for the tau=alpha/2 head and once for the tau=1-alpha/2 head, and sum the two losses together.

Conformalizing the Quantiles

Quantile regression on its own comes with no coverage guarantee at all. We still need conformal prediction to give us those nice guarantees. We define the new nonconformity score by how far the truth fell outside the predicted band

We then take the same finite-sample-corrected quantile of these scores:

and form the final interval for a new point by adjusting each predicted quantile:

There is still one thing to keep in mind, however. While CQR fixes the shape of the interval, it still carries along the problem with exchangeability. We will fix that next.

Adaptive Conformal Inference (ACI)

The idea is to stop treating the miscoverage level as a fixed number. So far, alpha has been set once (0.1 for 90% coverage) and used as is forever. ACI makes it a variable that adjusts itself in response to how well we’ve actually been covering. If the intervals have been missing too often lately, we push the level to make them wider, and if they’ve been covering more than we asked for, we tighten them. Concretely, we track whether each interval covered its outcome

which is 1 when the interval missed and 0 when it covered. We then update the level using the following rule:

Here alpha is still our target (0.1), while alpha_t is the target we actually feed to the conformal prediction to compute the quantiles

and gamma > 0 is a step size that controls how quickly we adapt.

What this gives us

ACI requires no distributional assumptions, no exchangeability, no stationarity, and nothing about how the data is generated. Under arbitrary distribution shift, the realised long-run miscoverage converges to the target you asked for:

What you lose is the finite-sample guarantee, which is now replaced by this long-run average. On any single day, the interval can be wrong, and the way ACI achieves its average is by reacting after it was wrong. It also introduces a parameter gamma, which you need to tune. There is a variant called AgACI, which removes it by running several values of gamma in parallel and aggregating them, though a single well-chosen gamma is enough for our purpose.

Implementation and Testing on Real Data

The rest of this article is built on top of the code from the following one:

Step 1 is defining the constants we will use and adding a second output to the NARX model:

ALPHA  = 0.10          # target miscoverage  ->  90% intervals
TAU_LO = ALPHA / 2     # 0.05  lower quantile
TAU_HI = 1 - ALPHA / 2 # 0.95  upper quantile

class QuantileForecastNet(nn.Module):
    def __init__(self, hparams):
        super().__init__()
        self.hparams = hparams
        self.n_d = hparams["n_d"]

        layers = []
        in_dim = hparams["input_dim"] + self.n_d
        for h in hparams["hidden_dims"]:
            layers.append(nn.Linear(in_dim, h))
            layers.append(nn.BatchNorm1d(h))
            layers.append(nn.ReLU())
            if hparams.get("dropout", 0.0) > 0:
                layers.append(nn.Dropout(hparams["dropout"]))
            in_dim = h

        self.features = nn.Sequential(*layers)
        self.output = nn.Linear(in_dim, 2)          # [q_lo, q_hi]
        self.detection_head = nn.Linear(in_dim, 1)

        self.apply(self._init_weights)

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            nn.init.kaiming_normal_(module.weight, nonlinearity='relu')
            nn.init.zeros_(module.bias)

    def forward(self, x, lags):
        z = self.features(torch.cat([x, lags], dim=-1))
        return self.output(z)                        # [B, 2]

    def forward_pretrain(self, x, lags_swapped):
        z = self.features(torch.cat([x, lags_swapped], dim=-1))
        return self.detection_head(z).squeeze(-1)

Next, we need to be able to train the model using pinball loss:

def pinball_loss(pred, target, tau):
    diff = target - pred
    return torch.mean(torch.maximum(tau * diff, (tau - 1.0) * diff))

def build_tensors(panel, feature_cols, hparams, device, with_target=True):
    panel = panel.copy()
    n_d = hparams["n_d"]
    lag_cols = [f"log_RV_lag{d}" for d in range(0, n_d)]
    for d, col in zip(range(0, n_d), lag_cols):
        panel[col] = panel["log_RV"].shift(d)
    panel = panel.dropna(subset=lag_cols)

    X = torch.tensor(panel[feature_cols].values, dtype=torch.float32).to(device)
    lags = torch.tensor(panel[lag_cols].values, dtype=torch.float32).to(device)
    y = None
    if with_target:
        y = torch.tensor(panel["target_log_RV_next"].values,
                         dtype=torch.float32).to(device)
    return X, lags, y

def train_model(model, train_panel, val_panel, feature_cols, hparams, device,
                name="default"):

    X_train, lags_train, y_train = build_tensors(train_panel, feature_cols, hparams, device)
    X_val,   lags_val,   y_val   = build_tensors(val_panel,   feature_cols, hparams, device)

    n_train = X_train.shape[0]
    batch_size = hparams.get("batch_size", 8192)
    n_batches = (n_train + batch_size - 1) // batch_size
    loss_cutoff = max(n_batches // 10, 1)

    model = model.to(device)
    optimizer = torch.optim.SGD(
        model.parameters(),
        lr=hparams["lr"],
        momentum=hparams["momentum"],
        weight_decay=hparams["weight_decay"],
        nesterov=True,
    )

    best_val_loss = float('inf')
    patience_counter = 0
    best_weights = None
    epoch = 0

    while True:

        model.train()
        train_losses = []
        perm = torch.randperm(n_train, device=device)

        pbar = tqdm(range(n_batches), desc=f"[{name}] epoch {epoch}", leave=False)
        for it in pbar:
            idx = perm[it * batch_size : (it + 1) * batch_size]
            xb, lagsb, yb = X_train[idx], lags_train[idx], y_train[idx]

            optimizer.zero_grad()
            q = model(xb, lagsb)                       # [B, 2]
            loss = (pinball_loss(q[:, 0], yb, TAU_LO)
                    + pinball_loss(q[:, 1], yb, TAU_HI))
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            optimizer.step()

            train_losses.append(loss.item())
            train_losses = train_losses[-loss_cutoff:]
            pbar.set_postfix(train_loss=f"{np.mean(train_losses):.4f}")

        model.eval()
        with torch.no_grad():
            qv = model(X_val, lags_val)
            val_loss = (pinball_loss(qv[:, 0], y_val, TAU_LO)
                        + pinball_loss(qv[:, 1], y_val, TAU_HI)).item()

        #tqdm.write(f"[{name}] epoch {epoch}  train {np.mean(train_losses):.4f}  val {val_loss:.4f}")

        if val_loss < best_val_loss:
            best_val_loss = val_loss
            best_weights = copy.deepcopy(model.state_dict())
            patience_counter = 0
        else:
            patience_counter += 1

        if patience_counter >= hparams["patience"]:
            break
        epoch += 1

    if best_weights is not None:
        model.load_state_dict(best_weights)
    else:
        print("Warning: training never improved past initial val_loss (likely NaN/divergence)")
    return model

def predict_quantiles(model, panel, feature_cols, hparams, device):
    X, lags, y = build_tensors(panel, feature_cols, hparams, device, with_target=True)
    model.eval()
    with torch.no_grad():
        q = model(X, lags).cpu().numpy()              # [N, 2]
    q_lo = np.minimum(q[:, 0], q[:, 1])               # enforce q_lo <= q_hi
    q_hi = np.maximum(q[:, 0], q[:, 1])
    return q_lo, q_hi, y.cpu().numpy()

And next, all of the conformal prediction machinery:

def cqr_scores(q_lo, q_hi, y):
    return np.maximum(q_lo - y, y - q_hi)

def cqr_quantile(sorted_scores, level):
    n = len(sorted_scores)
    k = int(np.ceil((n + 1) * level))
    if k < 1:
        return -np.inf
    if k > n:
        return np.inf
    return sorted_scores[k - 1]

def cqr_interval(cal_scores, q_lo_test, q_hi_test, alpha=ALPHA):
    s = np.sort(cal_scores)
    Q = cqr_quantile(s, 1 - alpha)
    return q_lo_test - Q, q_hi_test + Q

def aci_cqr(cal_scores, q_lo_test, q_hi_test, y_test, alpha=ALPHA, gamma=0.01):
    s = np.sort(cal_scores)
    T = len(y_test)
    lo = np.empty(T)
    hi = np.empty(T)
    covered = np.empty(T, dtype=bool)
    alphas = np.empty(T)

    alpha_t = alpha
    for t in range(T):
        alphas[t] = alpha_t
        Q = cqr_quantile(s, 1 - alpha_t)
        lo[t] = q_lo_test[t] - Q
        hi[t] = q_hi_test[t] + Q

        err = 0.0 if (lo[t] <= y_test[t] <= hi[t]) else 1.0
        covered[t] = (err == 0.0)

        alpha_t = alpha_t + gamma * (alpha - err)
        alpha_t = min(max(alpha_t, 0.0), 1.0)         # keep the working level in [0, 1]

    return lo, hi, covered, alphas

I will simply train this model using the same hyperparameters I picked for the single-point volatility forecast. If I were to run this in production, I would run full hyperparameter tuning again. For gamma, I just pick a standard value of 0.01, but you likely want to add it to the same hyperparameter tuning.

hparams = {
    "input_dim": len(feature_cols),
    "hidden_dims": [370],
    "dropout": 0.25505051838919374,
    "lr": 0.01619079458212241,
    "momentum": 0.9,
    "weight_decay": 1e-05,
    "patience": 1000,
    "batch_size": 4096,
    "n_d": 30,
}

device = 'cuda:0'
model = QuantileForecastNet(hparams)

split = len(val_data) // 2
cal_data  = val_data.iloc[:split]
test_data = val_data.iloc[split:]

model = train_model(model, train_data, val_data, feature_cols, hparams, device)

Now let’s finally test out our model:

cal_lo, cal_hi, cal_y = predict_quantiles(model, cal_data, feature_cols, hparams, device)
cal_E = cqr_scores(cal_lo, cal_hi, cal_y)

# test-time quantiles
test_lo, test_hi, test_y = predict_quantiles(model, test_data, feature_cols, hparams, device)

# CQR + ACI (log space)
aci_lo, aci_hi, aci_covered, aci_alphas = aci_cqr(cal_E, test_lo, test_hi, test_y,
                                                  alpha=ALPHA, gamma=0.01)

print(f"CQR+ACI marginal coverage : {aci_covered.mean():.3f}")

N = 300 

lo  = np.exp(aci_lo[:N])
hi  = np.exp(aci_hi[:N])
act = np.exp(test_y[:N])
mid = np.exp(0.5 * (aci_lo[:N] + aci_hi[:N]))

t = np.arange(N)

fig, ax = plt.subplots(figsize=(14, 5))
ax.fill_between(t, lo, hi, color="steelblue", alpha=0.25,
                label=f"{int((1-ALPHA)*100)}% CQR+ACI interval")
ax.plot(t, mid, color="steelblue", lw=1.0, label="predicted (band midpoint)")
ax.plot(t, act, color="black", lw=0.8, label="actual RV")

ax.set_xlabel("test step")
ax.set_ylabel("realized variance")
ax.set_title(f"CQR + ACI volatility intervals — first {N} test steps  "
             f"(overall coverage {aci_covered.mean():.1%}, target {1-ALPHA:.0%})")
ax.legend(loc="upper right", framealpha=0.9)
ax.margins(x=0)
plt.tight_layout()
plt.show()

Those forecasts look absolutely fantastic! Between 200 and 250 you can see very tight intervals where the model is sure volatility is low, and around 50-150 where there is a ton of volatility, the model is naturally uncertain and outputs large intervals.

Conclusion

We’ve taken our NARX volatility forecaster that already beats baseline models, and made it into something even greater! From here, there are still many directions you can take:

1. Worst-case risk analysis using the upper quantile.

2. Adding back a point forecast as a third head, rather than taking the mean of the two heads.

3. Properly tuning gamma or using a method like AgACI.

4. Improving the underlying model itself further.

Other Articles You Would Enjoy

Quant Corner

Join Quant Corner, a community that actually does things. Tournaments, live Q&As, open discussions, and the best place to shape what gets built next.

Join here: https://discord.gg/X7TsxKNbXg

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Common mistakes people make in quantitative research, and why they happen.

• How to diagnose if you’ve fallen victim to any of the mistakes.

• How to concretely mitigate the mistakes.

Read more

Two time series can move together for a lot of reasons that have nothing to do with one driving the other. Ice cream sales and drowning incidents both rise every summer without either one causing the other; they’re both just downstream of the same thing: hot weather.

We start with Bayesian networks and how they tell us which variables can influence which. From there, we move to structural equation models, which let us distinguish between observing a variable and actually intervening on it. We then bring in Granger causality, the classical way of asking whether one time series helps predict another, and its more general, information-theoretic counterpart, directed information. In the end, we take a look at the hardest and most practically relevant case: what happens when some of the variables actually driving the system were never observed at all, and how much of the true causal structure can still be recovered despite that.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• What it actually means for one variable to cause another, and why a Bayesian network’s conditional independencies alone can’t answer that.

• Why conditioning on the wrong variable can create a dependence between two variables that never existed.

• How the intervention operator do(.) formally separates observing X=x from forcing X=x.

• How Granger causality turns “does X cause Y” into a precise, testable statement about prediction, and what directed information has to do with it.

• Why a naive causality test across many time series can show a relationship between two variables that don’t actually influence each other at all, and what conditioning on the rest of the system has to do with fixing it.

• What happens when hidden variables enter the system, and under what conditons the true causal structure can still be recovered anyway.

Read more

When pricing options using Monte Carlo methods, on the other hand, you often fit something like Brownian motion to the underlying as well as possible and take the average payoff of those simulated paths, knowing full well that they behave nothing like the real asset.

GAN's solve both of those issues by learning the data-generating process directly from real market data with no assumptions about normality, no hand-crafted volatility models, and no direct parameters. The generator learns to produce synthetic return paths that are statistically indistinguishable from the real thing, preserving heavy tails, volatility clustering, the leverage effect, etc. that are present in real markets and that parametric models consistently get wrong.

In this article, we build up the full framework from scratch. We start with the original GAN, work through the training instabilities that make it difficult to use in practice, and then introduce methods to fix this training instability. In the end, we will take a look at one specific architecture called Quant GANs that is specifically designed for financial time series.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• What GANs are and why they’re so hard to train in practice.

• How the Wasserstein distance fixes the core mathematical failure of the original GAN objective.

• How Mescheder’s gradient penalty stabilizes training.

• How the Quant GAN architecture works and how it reproduces stylized facts of financial returns like heavy tails, volatility clustering, and the leverage effect.

• How to evaluate a generative model for financial data, and why standard metrics like loss curves tell you almost nothing.

Read more

Neural networks have a bad reputation in the quant finance community, and I constantly see people skip papers with titles like “Volatility Forecasting using Neural Networks” immediately. The reason is usually some version of: black-box, overfits, needs more data than any of us actually have, traditional models are more robust.

First things first, we will be clearing up some of the common misconceptions and giving them a chance, like proper researchers. Next, we will talk about the specific model architecture used in this article, why forecasting in log-space turns out to improve performance, and how we selected a final model without overfitting. Finally, we’ll put the model to the test against a simple traditional baseline model commonly used for volatility forecasting.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why the usual complaints about neural networks in finance hold up far less than people assume, and where they’re actually right.

• How a NARX architecture works, and how to apply it to volatility forecasting.

• Why forecasting realized volatility in log-space improves training stability and accuracy, even when your loss function (QLIKE) is already designed to be scale-robust.

• How to tune and select a model, including catching when a network is overfitting.

• How to interpret a trained network’s predictions using SHAP, instead of treating it as an unreadable black box.

• A full comparison against a tuned EWMA and persistence baseline.

Read more

The moment you care about speed, like when quoting live, or when calibrating a pricing model where you need to reprice options thousands of times, Monte Carlo quickly becomes unusable.

This article presents three Fourier-based methods that solve this problem, reducing pricing from seconds to microseconds. We’ll use the Heston model as our running example throughout, and achieve a speedup of 30,000x against Monte Carlo!

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why Monte Carlo breaks down for live pricing and model calibration, and what makes it fundamentally unsuitable for production options trading.

• How the Fourier Transform and FFT work and why they matter for options pricing.

• What the characteristic function is and how it enables us to price options even when the risk-neutral density of the model has no known closed-form analytical solution.

• How Carr-Madan transforms option pricing into an FFT problem.

• How Lewis reformulates the same problem via contour integration, eliminating the dampening parameter that makes Carr-Madan brittle.

• How the COS method takes a completely different approach, approximating the density via a cosine series, and why it converges exponentially fast.

• Python implementations for all methods, together with speed comparisons.

This article is free to read, so I would appreciate it if you shared it with someone who would also like it!

The Characteristic Function

To price options using Fourier methods, we need one thing from our model: the characteristic function of log S_T where S_T is the price at expiry.

For a random variable X, the characteristic function is defined as:

For log S_T we write:

where Q is the risk-neutral measure.

Just like the probability density function, it completely characterises the distribution of X. But while the density q(x) often has no closed-form expression, the characteristic function often does and is much easier to find.

In fact, the characteristic function and the density are a Fourier transform pair. Given the characteristic function, you can recover the density via the inverse Fourier transform:

This is the core idea behind all Fourier pricing methods. You don’t need to know the distribution of log S_T explicitly; you just need the characteristic function.

Why does the characteristic function always exist?

For any model where log S_T is a well-defined random variable, the characteristic function is guaranteed to exist for all real u, since

and thus the expectation is always finite.

Pricing in Fourier Space

In theory, you could now go ahead and compute the distribution from the characteristic function numerically, and then integrate against the payoff to get the fair value of the option. In practice, Fourier pricing methods skip this step entirely and price directly in Fourier space, which is both faster and more numerically stable.

Two Examples

For Black-Scholes, log S_T is Gaussian, so the characteristic function is that of a Gaussian random variable, with appropriate mean and standard deviation plugged in:

For Heston, log S_T has no nice closed-form density expression. But the characteristic function is still available in closed form:

where:

with

These are derived by applying Itô’s lemma to log S_t, exploiting the affine structure of the Heston dynamics, and solving the resulting Riccati ODEs in closed form.

How to properly derive this will be the subject of future articles.

Fourier Transform and FFT

Before deriving any pricing formulas, we need to understand the Fourier Transform and the Fast Fourier Transform algorithm.

The Fourier Transform

For a real square integrable function f(x), the Fourier transform is:

This function lives in frequency space; It tells you how much of each frequency xi is present in f. You recover f from hat{f} via the inverse Fourier transform:

Two real functions f and g that are both square integrable have a well-defined inner product. Parseval’s theorem says this inner product can also be computed in Fourier space:

This will be used later on in the Lewis method.

The Discrete Fourier Transform

In practice, you work with a finite grid of N points rather than a continuous function. The Discrete Fourier Transform (DFT) of a sequence x_0, x_1, …, x_{N-1} is:

Each output X_k is the inner product of the input sequence against a complex exponential at frequency k. Computing all N outputs naively requires evaluating N terms for each of the N outputs, a total of N^2 multiplications.

The Fast Fourier Transform

For N a power of 2, the Fast Fourier Transform, introduced by Cooley and Tukey, reduces the cost from O(N^2) to O(N log N) by exploiting the fact that the DFT of a sequence of length N can be written as the sum of two DFTs of length N/2.
Applying this decomposition recursively, splitting each half into its own two halves, reduces the total number of multiplications to O(N log N). For N = 4096, this is roughly a 340x speedup over the naive approach!

More formally, the DFT of a sequence of length N can be written as the sum of the DFT over the even-indexed elements and the DFT over the odd-indexed elements:

where E_k and O_k are the DFTs of the even and odd subsequences.

Carr-Madan

The Carr-Madan method (1999) is one of the most popular option pricing methods using the Fast Fourier Transform.

Let s = log S_T and k = log K. The risk-neutral density of s is q_T(s), with characteristic function

The call price as a function of log strike is:

We want to Fourier transform C_T(k). The problem is that C_T(k) is not square-integrable, since as k→ -infinity, C_T(k) → S_0 rather than decaying to zero. So its Fourier Transform doesn’t exist in the classical sense.

The fix is to multiply by an exponential dampening factor. Define the modified call price:

For suitable alpha, c_T(k) is square integrable, since the e^{alpha k) term decays fast enough as k → -infty to kill the blow-up. Therefore, this has a well-defined Fourier transform:

Our goal now is to write an analytical expression of Psi_T in terms of phi_T, and then obtain call prices numerically using the inverse transform.

Deriving Psi_T

Substitute the definitions of c_T(k) and C_T(k):

The region of integration is

which is equivalent to:

The integrand is non-negative, so we can apply Tonelli’s theorem and swap the order of integration:

Expand the inner integrand:

For the first term:

For the second term:

Substituting back:

Factor out

and recognise that since

we have:

So:

After simplifying, we finally get:

This is expressed entirely in terms of phi_T, the characteristic function of log S_T under our model of choice, which we said earlier we often have a closed-form solution for, so this whole thing is closed-form!

Recovery Formula

Since Psi_T is the Fourier transform of the dampened call price, standard Fourier inversion gives:

Dividing both sides by e^{alpha k}:

Since C_T(k) is real, we have

so the integrand over (-infty, 0) is the complex conjugate of the integrand over (0,infty), and their sum is twice the real part:

Choice of alpha

Two conditions constrain alpha. First, for Psi_T(0) to be finite, we need

Another condition is alpha not being equal to 0, to avoid a singularity in the denominator at v = 0. For Black-Scholes and Heston, all positive moments exist, so alpha is unconstrained. Carr and Madan suggest using one quarter of the upper bound implied by the moment condition, and find alpha = 1.5 works well in practice.

Truncation bound

The recovery integral runs from 0 to infinity. We need to verify it converges and can be safely truncated at some finite value a. For that, let’s first look at the numerator of |Psi_T(v)|.

It’s easy to see that

which is independent of v.

The denominator of Psi_T(v) grows like v^4 for large v, so:

for some constant A. Therefore

and the tail integral from any truncation point a satisfies:

Thus, the truncation error is bounded by:

To achieve a target error epsilon, choose:

FFT discretization

Discretize the recovery integral with N points at spacing eta, setting

Applying the trapezoid rule:

The effective upper limit of integration is a = N*eta. We want to evaluate this at a grid of N log-strikes

which gives us log strike levels ranging from -b to b, centred around ATM by setting

Substituting v_j and k_u:

For the sum to match the DFT structure

we need

This is the grid coupling constraint; Fine integration grid (eta small) forces coarse strike spacing (lambda large) and vice versa. The sum becomes:

which is exactly the DFT of the sequence

A single FFT call evaluates this for all N strikes simultaneously.

So you build the input vector

once and then call the FFT algorithm. Then multiply the u-th output by

to get the final call prices for each strike.

Simpson’s rule

The trapezoid rule has errors O(eta^2). To get accurate integration with larger eta, which, via the grid coupling constraint, means finer strike spacing, we replace the uniform weight eta with Simpson’s rule weights:

where delta_{j-1} is the Kronecker delta, equal to 1 only at j=1. The error improves from O(eta^2) to O(eta^4), and the FFT structure is preserved since replacing eta with w_j just multiplies each term by a scalar before applying the FFT.

The final formula is:

The weakness of Carr-Madan

The grid coupling constraint is the fundamental limitation; you cannot independently choose integration accuracy and strike grid density. Additionally, alpha requires model-specific tuning. Both issues are addressed by the methods that follow.

Python Implementation

We will be working with Heston in this article. Let’s first implement Monte Carlo pricing using Euler-Maruyama as a ground truth (note that Monte Carlo will give you slightly wrong values since we did not account for the discretisation, but it’s a method that is easy to implement and thus almost true, very likely) and for speed comparison:

import numpy as np
from scipy.stats import norm
import time
import matplotlib.pyplot as plt
from scipy.integrate import quad

def heston_mc(S0, K, r, q, T, kappa, theta, eta, rho, v0,
              n_steps=200, n_paths=1_000_000, seed=42):
    rng = np.random.default_rng(seed)
    dt  = T / n_steps
    S   = np.full(n_paths, float(S0))
    v   = np.full(n_paths, float(v0))

    for _ in range(n_steps):
        Z1    = rng.standard_normal(n_paths)
        Z2    = rho * Z1 + np.sqrt(1 - rho**2) * rng.standard_normal(n_paths)
        v_pos = np.maximum(v, 0)
        S    *= np.exp((r - q - 0.5 * v_pos) * dt + np.sqrt(v_pos * dt) * Z1)
        v    += kappa * (theta - v_pos) * dt + eta * np.sqrt(v_pos * dt) * Z2

    return np.exp(-r * T) * np.mean(np.maximum(S - K, 0))

Next, we implement the Heston Characteristic Function:

def heston_cf(omega, S0, r, q, T, kappa, theta, eta, rho, v0):
    x  = np.log(S0)
    xi = kappa - 1j * rho * eta * omega
    d  = np.sqrt(xi**2 + eta**2 * (omega**2 + 1j * omega))
    g  = (xi - d) / (xi + d)

    C = ((r - q) * 1j * omega * T
         + (kappa * theta / eta**2) * (
             (xi - d) * T - 2 * np.log((1 - g * np.exp(-d * T)) / (1 - g))
         ))
    D = ((xi - d) / eta**2) * (1 - np.exp(-d * T)) / (1 - g * np.exp(-d * T))

    return np.exp(C + D * v0 + 1j * omega * x)

and finally, the FFT implementation of Carr-Madan using Simpson weights:

def carr_madan_fft(S0, r, q, T, kappa, theta, eta, rho, v0,
                   alpha=1.5, N=4096, eta_grid=0.25):
    lam = 2 * np.pi / (N * eta_grid)
    b   = 0.5 * N * lam

    j   = np.arange(1, N + 1)
    v_j = eta_grid * (j - 1)

    cf  = lambda omega: heston_cf(omega, S0, r, q, T, kappa, theta, eta, rho, v0)

    psi = (np.exp(-r * T) * cf(v_j - (alpha + 1) * 1j)
           / (alpha**2 + alpha - v_j**2 + 1j * (2 * alpha + 1) * v_j))

    w   = (eta_grid / 3) * (3 + (-1)**j - (j == 1).astype(float))
    k_u = -b + lam * (np.arange(1, N + 1) - 1)

    x       = np.fft.fft(np.exp(1j * b * v_j) * psi * w)
    calls   = (np.exp(-alpha * k_u) / np.pi) * np.real(x)
    strikes = np.exp(k_u)

    return strikes, calls

Let’s compare MC to Carr-Mada:

# ── parameters ───────────────────────────────────────────────────────────────

params = dict(S0=100, r=0.05, q=0, T=1.0,
              kappa=2.0, theta=0.04, eta=0.3, rho=-0.7, v0=0.04)

strikes_to_price = np.arange(60, 141, 5, dtype=float)

# ── MC ground truth ───────────────────────────────────────────────────────────

t0 = time.perf_counter()
mc_prices = np.array([heston_mc(**params, K=K) for K in strikes_to_price])
mc_time   = time.perf_counter() - t0

# ── Carr-Madan ────────────────────────────────────────────────────────────────

t0 = time.perf_counter()
cm_strikes, cm_calls = carr_madan_fft(**params)
cm_time = time.perf_counter() - t0
cm_prices = np.interp(strikes_to_price, cm_strikes, cm_calls)

# ── print results ─────────────────────────────────────────────────────────────

print(f"{'Strike':>8} {'MC':>10} {'Carr-Madan':>12} {'Error':>10}")
print("-" * 44)
for K, mc, cm in zip(strikes_to_price, mc_prices, cm_prices):
    print(f"{K:>8.0f} {mc:>10.4f} {cm:>12.4f} {abs(mc-cm):>10.4f}")

print(f"\nMC time:          {mc_time:.2f}s  ({len(strikes_to_price)} strikes, 100k paths each)")
print(f"Carr-Madan time:  {cm_time*1000:.2f}ms (all strikes simultaneously)")

# ── plot ──────────────────────────────────────────────────────────────────────

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(strikes_to_price, mc_prices, 'o', label='Monte Carlo', color='steelblue')
ax1.plot(strikes_to_price, cm_prices, '-', label='Carr-Madan', color='tomato', linewidth=2)
ax1.set_xlabel('Strike')
ax1.set_ylabel('Call Price')
ax1.set_title('Heston Call Prices')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.bar(['Monte Carlo\n(100k paths, 17 strikes)', 'Carr-Madan FFT\n(all strikes)'],
        [mc_time, cm_time],
        color=['steelblue', 'tomato'])
ax2.set_ylabel('Time (seconds)')
ax2.set_title('Computation Time')
ax2.grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

  Strike         MC   Carr-Madan      Error
--------------------------------------------
      60    43.0926      43.0487     0.0439
      65    38.4447      38.4019     0.0428
      70    33.8703      33.8281     0.0423
      75    29.4033      29.3614     0.0420
      80    25.0861      25.0447     0.0415
      85    20.9701      20.9298     0.0403
      90    17.1172      17.0758     0.0415
      95    13.5838      13.5438     0.0401
     100    10.4286      10.3950     0.0336
     105     7.7081       7.6798     0.0283
     110     5.4500       5.4306     0.0194
     115     3.6676       3.6572     0.0104
     120     2.3451       2.3334     0.0117
     125     1.4146       1.4061     0.0085
     130     0.8051       0.8002     0.0049
     135     0.4343       0.4309     0.0034
     140     0.2238       0.2206     0.0031

MC time:          6.46s  (17 strikes, 100k paths each)
Carr-Madan time:  1.05ms (all strikes simultaneously)

The results speak for themselves! Even at 100k paths, which isn’t all that much, and gives somewhat inaccurate results, running MC takes around 6.5 seconds, while Carr-Madan only takes a millisecond. But we can do even better!

Lewis

Lewis (2001) reformulates option pricing as a complex contour integral. The result is a cleaner formula than Carr-Mada; no dampening parameter alpha to tune, and the strip condition is easily satisfied for essentially all models used in practice.

The core idea

The option price is a discounted expectation under Q (We are restating this since the paper has slightly different notation):

where x = log S_T, w(x) is the payoff function, and q_T(x) is the risk-neutral density of log S_T. This is an inner product of w and q_T. Parseval’s theorem says inner products are preserved under Fourier transformation:

Since q_T(x) is real, we have

So:

The problem

That formula looks clean, but there’s a catch. For a call, the payoff is

which grows like e^x as x → infty. Its ordinary Fourier transform diverges!

Carr-Madan’s fix was to multiply the payoff by a dampening factor before transforming. Lewis takes a cleaner route: allow the transform variable to be complex.

Generalized Fourier Transform

Let z = u+iv be a complex number. Define the generalized Fourier transform of the payoff:

Now e^{izx} = e^{iux-vx}, and the factor e^{-vx} provides the decay that kills the growth of w(x), as long as v is large enough. For the call:

This does not converge unless Im(z) > 1. With Im(z) > 1 we have:

We call this area the strip of regularity for the call payoff:

You can refer to the following table from the paper for other options:

The characteristic function has its own regularity strip. For the characteristic function of X_T to exist, we need

The set of valid complex v with this property forms the strip S_X.

Define the reflected strip

This is where phi_T(-z) is regular, which is what appears in the pricing formula.

Shifting the contour

By Cauchy’s theorem, the integration contour may be shifted to any horizontal line inside the common strip of regularity:

The Lewis Formula

The option price finally becomes:

where v = Im(z) in S_V and Y = ln S_0 + (r-q)T is the log forward price, and phi denotes the characteristic function of the centred log-return X_T = log S_T - Y.

Call Pricing Formula

Substituting the call payoff transform hat{w} from above:

where b is some model-dependent upper bound below which the characteristic function stays regular. Lewi’s preferred contour is v=1/2, obtained by shifting the contour from (1,b) down to (0,1) via the residue theorem and picking up the pole at z=i. That gives

where k = ln(S_0/K) + (r-q)T is the log-moneyness. The contour v=1/2 sits symmetrically between the poles at z=0 and z=i, and requires only

satisfies by practically every standard model, and is a much weaker requirement than

that Carr-Madan needs for alpha > 0.

Python Implementation

Lewis uses the characteristic function of X_T:

def heston_cf_xt(omega, S0, r, q, T, kappa, theta, eta, rho, v0):
    """CF of X_T = log(S_T/S_0) - (r-q)T under Heston."""
    xi = kappa - 1j * rho * eta * omega
    d  = np.sqrt(xi**2 + eta**2 * (omega**2 + 1j * omega))
    g  = (xi - d) / (xi + d)

    C = (kappa * theta / eta**2) * (
            (xi - d) * T - 2 * np.log((1 - g * np.exp(-d * T)) / (1 - g))
        )
    D = ((xi - d) / eta**2) * (1 - np.exp(-d * T)) / (1 - g * np.exp(-d * T))

    # no (r-q) drift term since X_T is already centered
    return np.exp(C + D * v0 - 0.5 * 1j * omega * eta**2 * 0)

Now since Lewis doesn’t use FFT by design, we have to compute strikes one by one using the contour integral, or we are smart and vectorize:

def lewis_calls_vectorized(S0, strikes, r, q, T, kappa, theta, eta, rho, v0,
                           N=4096, eta_grid=0.25):
    u_j = eta_grid * np.arange(0, N)
    
    # Simpson weights
    w = np.ones(N) * 2
    w[1::2] = 4
    w[0] = w[-1] = 1
    w *= eta_grid / 3

    phi = heston_cf_xt(u_j - 0.5j, S0, r, q, T, kappa, theta, eta, rho, v0)
    phi /= (u_j**2 + 0.25)

    # k for each strike: shape (n_strikes,)
    k = np.log(S0 / strikes) + (r - q) * T

    # outer product: shape (n_strikes, N)
    phase = np.exp(1j * np.outer(k, u_j))

    I = np.real(phase * phi[np.newaxis, :]) @ w / np.pi

    return S0 * np.exp(-q * T) - np.sqrt(S0 * strikes) * np.exp(-(r + q) * T / 2) * I

And the comparison:

t0 = time.perf_counter()
lw_prices = lewis_calls_vectorized(strikes=strikes_to_price, **params)
lw_time = time.perf_counter() - t0

print(f"{'Strike':>8} {'MC':>10} {'Carr-Madan':>12} {'Lewis':>10} "
      f"{'CM Err':>10} {'LW Err':>10}")
print("-" * 64)
for K, mc, cm, lw in zip(strikes_to_price, mc_prices, cm_prices, lw_prices):
    print(f"{K:>8.0f} {mc:>10.4f} {cm:>12.4f} {lw:>10.4f} "
          f"{abs(mc-cm):>10.4f} {abs(mc-lw):>10.4f}")

print(f"\nMC time:          {mc_time:.2f}s")
print(f"Carr-Madan time:  {cm_time*1000:.2f}ms")
print(f"Lewis time:       {lw_time*1000:.2f}ms")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(strikes_to_price, mc_prices, 'o', label='Monte Carlo', color='steelblue')
ax1.plot(strikes_to_price, cm_prices, '-', label='Carr-Madan', color='tomato', linewidth=2)
ax1.plot(strikes_to_price, lw_prices, '--', label='Lewis', color='seagreen', linewidth=2)
ax1.set_xlabel('Strike')
ax1.set_ylabel('Call Price')
ax1.set_title('Heston Call Prices')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.bar(['Carr-Madan\nFFT', 'Lewis\nVectorized'],
        [cm_time, lw_time],
        color=['tomato', 'seagreen'])
ax2.set_ylabel('Time (seconds)')
ax2.set_title('Computation Time (excl. Monte Carlo)')
ax2.grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

  Strike         MC   Carr-Madan      Lewis     CM Err     LW Err
----------------------------------------------------------------
      60    43.0926      43.0487    43.1459     0.0439     0.0533
      65    38.4447      38.4019    38.5020     0.0428     0.0573
      70    33.8703      33.8281    33.9311     0.0423     0.0608
      75    29.4033      29.3614    29.4673     0.0420     0.0640
      80    25.0861      25.0447    25.1536     0.0415     0.0674
      85    20.9701      20.9298    21.0417     0.0403     0.0716
      90    17.1172      17.0758    17.1902     0.0415     0.0730
      95    13.5838      13.5438    13.6612     0.0401     0.0774
     100    10.4286      10.3950    10.5150     0.0336     0.0864
     105     7.7081       7.6798     7.8025     0.0283     0.0944
     110     5.4500       5.4306     5.5570     0.0194     0.1070
     115     3.6676       3.6572     3.7858     0.0104     0.1182
     120     2.3451       2.3334     2.4652     0.0117     0.1201
     125     1.4146       1.4061     1.5412     0.0085     0.1266
     130     0.8051       0.8002     0.9381     0.0049     0.1329
     135     0.4343       0.4309     0.5717     0.0034     0.1375
     140     0.2238       0.2206     0.3646     0.0031     0.1409

MC time:          8.09s
Carr-Madan time:  1.18ms
Lewis time:       2.49ms

As you can see, when using the same N and eta as with Carr-Madan, Lewis starts diverging pretty quickly and is still over twice as slow. If we increase N and reduce eta, we are quickly multiple times slower than Carr-Madan when pricing many strikes at once. On the other hand, we get rid of the dampening parameter alpha, which can cause real problems.

Let’s finally look at our method of choice, which performs better than both Lewis and Carr-Madan.

The COS Method

Both Carr-Madan and Lewis price options by evaluating a Fourier integral numerically. The integrand is oscillatory, which is expensive to integrate accurately with equally spaced quadrature.

The COS method (2008) takes a different route. Instead of integrating the characteristic function directly, it expands the risk-neutral density in a cosine series and reads the series coefficients directly off the characteristic function, with those coefficients often being available analytically.

Fourier-Cosine Series Expansion

Starting from risk-neutral pricing (once again, slightly different notation!):

where x=ln(S_0/K), y = ln(S_T/K), v(y,T) is the payoff, and f(y|x) is the risk-neutral transition density of the log-return.

We truncate to a finite interval [a,b]:

Now expand f(y|x) in a cosine series on [a,b]:

where the ‘ in the sum means the k=0 term is weighted by 1/2, and the cosine coefficients are:

Substituting into the pricing integral and swapping sum and integral (which we can do thanks to Fubini):

where:

Note that the V_k are the cosine series coefficients of v(y,T) in y. Thus we have transformed the product of two real functions, f(y|x) and v(y,T), to that of their Fourier-cosine series coefficients.

Coefficients from the characteristic function

The density f(y|x) is unknown, that’s the whole reason we use Fourier methods in the first place. Comparing the definition of A_k(x) with the characteristic function phi_T(w), one finds that:

where phi_1 is the characteristic function computed over the truncated range [a,b]. Since [a,b] was chosen to capture essentially all the density mass, phi_1 is approx phi and so:

The COS Formula

Finally, replacing A_k by F_k in the formula for V(x, t_0), we obtain:

This is the COS formula.

Payoff coefficients for calls and puts

For a call, v(y,T) = K(e^y-1)^+, the integration domain reduces to [0,b] since the payoff is zero for y < 0. The coefficients are:

where chi and psi have closed forms:

For a put the integration domain is [a,0], and

In practice, you can price puts or calls, and recover the other via put-call parity

Truncation Range

The interval [a,b] must contain enough of the density without being so wide that the cosine approximation needs excessive terms to resolve the tails. Fang & Oosterlee propose using cumulants of ln(S_T/K):

where c_1, c_2, c_4 are the first, second and fourth cumulants of ln(S_T/K). Including c_4 matters for short maturities and fat-tailed processes where the density is sharply peaked.

You can additionally also include a c_6 term as well for extremely short maturities, but the sixth cumulant can be difficult to derive for many models. You can also go the other direction and only include the c_1 and c_2 terms.

Benefits of the COS method

For Carr-Madan and Lewis, accuracy is limited by how well a trapezoidal or Simpson sum approximates a slowly decaying oscillatory integral. The COS method avoids this entirely. For smooth densities like GBM, Heston, and many Lévy processes at moderate maturities, the density is C^{infty} on [a,b] and the cosine coefficients decay exponentially, yielding exponential convergence.

The computational complexity is O(N) for a single strike. For a vector of strikes, the F_k are the same across all strikes, and the V_k differ only through the x-dependent phase; pricing M strikes simultaneously costs O(MN) via matrix-vector multiplication.

Python Implementation

We need the first and second cumulants of log(S_T/K) under Heston:

def heston_cumulants(r, q, T, kappa, theta, eta, rho, v0):
    """First and second cumulants of log(S_T/K) under Heston."""
    c1 = (r - q) * T + (1 - np.exp(-kappa * T)) * (theta - v0) / (2 * kappa) - 0.5 * theta * T

    c2 = (1 / (8 * kappa**3)) * (
        eta * T * kappa * np.exp(-kappa * T) * (v0 - theta) * (8 * kappa * rho - 4 * eta)
        + kappa * rho * eta * (1 - np.exp(-kappa * T)) * (16 * theta - 8 * v0)
        + 2 * theta * kappa * T * (-4 * kappa * rho * eta + eta**2 + 4 * kappa**2)
        + eta**2 * ((theta - 2 * v0) * np.exp(-2 * kappa * T)
                    + theta * (6 * np.exp(-kappa * T) - 7) + 2 * v0)
        + 8 * kappa**2 * (v0 - theta) * (1 - np.exp(-kappa * T))
    )

    return c1, c2

Fast vectorized implementation of the COS method:

def cos_calls(S0, strikes, r, q, T, kappa, theta, eta, rho, v0, N=128, L=12):
    c1, c2 = heston_cumulants(r, q, T, kappa, theta, eta, rho, v0)
    a = c1 - L * np.sqrt(abs(c2))
    b = c1 + L * np.sqrt(abs(c2))

    k     = np.arange(N)
    omega = k * np.pi / (b - a)

    def varphi(omega):
        xi = kappa - 1j * rho * eta * omega
        d  = np.sqrt(xi**2 + eta**2 * (omega**2 + 1j * omega))
        g  = (xi - d) / (xi + d)
        C  = ((r - q) * 1j * omega * T
              + (kappa * theta / eta**2) * (
                  (xi - d) * T - 2 * np.log((1 - g * np.exp(-d * T)) / (1 - g))
              ))
        D  = ((xi - d) / eta**2) * (1 - np.exp(-d * T)) / (1 - g * np.exp(-d * T))
        return np.exp(C + D * v0)

    def chi(c, d):
        kpi = np.where(k != 0, omega, 1.0)
        val = (np.cos(kpi * (d - a)) * np.exp(d)
             - np.cos(kpi * (c - a)) * np.exp(c)
             + kpi * np.sin(kpi * (d - a)) * np.exp(d)
             - kpi * np.sin(kpi * (c - a)) * np.exp(c)) / (1 + kpi**2)
        val[0] = np.exp(d) - np.exp(c)
        return val

    def psi(c, d):
        val      = np.empty(N)
        val[0]   = d - c
        val[1:]  = (b - a) / (k[1:] * np.pi) * (
                       np.sin(omega[1:] * (d - a)) - np.sin(omega[1:] * (c - a))
                   )
        return val

    # precompute strike-independent quantities
    basis    = varphi(omega) * np.exp(-1j * omega * a)  # vphi * e^{-i*omega*a}
    A        = np.real(basis)
    B        = np.imag(basis)
    Uk       = (2 / (b - a)) * (-chi(a, 0) + psi(a, 0))

    # cache constants
    disc     = np.exp(-r * T)
    fwd_S0   = S0 * np.exp(-q * T)
    disc_K   = np.exp(-r * T)

    # vectorize over strikes
    x        = np.log(S0 / strikes)                    # shape (M,)
    theta_mx = np.outer(x, omega)                      # shape (M, N)
    F        = A * np.cos(theta_mx) - B * np.sin(theta_mx)  # shape (M, N)
    F[:, 0] *= 0.5

    puts  = disc * strikes * (F @ Uk)
    calls = puts + fwd_S0 - strikes * disc_K
    return calls

And last but not least, the comparison code:

t0 = time.perf_counter()
cos_prices = cos_calls(strikes=strikes_to_price, **params)
cos_time = time.perf_counter() - t0

print(f"{'Strike':>8} {'MC':>10} {'Carr-Madan':>12} {'Lewis':>10} {'COS':>10} "
      f"{'CM Err':>10} {'LW Err':>10} {'COS Err':>10}")
print("-" * 84)
for K, mc, cm, lw, cos in zip(strikes_to_price, mc_prices, cm_prices, lw_prices, cos_prices):
    print(f"{K:>8.0f} {mc:>10.4f} {cm:>12.4f} {lw:>10.4f} {cos:>10.4f} "
          f"{abs(mc-cm):>10.4f} {abs(mc-lw):>10.4f} {abs(mc-cos):>10.4f}")

print(f"\nMC time:          {mc_time:.2f}s")
print(f"Carr-Madan time:  {cm_time*1000:.2f}ms")
print(f"Lewis time:       {lw_time*1000:.2f}ms")
print(f"COS time:         {cos_time*1000:.2f}ms")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(strikes_to_price, mc_prices, 'o', label='Monte Carlo', color='steelblue')
ax1.plot(strikes_to_price, cm_prices, '-', label='Carr-Madan', color='tomato', linewidth=2)
ax1.plot(strikes_to_price, lw_prices, '--', label='Lewis', color='seagreen', linewidth=2)
ax1.plot(strikes_to_price, cos_prices, ':', label='COS', color='purple', linewidth=2)
ax1.set_xlabel('Strike')
ax1.set_ylabel('Call Price')
ax1.set_title('Heston Call Prices')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.bar(['Carr-Madan\nFFT', 'Lewis\nVectorized', 'COS'],
        [cm_time, lw_time, cos_time],
        color=['tomato', 'seagreen', 'purple'])
ax2.set_ylabel('Time (seconds)')
ax2.set_title('Computation Time (excl. Monte Carlo)')
ax2.grid(True, alpha=0.3, axis='y')

plt.tight_layout()
plt.show()

  Strike         MC   Carr-Madan      Lewis        COS     CM Err     LW Err    COS Err
------------------------------------------------------------------------------------
      60    43.0926      43.0487    43.1459    43.0487     0.0439     0.0533     0.0439
      65    38.4447      38.4019    38.5020    38.4019     0.0428     0.0573     0.0428
      70    33.8703      33.8281    33.9311    33.8280     0.0423     0.0608     0.0424
      75    29.4033      29.3614    29.4673    29.3612     0.0420     0.0640     0.0421
      80    25.0861      25.0447    25.1536    25.0446     0.0415     0.0674     0.0416
      85    20.9701      20.9298    21.0417    20.9297     0.0403     0.0716     0.0403
      90    17.1172      17.0758    17.1902    17.0753     0.0415     0.0730     0.0419
      95    13.5838      13.5438    13.6612    13.5434     0.0401     0.0774     0.0404
     100    10.4286      10.3950    10.5150    10.3942     0.0336     0.0864     0.0344
     105     7.7081       7.6798     7.8025     7.6788     0.0283     0.0944     0.0293
     110     5.4500       5.4306     5.5570     5.4303     0.0194     0.1070     0.0197
     115     3.6676       3.6572     3.7858     3.6562     0.0104     0.1182     0.0114
     120     2.3451       2.3334     2.4652     2.3326     0.0117     0.1201     0.0125
     125     1.4146       1.4061     1.5412     1.4057     0.0085     0.1266     0.0089
     130     0.8051       0.8002     0.9381     0.7996     0.0049     0.1329     0.0055
     135     0.4343       0.4309     0.5717     0.4303     0.0034     0.1375     0.0039
     140     0.2238       0.2206     0.3646     0.2203     0.0031     0.1409     0.0035

MC time:          8.09s
Carr-Madan time:  1.18ms
Lewis time:       2.49ms
COS time:         0.34ms

Awesome! We are more than 3 times faster than Carr-Madan, while getting rid of annoying parameters and gaining exponential convergence!

The method naturally vectorizes into dense linear algebra operations, meaning you could still make this way way faster if you optimized it properly.

Conclusion

Apologies for the delay since the last article! This one took a little longer than I expected since I had to verify all the math, and turn 3 papers into one article instead of the usual one paper.

I hope you enjoyed this rather math-heavy article anyway! I plan on covering more options content soon.

Other Articles You Would Enjoy

Quant Corner

Join Quant Corner, a community that actually does things. Tournaments, live Q&As, open discussions, and the best place to shape what gets built next.

Join here: https://discord.gg/X7TsxKNbXg

The problem is straightforward: When you post a limit order, you know your price and your size, but you don’t know where you actually sit in the queue and how much volume is ahead of you before you get filled. An order at the front of a 10 BTC queue has a completely different fill probability than one at the back of it.

There is one particular blog post by Rigtop (2013) that proposed splitting observed volume changes proportionally between ahead and behind. It works as a heuristic, but it has a problem: it treats cancellations as events where some volume gets cancelled ahead of you and some behind, but in reality, each cancellation is either fully ahead of you or fully behind you. This means it also only gives you a point estimate of where in the queue you are, with no measure of uncertainty.

We do something more powerful. We maintain a distribution over all possible queue positions using a particle filter, which gives us a whole distribution of where we likely are in the queue.

We derive the framework from scratch, implement it live on Bybit, and show the particle distribution evolving live in real time.

I write about quantitative trading the way it’s actually practised:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why market-by-price data makes queue position a latent variable, and why the naive proportional heuristic isn’t enough.

• How to model queue position as a distribution rather than a point estimate, and why that matters for execution and market making.

• How to derive a particle filter for queue position from scratch, handling trades, cancellations, and fills correctly.

• How to implement it live with Bybit’s WebSocket feed in Python.

• What the queue position distribution actually looks like in real life, and how it evolves as the market moves.

Read more

The models we used to forecast returns were themself not “online” though. They were trained on a warm-up set and then kept frozen. The first improvement that comes to mind is simply retraining them periodically, but it turns out there is a more powerful method that lets you learn every single timestep and not miss out on crucial shifts.

That’s what we’ll be implementing in this article.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn

• Why rolling window retraining is fundamentally limited and what assumptions it silently makes about your data.

• How AROWR works: a second-order Bayesian online regression algorithm that maintains a full posterior over model weights and updates every single timestep.

• How ARCOR extends AROWR with a principled covariance reset mechanism that detects when the market has changed and restores the model's ability to adapt.

• How to implement AROWR and ARCOR from scratch in Python and apply them to any regression problem that updates in real time.

• How we applied ARCOR to BTC beta estimation across 355 crypto assets over 4 years, significantly improving performance over rolling OLS, especially during LUNA, FTX, and similar situations.

Read more

Maybe one is a linear autoregressive model that performs well in trending markets. Another is a mean-reversion model, and a third is a more complex machine learning model that captures nonlinear patterns. Each has its strengths and weaknesses, and most importantly, you don’t know in advance which one will be best tomorrow.

The naive approach is to pick the best model in-sample and deploy it. Another naive approach is equal weighting. But those two approaches ignore everything you learn as new data arrives.

What if there exists a principled way to combine your models that:

• Allocates a lot of weight to models that currently perform well and less weight to models that perform poorly.

• Requires no retraining, no hyperparameter tuning, no rebalancing decisions.

• Comes with a mathematical guarantee that you perform nearly as well as the best model.

In this article, we present an algorithm that is able to achieve all of the above, demonstrate its behavior, and show how this model shines when there are different regimes where different models perform best.

I write about quantitative trading the way it’s actually practiced:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

Topics I write about include portfolio construction, market making, risk management, research methodology, and more.

If this way of thinking resonates, you’ll probably like what I publish.

What you’ll learn:

• The theoretical framework of online learning and how it applies to forecast combination.

• What regret is, why minimizing it is the right objective, and how it differs from standard in-sample loss minimization.

• Why square loss has a special property called 2-mixability that enables a provably optimal combination algorithm.

• How the aggregating forecaster works, why it is minimax optimal, and how to implement it from scratch in Python.

• How to extend the algorithm with polynomial discounting to handle non-stationary markets with regime changes.

• Why the optimal discount rate depends on regime persistence, and how to tune it empirically.

Read more

What is the point of such a measure?

Let’s say you are testing a moving-average crossover strategy, and you test 20 different combinations of moving-average lookbacks. Even for different parameter settings, the strategy returns will be correlated, so you didn’t really test 20 different strategies.

Let’s now dive into ways of measuring this and why it’s incredibly useful.

I write about quantitative trading the way it’s actually practiced:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

What you’ll learn:

• Why the raw number of tested strategies dramatically overstates overfitting risk.

• Why dependence geometry determines multiple-testing complexity.

• How expected chi-squared maxima define a continuous effective number of tests.

• How to compute K_eff numerically.

The 5 Axioms

If I think of a measure of how many strategies were effectively tested, I think of a few intuitive axioms that such a measure should follow:

• 1 <= K_eff <= K: You can’t test fewer than one strategy, and you can’t test more strategies than you’ve actually tested.

• K_eff = 1 iff all strategies are perfectly correlated/anticorrelated: If all your strategies are perfectly correlated/anticorrelated, then you’ve effectively only tested a single strategy.

• K_eff = K iff all strategies are uncorrelated: If all your strategies are perfectly uncorrelated (or independent, which is the same in the Gaussian world), then you can’t get more information; you really tested as many strategies as you got.

• K_eff is non-decreasing in K: If we test more strategies, our effective number of strategies that we tested can’t decrease; it should only increase, or stay constant if the new strategy is perfectly correlated/anticorrelated to a previously tested strategy.

• The expectancy of the maximum of the absolute Z-scores of your K strategies grows asymptotically like the square root of twice the logarithm of K_eff: Okay, this one may not be so intuitive… Let me explain! Let Z_i be the Z-score of strategy i. We assume the Z’s to be standard Gaussian random variables with covariance matrix Sigma. A standard result in extreme value theory is that:

  We want K_eff to be defined so that your correlated portfolio of K strategies looks exactly like K_eff independent strategies in terms of expected best absolute Z-score.

Why Popular Measures Fail

I’m not the first one to come up with a notion of “effective number of strategies tested”, so let’s look at some of the existing definitions and where they ultimately break down.

Spectral Participation Rate

Spectral Participation Rate is the measure that was used in the following article:

It’s defined as follows:

where λ_1, …, λ_K are the eigenvalues of Sigma, which is the correlation matrix of our Z-scores.

By construction, K_eff = K under independence and decreases toward 1 as configurations become collinear. And there you have your problem, K_eff decreases toward 1 as configurations become collinear. Let’s say you have strategies A and B, which are completely independent, so K_eff = 2. Now imagine you were to test strategy B over and over again. You would want K_eff to stay 2, but something else happens.

If you keep adding copies of strategy B, your correlation matrix becomes:

The eigenvalues of Sigma are:

So:

And as K goes to infinity, K_eff converges towards 1!

While this is an extreme example, it happens in reality with adaptive search as well. In Bayesian optimization, you explore parameter areas that previously gave you good results more thoroughly, which in turn gives you a highly correlated strategy and causes K_eff to decrease.

Axiom 4 is clearly violated, and thus, this is not usable for us.

Effective Rank

Effective rank is defined in Roy & Vetterli (2007), and measures effective dimensionality by providing a real-valued extension to the rank of a matrix.

Consider the singular value decomposition (SVD) of Sigma:

where U and V are unitary matrices, and D is a diagonal matrix containing the singular values

Let the singular value distribution be

where sigma is the vector of singular values and ||.||_1 the l1-norm.

The effective rank of the matrix Sigma is defined as

where H is the Shannon entropy given by

We will use this effective rank as our K_eff.

Consider again the scenario of independent strategies A and B, and testing strategy B over and over again. Since Sigma is SPD, the eigenvalues and singular values are the same:

The l1 norm is:

And the singular value distribution is:

With this, erank(Sigma), or K_eff, becomes

This suffers from the same problem as the spectral participation rate! The problem is that those two measures only look at the shape of the spectrum, not its scale. Adding more copies of B drowns out the contribution of A in the normalized distribution, pulling K_eff towards 1.

Bailey & López de Prado

In the paper "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality", Bailey and López de Prado present another method of measuring the effective number of strategies tested.

They define

where the equal-weighted average pairwise correlation is

Here is an example where this measure also breaks down:

Let A and B be two perfectly anti-correlated strategies. Then

and the average pairwise correlation is -1.

We get

With this definition, our effective number of strategies tested is 5, even though we only tested 2 strategies. It also violates a bunch of other axioms.

A Definition That Works

Here is what I propose. Let M(x) be the expected maximum of x i.i.d. squared standard normal random variables:

Equivalently, since squaring a standard normal random variable yields a chi-squared random variable, this is the expected maximum of x i.i.d. chi-squared random variables. Using the PDF and CDF, we write the expected maximum as:

M is strictly increasing and continuous, so it has a well-defined inverse

We then define the effective number of tested strategies as:

where Z_1, …, Z_K are the actual strategy Z-scores with correlation matrix Sigma. In other words, K_eff is the number of independent strategies that would produce the same expected best squared Z-score as your actual correlated portfolio of K strategies.

Now, let’s prove that this definition actually follows the 5 axioms.

Axiom 1

Since M is strictly increasing, it suffices to show that

Lower bound:

Upper bound:

By Šidák’s inequality for centered Gaussian vectors,

Since each Z_i is standard normal, we have

and hence

where st means stochastically dominated. And because x → x^2 is increasing,

Taking expectations gives

Axiom 2

If all strategies are perfectly correlated or anticorrelated, then for every i,j,

Since each Z_i is standard Gaussian, this implies there exists a single standard normal random variable X and signs epsilon_i, either -1 or 1, such that

Therefore,

so

Taking expectations,

Applying the inverse of M,

Now the other direction. Assume K_eff = 1. Since M(1) = 1,

Since each Z_i is standard Gaussian, we have E[Z_i^2] = 1.

Since

the equality

can occur only if

Thus

For jointly Gaussian variables, this implies

and therefore

Axiom 3

If all strategies are uncorrelated, we have

which implies that

Therefore,

Applying the inverse of M, we get

Now the other direction. Assume K_eff = K. Since M is strictly increasing,

Now let

From the proof of Axiom 1, we know that

i.e.

Using the tail-integral formula,

The integrand is nonnegative everywhere, and the left-hand side is zero since E[X] = E[Y]. Hence

Thus, X and Y have the same distribution. Equivalently,

By the equality case of Šidák’s inequality, this implies that Z_1, …, Z_K are independent, which for Gaussian random variables is equivalent to being uncorrelated.

Axiom 4:

We have

Taking expectation

and applying the inverse of M, we get

Axiom 5:

Let

Then

A standard result from Gaussian extreme-value theory states that

Moreover,

is uniformly integrable. Hence, convergence in probability implies convergence of expectations:

Hence

Substituting k = K_eff yields

Since

we obtain

Since Gaussian maxima concentrate sharply, the first and second moments are asymptotically equivalent up to square-root scaling. Therefore,

Simulation

Below is the code used for computing K_eff:

import numpy as np
from scipy.stats import chi2
from scipy.optimize import brentq


def M(x: float, n_quad: int = 4000) -> float:
    """
    Compute M(x) = E[max of x i.i.d. chi^2_1 variables] via numerical
    quadrature, working in log space to avoid underflow.

    Parameters
    ----------
    x       : effective count (any real >= 1)
    n_quad  : number of quadrature points

    Returns
    -------
    float : E[max_i Z_i^2] for x i.i.d. N(0,1) variables
    """
    # Adaptive upper limit: tail of chi^2_1 maximum decays beyond this
    hi = chi2.ppf(1 - 1e-10 / x, df=1) if x > 1 else chi2.ppf(1 - 1e-8, df=1)
    t  = np.linspace(0, hi, n_quad + 1)[1:]  # avoid t=0 where pdf diverges

    log_F   = chi2.logcdf(t, df=1)
    log_f   = chi2.logpdf(t, df=1)
    log_ker = np.log(t) + np.log(x) + log_f + (x - 1) * log_F

    # Zero out terms that underflow (negligible contribution)
    mask       = log_ker > -700
    integrand  = np.zeros_like(t)
    integrand[mask] = np.exp(log_ker[mask])

    return float(np.trapezoid(integrand, t))


def M_inv(val: float, tol: float = 1e-8) -> float:
    """
    Compute M^{-1}(val) via Brent's method.

    Parameters
    ----------
    val : target value, must be >= M(1) = 1
    tol : tolerance for root finding

    Returns
    -------
    float : x such that M(x) = val
    """
    if val <= 1.0:
        return 1.0

    # Bracket: expand upper bound until M(hi) > val
    hi = 2.0
    while M(hi) < val:
        hi *= 2.0

    return brentq(lambda x: M(x) - val, 1.0, hi, xtol=tol)


def K_eff(Sigma: np.ndarray, n_sim: int = 200_000, seed: int = 42) -> float:
    """
    Compute K_eff for a portfolio of K strategies with correlation matrix Sigma.

    Uses Monte Carlo to estimate E[max_i Z_i^2], then inverts M analytically.

    Parameters
    ----------
    Sigma : (K, K) correlation matrix
    n_sim : number of Monte Carlo samples
    seed  : random seed

    Returns
    -------
    float : K_eff in [1, K]
    """
    K   = Sigma.shape[0]
    rng = np.random.default_rng(seed)
    L   = np.linalg.cholesky(Sigma)
    Z   = (L @ rng.standard_normal((K, n_sim)))  # (K, n_sim)
    E_max_Z2 = float(np.mean(np.max(Z**2, axis=0)))
    return M_inv(E_max_Z2)

And now let’s simulate how K_eff evolves as K grows for different families of strategies:

import matplotlib.pyplot as plt


# ============================================================
# Helper: build equicorrelation matrix
# ============================================================

def equicorr_matrix(K, rho):
    """
    Build K x K equicorrelation matrix with off-diagonal rho.
    """

    Sigma = np.full((K, K), rho, dtype=float)
    np.fill_diagonal(Sigma, 1.0)

    return Sigma


# ============================================================
# Sequential simulation
# ============================================================

def sequential_keff(mode, K_max=100):

    K_vals    = []
    Keff_vals = []

    # Persistent storage for sequential random strategies
    random_vectors = []

    rng = np.random.default_rng(123)

    for K in range(1, K_max + 1):

        # ----------------------------------------------------
        # Independent
        # ----------------------------------------------------
        if mode == "independent":

            Sigma = np.eye(K, dtype=float)

        # ----------------------------------------------------
        # Perfect dependence
        # ----------------------------------------------------
        elif mode == "dependent":

            Sigma = np.ones((K, K), dtype=float)

        # ----------------------------------------------------
        # rho = 0.5
        # ----------------------------------------------------
        elif mode == "rho_0.5":

            Sigma = equicorr_matrix(K, 0.5)

        # ----------------------------------------------------
        # Maximally negative equicorrelation
        # ----------------------------------------------------
        elif mode == "negative_boundary":

            if K == 1:
                rho = 0.0
            else:
                # PSD boundary:
                # rho >= -1/(K-1)
                rho = -0.9 / (K - 1)

            Sigma = equicorr_matrix(K, rho)

        # ----------------------------------------------------
        # Sequential random correlated strategies
        # ----------------------------------------------------
        elif mode == "random":

            # Add ONE new latent-factor strategy
            v = rng.standard_normal(5)

            # Normalize
            v /= np.linalg.norm(v)

            random_vectors.append(v)

            X = np.stack(random_vectors)

            # Correlation matrix
            Sigma = X @ X.T

        else:
            raise ValueError(f"Unknown mode: {mode}")

        # ----------------------------------------------------
        # Numerical stabilization
        # ----------------------------------------------------

        Sigma = Sigma.astype(float)

        Sigma += 1e-10 * np.eye(K)

        # ----------------------------------------------------
        # Compute K_eff
        # ----------------------------------------------------

        keff = K_eff(Sigma)

        K_vals.append(K)
        Keff_vals.append(keff)

        print(
            f"{mode:20s} | "
            f"K = {K:3d} | "
            f"K_eff = {keff:8.3f}"
        )

    return np.array(K_vals), np.array(Keff_vals)


# ============================================================
# Run all simulations
# ============================================================

modes = [
    "independent",
    "dependent",
    "rho_0.5",
    "negative_boundary",
    "random"
]

results = {}

for mode in modes:

    K, Keff = sequential_keff(mode, K_max=100)

    results[mode] = (K, Keff)


# ============================================================
# Plot
# ============================================================

plt.figure(figsize=(12, 8))

for mode in modes:

    K, Keff = results[mode]

    plt.plot(K, Keff, linewidth=2, label=mode)

# Reference line
plt.plot(
    np.arange(1, 101),
    np.arange(1, 101),
    linestyle="--",
    linewidth=2,
    label="K_eff = K"
)

plt.xlabel("Raw Number of Tested Strategies (K)")
plt.ylabel("Effective Number of Strategies (K_eff)")

plt.title(
    "Sequential Growth of K_eff Under Different Correlation Structures"
)

plt.legend()
plt.grid(True)

plt.show()

Conclusion

K_eff is not just theoretical; it has immediate practical applications.

The most direct one is improving the Deflated Sharpe Ratio presented by Bailey and De Prado. More broadly, K_eff can be used anywhere you need to correct for multiple testing across correlated strategies.

It can also directly guide you in determining when to stop searching while tuning hyperparameters.

I will explore those and many other things K_eff is capable of, and the math behind K_eff in more detail in future articles, which I then plan to all combine into a research paper!

Join Quant Corner: https://discord.gg/X7TsxKNbXg

Quick reminder that I’m now open for consulting and short-term engagements. If your team needs help with anything in the quant space, like strategy research, portfolio construction, execution analysis, signal development, or just a second brain on a hard problem, I'm available. You can DM me on Discord or email me at vertoxquant@gmail.com.

I have some news to share. As of today, I'm going fully independent and dedicating all of my time to VertoxQuant.

For the past few years, I've been splitting my hours between my work as a quant researcher and the newsletter. That chapter is now closed, which means every hour I would have spent on industry work is now going straight into research, writing, and building things here.

In practice, that means more articles, deeper dives, and more time to explore the corners of quant finance I actually find interesting, not just the stuff that's immediately commercially useful.

For those newer here: my background is in quantitative research and systematic trading, with experience across strategy research, portfolio construction, execution analysis, and signal development. I also hold a degree in mathematics, though I tend to approach problems very practically while maintaining the right level of mathematical rigor. VertoxQuant started as a place to share ideas and research publicly, and it has grown into a community of people who care deeply about quantitative finance.

I'm also now open for consulting and short-term engagements. If your team needs help with anything in the quant space, like strategy research, portfolio construction, execution analysis, signal development, or just a second brain on a hard problem, I'm available. You can DM me on Discord or email me at vertoxquant@gmail.com.

I’m incredibly excited for what comes next.

-VertoxQuant

Rather than trying to fix a method, you can also go down the route of using specially designed portfolio optimization techniques that don’t suffer from instability as much. Hierarchical Risk Parity (HRP) is one of the most popular such techniques, replacing the covariance inverse with a recursive clustering procedure that is more robust by construction. But even HRP has its limitations; It’s designed to be long-only, and it takes no signal input whatsoever. It’s a pure risk allocation method. Any alpha you’ve worked so hard to generate is completely ignored by it.

This matters enormously for stat-arb and long-short strategies, where both the signal and the risk structure are crucial. What you want is a portfolio construction method that uses market structure to shape position sizes while preserving signal direction. That is exactly what Topological Risk Parity (TRP), introduced by Nayar, Ainasse, and Kulkarni (2026), is designed to do.

In this article, we implement TRP from scratch in Python on a crypto universe. We also implement the Semi-Supervised variant of TRP, which allows you to impose an economic prior on the hierarchy, like forcing BTC to be the market root and major L1s as the second layer. We then apply Random Matrix Theory (RMT) to further improve TRP across every performance metric!

What you’ll learn:

• What Topological Risk Parity is and why it outperforms HRP for long-short strategies

• How to build a Minimum Spanning Tree from a correlation matrix and what it reveals about market structure

• How Random Matrix Theory separates genuine economic signals from statistical noise in correlation matrices

• How to denoise a correlation matrix using Ledoit-Wolf shrinkage and hard thresholding

• How to implement TRP from scratch in Python, including the Semi-Supervised variant with economic anchoring

I write about quantitative trading the way it’s actually practiced:

Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Mapping the Market with Correlation Networks

Read more

One of your first thoughts when looking at a stranger’s backtest is probably that it’s overfit, or that there is some look-ahead somewhere.

When you go a step further, you are probably constantly worried about overfitting your own backtests too!

In this article, we will introduce a framework that allows you to identify both! It’s a two-stage approach introduced in D. Nikolopoulos (2026). We will introduce the method, develop the two-stage approach, test the methodology empirically on a crypto dataset, look at some limitations, and also add some improvements of my own!

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

How Backtests Lie

Read more

Everyone talks about alpha decay, but no one tells you how to actually quantify it. In this article, we’ll figure out how to do so and build a warning system that tells you when it’s time to pull the plug on a strategy.

We do this by implementing Minimum Regime Performance (MRP), a framework introduced by Alexander and Fabozzi (2026) that measures how a strategy holds up across structurally distinct market regimes. We will apply this method to a universe of crypto factors.

Beyond signal monitoring, MRP is also relevant in portfolio construction. Just as you would punish strategy tail risk like Value at Risk, you can also punish strategies that have a high tendency to decay.

The article will use the factors discussed in the following article:

VertoxQuant

The Myth of Factor-Free Crypto

People keep telling you to take what works in equities and apply it to crypto…

Read more

8 months ago · 9 likes · Vertox

Knowledge of regime switching models is NOT required.

I write about quantitative trading the way it’s actually practiced:
Robust models and portfolios, combining signals and strategies, understanding the assumptions behind your models.

More broadly, I write about:

• Statistical and cross-sectional arbitrage

• Managing multiple strategies and signals

• Risk and capital allocation

• Research tooling and methodology

• In-depth model assumptions and derivations

If this way of thinking resonates, you’ll probably like what I publish.

Read more

• 1 referral → 20% off

• 5 referrals → 1 month free

• 25 referrals → 6 months free

If you’ve gotten value from it, help me grow by recommending it. It costs you nothing!

You can find your personal referral link at the bottom of this email or right here:

Leaderboard

Thanks!

-Vertox