Train/test split & out-of-sample

The simplest, strongest defence: cut history in two, tune only on the first half, and grade the strategy exactly once on the second — the one honest look you get.

If overfitting is the disease, the train/test split is the first and cheapest cure. The idea is borrowed straight from machine learning and it is almost embarrassingly simple: divide your history into two chunks. Do all your thinking, tuning, and experimenting on the first chunk. Then, at the very end, run the finished strategy *once* on the second chunk to get an honest estimate of how it behaves on data it has never seen.

The split. You optimise freely on Train (in-sample). Test (out-of-sample) is sealed — touched exactly once, at the end, to grade the result. Its whole value is that the strategy never got to fit it.

Why one look, and only one

Here is the subtle, crucial point that people get wrong. The test set is only honest *while it is unseen*. The moment you peek — "hmm, it did badly on the test, let me adjust the stop and try again" — you have used the test data to make a decision, and it has silently become part of your training data. Do that five times and your "out-of-sample" result is just a slower, sneakier form of overfitting.

What a fair split looks like in code

pythonFit on train, judge only on test

import pandas as pd

df = load_prices()                       class="c"># a long price history, time-ordered

class="c"># class="n">1. Split by TIME, never randomly — markets have order and autocorrelation.
cut   = int(len(df) * class="n">0.70)
train = df.iloc[:cut].copy()             class="c"># do ALL tuning here
test  = df.iloc[cut:].copy()             class="c"># sealed until the very end

class="c"># class="n">2. Optimise the parameter ONLY on train.
def sharpe_of(data, lookback):
    ma     = data[class="s">'close'].rolling(lookback).mean()
    signal = (data[class="s">'close'] > ma).astype(int).shift(class="n">1)   class="c"># lagged: no look-ahead
    strat  = signal * data[class="s">'close'].pct_change()
    return strat.mean() / (strat.std() + 1e-class="n">9) * (class="n">252 ** class="n">0.5)

best_lb = max(range(class="n">10, class="n">200), key=lambda lb: sharpe_of(train, lb))
print(fclass="s">"Chosen on TRAIN: lookback={best_lb}, sharpe={sharpe_of(train, best_lb):.2f}")

class="c"># class="n">3. ONE honest evaluation on test, using the parameter chosen on train.
oos = sharpe_of(test, best_lb)
print(fclass="s">"Out-of-sample TEST sharpe: {oos:.2f}")

class="c"># Decision rule: if the test Sharpe is a small fraction of the train Sharpe,
class="c"># you overfit the train set. A robust edge holds up (roughly) out of sample.

Notice two disciplines baked in. First, the split is by time, not random rows — shuffling would let the strategy learn from future days interleaved with past ones (a look-ahead leak, and it destroys the natural autocorrelation of markets). Second, best_lb is chosen using only train, then frozen before test is ever touched.

Reading the result honestly

Test ≈ Train: the best outcome. Your edge generalises. This is what you are hunting for.
Test << Train: classic overfitting. The train result was luck you sculpted. Simplify the strategy (fewer parameters) and try again on fresh data.
Test > Train: be suspicious, not delighted. Usually it means the test period simply happened to suit the strategy — do not bank on it. It is a smaller sample; treat it as "not falsified" rather than "proven".
Both mediocre: at least it is honest. A real, small edge beats a fake, large one every time.

The train/test split gives you one clean snapshot of out-of-sample behaviour. But a single split wastes most of your history and depends on where you happened to cut. The next lesson generalises it into the technique the professionals actually trust: walk-forward analysis.

← Previous

Look-ahead & survivorship bias

Walk-forward analysis