Quant Basics· Lesson 5 of 8· 6 min

Out-of-Sample Testing: Don't Get Fooled by Backtests

Picking the best settings on historical data is easy. The hard part is whether they still work on data they've never seen. Walk-forward testing mimics the real process of choosing settings first, then trading.

This lesson is free: sign up for an account to use the interactive simulation

Every lesson has an interactive simulation you can drag and play, plus a quick quiz. Sign up free to use them and save your progress automatically. Here is the text version of this lesson.

Not sure yet? Try the first lesson (no sign-up needed)「Building Strategies from Rule Blocks」。

Text version · Quant Basics, lesson 5: “Out-of-Sample Testing: Don't Get Fooled by Backtests”

Pick parameters first, then play

In real trading you can only pick parameters using past data. walk-forward analysis simulates this: pick the best parameters in-sample, test them on the out-of-sample data that comes right after, then roll the whole thing forward.

The fluff in a backtest

The “best parameters” picked using all the data are like doing a test with the answer key. The gap between that and out-of-sample performance is the fluff in the backtest. Being able to pick great-looking parameters even in pure noise is the most direct evidence of overfitting.

Look for plateaus, not spikes

Around a reliable parameter, nearby parameters do well too. If one parameter is especially good and its neighbors are all bad, it's most likely luck and will stop working in a different market.

Key takeaways

Pick settings in-sample and test out-of-sample; only out-of-sample performance is close to the real thing.
Walk-forward: each time, choose settings on a past stretch of data, test them on the next stretch, then roll the whole window forward.
Great in-sample and poor out-of-sample usually means the strategy is fitting noise.
Settings near a good setting should also do well. If only one "spike" setting stands out, it's probably a coincidence.

Try it yourself

You can do these exercises in this lesson's interactive simulation:

In the “Style rotation” market: what's the best full-sample return? And the out-of-sample total?
Switch to “Pure noise”: what's the average in-sample return per window? And the out-of-sample total?
Go back to “Style rotation” and change the training window from 120 bars to 200: are there fewer parameter changes? Is the out-of-sample total better or worse?
Switch to “Long-term trend”: is the out-of-sample total still positive? How far is it from “Best on full sample”?

Quiz and explanations

1. A setting returns 40% a year in-sample and −5% a year out-of-sample. What does this most likely mean?: Answer: It overfit historical noise.Doing well only on data it has "seen" is the classic sign of overfitting.
2. How does walk-forward testing work?: Answer: Choose settings on past data, test on the next stretch, then roll forward.It mimics the real situation of only being able to decide with past information.
3. You tested 200 setting combinations and picked the best backtest, at 35% a year. What's the most sensible next step?: Answer: Test it on out-of-sample data that wasn't used in the selection.The more combinations you try, the better the chance of picking one that "happens to look good", so you must test on data it hasn't seen.

Terms in this lesson

in-sample (In-Sample): The stretch of historical data used to tune parameters.
out-of-sample (Out-of-Sample): Data kept out of tuning and used only to test the strategy.
overfitting (Overfitting): Tuning parameters so tightly to past data that the strategy does badly on new data.
walk-forward analysis (Walk-Forward Analysis): Repeatedly picking parameters on one stretch of past data, testing them on the stretch right after, then rolling forward, to mimic real trading.