Sample Size: Is 7 Wins out of 8 Reliable?

Too few trades, so luck calls the shots

Flip a fair coin 8 times and getting 6 heads is nothing unusual. When a backtest has only a dozen or so trades, the win rate can likewise be pushed up or down by luck: with too small a sample size, the conclusion doesn't hold up.

Put a range on the win rate

Instead of looking at a single win-rate number, ask "roughly what range is the true win rate in?" This confidence interval narrows as the number of trades grows: 7 wins out of 8 gives a very wide range, with a lower bound only a bit above 50%.

The more you try, the easier it is to hit a good-looking result

Try dozens of rules on the same data and one will always happen to look amazing. That's data snooping. To trust a result you need enough trades, ideally tested on fresh data you've never used.

Key takeaways

• A backtest's win rate is just one sample. The fewer the trades, the bigger the role of chance.
• 7 wins out of 8 (87.5%) gives a 95% confidence interval for the true win rate of about 53%–98%, with the lower end only just above 50%.
• The more trades, the narrower the interval: the same 60% win rate is far more credible over 100 trades than over 20.
• Even a good-looking interval only describes that past data. You still need to consider costs, changing markets, and the bias from trying many settings.

Try it yourself

You can do these exercises in this lesson's interactive simulation:

1. Click "Example: 7 wins out of 8": what's the lower bound of "My win-rate range"? Then look at the histogram: when the true win rate is only 50%, what share of backtests count as "Looks great" (win rate ≥ 60%)?
2. Keep the true win rate at 50% and drag the number of trades from 8 to 200: does the histogram narrow? How does the share that "Looks great" change?
3. Set the true win rate to 55% and the number of trades to 30: how many red bars (observed win rate below 50%) are still in the histogram? A strategy with a real edge can still look like a loser in a backtest.
4. Set the trades to 20 and drag "Wins": how many wins do you need before the lower bound of the range is above 50%? Try again with 100 trades.

Quiz and explanations

1. A strategy with a true 50% win rate and 1:1 payoff (no edge) is backtested over only 10 trades. How likely is it to show a 70% win rate?

Answer: Not unusual: about 1 in 6 backtests would show 70% or more.The chance of winning 7 or more out of 10 is about 17%, so luck alone can produce a great-looking record.

2. All else equal, if you increase the number of backtest trades from 20 to 200, the win rate's confidence interval will…

Answer: Narrow.The bigger the sample, the more precise the estimate and the narrower the interval.

3. Two strategies both show a 60% win rate: A from 15 trades, B from 300 trades. Which conclusion is more credible?

Answer: B, because the larger sample means less chance involved.For 300 trades, the 60% interval is about 54%–65%; for 15 trades, it's wide enough to include values below 50%.

Terms in this lesson

• confidence interval (Confidence Interval): A range estimated from a sample that very likely contains the true value; the smaller the sample, the wider the range.
• data snooping (Data Snooping): Trying lots of rules or parameters on the same historical data will always turn up a set that looks great by chance, but it's usually just luck.
• expectancy (Expectancy): What you make per trade on average over the long run: win rate × average win − loss rate × average loss; only worth trading if positive.
• sample size (Sample Size): How many data points a statistic is based on, such as the number of trades in a backtest; the fewer the trades, the more luck drives the win rate and returns.
• win rate (Win Rate): The share of all trades that made money.