Parameter Optimization and Overfitting
Tuning settings to the historical "best" often just memorizes noise. Out-of-sample testing is the real test.
Every lesson has an interactive simulation you can drag and play, plus a quick quiz. Sign up free to use them and save your progress automatically. Here is the text version of this lesson.
Not sure yet? Try the first lesson (no sign-up needed)「Indicator Confluence」。
Text version · Indicators in Practice & Backtesting, lesson 4: “Parameter Optimization and Overfitting”
Tuning settings to the historical "best" often just memorizes noise. Out-of-sample testing is the real test.
Spikes collapse, plateaus hold
A setting that shines only at one "magic value," while its neighbors are mediocre, has usually memorized noise: overfitting. A "plateau" where a whole area of settings does well is more reliable.
Keep some data back as the exam
Pick settings on in-sample data, then test them on out-of-sample data you've never seen. Only if they pass the exam too are they worth researching further.
The more you try, the more careful you need to be
Try hundreds of settings on the same data and the best one is very likely just the luckiest. That's data snooping. Tune as little as possible, and prefer settings whose logic makes sense and whose neighbors also do well.
Key takeaways
- Overfitting: settings fit historical data perfectly but perform badly on new data.
- Optimize on one stretch of data (in-sample) and test on another stretch you've never used (out-of-sample).
- A robust strategy works well across a whole region of nearby settings (a plateau), not only at one "magic number" (a spike).
- The more setting combinations you try, the easier it is to stumble onto a good-looking result by chance, and the more important out-of-sample testing becomes.
Try it yourself
You can do these exercises in this lesson's interactive simulation:
- Start with the heat map: which settings are the in-sample "best" amber cell? What color are the cells around it?
- Click "Test out of sample": what did the spike's positive return turn into? It's the same thing as the example's "+80% in sample, −20% out of sample."
- Compare the in-sample and out-of-sample averages of the dashed cyan box (the plateau): it got worse, but did it collapse?
- Click a few cells inside the plateau, then a few next to the spike, and compare the returns of the two stretches.
Quiz and explanations
- 1. A setting returns +80% in-sample and −20% out-of-sample. What's the most likely cause?
- Answer: Overfitting.It memorized historical noise.
- 2. To judge whether a setting is robust, you should look at…
- Answer: Whether the region around it all performs well.A "plateau" is more reliable than a "spike".
- 3. You tried 500 setting combinations on the same data and picked the best one. What should you do next?
- Answer: Test it on out-of-sample data that wasn't used for optimization.The more you try, the easier it is to hit a good result by luck. Only data it has never seen can tell you whether it was luck.
Terms in this lesson
- data snooping (Data Snooping): Trying lots of rules or parameters on the same historical data will always turn up a set that looks great by chance, but it's usually just luck.
- in-sample (In-Sample): The stretch of historical data used to tune parameters.
- out-of-sample (Out-of-Sample): Data kept out of tuning and used only to test the strategy.
- overfitting (Overfitting): Tuning parameters so tightly to past data that the strategy does badly on new data.