Backtest Truth, Part 4: The Year That Looks Perfect
One rule made about +22% in the last year, then lost about 27% over 2 years. A stronger setup was about +77% on 2020–2024 data, with a 65% drawdown.
On this page
This is education only and not financial advice. Trading involves risk, and you can lose money.
“In-sample” is the period you used to build or choose a strategy. “Out-of-sample” is fresh data you did not tune on. If a rule only works on the dates you already studied, it may be fitted to the past, not ready for the future. Part 4 is that lesson, with the windows these tests actually printed.
Why the first good window feels like proof
This is easy to miss, because the first good window feels like proof. You scroll a chart, the last year looks kind, and the brain treats that year as the whole market. A backtest that only reports that year is telling the truth about that year. It is not telling the truth about the years beside it.
One 2% Renko flip rule made about +22% in the last year, with a 28% drawdown. That looked useful. Over 2 years it lost about 27%, with a 51% drawdown. The year before the “good” year was a deep loss. The short window was not a lie. It was incomplete. A 28% drawdown inside a +22% year is already a hard path. Stretch the window to 2 years and the ending number flips from a gain of about 22% to a loss of about 27%, with a 51% drawdown. The extra year was not a footnote. It was the year that changed the story.
A similar ATR Renko rule made about +25% in 1 year, then lost about 10% over 2 years. ATR is a common way of measuring how wide price has been moving. You do not need the formula to see the pattern. One year was about +25%. Two years was about a 10% loss. Same family of idea, longer window, the headline changed sign.
A stronger setup, then older data
The same lesson showed up on a stronger setup. A 1-hour pullback rule, taken only in the direction of a 4-hour trend and the forming weekly candle, looked excellent on recent data: about +90% in 1 year and +245% in 2 years, with drawdowns around 24–25%. Those are the same recent figures Part 2 used when it talked about fees at 0.07% per side. Recent strength is real as a description of that window. It is not a certificate.
Then the same rules were run on older data, 2020 to 2024, which had not been used to pick the idea. That is the out-of-sample check: leave the rules unchanged, and ask a period you did not shop inside. It was still profitable, about +77%, but the drawdown was 65%. It lost about 19% in 2021 and about 38% in 2022. At a slightly higher fee, that older-data result shrank to about +8%.
Sit with the years, not only the total. About +77% across 2020 to 2024 can hide a year of about −19% and a year of about −38%. A 65% drawdown means the path down was deep even though the period ended in profit. At the slightly higher fee, about +8% is a thin ending for a path that included those losing years. Part 2’s point and this part’s point meet: cost and the choice of years can both shrink a story that looked excellent up close.
What Bitcoin’s mixed years mean for the test
Recent strength and long-run safety are not the same thing. Bitcoin had very different years inside 2020–2024, including a fierce bear market. A test that never leaves a friendly window cannot show you that. A bear market is a long period when prices fall and rallies fail. If your test window never includes one, you have not seen how the rules behave when the background wind is against them.
The practical habit is simple. Choose rules on one period. Then leave them unchanged and test them on data you have not used. Look at each year, not only the final total. If the plan needs one special window to look good, it is not yet a plan. It is a memory of that window.
For an OFW looking at a screenshot in a group chat, this is the question to ask before any talk of copying a method: which years are in the picture, and which years were used to choose the rules? If the answer is “the last year, and we liked it, so we kept it,” you are looking at in-sample comfort. You have not yet seen the out-of-sample years.
Key takeaways
- A strong result on the period you tuned is not enough. Test again on untouched data.
- One good year can hide a bad year just beside it.
- Even a profitable out-of-sample test can have deep losing years and a much larger drawdown.
FAQ
What is the difference between in-sample and out-of-sample?
In-sample is the period you used to build or choose a strategy. Out-of-sample is fresh data you did not tune on. A rule that only works on the dates you already studied may be fitted to the past.
How did the 2% Renko flip rule change when the window got longer?
It made about +22% in the last year, with a 28% drawdown. Over 2 years it lost about 27%, with a 51% drawdown. The year before the good year was a deep loss. A similar ATR Renko rule made about +25% in 1 year, then lost about 10% over 2 years.
What did the pullback rule do on 2020 to 2024 data?
On recent data it was about +90% in 1 year and +245% in 2 years, with drawdowns around 24–25%. On 2020 to 2024 data, which was not used to pick the idea, it was still about +77%, but the drawdown was 65%. It lost about 19% in 2021 and about 38% in 2022. At a slightly higher fee that older result shrank to about +8%.
Does a profitable out-of-sample test mean the hard part is over?
No. The 2020 to 2024 run was still profitable and still had a 65% drawdown, plus losing years. Look at each year, not only the final total.
This is education only and not financial advice. Trading involves risk, and you can lose money.