Myth: A Good Backtest Means a Good System

Every trading forum has one: the screenshot of a backtested equity curve rising in a clean 45-degree line, captioned “5 years, 74% win rate”. The myth is not that backtesting is useless — it is that a good-looking backtest is evidence of a good system. Most of the time it is evidence of something much cheaper: selection.

The arithmetic of finding fool’s gold

Run this thought experiment. Generate 200 strategies that are pure coin flips — no logic, random entries, random exits — and backtest each over 250 trades. The average one wins 50% of the time, obviously. But you are not going to show anyone the average one. You are going to show the best one.

The spread of pure luck over 250 trades is about ±3.2 percentage points (one standard deviation). The best of 200 random tries will typically sit almost three standard deviations above fair — a win rate near 59%, produced by nothing at all. Now add the usual practice of tuning parameters — test 20 stop sizes, 20 lookback periods, 3 sessions — and a single “strategy” quietly becomes those same hundreds of tries. The optimizer is not discovering your edge. It is discovering your noise, one parameter at a time.

A backtest cannot tell you how good the idea is until you tell it how many ideas you threw away first.

Four leaks that flatter every backtest

1. Overfitting. The more parameters a system has, the more perfectly it can memorise the past — and the past does not repeat at parameter precision. A rule of thumb that has aged well: every added parameter should have a reason from market mechanics, not from the optimizer. If the stop is 7 ticks because 7 outperformed 6 and 8, the stop is decoration.

2. Costs and slippage. Backtests fill you at the print; markets do not. On a scalping system, one tick of slippage per side on MES is $2.50 per round trip — across 500 trades, that is $1,250 removed from the equity curve before reality has even started charging for news spikes and thin books. Many published curves die from this line item alone.

3. Look-ahead and survivorship. Using today’s settled data to “decide” yesterday’s trade, testing only on instruments that are still alive and liquid, starting the test conveniently after 2022’s drawdown — none of these feel like cheating while you do them. All of them are.

4. Regime dependence. A system tuned on two years of trending markets has never met a balance. It is not robust; it is specialised — and it does not know which regime it is specialised for. This is why curves break precisely when the market character changes, which is precisely when you need them most.

What a backtest is actually for

Used honestly, a backtest is a falsification tool, not a promise machine. It can tell you an idea is bad; it can never certify that an idea is good. The practical checklist:

• Fewer parameters, more mechanics. An entry that exists because real size defends a level needs no optimizer to justify it.
• Hold out data. Tune on one period, verify untouched on another (walk-forward). The out-of-sample result is the only one that counts.
• Stress the costs. Double your assumed slippage and commissions. A real edge survives; a curve-fit one dies instantly.
• Count your tries. If the winning variant came from 100 attempts, judge it as the best of 100 random draws — because in part, it is.

This is also why the Conflux approach starts from reasons — who is positioned where, what volume actually executed, what the options market has priced — rather than from patterns an optimizer can memorise. Evidence that has a mechanism does not need 200 tries to look good.