Backtest vs forward test: overfitting explained for EA developers

The backtest is not a prediction, it is a fit

The uncomfortable truth about algorithmic trading strategy development is that the MT4/MT5 strategy tester does not tell you how a trading bot will perform. It tells you how well a set of rules describes the past you fed it. Those are only the same thing if the rules were chosen without looking at that past, and they never are. You looked. You adjusted the stop. You tried 14, 20 and 50 for the moving average and kept the one that worked. Each adjustment bent the rules a little closer to the data, and every bend that fits noise rather than signal is overfitting.

A forward test is the only clean answer, because it runs on data that did not exist when you made the decisions. Everything else in this guide is about making your backtest less of a lie and measuring how much of a lie it was.

Degrees of freedom: how much fitting you have done

Count the knobs. Every optimisable input is a degree of freedom, and every degree of freedom lets the optimiser explain a bit more of the past by accident. A rough rule from statistics: you want many times more trades than parameters, and the ratio gets worse fast. Worked example: an EA with 6 inputs, each tested at 10 values, is 1 million combinations. Run that over 300 historical trades and the best combination will look brilliant even if the underlying idea is worthless, because among a million random fits, some fit.

The same arithmetic applies to the inputs you "just set" by eye. If you tried four stop-loss values by hand before the optimiser ran, that is a fifth parameter. Be honest in the count.

• Under 3 free parameters and a few hundred trades: the backtest might mean something.
• 5 or more parameters optimised together: assume the report is mostly noise until forward tested.
• Any parameter whose neighbours perform much worse than the chosen value: the chosen value is a fit, not a feature.

Out-of-sample and walk-forward

The minimum discipline is to split the history. Optimise on the first 70%, then run the frozen forex robot on the final 30% once, and only once. If you go back and change something after seeing the out-of-sample result, the out-of-sample segment is now in-sample and you need fresh data.

Walk-forward testing is the same idea on a rolling basis: optimise on a window, test on the next slice, roll both forward, repeat. Stitch the test slices together and you get a track record made entirely of data the parameters never saw. The walk-forward efficiency, the out-of-sample profit per period divided by the in-sample profit per period, is the number to watch. Above about 50% is a durable edge; below 30% the optimiser was fitting noise. Say in-sample expectancy is $9 per trade, stitched out-of-sample expectancy $4. Efficiency is 44%. Usable, but plan your sizing around $4, not $9.

The four ways a strategy tester flatters you

Even a perfectly out-of-sample backtest runs in an environment that is kinder than a broker. Know the gaps so you can price them in before forward testing.

• Fixed spread. The tester often uses one spread for the whole run. Real spreads widen at session opens, news and rollover, which are exactly the moments a breakout EA trades. The Execution panel on a live account shows the actual spread histogram by hour in points.
• Zero slippage. Every order fills at the requested price. On MT5 live accounts the connector reports open and close slippage from history per deal; on MT4, stop-loss and take-profit closes. Signed, in points. Expect negative.
• Modelling quality. Tick data that was interpolated from minute bars can show fills that never existed, especially for scalpers with targets under 10 points.
• Survivor data. The instrument list, broker conditions and contract specs you tested are the ones that survived to today.

Measuring degradation honestly

Once the Expert Advisor is forward testing on a demo or a small real account, compare the same statistics side by side over the same window length. Not net profit, which depends on lot size; use the size-independent rows of the metric matrix from EA statistics that actually matter: expectancy in points, profit factor, payoff ratio, win rate, average holding time, and maximum drawdown as a percentage.

Worked example: backtest profit factor 1.9, forward 1.3; backtest expectancy 14 points, forward 6; win rate 58% backtest, 51% forward; average holding time identical. That pattern, similar behaviour but roughly half the edge, is classic cost-plus-mild-overfit degradation and is survivable if 6 points clears your costs. A different pattern, win rate unchanged but average loss doubling, means the stop is being slipped through live, and that is a broker or execution problem rather than a statistics one.

A 30 to 50% haircut on the backtest edge is normal. An 80% haircut means the backtest was mostly fit. Zero haircut on a demo is suspicious; demo servers fill generously.

Forward testing in public, for free

There is one more reason to forward test on a connected account rather than in a spreadsheet: you cannot fool yourself. ConfirmedTrades live-syncs trades, balance and equity from the MetaTrader terminal, computes every metric from that stream, and does not let you edit or trim it. The AI review attaches a short-record flag below 30 trades and reports Insufficient data below 15, so the page itself tells you when you are reading noise. Connecting is free on any plan; sign up and run the connector on the forward-test terminal.

The drawdown ledger also records the worst balance and equity drawdown per broker login permanently, so a forward test that blew up and was restarted still shows the blow-up. For a developer that is a feature: the point of a forward test is to find out.

A pre-launch ritual

• Count your degrees of freedom, including the ones you set by hand.
• Keep a final out-of-sample segment you have looked at exactly once.
• Run walk-forward; write down the efficiency ratio.
• Add realistic spread and at least a few points of slippage to the backtest and re-check profit factor.
• Forward test on a live-synced account for long enough to matter; how many trades before you trust your EA tells you how long that is.
• Size from the degraded numbers, never the backtest ones.