On data with no edge, the best of a thousand backtests still looks excellent. That is not luck. It is arithmetic.

A backtest is a measurement. The number that reaches you is almost never the measurement — it is the largest of the measurements you took, and the difference between those two things is the subject of this page.

The maximum of many results is large even when every result is drawn from a process whose average is exactly zero. That is not a market phenomenon or a psychological failing; it is a property of maxima, and it can be quantified. The average largest of 100 independent tries sits 2.51 standard deviations above zero, which is a one-sided p-value near 0.006. A hundred tries is a quiet afternoon.

To show it rather than assert it, this page builds a price series from independent random numbers with a stated generator and a stated seed, so that the true edge in the data is known to be zero by construction. A search over 1,200 moving-average variants on that series produced a best result of +61.0 points with a t-statistic of 2.34. Run forward on the untouched second half of the same series, the same rule returned −6.2 points at t = −0.23.

One market proves nothing, so the whole experiment was repeated on 200 independent markets. The winning rule averaged t = 1.64 in sample and t = 0.04 out of it; 52 of the 200 winners cleared t = 2 in sample and 4 did so forward, which is what chance alone produces. Forward, 110 of 200 made money. The honest description is not that fitted strategies collapse. It is that they become coin flips, and a coin flip is exactly what you were promised nothing about.

Because the backtest reported the best of everything that was tried, and a best is large even when the average is zero. The forward result is not the same quantity decaying — it is a different quantity, drawn fresh, with the selection removed.

The cleanest way to see this is to remove every possible edge from the data and check what remains. Let the price change at each bar be r, drawn independently with mean zero. Let a rule hold position p at each bar, where p may be any function whatsoever of information available before that bar — any indicator, any parameter set, any rule you would like, chosen however you like:

The second line is the whole argument, and it holds for every rule, including the one you spent three weeks selecting. Because pₕ uses only information from before bar t, it is independent of rₕ however cleverly it was chosen. Selection can make the past look like anything. It cannot reach forward.

So on data of this kind there is exactly one explanation available for a beautiful backtest, and it is not partially true or mostly true: the entire in-sample result is selection. The figure below is one such result. The left half is the half the parameters were chosen on. The right half is the same rule, the same market, the same generator, and nothing chosen.

A fitted backtest and its forward result are not the same measurement taken twice. One is a maximum over many draws; the other is a single draw. The gap between them is not decay — it is the selection being taken back.

Large, quickly, and then almost flat. Take N independent results from a process with mean zero and unit spread; the expected maximum is the table below, computed by numerical integration of E[max] = ∫ (1 − Φ(x)^N) dx over the positive axis less the matching integral over the negative one.

Two readings of that table matter, and the second is the one nobody expects.

First: 100 tries is enough. The average best of a hundred sits at 2.51 standard deviations, which any statistics package will label significant at the one per cent level. A hundred variants is not an industrial data-mining operation; it is one person, one afternoon, one spreadsheet.

Second: searching harder barely helps. Going from 100 tries to 100,000 — a thousandfold increase in effort — moves the expected best from 2.51 to 4.38. The maximum grows like the square root of the logarithm of N, which is about as slowly as anything grows. The practical consequence is uncomfortable: the size of a reported backtest result tells you almost nothing about how hard someone searched, so you cannot infer the second from the first.

The illusion is cheap to manufacture and expensive to detect. A hundred tries buys a result that reads as significant at one per cent, and a thousand times more work buys less than twice as much.

The table above is computed here rather than taken from anywhere, because the integral is short and a reader can check it. It is worth checking against a published statement of the same result: Proposition 1 of Bailey, Borwein, López de Prado and Zhu in the Notices of the American Mathematical Society gives the expected best Sharpe ratio over N independent trials in closed form, notes that √(2 ln N) is an upper bound on it, and reports 1.57 at N = 10. The exact integration used here gives 1.539; their figure comes from the limiting approximation, which overstates by 0.036 at N = 10 and by 0.007 at N = 100,000. Two independent routes to the same table is a better reason to believe it than either one alone.

Every parameter value, every rule variant, every filter switched on and off, and every idea abandoned before it was written down. They multiply, they are almost never counted, and the ones that were never recorded count exactly as much as the ones that were.

The grid in the laboratory below is deliberately modest — a crossover of two moving averages with a threshold, which is the plainest rule in the literature. Its size is not modest:

Nobody set out to run 38,400 backtests. They ran one rule and had three ideas about it. The word “also” is what does the damage: it reads like an addition and behaves like a multiplication.

The choice of a moving-average crossover here is not incidental. It is the example Bailey, Borwein, López de Prado and Zhu reach for as well, on the grounds that the sample size caps how many pairs of lookbacks exist, which makes it “relatively easy to determine the pair that maximizes the backtest's performance”. Their conclusion is the one this clause is arguing towards from the other direction — that it takes a surprisingly small number of trials to produce a spuriously good backtest.

Now the honest complication, which is measured rather than assumed. On the market used throughout this page, enlarging the grid from 1,200 to 2,400 to 9,600 combinations did not improve the best result at all — it stayed at +61.0 points, because the added variants were near-copies of ones already tried. Only the fourth enlargement, to 38,400, found something better, and only by 11.6%. Against that, the independent-tries table predicts a 22% rise over the same range. Roughly half the predicted gain appeared, which is what a correlated grid looks like.

The reading is not that grid size is harmless. It is that rows are not tries. What inflates a backtest is the number of genuinely different things attempted, and that number is raised far more by “let me add a filter” than by “let me step the lookback by one instead of five”. This is a different mechanism from slicing a completed record after the fact — that one belongs to the journal article, and it applies to evidence you already hold rather than to rules you are still building.

Degrees of freedom multiply and are almost never counted. But the count that matters is of distinct ideas, not of grid rows — and a filter is a distinct idea in a way that a lookback of 31 instead of 30 is not.

Yes, and that is the point of the next few minutes. The market below is not a market. It is a running sum of independent draws from a stated generator with a stated seed, so its true edge is known — it is zero, exactly, by construction.

Move the three parameters and watch the equity curve of the first half. When you find one you would trade, press Run it forward and the second half appears. Or skip the manual work and press Search this grid, which tries every combination and hands you the winner. The instrument will not show you the forward half until you have committed to a rule, because committing first is the entire discipline being illustrated.

On data with no edge in it, no. The whole experiment was repeated on 200 independent markets from seeds 1 to 200: search 1,200 variants on the first half, keep the winner, run it on the second. The correlation between how good the winner's backtest was and how it then did forward is −0.09.

Read the last three lines together. Roughly one search in four produced a backtest that a trader would call statistically significant. Forward, that fell to four in two hundred — which is not a collapse but a return to the rate chance produces on its own, since a one-sided t > 2 occurs about 2.3% of the time by definition. And the forward mean, +1.4 points against a standard error of 1.75, is indistinguishable from zero, exactly as clause 01 requires.

The number worth sitting with is 110 of 200. Fitted rules do not reliably lose money forward; that would at least be informative. They win about half the time, which means the first month of live trading cannot tell you whether you were fooled. You will get a positive month roughly as often as a negative one, and neither is evidence.

A searched backtest and its forward result are uncorrelated when there is nothing there. The failure mode is not a strategy that stops working — it is a strategy that never started, and a first month that cannot tell you which.

It is enough for exactly one look. The protection is not a property of the data being held back; it is a property of the data never having influenced a decision. The first time a disappointing forward result sends you back to change something, that block has influenced a decision, and it is in-sample from then on.

The arithmetic is the same arithmetic as clause 02, applied to the reserve. Look at the held-out block k times, adjusting between looks, and you have run a search of size k on it. The expected best of those k looks follows the same table: five looks is already worth about 1.16 standard deviations of free flattery, and ten is 1.54. Nothing announces this. The reserve looks untouched, because what was consumed was not the data but its independence.

There is a second, quieter weakness. A single split gives you one number, and that number has a standard error of its own. In the 200-market experiment the forward results of the winners had a spread of 24.8 points around a true value of zero. A single forward test landing at +25 points is therefore an entirely ordinary outcome for a rule with no edge at all. One split cannot separate a real small edge from that spread, which is a resolution problem rather than a discipline problem, and no amount of care fixes it.

This is also where “more data” stops being a reliable answer. Extra history raises the evidence only if it did not help choose the rule — and in practice the longer the history, the more variants get tried on it, the more regimes there are to select a rule to suit, and the more attractive it becomes to shorten the window to the years that behave. Data used to choose is not available to test, however much of it there is.

A held-out block is not a shield, it is a budget. Every look spends some of it, the balance is never displayed, and the arithmetic of spending it is the same arithmetic that inflated the fit in the first place.

Two things, and neither is a matter of judgement or discipline. They are defects in the data or the code, they are present before the first parameter is chosen, and one of them is immune to every protection described so far.

Look-ahead. A signal that uses a value which was not knowable at the moment it claims to trade. The famous cases are subtle: a settlement price used to trade the session it settles, a restated economic figure used at its original release date, an indicator array indexed one step ahead of the return array it is multiplied by. The last of those is an off-by-one, and it is a single character.

The laboratory has a switch for it. On the same synthetic market, with the same 1,200 variants and no edge in the data whatsoever, shifting the signal one bar forward moves the whole grid: its average in-sample profit rises from +0.5 to +19.6 points, and the best variant reaches t = 5.68.

Then note the number that matters. Out of sample, the same rule scored t = 4.05. The look-ahead is in both halves of the data, so the held-out block confirms the result instead of contradicting it. Every defence in clause 06 is defeated: the split is honest, the search was fair, the forward test passed, and the strategy does not exist.

Survivorship. A backtest is run on the instruments that exist now. The contracts that were delisted, the pairs that were depegged, the venues that closed and the products whose specifications changed do not appear in the file, and every one of them disappeared for a reason correlated with performance. The effect is not a bias you can estimate from the data you have — the missing rows are missing precisely because of what happened in them. The practical test is a question about provenance rather than about statistics: does this file contain the things that stopped existing? For most conveniently downloadable histories the answer is no, and nothing in the file says so.

An out-of-sample split defends against search and against nothing else. A look-ahead error is present in the reserve too, so it passes every test you have — which makes it the one failure that a careful process makes more convincing rather than less.

More than the preceding clauses might suggest, but not the thing it is usually run to prove. A backtest is a strong instrument for falsification and a weak one for confirmation, and the asymmetry is not a matter of degree.

• Falsification, at full strength. A rule that fails on the very data it was fitted to is finished, and no further testing is required. This is the one direction in which a backtest gives a clean answer, because a maximum that is still bad is genuinely bad.
• Implementability. Whether the rule can be stated precisely enough to execute, how often it trades, how long it holds, how much capital it uses and how much it would have paid in costs. These are properties of the rule and the data, not inferences about the future, and a backtest measures them exactly. Notice what this shares with falsification: both are statements about what the rule and the data already did.
• Behaviour in a named episode. What this rule would have done in a specific historical stress is a well-posed question with a determinate answer. It is evidence about that episode, and it is not evidence about the next one.
• Ranking a small, pre-declared set. If three candidate rules were written down before the data was touched, the backtest ranks three things and the arithmetic of clause 02 applies with N = 3 — an expected inflation of well under one standard deviation. The discipline that makes the number usable is the declaration, not the test.

Those four have one property in common, and it is the property that makes them safe: not one of them is a forecast. Three are statements about what the rule and the data already did; the fourth is a statement about how many things you declared before you looked.

What it cannot do is convert a searched maximum into a forward expectation. That is not a limitation of the software or of the data; it is clause 01, which says the forward expectation of any rule chosen from the past is zero when there is nothing there. A backtest can tell you a rule is dead. It cannot tell you a rule is alive.

A backtest answers “is this ruled out?” with authority and “does this work?” with a number that was chosen to be flattering. Almost every practice that improves backtesting amounts to asking it the first question instead of the second.

Everything numeric on this page was produced by the code that ships inside it. There is no market data anywhere in this article, and no claim about any real instrument.

The generator. A 32-bit mulberry32 counter PRNG, converted to normal deviates by the Box–Muller transform, summed to give a price path. The increments are therefore independent by construction, with mean zero and unit standard deviation, and the true edge in the series is exactly zero — not approximately, not empirically, but as a property of how the numbers were made. The laboratory prints the realised mean increment and the lag-one autocorrelation of the series on screen, with their two-standard-error bands, as a reproducibility check on the generator rather than as a discovery.

The market. 1,700 bars. The first 200 are warm-up for the longest moving average and are never traded. Bars 200–949 are the fitting half; bars 950–1,699 are the forward half. The figures in clauses 01, 02 and 07 use seed 20260727, fixed in advance as the publication date rather than chosen for its results. Clause 05 uses seeds 1 to 200.

The rule. Position is long when the fast average exceeds the slow average by more than the threshold, short when it falls below by the same margin, flat otherwise; optionally long-only, optionally gated by a trend filter, optionally exited after a maximum number of bars. Profit is position × next bar's change, with no costs of any kind — a deliberately generous assumption.

The statistic. Total profit in points, and the naive t-statistic mean ÷ (sd ÷ √n) over bars. The naive form overstates significance for a rule that holds positions across many bars, so it is quoted as the number a backtest report would show, not as a defensible test.

What is illustrative. Every profit figure, every t-statistic and every equity curve on this page. They describe synthetic series and nothing else. The two claims that are not illustrative are the derivation in clause 01 and the expected-maximum table in clause 02, both of which are mathematics.