A man fires forty rounds at the side of a barn, walks over to the tightest cluster of holes, paints a target around it, and calls himself a marksman. Statisticians call this the Texas sharpshooter fallacy. Traders call it a backtest.
That is unfair to good backtesting, but only slightly. When you backtest a trading strategy, you are asking whether a fixed set of rules would have made money on data you have already seen. The honest version of that question is hard to answer. The dishonest version, where you keep tweaking until the equity curve smooths out, is easy. AI has made the dishonest version nearly free.
A language model can now write a strategy script, run it, read the results, and revise the logic in under a minute. You can paint a hundred targets around a hundred bullet clusters before lunch. Speed only helps if you also speed up the part where you try to destroy your own result.
This guide covers what a backtest actually measures, how to run one with AI without fooling yourself, the specific numeric thresholds that separate a real edge from curve-fit noise, and the mistakes that turn a beautiful backtest into a losing live account.
Current conditions make the trap easier to fall into.
The S&P 500 sits near 7,737 with the VIX around 14, a calm, trending regime that flatters almost any long-biased rule set you point at it.
What Is Backtesting a Trading Strategy?
Backtesting is the process of applying a fixed set of trading rules to historical price data and recording what would have happened. Entry conditions, exit conditions, position size, and costs go in. A trade log and a performance summary come out.
The output is not a prediction. It is a measurement of one specific history. A strategy that returned 40% annualized from 2020 to 2026 tells you what worked across a pandemic crash, a zero-rate boom, an inflation shock, and an AI capex cycle. It does not tell you what happens next.
Three things must be present for that measurement to mean anything.
- Clean data. Open, high, low, close, and volume at the timeframe you actually trade. Survivorship-free equity data if you are testing stocks, so delisted names are still in the universe.
- Realistic costs. Commission, spread, slippage, and financing. A fast mean-reversion strategy can look profitable at zero cost and lose money at one basis point per trade.
- Enough trades. A backtest with 18 trades is an anecdote. Practitioner guidance is a minimum of 100 trades for basic statistical significance, and 200 to 500 for real confidence.
Out-of-sample data is the portion of history you deliberately never look at while building. It is the only part of the test that carries information about the future, and it is the part most traders quietly spend early.
What a backtest quietly leaves out
- Slippage on illiquid names | fills assumed at the requested price | real cost scales with order size versus average volume
- Borrow cost and availability | short entries assumed free and always fillable | hard-to-borrow names can cost double digits annually
- Survivorship bias | delisted companies dropped from the universe | inflates historical equity returns
- Regime change | only regimes inside your window get tested | a 2023 to 2026 window contains no sustained bear market
- Your own behaviour | assumes every single signal is taken | real traders skip trades after a drawdown
Why Backtesting a Trading Strategy With AI Changes the Risk
The mechanics of backtesting have not changed. The economics of searching have.
Before AI assistants, writing a strategy in Pine Script or Python was a bottleneck. That friction was accidental protection. You tested five ideas, not five hundred, so the odds that your best result was pure luck stayed manageable.
Now the bottleneck is gone. Ask a model to convert a plain-English idea into a TradingView strategy script and you get runnable code in seconds. Ask it to try fourteen variations of the lookback period and it will, without complaint and without telling you that it just ran fourteen experiments on your behalf.
Over 300 published factors claimed to predict stock returns by 2012, according to Harvey, Liu and Zhu's survey of the academic literature. Most were the product of exactly this kind of search, run by professionals with far more discipline than the average retail trader.
This is the multiple-comparisons problem. Test enough rules against the same history and some will look brilliant by chance alone. The more tests you run, the higher the bar a result must clear before you should believe it.
The practical fix is not to test less. It is to log every test, keep a strict out-of-sample holdout, and raise your significance threshold in proportion to how many variants you tried. AI is genuinely useful for the boring half of that discipline. It will happily maintain the experiment log, compute corrected thresholds, and remind you what you already tried and rejected.
How to Backtest a Trading Strategy in Six Steps
Step 1: Write the thesis before the code
State in one sentence why the edge should exist. "Gold miners lag the gold price by two days because fund flows rebalance on a delay" is a thesis. "The 14-period RSI crossing 30 works" is a pattern with no mechanism attached. Patterns without mechanisms are the ones that vanish the moment you fund the account. If your thesis is valuation-driven, start from something concrete like valuing a stock with the PE ratio.
Step 2: Turn the thesis into unambiguous rules
Every rule needs a number and a timestamp. Entry, exit, stop, position size, universe, timeframe, and rebalance schedule. If a human could read your rules two different ways, so can the backtest engine. Have the AI restate your rules back as pseudocode, then hunt for the places where it filled a gap with a guess.
Step 3: Split the data before you look at it
Reserve the most recent 30% of your history as an untouched holdout. Do not run a single test on it. If you are working with 2015 through 2026, build on 2015 to 2022 and lock 2023 to 2026 in a drawer.
Step 4: Build and run on the in-sample window only
Use the AI to write the script, then read every line yourself. The two errors models make most often in strategy code are lookahead bias, where a bar's close decides a trade taken at that same close, and repainting indicators that recalculate after the fact. Both produce gorgeous, impossible results.
Step 5: Run walk-forward analysis
Optimise on a window, test on the window immediately after, roll both forward, repeat. Then compute walk-forward efficiency, the annualised out-of-sample Sharpe divided by the annualised in-sample Sharpe. A healthy band is 0.5 to 0.7. Below 0.5 your parameters are memorising rather than learning.
Step 6: Spend the holdout exactly once
Run the final, frozen strategy on the reserved data. One run, no adjustments. If out-of-sample Sharpe lands more than 30% below in-sample, treat the strategy as overfit. If it survives, you have earned the right to paper trade, not the right to go live.
0.5 to 0.7 is the walk-forward efficiency band that separates a robust parameter set from a memorised one.
A four-parameter strategy on 41 trades fails most standard validation thresholds, no matter how good the Sharpe ratio looks.
Backtest health check thresholds
- Number of trades | 100 minimum, 200 to 500 for confidence | under 50 is an anecdote
- Trades per tuned parameter | 50 to 100 | under 20 means the parameters are fitting noise
- Walk-forward efficiency | 0.5 to 0.7 | under 0.5 means memorisation
- Out-of-sample Sharpe degradation | under 30% | over 50% is near-certain overfit
- Backtest Sharpe ratio | 1.0 to 2.0 is credible | above 3.0 is a red flag
- Win rate | any level, if the payoff ratio supports it | above 80% usually hides unmodelled costs
Real Examples
Two readings of the same kind of numbers.
Consider a simple momentum rule tested on the S&P 500 from 2015 to 2022. Buy when price closes above the 200-day moving average, sell when it closes below. In-sample it produces a Sharpe of 0.9 across 41 trades. Rolled forward into 2023 to 2026, a stretch the parameters never saw, it produces a Sharpe of 0.7. That is a walk-forward efficiency near 0.78. Only 41 trades, so confidence stays thin, but nothing here smells like curve fitting: one parameter, an obvious economic story, and modest degradation.
Now consider what an over-eager optimiser produces from the same idea. A 187-day moving average, a 2.3% confirmation buffer, a filter that skips signals in the first week of any month, and a volatility switch. In-sample Sharpe jumps to 2.6. If you want a cleaner way to run this kind of head-to-head, see how to compare stocks side by side with Claude AI.
That number is the warning, not the reward. A backtest Sharpe above 3.0 is treated as a red flag by most practitioners, and 2.6 built from four tuned parameters across 41 trades works out to roughly ten trades per parameter, against a working guideline of 50 to 100.
The second strategy has the better backtest and is the worse strategy. That is the whole lesson.
Common Mistakes
Mistake 1: Reusing the holdout
You run the frozen strategy on reserved data, it disappoints, you adjust one rule and run it again. The holdout is now in-sample and no longer tells you anything. There is no way to un-see data. Once it is spent, the only honest test left is time you have not lived through yet, which means forward testing.
Mistake 2: Ignoring costs and slippage
Backtest engines fill at the price you ask for. Real brokers do not. Model commission, half the bid-ask spread on each side, and additional slippage on any order that is large relative to average volume. High-turnover strategies die here more often than anywhere else.
Mistake 3: Testing a single market regime
A strategy built entirely on 2023 to 2026 has never seen a sustained bear market, a rate-hike shock, or a liquidity crisis. Stretch the window back far enough to include at least one regime that should hurt the thesis, or state plainly that the strategy is regime-dependent and size it accordingly.
Mistake 4: Trusting AI-generated code without reading it
Models write plausible code quickly. Plausible is not the same as correct. Ask directly: does this script use any information that would not have been available at the moment of the trade? Then verify by hand against three sample trades pulled from the log.
Mistake 5: Confusing a good backtest with an edge
Every profitable strategy has a good backtest. Most good backtests never become profitable strategies. A backtest is a filter that removes obviously bad ideas. It is not evidence that a surviving idea is good.
The gap between a clean equity curve and a real brokerage statement is mostly made of costs the engine never charged you.
Frequently Asked Questions
How many trades do you need for a valid backtest?
At least 100 for basic statistical significance, and 200 to 500 for genuine confidence. A second constraint applies per parameter: aim for 50 to 100 trades for every free variable you tuned. Four tuned parameters across 60 trades is not a backtest, it is a coincidence with a chart attached.
What is walk-forward analysis in trading?
Walk-forward analysis optimises a strategy on one slice of history, tests it on the slice immediately after, then rolls both windows forward and repeats. It mirrors the actual sequence a live trader faces, where you only ever tune on the past and trade an unknown future. Walk-forward efficiency above 0.5 suggests the parameters carry real signal.
Can AI backtest a trading strategy for me end to end?
It can write the script, run it, tabulate results, and summarise them clearly. It cannot supply the economic thesis, choose an honest holdout, or resist the temptation to keep tweaking. Those three are exactly the parts that determine whether the result means anything.
Is a high Sharpe ratio always good?
No. Above 1.0 is solid and above 2.0 is strong, but backtest values above 3.0 usually point to overfitting, lookahead bias, or unmodelled costs rather than a discovery. Treat an implausibly good number as a bug report, not a result.
What to Watch Next
- v Does your out-of-sample Sharpe hold above half its in-sample value after three more months of forward data?
- v Does the VIX moving back above 20 break a strategy built entirely in the current low-volatility regime?
- v Do your realised fills match backtest assumptions within one basis point across the first 30 live trades?
- v Does adding one more year of older history change which parameter set your optimiser selects?
- v Does the strategy still work when you disable your single most-tuned parameter?
If the strategy does survive, the next question is position sizing, which is really a question about rebalancing a portfolio step by step.
The sharpshooter painted his target after the shots had already landed. The only way to find out whether he can actually shoot is to hand him a fresh wall. Your holdout data, your paper trading period, and your first fifty live trades are that fresh wall. Everything before it is decoration.
Key Takeaways
- A backtest measures one specific history, not the future. Treat it as a filter, never as proof.
- AI removes the friction that used to limit how many strategies you tested, which raises the odds your best result is luck.
- Write the economic thesis before the code. Patterns without mechanisms are the ones that stop working.
- Reserve 30% of your history as an untouched holdout and spend it exactly once.
- Target walk-forward efficiency of 0.5 to 0.7, and treat out-of-sample degradation beyond 30% as overfitting.
- Aim for 100 trades minimum, and 50 to 100 trades per tuned parameter.
- A backtest Sharpe ratio above 3.0 is a warning sign, not an achievement.
- Read every line of AI-generated strategy code for lookahead bias and repainting before you trust the output.
This article is educational and is not investment advice. Backtested results do not guarantee future performance.