Alex Vidovich has published an important working paper on the tendency of backtested investment results to produce “false positives”—i.e., methods that appear to improve investment performance but that really don’t.

Why the topic matters

Investment companies routinely offer investment products whose performance is validated through backtested results. In the investment world, backtesting refers to the practice of defining an investment strategy, then applying the investment strategy to a historical data set. If the strategy performs well against the historical data set, companies tend to claim that the strategy will perform well in the future.

Companies routinely publish graphs of performance that look as though their approach has been successful, but that really amounts to a new investment approach fitted to old data.

As one example, here is a graph of how my Magic Backfitted Portfolio has performed over the past 10 years:

As you can see, this portfolio has dramatically outperformed the S&P 500, returning a compounded annual growth rate of more than 35% vs. the S&P’s growth rate of 15%. The investor who invested $10,000 in the S&P 500 10 years has grown their investment to a meager $41,000, whereas the investor who put their money into my Magic Backfitted Portfolio would have grown their investment to $236,000.

What’s wrong with this picture?

What’s wrong with this picture is that I used my 2026 knowledge of the 10 stocks that grew the fastest from 2016-2026, and I constructed my Magic Backfitted Portfolio from those. Would I have picked those same 10 stocks in 2016, knowing what I knew then? That’s extremely unlikely.

I see graphs like the Magic Backfitted Portfolio graph routinely. Automated stock picking algorithms, derivatives, long/short portfolios, and structured products seem to rely especially heavily on backtesting.

The problem with backtested results is that it’s almost impossible for the human beings performing the backtesting to avoid assuming the conclusion—meaning the method is constructed to perform well against the historical data set, based on knowledge of what’s in the historical data set. When these methods are deployed, what tends to happen in practice is the first real new data departs from the patterns seen in the historical data, and all bets are off as far as future performance is concerned.

This is not unique to investment backtesting. I saw it in Covid forecasting, and the literature on backtesting contains numerous methods that are designed to avoid knowledge of the past contaminating construction and validation of the methods.

Summary of Vidovich’s paper

Vidovich’s paper was obviously written by Claude and contains a distracting amount of Claude-speak, which I personally find annoying, but that doesn’t diminish the results.

His headline finding was that, using a popular backtesting method (tuning the backtesting on the full data set), in his simulation, 36.9% of “pure null” methods (neutral methods that should have been negative results) were accepted as positive results—i.e., the rate of false positives was 36.9%.

Using a more restrictive but still popular backtesting method (fold-consistency), doesn’t help: the false-positive rate is 88%. This is a bit shocking, since fold-consistency is intended to be more rigorous but actually has a higher false-positive rate.

Using a strongly restrictive backtesting method (training on 70% of the data; trying as many variations as wanted, but testing only one time on the 30% that’s been held out), reduces false positives to 2.3%. However, this is where the paper becomes especially interesting.

Vidovich observes that, if researchers make multiple attempts with different methods on the 70% training data, comparing results to the test data set and discarding methods that are ineffective against that data set, then the result needs to be adjusted for the total number of trials, not just the trial that finally produced the desirable result that was frozen. If the test data set is reused (which is common), the overall effect is the same as the first method—36.9% false positives, because the data that was “held out” was not really held out—in effect, it becomes part of the training data set.

If I took a random sampling of 10 stocks from the S&P 500 in 2016 and kept testing their 10-year results until I found a set that significantly outperformed the S&P 500, I would eventually find a set with amazing performance. But that successful backtesting tells me nothing about how that method would perform in the future.

The conclusion is that the 2.3% false positive rate applies only if the test data set is used only once. You can imagine that if I took one random sampling of 10 stocks from the S&P 500 in 2016, it’s vanishingly unlikely to produce the Magic Backtested Portfolio. (The odds of choosing that portfolio randomly are approximately 4×10⁻²¹).

Vidovich makes an additional interesting comment that if the test data set is reused multiple times, and the number of trials is accounted for statistically, the false positive rate can be dropped to 0.4%. Unfortunately, the method used to achieve this low level of false positives is rarely, if ever, used in practice, and should be considered aspirational.

Analysis

Vidovich’s findings are based on his own Monte Carlo simulation. They are not based on an empirical study of backtesting used for investment purposes. As such, his specific false-positive numbers of 87%, 36.9%, 2.3%, 0.4%, etc. are relevant only to his simulation design assumptions and not usable as specific fault rates for real-world backtests.

That said, a finding that common backtesting methods produce false positives >10% on any data set is significant. Investment approaches that disclose specifics of their backtesting methods are a lot closer to the methods with false positive rates of 87% and 36.9% than they are to the methods with rates of 2.3% and 0.4%.

For context, I think it’s useful to recall the levels of statistical significance required for publication in professional journals. A common standard is that results must be significant at the p < 0.05 level, meaning, less than 5% probability that the results occurred solely by chance. More rigorous journals require results to be significant at the p < 0.01 level–less than 1% probability of the results occurring by chance.

Other studies have come to similar conclusions. Among the papers cited by Vidovich, a paper by Harvey, Liu and Zhu argued that extensive factor mining required a t-statistic above roughly 3 rather than 2. A paper by Hou, Xue and Zhang found that 65% of 452 published investment characteristics failed their testing hurdle at |t|>1.96 (usual practice) and 82% failed at |t|>2.78, which was the t value appropriate for the number of trials represented by the papers collectively.

In my work on software estimation, I found that, when a researcher wants a method to produce a particular result, the researcher will find a way to make the data produce that result. Bad faith is not required to produce invalid results; wishful thinking is enough. I see no difference between software estimation models and investment models in this regard.

Perspective

Even the most rigorous backtesting method that uses its test data set only once (2.3% false positives) would not clear the level of statistical significance required by the most rigorous journals (p < 0.01)—yet the companies that offer investments validated by backtesting are asking investors to make significant bets on those methods with their retirement savings or other personal assets.

For investment advisors regulated by Washington state (where I’m located), publishing hypothetical or backtested results is prohibited—it’s considered fraud under current Washington state law.

My own experience with software estimation research and Covid forecasting leads me to agree with Washington state’s prohibition: I believe that virtually any method can be validated via backtesting, and the weight that investors should put on investment offerings whose performance is validated primarily via backtesting should be effectively zero.