WHAT IS A TRUE POSITIVE, AND WHY YOU SHOULD CARE
You may recall from statistics class that testing a scientific hypothesis yields four types of answers:
1) True Negatives: things that look wrong and are wrong;
2) False Negatives: things that look wrong but are right;
3) False Positives: things that look right but are wrong; and…
4) True Positives: things that look right and are right.
The main goal of scientific research is to find True Positives. For example, geologists employ chemical tests to determine the amount of microscopic gold per cubic meter of soil. A True Positive is a find that becomes a viable mine. True Negatives are not profitable, but at least the mining company didn’t chase the wrong opportunity. False Negatives are missed opportunities, which may be captured by a competitor. The worst possible outcome is a False Positive, where the company wastes precious resources on an opportunity that was never there.
Unfortunately, the more True Positives you search for, the more False Positives you end up getting. In developing investment strategies, in particular, it is much easier to find a False Positive than a True Positive.
Let’s look at a simple example: You want to know if there is a particular day of the week, month, quarter or year when you should buy stocks, and a particular day of the week, month, quarter or year when you should sell stocks. So you try thousands of combinations of buy and sell dates, and conclude that it is optimal to buy stocks on the second Tuesday of each month, and sell them on the final Friday of each month.
Such finding is likely to be a False Positive. The reason is, if you try thousands of random strategies, there will always be a lucky one that beats the rest, even if there is no investment opportunity. Likewise, you could generate a random price series (hence unpredictable), and find misleadingly profitable trading strategies where obviously there are none.
Many firms produce nice-looking backtests, and most of them are False Positives. This phenomenon, widely-known as “backtest overfitting,” is one of the reasons why so many hedge funds fail. Eliminating False Positives is one of the most challenging problems in finance (otherwise, anybody with a positive backtest would be a billionaire!) Indeed, tests that isolate True Positives in finance are so hard to come by that the problem is largely ignored by academics and practitioners alike. As a President of the American Finance Association recently acknowledged, most claimed research findings are likely false.
Over the past decade, True Positive Technologies has developed multiple proprietary techniques that seek to prevent backtest overfitting. Our scientists were among the first to identify this industry-wide problem, and provide sound and practical solutions, including:
+ PBO: It computes the probability that the best in-sample strategy will underperform out-of-sample a randomly chosen alternative.
+ DSR: It deflates the predicted performance of a strategy by controlling for the number of trials involved in a discovery. In the example above, the “buy on Tuesday and sell on Friday” strategy would be discarded because of the large number of combinations tried.
+ OTR: This empirical procedure determines the optimal trading rule on a large amount of synthetic data generated by the distribution that characterizes the observed data.
+ CPCV: It derives the full distribution of the strategy’s performance (rather than a single path) under thousands of alternative scenarios.
These proprietary methods have been peer-reviewed, published in the leading academic journals, and are the subject of various patent applications owned in exclusivity by True Positive Technologies. More important, they have been tested over the years in multi-billion dollar investment vehicles, and shown to work as designed.