In this article, I interview two distinguished professors and practitioners of mathematical finance, David H. Bailey, Ph.D. of Lawrence Berkeley National Laboratory (retired) and the University of California at Davis, and Marcos Lopez de Prado, Ph.D., of Lawrence Berkeley National Laboratory and Chief Executive Officer of True Positive Technologies. In a series of recent blog posts and journal articles, they have made the case that we are in the midst of “the greatest financial fraud in history.” What is this crisis facing finance and how does it impact individual investors, as well as financial institutions? Here are eye-opening perspectives from professional mathematicians who apply their work to financial markets.
Imagine that a pharmaceutical company develops 1000 drugs and tests these on 1000 groups of volunteer patients. When a few dozen of the tests prove “significant” at the .05 level of chance, those medications are marketed as proven remedies. Believing the “scientific tests”, patients flock to the new wonder drugs, only to find that their conditions become worse as the medications don’t deliver the expected benefit. Some consumers become quite ill and several die.
Clearly, there would be a public outcry over such deceptive practice. Indeed, that is precisely the reason we have a regulatory agency and laws to help ensure that medications have been properly tested before they are offered to the public. According to Marcos Lopez de Prado, no such protections are offered to financial consumers, leaving them vulnerable to unproven investment strategies. Indeed, he and several co-authors contend that the majority of investment strategies promoted by academics and quantitative practitioners are false. As Lopez de Prado explains in his recent book, Advances in Financial Machine Learning, these strategies look at so many combinations of so many variables that they are likely to find models that seemingly work both in- and out-of-sample, only to fail dismally in the real world. These false positives are particularly misleading, as they are promoted by researchers with seemingly impeccable research backgrounds—and who do not employ the scientific tools needed to detect such false findings.
Nor is this crisis limited to the world of quantitative finance. David H. Bailey, in a post to the Mathematical Investor site, points to a wide range of financial providers, from banks to brokerages, that promote untested, unproven technical analysis strategies. These encourage traders and investors to put their money to work, while offering guidance with no objective information value. Indeed, Bailey and Lopez de Prado note that the most important plot in finance is not a technical analysis pattern, but a statistical distribution of Sharpe Ratios as a function of the number of backtests conducted. If one looks at enough combinations, it is quite possible to find technical patterns that will achieve seemingly astronomical risk-adjusted returns. The great majority of these are spurious.
Lopez de Prado and Bailey are not alone in calling attention to the crisis in finance. Peter Coy recently quoted Campbell Harvey’s presidential address to the American Finance in which Harvey noted that standard testing methods are often inadequate for answering the questions posed by research. Indeed, Harvey and Liu, in their paper on evaluating trading strategies, show that strategies built with random numbers can test as significant. Citing the potential for abuse of misleading research findings ,Harvey implored his professional finance audience to think of themselves as scientists, not salespeople.
What does this proliferation of false investment strategies mean for traders, investors, investment advisers, and institutions managing client capital? Professors Bailey and Lopez de Prado graciously agreed to answer a few questions:
- Five years ago, you brought this issue to public attention. Still, here we are with one brokerage firm after another, one media outlet after another, dispensing advice from chart patterns, wave structures, technical indicators, and other unproven methods. How might we better protect financial consumers?
Bailey: Yes, it is very distressing that after all of the research published by ourselves and many others, numerous unproven and technically dubious techniques (“head and shoulder patterns”, “b-waves”, “technical indicators”, “support levels”, etc.) are still widely featured in the financial press, and thus presumed by millions of consumers and professionals alike to be the “scientific” way of handling one’s investments. Our own research has amply confirmed that these methods do not work. For example, in our recent study of market forecasters, we found that those forecasters who rely on charting, technical analysis and/or wave analysis averaged only 44% correct predictions, which is even less than the disappointing score (49%) achieved by the full collection in our study. It may well be that the vast majority of persons featuring and promoting these techniques do not realize that they are scientifically ill-founded, but that does not excuse those who do.
Lopez de Prado: When we have confronted financial firms, a typical response it that they are merely satisfying customers’ demands. The reality, of course, is quite different. For decades, these firms have told retail investors that technical analysis, fundamental analysis and academic papers “work.” At this point, investors do not even challenge the notion that those claims may be false. They assume that, if those claims were fraudulent, regulators would have taken action, just as the FDA would not allow pharmaceutical companies to sell crack-cocaine. But the SEC does not have the oversight powers of the FDA. Investors do not receive the level of malpractice protection that patients enjoy.
Regulatory agencies cannot protect retail investors unless there is a public outcry. And the public outcry has not happened because of the successful misinformation campaign alluded earlier. Our best hope is to educate investors.
Bailey: The challenge here can be seen in some recent studies of individual investor behavior. For example, the 2017 DALBAR report found that over the 30-year period ending in 2016, individual equity fund investors averaged only a 4% annual return, compared with 10.2% that could be had by simply investing in a low-cost S&P 500 index fund. As we have pointed out, most of the shortfall is due to panic selling in downturns, lack of diversification, not understanding the basics of long-term compounded returns, and, in the end, a failure to establish a rational financial plan and stick with it over the long term.
If most consumers do not understand, much less practice, the basics of rational investing, and if even fewer understand that reliance on charts, graphs and frequent trading is not the answer, then it is much less likely that individual consumers can understand difficulties with backtest overfitting and other potential pitfalls that unfortunately pervade the investment world. Although improved consumer education would undoubtedly help, I personally see no alternative to some significantly higher level of government regulation.
- You’ve also written on the topic of evaluating the probability of backtest overfitting. I find that most professional money managers are not familiar with these methods and simply use in-sample and out-of-sample backtesting to establish the validity of an investment strategy. Can you address why traditional backtests are not sufficient and how they can be made more rigorous?
Lopez de Prado: Statistical tests assume a certain false discovery probability. In the context of investing, that’s the probability of saying a false strategy will make money. That probability is generally accurate if the test is dispensed only once. However, if a researcher dispenses 20 times a test with let’s say a 5% false positive probability, it is all but guaranteed that one false strategy will be produced. In modern finance, researchers conduct millions of tests. The implication is that most discoveries in finance are likely false, and that most investors are putting their money on losing strategies. This methodological error is called selection bias under multiple testing. One reason financial researchers get away with this scientific fraud is because we don’t have laboratories where discoveries can be challenged based on new evidence. It will take decades to collect the new evidence needed to debunk each of these claims, and by then new false claims will be marketed to investors.
All forms of backtesting are susceptible to this flaw: Cross-validation, out-of-sample walk-forward, hold-out, etc. As long as researchers do not control for the number and variance of the trials involved in a discovery, false positives are almost certain.