A Backtesting Protocol in the Era of Machine Learning (2019)
Rob Arnott, Campbell R. Harvey, Harry Markowitz
The Journal of Financial Data Science Winter 2019, URL/SSRN/PDF
I have already written about the pitfalls of research in asset management and the importance of good research practices for the application of machine learning. This week’s AGNOSTIC Paper takes this idea even further and provides a seven-point protocol for empirical research in finance.
Everything that follows is only my summary of the original paper. So unless indicated otherwise, all tables and charts belong to the authors of the paper and I am just quoting them. The authors deserve full credit for creating this material, so please always cite the original source.
Setup and Idea
Protocols are well-established in many fields and some very successful fundamental investors reportedly also use checklists. In his outstanding book, Atul Gawande explains how checklists improved outcomes across many disciplines. So why not also use one when doing empirical financial research? Checklists and protocols help in complicated situations where it is easy to miss important details. For example, all experienced researchers know that outliers can materially affect results. Outlier management, however, is just one of many small steps in a research project. Without a checklist, it is easy to forget it or to do it in a way that leads to potentially biased results.
Systematic processes and productive habits are one of the best weapons against psychological biases, so using a checklist or protocol in empirical research is just logical.
Important Results and Takeaways
A seven-point protocol for empirical finance research
The key result of the paper is the following seven-point protocol for empirical research in finance. The authors develop it with the explicit goal of eliminating false positives. Needless to say, false discoveries are very problematic for asset managers because a false positive strategy will most likely disappoint in live trading.
Conclusions and Further Ideas
I really like the idea of a comprehensive investment checklist. As you know, I believe in systematic processes and I know from my own experience that investing is way too hard to make decisions case-by-case without a good underlying framework.
The points in the authors’ research protocol are already helpful but quite narrowly focused on statistical issues of quantitative research. However, the paper motivated me to further sharpen my own investment process and actually summarize it in a protocol.1This is going to be a very long-term evolutionary project… Maybe I will do it here on the website… In addition to that, I will also try to incorporate the authors‘ points into future reviews of AGNOSTIC Papers. Nobody likes people who give themselves credit, but I do believe that some of the prior posts were already not bad in this respect. At least in my perception, I often highlighted the importance of statistical robustness and out-of-sample tests…
- AgPa #76: ESG Myth Debunking
- AgPa #75: Optimal Investment Committees
- AgPa #74: Peer-Reviewed Research is Not Helpful to Predict Returns – Really?
- AgPa #73: Country and Industry Momentum
This content is for educational and informational purposes only and no substitute for professional or financial advice. The use of any information on this website is solely on your own risk and I do not take responsibility or liability for any damages that may occur. The views expressed on this website are solely my own and do not necessarily reflect the views of any organisation I am associated with. Income- or benefit-generating links are marked with a star (*). All content that is not my intellectual property is marked as such. If you own the intellectual property displayed on this website and do not agree with my use of it, please send me an e-mail and I will remedy the situation immediately. Please also read the Disclaimer.
Endnotes
| 1 | This is going to be a very long-term evolutionary project… Maybe I will do it here on the website… |
|---|