Know Your System! Turning Data Mining from Bias to Benefit Through System Parameter Permutation

Dave Walton
2014 Social Science Research Network  
This paper is targeted towards active traders who follow a systematic approach to alpha generation and wish to thoroughly understand the potential risks and rewards expected from a trading system prior to allocating capital. The conclusions are applicable to all timeframes and parameter-based systems though the example specifically discussed herein is an end-of-day trading system. The goal of this paper is to assist the trader in answering two questions: 1) "What is a reasonable performance
more » ... ble performance estimate of the long-run edge of the trading system?" and, 2) "What worst-case contingencies must be tolerated in short-run performance in order to achieve the long-run expectation?" With this information, the trader can make probabilistic, data-driven decisions on whether to allocate capital to the system and once actively trading, whether the system is "broken" and should cease trading. After explaining assumptions, definitions, and methods, the paper discusses how and why traditional trading system development processes lead to positively biased performance estimates due to the data mining bias (DMB). Without understanding the substantial effects DMB has on historical simulation results, inappropriate or inaccurate conclusions may be reached. Several existing methods of DMB mitigation are briefly 1 http://finance.yahoo.com/ 2 From the current (January 2014) schedule of fees for margin interest from Interactive Brokers <$500,000 borrowed. 3 http://www.newyorkfed.org/ 4 Defined as expected active return (system return -benchmark return) divided by tracking error: �� = ��� � �� � � ������ � �� � � .
doi:10.2139/ssrn.2423187 fatcat:h3zftoz6pbhvxifbs5lj26azii