When coming up with a trading idea or a trading strategy, algorithmic traders will model their idea into a specific algorithm. They then use historical data to test whether an algorithm is profitable or not. They evaluate how this profitability will likely continue in the future to make investment decisions.
In a real trading environment, however, the target market often changes. This can lead to many different scenarios. Some algorithms are only profitable when the market trends upward. Some yield profits in trending markets but undergo big losses when the market does not show any particular trends.
In addition, it’s possible the random test data is perfect for the algorithm, leading to highly positive test results. In practice, it may be difficult to repeat the same scenario. Therefore, the process of developing and evaluating an algorithm should try to test many different scenarios to objectively assess the algorithm’s potential before real trading.
Figure 15 illustrates the process that combines past future data in the algorithm development and evaluation process.
-
In-sample data: a part of past data, also known as in-sample past data, to test and optimize the algorithm. Typically 70% of past data will be used for this purpose.
-
Out-of-sample data: includes out-of-sample past data and future unseen data. The purpose is to test an algorithm after the optimization phase whether it can generate stable profits. Out-of-sample data and in-sample data should have no overlap.
For example, it’s necessary to test algorithms with the recent 10-year price and volume data from 2012 to 2022. The data from 2012 to 2019 will be the in-sample past data and the rest is out-of-sample past data.
The backtesting process uses both in-sample and out-of-sample data. Normally this data set is sufficient to evaluate algorithms. However, since the testing only uses past data, there’s a possibility of common errors such as skipping slippage and overfitting.
To avoid common errors, the evaluation process takes an extra step to test future data. Forward testing, known as real-time algorithm testing, uses out-of-sample data that is not available at present. Its purpose is to add objectivity to the evaluation process while evaluating the algorithm’s feasibility.
Forward testing has two main stages. The first is paper trading, which is similar to real trading but no actual transactions are made. All transactions are simply recorded for further evaluation. The second stage is performed in a real environment with a small amount of capital. It is to finally test the effectiveness of the algorithm, its hypothesis, and any technical issues in the system. This stage is thus called a small account test.
Backtesting and forward testing are both essential in the algorithm development process. Neither is better than the other. High profits and similar results in both types of tests are important criteria to decide whether to use a particular trading algorithm.