In finance, it’s difficult to distinguish between skills and luck. In the short term, luck can be up to 95% of total results. In the long term, however, skills will be 95% of the total results instead. Proven algorithms eliminate luck or randomness in the long run and they are important milestones for   algorithmic traders to succeed.

Here are the 3 price charts of stocks A, B, and C over the last 1000 days. In your opinion, which stock will trend up, trend down, or trend sideways?

Mark your answer below and see the right answer.

Do you think stock A will go sideways or trend down without  recovery?

Do you believe stock B will rise strongly? Do you stock C will move into a correction in the near term?

The final answer for all three stocks cannot be predicted. It’s because the three graphs above are purely from a lab simulation. It simulates the next day’s price is normally distributed around the previous one, ranging from -7% to 7% (statistics from HOSE standard deviation).

The truth is the next day’s price is random, unpredictable, and independent of the previous 1000 days in all graphs above. This is a prime example of a striking similarity when comparing price change by model and by randomness.

The Danger of Confusing Patterns and Randomness

If an algorithmic trader finds a winning model from thousands of stocks like A, B, and C, what would the long-term outcome be? According to the law of large numbers, no matter what models were discovered, investors will have their assets slowly decline due to the effect of fees, taxes, and slippage. This is the main reason why most passive trading funds are more efficient than active funds, with management and trading fees included. In other words, in a stochastic market, excessive trading can only lead to excessive losses with unmeasurable fees.

In the financial world, the biggest reason that causes the confusion between a predictive model and randomness is that wrong investments can still make profits and vice versa. In other professions such as musicians, pilots, engineers, and artists, the right decision will usually have corresponding positive feedback and vice versa. In finance, the feedback typically has lots of noise. Following a few specific cases can lead investors in the wrong direction. What if you make a bad investment but still get profits five times in a row? Or conversely, what if you make a great decision but suffer losses five times in a row?

This confusion can bring extremely negative results in the long run. As an example, inexperienced investors with no knowledge of the stock market can still make profits as large as multiple times of their investment assets. At that point, they may feel like they already identified the right investment model and start using margin and debt instruments, or receive investment trusts. However, when a catastrophe happens, these investors may lose all the profits accumulated from the previous period.

This investment model has been widely popular during the Covid period when new investors entered the market and accidentally made huge profits in just a very short time. This pattern will likely continue in the future and is not limited to the stock market.

How to Identify Predictive Models

Distinguishing patterns from randomness is a complicated process since they both appear exactly the same. Even randomness may yield better results in the short term. There are two main methods to identify patterns and avoid randomness in algorithmic trading:

First, use multiple rounds of rigorous testing with a large number of transactions to ensure that the odds to gain long-term profits after all fees and taxes are much greater than the 50% threshold. Algorithmic traders need to understand the steps of backtesting, optimization, and forward testing for detailed implementation.

Second, use economic fundamentals in forming the algorithm hypothesis. A profitable algorithm is not everything. It needs a solid explanation. For example, a stock from a well-performing company with trustworthy reports is much better than a sharply rising stock for unknown reasons. In the long term, algorithms using machine learning may grow beyond human understanding. However, in the long term, understanding the source of profits will ensure the system operates stably and builds sustainable growth.