Do not trust ML (Machine-Learning) blindly

It is easy to get started with ML (Machine Learning), but it’s a huge challenge to get results that are useful and can be used in the future. I have worked intensively with ML, and I have come to the strange conclusion that ML is so good at finding patterns that you can often not trust the results – at least not blindly.

My education as well as my work with ML over the past years has been related to the financial world. I therefore refer to this context in the following.

Share recommendations, FX recommendations and analyzes often bear the same challenges that I have encountered with ML. When experts hear about why a given share has fallen, the experts can quickly give x different factors to be encountered. But when you ask the same experts to come up with an assessment of the near future based on the same analytical methods, the result is highly doubtful.
At FXStreet, the world’s best experts in forex trading give their assessment of the coming week. They are, for example, Citi, Barclays, Goldman Sachs, UOB, Wells Fargo, etc. Following their weekly Forecast Poll, these experts over a period of 3 months are not even close to guessing 50% correctly. In fact, they are down 27%. None of the “big stars” are significantly better than the average and no one is close to 50%.

Even though the market has been difficult for the last 3 months, the result provides a strong clue that we may not be able to express ourselves with such high degree of uncertainty about the market as the analyzes cannot be used in the future. In other words, we should also doubt our backward analyzes.

ML works a bit the same way. It’s just like experts quickly finding the causal relationships, and with good ML tools, one can quickly create a model that matches the experts. The problem is just that if you use the same ML model to predict the coming week, ML will probably not do better than the above-mentioned experts.

A partial conclusion could be that we can always find correlations, but we cannot accurately identify the “true” causality. Because if we found the right reasons, we would also be able to predict the effects and thus the direction of the market.

An example of why ML goes wrong

To illustrate the limitations in ML, I took a trip to McDonalds. Here I would try to find a pattern between the color of shirt on the guests, a coin box and the last-minute direction of the EUR vs. USD. I divided the shirt colors into different color categories, so for example all shades of blue were categorized as “blue”.

When someone entered the door, I registered, for example, blue in the first column. Next column I threw the coin and recorded either 0 or 1. The same did I do in the third column relative to the direction of EUR vs USD within the last minute. A falling ratio in the course gave the value 0, and an increasing ratio got the value 1.

I conducted measurements of 200 people and based on the data from here I made 2 different ML models. One should be able to predict the color of the shirt for the next customer who came into McDonalds, and the other could predict from the color of the shirt if EUR vs. USD would decrease or increase the next minute.

Not surprisingly, I was able to predict with great precision the future person’s color selection of shirt by simply throwing a coin and looking at developments in EUR vs. USD.

Why is this not surprising?

This is because almost everyone had black shirts on at this day. So, the ML model guessed more or less every time the shirt would be black. If you make an analysis of EUR vs. USD in a period when the market is primarily declining, the ML model will most likely always be on a decline. The pattern is correct. The highest success rate is obtained from the trained amount of data by waging a fall.

In this example it is obvious that the ML model is not wrong, but that the model cannot be used to predict the future. The problem is just that if we were waiting for 10,000 guests at McDonalds, then the ML model would still be able to find a pattern. We always find correlations.
Therefore, never blindly trust the ML result, and simply because ML has an accuracy of 80-90% does not mean that the identified correlations also give a high forward-looking causality.

The solution is a model for validating ML

Until recently, I continued to encounter the above challenges with ML. I then tried to develop a cube model, with the thesis that ML may not always find a valid pattern. If a pattern is to be valid, the pattern must have been identical to one or more times in the past. It is not enough, however, to find a pattern, the pattern must also fit into one or more of the previous boxes / patterns.

Out of this I have developed a Pre-Validation Curve. The Pre-Validation Curve can also be compared to a trend curve. When the trend is increasing, ML works while the ML does not work when the trend is falling. By not just looking at a high ML score and thus focusing more on the Pre-Validation Curve, I’ve improved the success rate somewhere between 7-18 percentage points.

This is comparable to the fact that trading experts failed to come up with a weekly Forecast Poll, but just came with a poll when they had seen something similar before. This approach would probably also significantly increase the experts’ success rate. The problem is just that the experts are “pushed” to predict each day on various stocks and foreign exchange markets.

Pre-Validation Curve user for Forex

Over the last 4 years I have worked with EA (Expert Advisor), which means automation of trading strategies. The interesting thing about this area is how often the world changes and how complex it is. There is almost an infinite number of factors, both fundamental and technical, that can affect how a currency is strengthened or weakened. Technical tools are many, and when you put them together individually, there are many different angles. The corresponding applies to fundamental analysis.

The reason I mention this complexity is because Machine Learning really comes to a test here. What factors do you need to train ML in, now that there are almost never many factors to pick from. These have been my initial major considerations in relation to ML.

I have driven several thousand different simulations through different parameters and factors to reach the conclusion that I looked in the wrong direction. As mentioned earlier, ML always finds a pattern.

A future prediction based on an ML model’s success rate / quality has hardly any correlation to future success rates. Patterns come, and patterns go. It may be one pattern today and another tomorrow, but it seems in the financial world that the market is moving in waves or in trends. These “pattern” waves can create a curve, and this is where I have developed my model, which I call the Pre-Validation Curve. With the Pre-Validation Curve, a given success rate is improved by many percentage points.

I have tested the Pre-Validation Curve in many different markets, trading strategies and for a wide variety of time periods. Even from hundreds of different scenarios, applying the Pre-Validation Curve has always, without exception, improved the final success rate.
My Pre-Validation Curve has been and is a great model for me to make the most of Machine Learning. I do not currently have the great confidence in ML itself because it cannot stand independently in a switchable market as Forex. But when I put my ML models together with a robust Pre-Validation Curve, I almost “blindly” trust the result, as I’m sure they will come out with a far better rating than I can ever come until.

Get started with Machine Learning

Whether you use your own ML algorithm, or one from Amazon, Google, Microsoft or IBM, you’ll always have the above challenge. Therefore, I’ve used my resources 100% to understand existing models’ limitations and capabilities. My choice has fallen on Amazon’s AWS, which is also the world’s largest in the field. AWS can be accessed with more or less all programming languages, so you can continue with the language you prefer.

I have not compared prices directly, but with the Amazon approach to pricing, I’m not sure they are competitive. Amazon has a small settlement per ML model, as well as settlement for each ML real-time prediction. AWS offers 90 different services in all parts of the world, including computing, storage, networking, database, analytics, application services, deployment, management, mobile, developer tools, and tools for Internet of Things. So immediately no restrictions.

AWS also has some good visual solutions, so you can quickly get an insight into the possibilities, but if you are to work seriously with ML, you need to access the API, where they have a very comprehensive SDK (Software Development Kit). The API is flexible and there are many things you can do via the API, which are not possible with the visual solution. Finally, you first understand the ML model options when you have automated the entire ML process 100% from Amazon’s API.

I have explained some of my own experiences with Machine Learning. From AiProSoft we would like to offer our expertise. Please contact us if you want an assessment of how Machine Learning can help your business to get a competitive edge.


Leave a comment

Your email address will not be published. Required fields are marked *