My Backtesting methodology (Patreon)

This is a repost of one of my earliests posts, and probably one of the most important posts I made (also the one that is most forgotten or neglected by others).

It’s a boring subject about principles, processes, determining how to value backtest results and why I try to keep changes of this methodology to a minimum. Nonetheless I will put the effort in republishing  this post on this platform because it will help us keep on track on the backtesting principles I use.

A lot of people ask me how I do stuff, which pairs do I use, timeframes, timeranges and more. For these people I always refer to this article since it explains the ideas behind my methodology. Also this post acts as a guide for myself and I can revisit it sometimes if I make an error that I have to correct.

A scientific approach

I have a background as a Master of science (actually MscBA). During my study I learned that you never can compare results of multiple tests if these tests do not have a common starting point. It’s like comparing the performances of different cars under different circumstances. You will probably guess that comparing the performance of car A during a mild summer day cannot be compared with the performance of car B when it’s storming and raining during winter. Even testing the same car during these different circumstances can give different results.

A short rant

The same thing goes when testing multiple strategies. You cannot compare strategies when they all use different pairs, different timeframes and/or different timespans. That is why I am always suspicious (and you should be too btw.) about these Youtube videos where the presenter has found a new strategy with 90% win rate or 5000% returns over a certain period. Sure, if the presenter has a good idea to show, I’m always interested. But a good video also shows that this performance was on a specific asset pair, on a specific timeframe and during a specific (random…?) time period. So that you know that this strategy can only be used under these same similar circumstances and probably nowhere else.

Making a difference

I wanted to make a difference here by testing each strategy under the same circumstances so that you can expect outcomes that you can compare with each other. You then have the knowledge that you can reasonably trust this strategy on your favorite asset pairs.

However you still have to do your own research before putting your hard earned money on the line. If you ever have second thoughts of the results I get and if they are valid or not,  then you should definitely always do your own research (DYOR) before ever using these strategies that I test! Just remember that I got these results on my specific setup and it can well be that this strategy has totally different results on your machine. If you can validate my results, then start dry-running to find out if a strategy also works in the real world.

I try to be as transparent as I can. That’s why I always add my strategy results and all other output to the posts I make. This way you can also do a deep dive into my results to see what’s going on.

To be fair, I only test an algorithmic strategy for crypto trading and show these results to the screen. These results are what I get and I have no interest in making them nicer or more profitable then they are. I have nothing to gain from that. Even worse, presenting a strategy's results as “the only ultimate strategy you'll ever need” will only make us use a losing strategy. And losing money is the last thing I want…

The initial setup for each strategy backtest

For my videos and blog posts, where I backtest an algorithmic trading strategy I use the following setup:

• The bot starts with 1000 USDT as starting capital
• I have 10 open trades maximum
• I use the top 50 coins of January 2022. See https://coinmarketcap.com/historical/20220102/
• I use the Binance exchange so the pairs should be available there. Otherwise I take the next available pair.
• I test multiple timeframes (5m, 15m, 1h, 4h, 1d).
• The backtest date range is “2017-08-17” – “2023-01-01” (pairs that were available on the Binance exchange after Jan 2017 will have shorter data ranges of course).
• I use the Freqtrade trading bot for automatic backtesting and hyperparameter optimization

The main point is that I want to keep things simple so that I can take a strategy (which can be very complex by itself) and compare its results with other strategies.

You can discuss some of the choices I made in the settings above. Some people on Youtube ask “why use this pair and not that pair”, “why use 50 pairs and not 100” or “why use this time range and not that time range”.

And yes, some of these choices are arbitrary, like the amount of open trades at one time, the pairs and date range. Or the choice between compounding stake amounts or using the same stake for each and every trade. I had to make choices between being as extensive as possible and being practical too. Besides, for example, if a strategy does not work well over 50 pairs, then does using 100 pairs really make a positive difference here..?

And everybody is welcome to test each strategy with their own preferences of course. My tests are just a baseline and further investigation on specific setups are always needed. I’m not asking other people why they chose their specific pairs or their preferred timeframes…

Mechanical circumstances too

Also, the whole idea behind this setup is that the strategy should be tested in a ‘generic way’ under ‘mechanical circumstances’ and no further human interpretation should be necessary. This way I’m also taking all emotions out of trading and hope to find out the raw performance of the investigated strategy.

Secondly, by finding out if a strategy is performing under mechanical circumstances, you can turn this strategy into programming code and use it with a trading bot. With a bot you are always ready for a trading opportunity because crypto trading is a 24/7/365 business.

The last reason is that this way you can backtest your strategy in a mechanical way.
This takes the grinding out of the process and hopefully will give you a good indication on the strategies performance

The backtesting process

The backtesting and judging process is pretty straightforward:

Every time I test a strategy I will:

• Test multiple timeframes with the strategy
• Choose the timeframe with the best results
• Optimize it one more time with hyperparameter optimisation
• Backtest one more time with optimized parameters
• Investigate the differences in performance
• Compare the endresults with earlier tested strategies

Each strategy will be tested on its performance by trading the 50 digital asset pairs over their largest possible backtest period.

I will test the strategy on multiple timeframes, even when it is advised to use it only on one timeframe
Then I will choose the timeframe with the best results. This ‘best results’ valuation is a combination of factors and weights I give to the specific backtest output.

Next, I will optimize it one more time with hyperparameter optimization, and then I will backtest one more time with these optimized parameters.
Beware that, at this stage the strategy will most likely suffer from curve fitting or overfitting. But nonetheless it will give a (sometimes surprising) view on what is possible with the tested strategy.

I will take the ‘best result valuation’ of all these backtests (sometimes hyperotimizaton can give worse performance results) and compare it with other strategies which I have tested earlier. This will be done in a “Strategy League” which I created. It is a ‘one glance’ overview of all the strategies I tested so far and their corresponding ‘best’ performances.

I explained how I ‘score’ a strategy in an earlier post on this platform.

This way I hope to find the best strategy for cryptocurrency trading with a trading bot to attain a nice side income.

Final words

Finally, I assume that you have some knowledge on programming in Python. Can use data science tools like Jupyter notebooks, Numpy and Pandas. Also a good foundation about Trading and its principles is welcome if you like to do the stuff that I do.

But MOST OF ALL it is important that you are willing to learn and to make mistakes. Want to investigate time and money (to gain more in the future). And are enthusiastic and not willing to give up easily because you can spend a lifetime finding the optimal algo trading strategy.
In the end I am sure that we all will find our favorite strategy that will give us a good income to attain the life we want to live.