So I improved the Strategy League, again - 2023 version... (Patreon)

The last couple of weeks were quite busy because I wanted to prepare myself for the upcoming strategy testing sessions/videos. 

To start with the beginning: After my extensive video tutorial about building a algotrading bot with strategy creation and testing, I also wanted to update my own personal bot with the latest Freqtrade trading software. This way I could enjoy the latest features and maybe dispose of some bugs. 

Then I found out that the development version for 2023 provided some very interesting new features that could help me analyse the trading strategies even better. Together with my 'fun video' on how the top 15 strategies would perform if you used them in 2022, I thought to myself "Why not do a full recalibration of all the strategies over all the backtesting period available to see if my initial tests were correct". At that time I did not realise that this was much more work than anticipated...

Updating the dataset

Updating the dataset proved not to be a problem because I already did this when I updated the config files for spot and futures trading. I just had to double check and make sure this really was the correct data. 

Gathering all the strategies tested

This was a slightly bigger challenge because during the last year I collected a LOT of different strategies. For some reason I did not kept the already tested strategies in a different folder (dump mistake here...). Walking trough my testing material, video, posts and more was necessary to collect all the code that I already tested. 

Luckily I already filtered out the code of the top 15 strategies. But now I had a decision to make...

Decision time 1: Can I compare newly tested strategies with the 2022 dataset included honestly with strategies that never suffered that fate?

Since I already backtested the top 15 strategies over 2023 for my video I could leave it at that and just start a new series of algo strategy testing videos. But now I had a dilemma. If I wanted to test a new strategy, should I just use the old dataset that did not include 2022 or should I just include it and continue as if nothing happened...

Now hence my dilemma: "can I really compare a algo strategy that includes data from a year's long bear market with one that did not suffer that fate?"
Is this fair against the newly tested strategy and can I honestly add these to a list of strategies that never experienced a recently year long drawdown?

So, therefore I decided to retest the top 15 strategies with the complete dataset. Including 2022!

Decision time 2: Should I redo only the top 15 best algo strategies or all strategies..?

The second question followed quickly...

Would it be fair to the top 15 strategies in the league if I only tested these strategies with the complete updated dataset. What if they would perform poorly and therefore had lower scores in comparison with the earlier tested 'bad' ones.
These would then have relative better scores in comparison with the best strategies and I'd still end up with the same problem.

So I became bold and collected all the strategies I tested earlier and retested all of these over the complete dataset.

Optimisation dilemma

Now things became somewhat problematic because (almost) all these old strategies were optimised with the hyperopt function. 

Should I redo all these optimisations also? 

Did I really want to put so much time and effort in re-optimising these strategies over the 2022 dataset as well?

Here I reached my limit of putting effort into the recalibration. This would take way too much time and it would not prove anything more than also curve fitting the strategy to last year's bear market as well. 

Therefore I made the following bold decision: I will backtest (recalibrate) all the strategies against the updated dataset and in the mean time test if earlier optimisations prove to be beneficial to the strategy as well. 

So keep in mind that all future tested strategies are optimised against the updated dataset but these older tested strategies are not!

Nonetheless, there is so much data available at this point that I assume that optimisation can only be so much. Maybe I will do an experiment later to test if an old strategy optimized against the new dataset will be that better. But again, this will take me into the realm of overfitting...

New freqtrade features

During these backtests I discovered that the Freqtrade developers were implementing some very interesting features that would help me compare these backtest results even better. This was somewhat of a blessing in disguise because if I did not made all this effort in decision making, then I probably would discover these features in the new stable version and had to redo everything again... :-S

But luckily for me I discovered that the following strategy performance analysis features become available in the future release of Freqtrade:

• Sharpe ratio
• Calmar
• CAGR
• Sortino
• Expectancy

These ratios and rating tools make comparison between multiple strategies more granular and finetuned. But on the other hand provides me with a luxury problem:

Which ratio to use for rating a strategies performance?

Now there are 5 additional rating tools to choose from besides my original profit factor, drawdown, winrate, Risk of ruin and applicability. 

What's more, some of these tools are derivatives or replacements over the other. For example: Sharpe and Sortino for example both evaluate the performance of an investment relative to the downside risk. But each in their specific own way with pro's and con's. 

I will add more information about these rating measures on my dutchalgotrading.com website as a source of reference. But for now keep in mind that these are in the Freqtrade development branch at the moment of this writing.

The big question with all these additional measures is if I should use one over the other, or both or none?

My solution

My solution to this problem is both simple & clever. But also highly subjective.

This idea is what i call the "rate the rating" principle...

if you are familiar with one of my backtesting video's, then at one point I rate a strategies performance by giving it a certain score and then place it on my "Strategy League". This scoring is done by giving each performance factor a "sub score" and then add all these numbers up to calculate the final score.

Now, in this post I will not go into much detail on this scoring. However I will make a post/video exclusively on my Patreon about this subject later this year. 

What I did was adding all these different additional metrics to my spreadsheet and rated them with a factor, accordingly to their performance and my preference to certain metrics. 

Then at the end I compounded all these scorings to a definitive number. Add with this final score I reorganized the strategies in the league.

If this sounds a little abstract, just know that just scored each score by weighing them with a certain factor. These factors are the subjective part because my preferences can be very different from yours. So my "League" can look different from yours. But in the end the best performing strategies always end up high in these rankings.

Recap

So to conclude this post about my recalibration of the strategy league. Just know that I did a backtest of all these strategies with the updated dataset (including 2022). Over 50 other pairs (jan 2022) than the previous test and that I added the new Freqtrade strategy performance metrics to the valuation as well.

Other slight changes I did not mention yet are: I removed my risk of ruin calculation because it proved to be inconsistend with compounding stake amounts, tested the "Slope is Dope" strategy without the buggy trailing stop loss and removed the "Golden / Death cross (LR)" strategy).

These new test results changed the order of the league slightly but not that much

The old Strategy League (pre 2023)

And the new updated 2023 Strategy League

The difference between the old and new order:

The most remarkable thing is that with these backtests and new ratings some lower scoring 'bad' strategies suddenly performed way better than tested earlier and vice versa for some better performing strats (the "slope is dope" strategy here was tested without the weird performing TSL so that one is explained).

At this moment I have no clue why this happened and maybe I will investigate this further. Maybe bear market circumstances apply better to these but that is just a guess at this point.

Conclusion

What can we take from this recalibration?

All in all, this excersise shows once again that well performing strategy can suddenly perform bad under different market circumstances. And a bad strategy can perform way better than expected under these same circumstances. 

Adding more metrics to determine if a strategy is good for use is not always the better option, but it helps me (us!) to put things more in perspective. And since other traders also use these metrics we have something as a reference to compare these with each other. 

However, I am not certain if I want to do this again next year with an updated dataset because it is way more work than anticipated. My main goal is to compare each strategy with each other and that is not possible when the data or metrics change each time because it can make a newly tested strategy perform better or worse in comparison with older tests under other circumstances.

Also, this time I had to retest 28 and If I double the amount of tests this year, it means that I have to test 62 strategies again. I wonder if that would really give us new insights...

Dataset

Espesially for you, my patrons, I also added all the output I got from these tests into one single file so that you can also check for yourself if and where I made mistakes. You can also use these datasets to check your own tests or want to analyse a certain strategy further if you are interested.

Thank you for your support and see you next time!