Install extra TA modules in Freqtrade Docker (Patreon)

Hi There, and welcome to this post where I will show you how you can  add additional python modules, like TA libraries, to your Freqtrade  Docker Image.

This is especially important imformation if you run scripts or  strategies that use Python modules that are not part of the default  Docker Image of the Freqtrade trading bot. In these cases the code will  stop with an error that it cannot find a module. And thus you cannot run  the strategy.

This is a full instruction post, so I will begin with the  installation of Docker, then install Freqtrade Docker and finally show  you how to install additional modules in a custom Freqtrade Docker image  to use with your bot!

I Fully installed this on a Linux Mint machine. I also tried to do  this installation on a Windows 10 WSL VM, but there were so many  problems that I eventually gave up on it. Windows is not my preferred OS  anyway...

There wil also be a video posted on my Youtube channel for those that  do not want to read this post or need visuals to guide them though  these steps.

Now let's begin...

Preparing the system with dependency packages

In this instruction I am going to install Docker Community Edition (CE) in Linux Mint, which uses apt based packages. Based on: https://computingforgeeks.com/install-docker-docker-compose-on-linux-mint/

Any other Linux distribution or Windows can have other instructions  and commands. So please refer to the documentation for these platforms  for the Docker installation.

First I am going to update the apt cache and then install some  additional required software that makes it possible to safely download  the packages from Docker.

To be sure I install the correct package versions, I can remove any already installed Docker packages on my distribution.

This step is not really necessary but can be used if you never  installed Docker or haev a brand new installation. But I consider it  personally as a hygiene factor. And as a way to clean up your system  from packages that are not used, like unused dependencies. And you  should do this only if you are sure that you

After this is done I can add the Docker keyring and repository GPG key for downloading the Docker packages.

Keyring

GPG key

Now I can check if the keyring is installed correctly

If this is the result, then the keyring is installed correctly:

Now update the apt index again with the Docker packages. After this is done, we can install the Docker software.

Download and install the Docker packages

After  we have prepared the system with the necessary Docker source and  keyring, it is time to install the Docker packages, including  Docker-compose, with the following command:

Verify the installation

Now that  we have installed the packages, we can verify that Docker Engine is  installed correctly by running the Docker hello-world image:

Docker daemon not running

It can be that the Docker deamon does not run after the installation and you get the following error:

The way you start the Docker daemon is partially dependend on your  system. In my specific case I had to start it with the following  command:

If everything does work right from the start, we will get the following response back:

Now we can proceed with the next step.

Add a user to the Docker group for management

It is unconvienient to run the Freqtrade Docker image with root privileges every time.

To solve this, we can add the current user to a Docker group which  has enough rights to run Docker containers without having to use sudo.

Execute the following commands to add the current user to the Docker group:

Now run hello-world again, but without the sudo part:

If this works, we can again proceed with the next step.

Docker container and image management

Checking containers

To check all the Docker containers that are, or were in use, execute the next command:

For running containers

And if you don't see any output, for all containers, including the stopped ones:

Checking images

To check the docker images that are downloaded, and take up space on your system, use:

Removing containers

To remove a container that is not in use and stopped, use the command:

The first four digits of the containernumber are enough to identify which container to remove here.

Removing images

Before removing an image, the corresponding container should be stopped and removed first.

Then we can remove the unused image and free up diskspace again:

Here we can use the container name.

Preparing for Freqtrade

Now that  we installed Docker and Docker compose, have checked if it works and  know a little bit about container and image management, it is time to  finally install the Freqtrade Docker image.

Create user_data directory

let's first create a directory that will contain the docker-compose file and all the freqtrade data.

Download the docker-compose file from the repository

Now it is time to download the freqtrade docker-compose file into this directory.

This file will be used by docker to download the correct image and that has all the configurations for a correctly working bot.

Pull the freqtrade image

Now we  can download the Freqtrade docker image to our system. Be aware that,  by default, this will download the bot's most current docker image.

With the docker images command we can check the downloaded image:

Create user directory structure with the necessary freqtrade directories and files

Now  we can really do business and create the userdirectory that the bot  will use for all it's configurations, strategies, data, logs and other  files.

And we can also see which bot version will be downloaded here.

By the way, I use the --rm switch here that will remove  the container after it has been ran. Otherwise Docker will create a  container each time you run a docker command and this can easily build  up to many unused containers on your system.

Create the Freqtrade configuration - Requires answering interactive questions

The next thing to do is to create a new configuration file with the new-config sub command.

Add a pair to the config

After  the config has been created, we have to edit the config file so that it  contains at least one pair for downloading data. The bot cannot backtest  without at lease having some pairs data.

I am going to add the Bitcoin USDT pair in this section:

And I also have to edit this section so that the bot knows that I use a static pairlist.

From:

To:

If you want, you can use a higher heartbeat interval while testing stuff out:

Download pair data from exchange

Now  we are going to test out if the bot can download data from the exchange  by using the following command to download backtest data:

Again, the --rm switch will remove the container after completion otherwise you'll end up with many closed containers.

If you see the following appear on your screen, then you are good to go:

Backtest a test strategy

After the data is downloaded, we can check if it exists in the following directory:

And now it is time to do our first backtest with the bot and the  default strategy that was installed when we created the user directory.

We know now that the bot can download data and do a backtest with a strategy that is located in the strategy folder.

Running the Freqtrade container as a Daemon

let's  now do a quick demonstration where we run the bot in dry run mode.  Which means that it does paper trading with the selected strategy and  config settings.

Edit the docker-compose file

First  we have to check the compose.yml file in the ft_userdata directory. We  do not want to accidentally trade with live money, do we..? It is good  practice anyway to always check this.

First check which config the bot will use when starting the docker container:

You can specify the configuration, strategy and all other freqtrade command line parameters under the "command" section:

Here is says that the config.json in the user_data directory is used.  And that is correct because we just created and configured that file.

It seems that the dry_dun option is set to true. And that there is no key and secret set. So it seems that we are safe here.

Starting the Docker container

To start dry run trading with the freqtrade bot in deamon mode, just use the command:

The -d switch causes Docker to run the container in the background. Hence the daemon mode.

This command also automaticalle reads the compose file in the  directory, with the commands and other parameters set. That's why it is  important to not change the docker-compose.yml filename here.

Checking the log file

To see if Freqtrade is working, use the next commands:

The Docker processes running with:

The Docker container logs:

The Freqtrade log:

It looks like the bot is working great here.

And we can even see some trades happening as we speak.

So now we are ready for the next phase of this video and that is to  add additional python modules to this Docker image so that is also can  run strategies that depend on packages that are not installed by default  in the image.

Stopping the container

To stop the container with the running freqtrade bot, we first have to get the container ID that is running:

Once figured out, use it in the command to stop the bot. You do not  have to specify the complete container ID, the first 4 or 5 characters  will do as well:

Check if the instance is stopped:

Running a strategy with additional library requirements

'This is where the fun begins'

Sometimes a strategy requires additional libraries like finta or ta. These are not installed by default in the Docker image.

Lets first install a strategy of which I know it uses the technical  library 'ta' that is not installed by default in the Freqtrade  container.

To get a first glimpse of the unavalable module, let's do a list strategies to see which strategies we have available.

In this example the GodStra strategy has status LOAD FAILED.

The log also gives a warning that GodStra cannot be loaded because of a missing 'ta' module.

We can also do a backtest with this strategy and the original Docker container:

But as you can see, it exits with the error that it cannot find the ta-module.

And naturally you also cannot run this strategy in dry-run or live trading mode ofcourse.

Installing modules on the host system

Installing this library on the host system with pip install ta does not solve this problem. The Docker container is an isolated  environment and does not use its hosts system resources or modules.  The  only solution here is to install the module within the Docker image and  then run the container with that newly installed module.

Adding/Installing additional modules in image

To install additional modules in your image do the following:

Preparing the files

First we are  going to modify the docker-compose file so that it creates a custom  image and uses an additional file which I call Dockerfile.custom.

I will first make a backup of my docker-compose like this.

Then I will edit the docker-compose. I will call the new Docker image  "freqtrade_custom" and point to a file called ./Dockerfile.custom.

Now it is time to create the Dockerfile.custom. The reason why I use  ./ is that I prefer to keep the docker-compose file and all  customiztions in the same location. Which is easier for me to manage.

The Dockerfile.custom file will contain the following lines:

The first line tells Docker where to get the initial Docker image.  The second line is used to install additional modules in that image.

The second line will install two additional Python modules that are  often used with other trading strategies and that are not installed by  defaultin the Freqtrade image.

Create the Docker image

Now that the files are ready to use, lets use the following command to create the new custom Docker image:

If you watched closely, you could see that during the build these modules were installed additionally.

We can now check the existence of our own custom image with docker images

Checking for success

Now we can try  the list-strategies command again to see if the strategy now will be  accepted, since the ta library is also installed in the image.

And behold, no error anymore. The freqtrade_custom container is used  with the additional modules and freqtrade can find these to run and  check the strategy:

Now we can also run a backtest which will not error out on us anymore.

And as you can see, backtesting the strategy works as well here as well as dry-run trading. As you can see next:

Finish

And with this we are at the end of this  post. I hope this was informative and you are now able to run strategies  that require more exotic, not by default, installed python modules in  the Docker container.

Thanks for reading and being my Patron and see you in the next post!

Goodbye!

Files

The full docker-compose.yml file

The full Dockerfile.custom file