Feature preview: concurrent downloads limit tripled, but please use it with caution! (Patreon)

Hello supporters,in the upcoming nightly build, there will be an important new feature for you: the concurrent downloads limit in the program will be tripled, increasing from a maximum of 10 to as many as 30.So, I decided to write a couple of personal and important notes about this new possibility.

First of all, why? Well, I increased the download limit because it had essentially become the most common request from subscribers, and I genuinely felt bad having to decline it every time. However, please read the following carefully and don’t make me regret this choice 😅

Each download operation may look trivial from the outside, leading you to believe it costs nothing — nothing could be further from the truth. Every download spawned by the program runs in its own thread with a long, complex routine that handles countless tasks: sending web requests to servers to retrieve images, often decrypting them (since most sites now use image protections), performing complicated image manipulation and calculation steps in memory, and finally writing them to disk — frequently converting formats (PNG, JPEG, PDF…) and applying compression (ZIP, JPEG, PDF…). Now multiply all of this by dozens (sometimes even hundreds!) of pages, depending on the chapter. As you can imagine, this puts a heavy load on your system resources (CPU, RAM, and disk). Starting 30 of these threads at the same time could easily slow down or even freeze your PC, especially on machines with very low resources.

On top of that, it should be obvious that sending 30 simultaneous web requests to the same site will likely trigger its protection mechanisms. Servers start returning HTTP errors (503, 404, 403, and similar), or in more extreme cases, ban your IP. And even if a site doesn’t block you — perhaps because it’s a small one — bombarding it with dozens of requests at once will still put it under significant stress, so please always think carefully about what you’re doing, and don’t be selfish. Additionally, please note that running 30 parallel background operations drastically increases the chances of encountering access/write or download errors, which may result in corrupted files on your disk. So keep in mind that downloading many files at the same time does NOT necessarily mean finishing the download procedure sooner. Due to system and network slowdowns, as well as download or write errors that may force you to restart some downloads from scratch, sometimes you end up taking even longer than you would by downloading just 4 or 5 at a time!

For all the reasons above, I strongly advise against using more than 5 simultaneous downloads. If you choose to go beyond that, please be aware that you may frequently encounter errors, corrupted files, unexpected program behavior or IP bans from the website you’re downloading from; it’s all at your own risk, I assume no responsibility for your actions or for any consequences that may result from them. For this reasons, I kindly ask you not to report network or download errors when you intentionally force more than 5 concurrent downloads. The program was never designed for such scenarios, and the potential causes of failure are numerous (as explained above). So before telling me "the program fails on this site", please make sure the issue isn’t simply caused by abusing this feature rather than a real problem with the framework.

Lastly, a final note on why there’s still a hard limit: I’ve tested even higher values, and you start encountering errors almost 100% of the time once you exceed that threshold (and with many websites, often even before!). Therefore, having more than 30 is completely pointless and counterproductive. This limit will not be removed or increased any further, so please refrain from requesting it.