Support as a front-end for multiple instances of a given torrent client.

[C84186]

Hi there.
I'm not certain about the feasibility of this, as maybe flood's infrastructure wouldn't be sufficient, but after a certain number of torrents, most client's web-uis (idk about their native UIs) tend to get really sluggish.
qBittorrent does this, on both the standard webUI & on alternate UIs.

For me, qBittorrent hits that wall at, idk, 5000 ubuntu isos. It's frustratingly slow.

From what I've seen, the only solution that seems to address this is horizontal scale-out by running concurrent instances

https://www.reddit.com/r/DataHoarder/comments/3ve1oz/torrent_client_that_can_handle_lots_of_torrents/

A few people on the above thread discuss their approach to this, and it seems as though many were running parallel instances of whatever torrent client they're using via docker-compose, apparently managing these instances via a couple of shell scripts.

That feels... annoying to me, and I was wondering if it would be possible for flood to aggregate multiple instances of a client?

If that feature doesn't sound appropiate for flood's roadmap, maybe sometime I'd implement my own proxy aggregator / hypervisor that translates single instance qBittorrent queries into multiple-instance queries, and aggregates the result so that a client like flood, or an nginx backed alternative api could just talk to that proxy (no idea if that'd solve anything tbh, but its a nice thought).

What do you think? Is the bottleneck for huge seed amounts the frontend, or something in the backend clients?

[jesec]

Flood implements state-of-the-art windowing, memoization and incremental data fetching, so it is probably more scalable than the bundled web UIs of torrent clients. Please let me know if you encountered severe performance issues with Flood.

It is unnecessary to implement such "aggregation". Flood supports multi-user with multi-client. You can simply add an additional user that connects to a different client.

Additionally, torrent clients generally have their own limitations. I actually did weeks of research in order to find the best fit for my use case.

In my experience, qBittorrent cracks under pressure. With a lot of torrents and long uptime, sometimes it just "stuck" with zero upload/download. As the number of torrents grows, its resource consumption (I/O, CPU, memory) skyrockets. Not mentioning that from time to time it randomly gives you "checking all torrents" heart attacks. And, it is almost impossible to gracefully shut it down in a reasonable amount of time when there is a lot of torrents. However, it does work quite well with smaller number of tasks, so I primarily use it for RSS/Sonarr and public torrents that I am not going to seed for an extended period of time.

As for the Transmission, the resource consumption is low and the runtime stability is quite good. However, its performance, when compared to other clients, is less than ideal. It often fails to proactively connect with other peers and compete to leech/seed data. It might be a good choice if you don't care about ETA or ratio, though.

rTorrent is the only torrent client that truly works (for me). It is known to be a bit difficult to configure, but I think those efforts pay off in long-term. rTorrent can easily handle hundreds of torrents with tens of terabytes of associated data, and it is known to be able to handle thousands (some people are seeding 15000 torrents simultaneously with 1 single instance of rTorrent). I am able to run rTorrent 24/7 continuously for months without being forced to sudo service qbittorrent-nox restart, and it is easy to find rTorrent users who are seeding for a whole year non-interrupted in the peer/seed section of a torrent. The most amazing part is that the performance remains very good and the resource consumption remains low. It is the most stable torrent client I have ever seen. It also has the best-in-class performance that does not fluctuate. Initially I was worry about the lack of new changes/releases, but now I am convinced that rTorrent is just stable. You should try it out if you are a heavy seeder.

[C84186]

Thank you for this in-depth explanation + discussion.
It sounds like what I need to do is fairly obvious, assuming I can run rTorrent in a container that funnels traffic to my VPN, whilst still exposing the webUI locally ( I use guillaumedsde/alpine-qbittorrent-openvpn presently).

I look forward to yet another torrent client migration, haha .
Hopefully there's some utility out there that makes it easy.

[jesec]

Flood’s supports adding completed downloads without hash checking. Flood also supports downloading .torrent file.

And, as you may know, Flood supports both rTorrent and qBittorrent. You can add two users that connects to qBittorrent and rTorrent respectively. You can either do a mass batch add via web UI or make some scripts that interact with Flood’s APIs directly.

[C84186]

or make some scripts that interact with Flood’s APIs directly.

I've done as much for qBittorrent, and written machine migration scripts that actually sftp the files to a remote machine, and then add the torrent to qBittorrent.

Migrating to rTorrent on the same filesystem is probably going to be easier than that. Thanks for the advice.

[mrhotio]

Thank you for this in-depth explanation + discussion.
It sounds like what I need to do is fairly obvious, assuming I can run rTorrent in a container that funnels traffic to my VPN, whilst still exposing the webUI locally ( I use guillaumedsde/alpine-qbittorrent-openvpn presently).

I look forward to yet another torrent client migration, haha .
Hopefully there's some utility out there that makes it easy.

https://hotio.dev/containers/rflood/

[C84186]

@mrhotio , that container looks very convenient, but unfortunately my VPN provider does not support wireguard, so I'll have to use some kind of openvpn container.

[C84186]

While we're discussing qbittorrent's performance WRT flood, I'd like to flag an issue & check if you think the reason is something besides client performance.

I run qbittorrent via guillaumedsde/alpine-qbittorrent-openvpn.

Some details:

• I have qBittorrent categories enabled, and use automatic torrent managment. I know this is not supported
• I have a lot of torrents, as we've discussed

I've noticed that flood is not really letting me interact with my torrent list (at this point)- I can see the speed graph, but not much else.

Do you have any idea why this might be? Do you think its related to the two issues mentioned, or its something else?

[C84186]

btw, here is my docker compose

  flood:
    <<: *restart
    <<: *logging
    image: jesec/flood:4.4.1
    hostname: flood.${root_domain}
    # cpus: 2.0
    mem_limit: 500m
    # read_only: true
    user: ${UID}:${GID}
    volumes:
      - ${docker_data_folder}/flood:/config
      - ${data_folder}:/data:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - traefik
    command:
      - --auth=none
      - --qburl=http://qbittorrent:8080
      - --qbuser=${QBT_USER}
      - --qbpass=${QBT_PASS}
      - --rundir=/config
      - "--allowed_path=/data"
    environment:
      HOME: /config
      UID: ${UID}
      GID: ${GID}
    healthcheck:
      test:
        - "CMD"
        - "wget"
        - "--quiet"
        - "--tries=1"
        - "--spider"
        - "http://localhost:3000/"
      interval: 10s
      timeout: 2s
      retries: 5
      start_period: 40s
    labels:
      <<: *traefik
      traefik.http.routers.flood.rule: Host(`flood.${root_domain}`)
      # service
      traefik.http.services.flood.loadbalancer.server.port: "3000"
      # Forward auth

[C84186]

Double check client connection settings. What’s your qBittorrent version?

qbittorrent is 4.3.1 , and it seems to connect okay, since flood will happily fetch the speed graph & generate the snapshots.

Flood doesn't seem to be writing to its own settings, however (in the /config mount point), or many other of its config files

[jesec]

It is --allowedpath not --allowed_path. Double check permissions. Check Flood's logs. Additionally, try with a clean qBittorrent instance and see if things work. Flood is tested against real torrent client instances, and this is covered by one of the tests. Such issue is often caused by misconfiguration.

[C84186]

Thank you for all your help.
Floods logs are uninformative in this case. I can test against another qbt instance.

I am sure it's a configuration issue, though the part that confuses me is where and how. I'll fix my --allowedpath flag

[C84186]

Hi @jesec , I just wanted to update you that, for now, forwarding my qbittorrent port to my LAN IP address, and then using that for --qburl in flood worked- It looks like there was some problem with the two containers routes, probably because of the very strict iptables rules on qbittorrent.

So you can safely conclude it wasn't your problem. Now I can verify your claim that:

Flood implements state-of-the-art windowing, memoization and incremental data fetching, so it is probably more scalable than the bundled web UIs of torrent clients.

WOW flood is so much more responsive than any other web client I've used. I've wished for a client that doesn't hang on every input, and finally one exists, with one of the most proactive maintainers I've encountered.

Thank you for your excellent work.

[C84186]

10 days later, I am lucky enough to experience first-hand qBittorrent's catastrophic failure in the face of system fluctuations.
Either because the download partition ran out of space, or because my server had a power failure, all but very few of my .fastresume files have now been replaced with empty files.

I will be accellerating the transition to rTorrent, I think