Information request about Webseed within this library (BEP 19)

[abhishek-das-gupta]

Hi @cenkalti,

I'm exploring options for a new torrent client library in Go to replace libtorrent, which I'm currently using. My codebase heavily relies on the performance of webseeding to distribute a 13 GB file within a cluster using the BitTorrent protocol.

Our cluster topology typically includes a master node with several worker nodes that report to the master at regular intervals via heartbeat signals. All nodes are located within one or more racks.

During torrenting, the master node acts as the webseed. A rack leader is selected in each rack to perform an HTTP(S) download from the webseed, while simultaneously distributing the pieces to its rack peers.

Currently, we're using anacrolix/torrent, but its webseeding is inefficient because it doesn't implement BEP 19. As noted in this comment, when I use this library in a rack-aware cluster, the torrenting process stalls at 0% progress. The rack leaders within each rack are unable to download pieces from the webseed (master node). This same operation works smoothly with libtorrent.

Could you please confirm whether your library efficiently implements BEP 19 for webseeding?

[cenkalti]

Hi @abhishek-das-gupta,

Yes, Rain implements BEP19. Although BEPs are called BitTorrent standards, they are not strict rules and leave most decisions to client developers.

I understand your issue with anacrolix/torrent. From the logs, it looks like it makes separate requests to download each piece. This can hurt the total download speed due to latency.

In Rain, I tried to reduce the number of requests to the webseed source when implementing Webseeds by using a similar algorithm to BEP19.

I think Rain is suitable for distributing a torrent from a single webseed source. Please try it out and let me know if it works.

Make sure you're using the latest version because I've just sent some improvements today.