A Question

[malixi94]

No description provided.

[malixi94]

Hi Denis,

I noticed the ElementumOrg/lt2http libtorrent-to-http C++ webserver. It has not been updated in three years, while development of the Elementum Go daemon is currently active. Has the lt2http functionality been fully subsumed into the Elementum server? Or what's going on there?

Thank you

[elgatito]

Has the lt2http functionality been fully subsumed into the Elementum server? Or what's going on there?

Opposite. Changes from plugin.video.elementum need to be moved into lt2http and catch up all the latest changes/fixes with settings.

I hope to complete update.

[malixi94]

Opposite. Changes from plugin.video.elementum need to be moved into lt2http and catch up all the latest changes/fixes with settings.

I don't quite understand the difference between the two projects.

Elementum - though it is a plugin for Kodi - seems to do all the things lt2http does. It (in main.go) starts up a web server that serves requested torrents as http streams (using elementum/bittorrent package). Why maintain two distinct projects in two different languages when you could reduce the effort to one?

What does lt2http have that Elementum does not?

[elgatito]

What does lt2http have that Elementum does not?

Eventually torrenting functionality will be removed from Elementum and lt2http will be used instead.

[malixi94]

What does lt2http have that Elementum does not?

Eventually torrenting functionality will be removed from Elementum and lt2http will be used instead.

Is it because you want to factor out the complexity of torrent handling from Elementum?

But then why write lt2http in C++ when you could also use Go like your other projects?

[elgatito]

Is it because you want to factor out the complexity of torrent handling from Elementum?

It is to make Elementum lighter, easier to support, ability to use torrenting externally/centrally.

But then why write lt2http in C++ when you could also use Go like your other projects?

Because libtorrent library (https://github.com/arvidn/libtorrent) is originally in C++ and Go is using Swig to operate with that library. It brings memory issues and maitanence issues as well. It is just easier to have it separately and natively (and it would work faster).

[malixi94]

But then why write lt2http in C++ when you could also use Go like your other projects?

Because libtorrent library (https://github.com/arvidn/libtorrent) is originally in C++ and Go is using Swig to operate with that library. It brings memory issues and maitanence issues as well. It is just easier to have it separately and natively (and it would work faster).

Okay then what about using anacrolix/torrent? It is actively maintained Go library that has a torrent Reader interface so you can use it for streaming without doing a lot of dev work, and you would be using Go for your torrent web server, making things easier.

[elgatito]

Okay then what about using anacrolix/torrent?

Elementum was using anacrolix/torrent in version 0.0.*, you can take a look and use it if you want.
There is no plan to return back.

#351

You can find information about reasoning and so on.

[malixi94]

Elementum was using anacrolix/torrent in version 0.0.*, you can take a look and use it if you want. There is no plan to return back.

Ah I did not know this!

#351

You can find information about reasoning and so on.

I could not find any reasoning for the change. Most of that discussion is not in English (Russian?)

In any case it's been 5 years now. A lot of things have changed. anacrolix has just landed BitTorrent v2 support and is actively maintained (this cannot be said of libtorrent.) And a lot of bugs have been fixed I'm sure. So if that was the reason for moving away from it as a buggy new project, why not reconsider? (Unless there were other reasons)

It is always prudent to continuously reassess things over time depending on the environment as that ensures the best outcome for the project. (Edit: There is now even a fork of qBitTorrent using anacrolix so it has come a long way!)

[elgatito]

For sure there will be no migration done by me.
There is always a possibility to fork/contribute.

[malixi94]

For sure there will be no migration done by me. There is always a possibility to fork/contribute.

What is the reason(s) though you moved away from anacrolix back then?

[elgatito]

What is the reason(s) though you moved away from anacrolix back then?

Bad speeds when working with files and when having many files in a session.

[malixi94]

What is the reason(s) though you moved away from anacrolix back then?

Bad speeds when working with files and when having many files in a session.

Ah ok, this is not something anacrolix/torrent would have likely improved over time, as it is in Go, with its characteristic concurrency model and garbage collection affecting performance (correct me if I'm wrong @anacrolix)

So Denis, in terms of porting things into lt2http: Over the past three years with the elementum/bittorrent package, have you added features to Elementum's torrent handling and streaming algorithm that make it materially better at torrent media serving than what was in lt2http, or is it just bug fixes in the Go implementation and compatibility updates to keep up with libtorrent over the years?

[anacrolix]

I appreciate the thought @malixi94. Denis was a very early adopter of anacrolix/torrent. I think Elementum stopped using it around 2018/2019. There were some huge performance improvements in 2021 that probably addressed many problems, so I don't know where it stands now relative to the issues Denis had back then.

[malixi94]

I appreciate the thought @malixi94. Denis was a very early adopter of anacrolix/torrent. I think Elementum stopped using it around 2018/2019. There were some huge performance improvements in 2021 that probably addressed many problems, so I don't know where it stands now relative to the issues Denis had back then.

@anacrolix what about when there are like a thousand torrents in a session though, doesn't the speed and memory consumption significantly deteriorate?

[elgatito]

@anacrolix Nice to see you here. Cheers! 🍺

@malixi94 That is great to hear about anacrolix/torrent improvements. I was continuing following the changes and work there.

The problem with migrating back from libtorrent is that needs huge amount of time and time to solve all incoming issues reported by users.

[malixi94]

@elgatito looking into the lt2http code, you define your own Reader class which manually cobbles byte buffers together using readv calls through a libtorrent custom storage interface, and then hands it to the web framework for serving.

This implementation presents problems in libtorrent 2.0, as was made clear in #6118 since running your readv-calling client code callbacks in the network thread is not safe, according to Arvid. He suggested you use read_piece() on the torrent handle instead, but that has its own problems.

Have you added features to the elementum/bittorrent package over the past three years that has made its torrent handling and streaming algorithm diverge from that of lt2http?

Hmm even if this is the case I'm thinking that the additional code that needs to be ported to lt2http is miniscule compared to porting the lt2http implementation itself to using libtorrent 2.0 (same goes for porting elementum/bittorrent to 2.0) Is my intuition correct Denis?

@anacrolix I searched in your torrent repo, there was an issue (#820) where having more than 10 torrents caused problems, but this seems to have been fixed by you, which is good.

But there was another issue (#839) - open for nearly a year now - where having 30 torrents caused apparently the same sort of problems of "high load". So is 10 to 20 torrents at a time the high mark for anacrolix/torrent? Because libtorrent-based clients do 1000 torrents in a session without issue.

[malixi94]

@elgatito how do you figure lt2http should be ported to libtorrent 2.0?

I don't know if you've thought about it, but can it safely somehow get the torrent data for streaming without using read_piece()? Or would libtorrent itself have to be changed, its interface, to accommodate apps like l2http, to make things easier?

What do you think is the best course of action?