Webtorrent support

[Zorlin]

It would be crazy cool if like Aquatic this supported webtorrent.

[Power2All]

@WarmBeer Can be closed, it's a protocol that is already supported by Torrust-Tracker.
WebTorrent is a protocol for "clients" in JavaScript for example, but does still announce and scrape a torrent server using HTTP/HTTPS, but I'm unsure if UDP is supported anyhow.

[Tails]

I came here because I was trying to use https://github.com/dmotz/trystero with Torrust. However, typical for Webtorrent is that the announce endpoint is available over a Websocket transport, which for Torrust seems not to be the case, so I would vote for re-opening the ticket. I would much prefer to be able to run Torrust over Aquatic.

[Power2All]

@josecelano There is no BEP for it, there is also no "draft" for the BEP either.
Besides, I firmly believe Websockets should never be used for Announce and Scrape requests. They're an overhead compared to HTTP/HTTPS and UDP announcements. As long as there is no BEP for it (or let stand a draft), I wouldn't bother using it yet. WebTorrent works with the normal URL's just fine.

[josecelano]

It seems there are other trackers implementing "webtorrent trackers":

• https://github.com/greatest-ape/aquatic/tree/master/crates/ws
• https://github.com/greatest-ape/aquatic/blob/master/crates/ws/README.md

And projects using these webtorrent trackers:

• https://github.com/ThaUnknown/miru (https://github.com/search?q=repo%3AThaUnknown%2Fmiru%20tracker&type=code)

It looks like the intention was to publish a BEP:

WebTorrent Trackers

• https://openwebtorrent.com/

WebTorrent Trackers URLs using websockets

wss://tracker.openwebtorrent.com',
wss://tracker.webtorrent.dev',
wss://tracker.files.fm:7073/announce',
wss://tracker.btorrent.xyz/

I can't find information about how to connect to them, but I guess I can check Aquatic implementation. Maybe @greatest-ape could clarify:

• If it makes sense
• What changes you need to implement for a webtorrent tracker
• Poting us to more formal official documentation

@Power2All do you know what was the reason why people started using them even if it's is an overhead? Even if it makes no sense, I would like to know the reason. I think BitTorrent has a LOT of missing documentation, missing official documentation BEP, lack of coordination between projects, etcetera.

[josecelano]

webtorrent/bittorrent-tracker#354

[Power2All]

@josecelano Reason is unclear, but I think it was because everything else used websockets for transferring blobs, why not also make announce and scrapes using the same protocol. However, it's a overhead, as a websocket first creates a HTTP/HTTPS connection, and then uses a switcher with a request to change the protocol into a websocket. Websockets are always active "tunnels" so to speak, but announcements and scrapes are only done ones every 30 or 45 minutes or so. There is no point in using Websockets for that reason.

It seems this was just an idea of someone randomly, and made it work, but since there is no official BEP, I think the BitTorrent document holders didn't seen any fit of using Websockets, probably also because of this useless overhead it creates. Besides, WebTorrent also works with normal HTTP announcements, so I see no reason why WSS should be used whatsoever. UDP is not supported by the browsers anyhow.

[ThaUnknown]

hey, webtorrent dev here

the reason for websocket trackers is quite simple, it needs to run in browser

we don't really have the kind of HTTP you need for trackers in browser, only fetch which doesn't allow low-level control that's needed for the communication you need

webtorrent does not work with normal http announcements, not in browser at least, in node.js and deno it does, but not in browser, as HTTP trackers return IP's, which as a concept don't exist in a browser, instead the WSS tracker returns WebRTC offers which is the only protocol we have in browser that's usable for P2P

there's no BEP because the H@ that knows pretty much everything about webtorrent, and the only person capable of writing a BEP is MIA

[greatest-ape]

I can't find information about how to connect to them, but I guess I can check Aquatic implementation. Maybe @greatest-ape could clarify:

• If it makes sense

I’m suspect that not a lot of people use webtorrent and implementing it takes quite a bit of work.

• What changes you need to implement for a webtorrent tracker

It differs quite a bit from UDP/HTTP BitTorrent. It is based around having the tracker pass WebRTC SDP “offers” and “answers” between peers instead of returning peer IPs in announce responses. (At the very least, this was how the reference implementation did it a few years back when I read it to implement aquatic_ws)

• Poting us to more formal official documentation

As far as I know, there is none. I had to resort to reading the (unfortunately quite messy) reference implementation.

[Power2All]

@ThaUnknown But, WebRTC and WebTorrent still retrieve IP's and ports from the same tree, since Bittorrent clients also start to support WebRTC out of the box, so it's kind of pointless to split that up anymore. When it first got introduced, would have been better that the clients would send a extra argument in the HTTP/HTTPS request that they only support WebRTC. Using WSS is just, a improper way. Adding something like webrtc=1 in the client request to trackers, would be a better way to deal with it.

NAT punching, IPv6 and encryption is also already supported in Bittorrent itself, and even more, so I don't know what else WebRTC improves over the current Bittorrent technique, it does support also TCP and UDP traffic using WebRTC (I didn't knew about WebRTC supporting UDP), which is pretty cool. Other then that, using websockets for just announcing and scraping, I would say, nahhhhh.
I'm waiting for the developers at WebTorrent what good reason they have for using it, but so far I'm still not convinced.

[SilentBot1]

@Power2All WebTorrent does not use the standard swarm for WebRTC peers. WebRTC peers must maintain their own swarm for any given infohash, as the tracker itself does not exchange IP:PORTs, it exchanges WebRTC SDP offers/answers, which while contain multiple IP:Port mappings in the offer (per-interface, IPv4 and IPv6), can not be created from only an IP:PORT without faking SDP offers.

WebTorrent as a library can speak to UDP and TCP clients when, and only when, operating within NodeJS or Deno, as it acts as a standard BitTorrent client. The WebTorrent library when running in a browser can not open arbitrary TCP or UDP ports, so can not speak with TCP or UDP peers, or TCP or UDP trackers, so acts as a WebRTC peer only client. WebTorrent in the browser can speak with HTTP (and HTTPs) trackers, depending on the browsers security context, but the current implementations of the trackers do not support SDP offers.

From my understanding in the beginning when designed by Feross, WebSockets were used for real-time communication as this immensely speeds up peer discovery for WebTorrent, and due to the short lived nature of the WebRTC SDP offers (WSS trackers announce by default every 120 seconds) which require two way communication, explicitly having a real-time announce + offer and answer process prevents wasting these short lived, limited, single use and expensive to generate SDP offers.

As a note, both ThaUnknown, and myself, are maintainers over at WebTorrent, so if you do have any further questions, please ask away.

[Power2All]

@SilentBot1 Thank you for your explanation.

Is the reason the announcements are so many in a short time frame, because WebTorrent is missing out the DHT functionality ? How come it needs to do a announce every 120 seconds, this feels like too fast in my opinion. Would like to know the reasoning behind it.

SDP offerings, isn't that just not the protocol what browsers/clients talk with each other, and is completely separated from how it speaks to the announce/scrape tracker ? If so, wouldn't the argument in an HTTP/HTTPS request be enough as well to only retrieve peers that are WebRTC related ?

Just trying to understand the thought about it.

[ThaUnknown]

no, it's because of the nature of the WebRTC protocol...

no it's not separated, the tracker connects the peers, so it exchanges the offers, it requires an unique offer AND answer, where as http trackers only have 1 non-changing IP, which is ONLY an answer

[SilentBot1]

@Power2All

Is the reason the announcements are so many in a short time frame, because WebTorrent is missing out the DHT functionality ? How come it needs to do a announce every 120 seconds, this feels like too fast in my opinion. Would like to know the reasoning behind it.

RTCPeerConnection offers rely on the underlaying RTCIceCandidate, which is used to perform NAT hole punching to allow two peers to directly communicate. Typically this hole punching is only valid for as long as your router performs connection tracking (keeping the port open), which for many routers varies between 300 and 600 seconds for TCP/UDP connections from my understanding.

Due to how short lived these offers can be, they are announced frequently, to ensure they have an optimal chance of being consumed, combined with the fact that the offer list is only updated with new offers, after an existing offer is answered and consumed (or expires!), only at the announce interval from my understanding. (Note: these new offers can still be returned to other peers directly via the tracker, before the announce interval using the announce/offer/answer messages).

SDP offerings, isn't that just not the protocol what browsers/clients talk with each other, and is completely separated from how it speaks to the announce/scrape tracker ? If so, wouldn't the argument in an HTTP/HTTPS request be enough as well to only retrieve peers that are WebRTC related ?

SDP is not related to how the client speaks with the tracker, this is WebSocket, but SDP offers are sent over the WebSocket connection to the tracker. They are the vital part establishing a direct connection between two peers within WebRTC, and are defined in this RFC. The WebRTC connection process itself is outlined here.

[Tails]

Hi, @Tails can you elaborate a little more on what you were trying to achive:

I came here because I was trying to use https://github.com/dmotz/trystero with Torrust. However, typical for Webtorrent is that the announce endpoint is available over a Websocket transport, which for Torrust seems not to be the case,

Do you mean that the tracker could accept an announce request on a websocket created on the server?

I see that this tracker: https://github.com/webtorrent/bittorrent-tracker:

Supports all mainstream tracker types:

* HTTP trackers

* UDP trackers ([BEP 15](http://www.bittorrent.org/beps/bep_0015.html))

* WebTorrent trackers ([BEP forthcoming](http://webtorrent.io/))

But I do not know what a "WebTorrent trackers". In fact, I have not seen any info about it. I will research.

As elaborated by the other replies here from much more authoritative sources, there are some things that a Webtorrent-tracker server does a bit differently that make Javascript-based Webtorrent clients want something over a Websocket connection. I'm no expert on the difference in protocols so I have to delegate that to the experts that joined the discussion here.

On the other hand, can you tell me:

* Why do you prefer to use Torrust over Aquatic? I just want to know what things we could improve to make this project better. I have nothing against Aquatic, in fact, we are using part of its UDP tracker types.

Should the question not be the reverse? Anyway, I am working on this thing: https://parture.org that creates a network of nodes which should collectively serve CRDT collaboration sessions, think a Google Docs-way of working but trustless and decentralized. To this end I wanted to use Trystero, the Javascript Webtorrent lib that uses Webtorrent sessions to start WebRTC channels for sharing CRDT objects. For the trustless network I want(ed) to create a single Rust binary that does everything (IPFS server, Avalanche node, Webtorrent tracker, etc).

When looking for crates that did bittorrent tracking I did notice Aquatic and Torrust both, but I tried Torrust first because it looked very well done, with clear documentation, and eye for detail. Torrust fit that perfectly and I managed to relatively straightforwardly compile it into my server-stack. It was only then that I read the fine line of Trystero and noticed it was incompatible with Torrust due to it "not supporting Webtorrent". As mentioned, Aquatic does do it but I'm having trouble with it because it tightly integrates with a Linux kernel version, but my development happens on Mac, I would need a Docker container to wrap my stack in, etc.

My needs on the technical aspect are not very deep beyond decentrally being able to support WebRTC sessions and maybe acting as a CDN, both of which in browser contexts seem to be tied to the WebTorrent idea so I can't help much on suggestions to make it better.

[josecelano]

Hi @Tails thank you for your reply. https://parture.org/ is a fantastic project!

Regarding my question, "Why do you prefer to use Torrust over Aquatic?" maybe there was a misunderstanding. I assumed you were using a WebTorrent client with the Aquatic Tracker because it supports the WebTorrent Tracker protocol. But you wanted to use Torrust instead of Aquatic because it may have some features you want to use. I just wanted to know which ones. We do not have that much feedback from Torrust users, and I really appreciate it.

I still have to read the other implementations to have a clear understanding of what the WebTorrent Trackers do, but it seems that we all finally agree that the WebSocket tracker for webtorrent clients makes sense, and it's something valuable to add to the Torrust Tracker.

I would propose these steps:

• Generate a TEP draft with the protocol specification, collecting as much information as possible.
• Implement it in the Torrust Tracker, improving the draft TEP.
• Release the final TEP
• Submit a PR to create the missing BEP.

I'm busy now finishing other stuff for the next Torrust major release, and this proposal is a lot of work. But I would like to write the draft TEP (if people with experience defining or implementing it cannot do it), and you all participate in validating the document. For me, the best way to contribute with BitTorrent now is by writing proper documentation instead of writing more code, improvements, etc. I think it's a significant barrier to entry for other developers.

[Power2All]

From what I could find, websockets are used to exchange data "live" (as a websocket stays alive until it's closed, so it's a active connection), and uses the offers and answers to exchange peers between the clients.

https://github.com/webtorrent/bittorrent-tracker/blob/master/lib/client/websocket-tracker.js
https://github.com/webtorrent/bittorrent-tracker/blob/master/server.js

It seems clients send information when a update is done, and webtorrent seems to be much more active then the usual normal bittorrent traffic.
Would be nice to have a chart that explains how the first init connection is handled for websockets, and how they are updated when they aren't stopping. I see also something like "to_peer_id", which is unclear to me what the use of it is.

Basically, what I'm trying to say, I see difference in announcing with WebTorrent, but it's pretty unclear how it works underwater, and would be nice if someone could explain what it does underwater when running, both for initial announcement (what exactly it's sending in a an array view), and what it does when it's renewing the announcement.

[josecelano]

I totally agree @Power2All. We should include some diagrams (sequence and architecture) and detailed data structures interchanged by web peers and the tracker.

Since I could not find at least a vague description, I tried asking ChatGPT since I need a basic understanding of web torrenting and some basic concepts, but I need to read about WebRTC and WebSocket from the sources (https://webrtc.org/, https://www.rfc-editor.org/rfc/rfc6455, https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Protocols, https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API)

WebRTC in Web Torrents:

1. Peer-to-Peer Data Transfer: WebRTC enables direct peer-to-peer communication in the browser. WebTorrent is a streaming torrent client for the browser that uses WebRTC for peer-to-peer transport. This allows browsers to directly download pieces of files from multiple peers, similar to how traditional torrent clients work.

2. Data Channels: WebRTC's data channels are used to transfer non-media data between peers. In the context of WebTorrent, data channels transfer the actual file data.

3. Tracker as a Signaling Server: In WebTorrent, trackers also play the role of signaling servers. When a WebTorrent client wants to find peers or announce its presence for a specific torrent, it communicates with a tracker over WebSockets. The tracker helps peers discover each other and initiate the WebRTC handshake.

WebSockets (WSS) in Web Torrents:

1. Tracker Communication: In traditional BitTorrent, UDP is often used for communication with trackers. However, browsers don't support raw UDP communication (outside of WebRTC). Instead, WebTorrent uses WebSockets to communicate with trackers for peer discovery. This allows the browser-based client to get a list of potential peers.

2. Signaling for WebRTC: As mentioned earlier, the tracker helps facilitate the WebRTC handshake by acting as a signaling server. It does this using the WebSocket connection. Once peers have discovered each other and the WebRTC handshake is completed, they can communicate directly via WebRTC, and the WebSocket connection to the tracker is just used for periodic updates or finding additional peers.

In summary, for web torrents:

• WebSockets (WSS) is primarily used for communication with the tracker, which in turn helps with peer discovery and acts as a signaling server for WebRTC.
• WebRTC is used for the actual peer-to-peer data transfer, allowing browsers to share pieces of files directly with each other.

Both protocols together enable a complete, browser-based torrenting experience without needing to download any software or extensions.

Signaling

In the context of WebRTC, "signaling" refers to the process of coordinating communication to establish a peer-to-peer connection. Signaling is used to exchange metadata between peers before the direct connection is established. This metadata includes information like:

• Session control messages to open and close connections.
• Error messages.
• Media metadata, such as codecs and codec settings, bandwidth, and media types.
• Key data for encrypting the media.
• Network data, such as IP addresses and ports (called ICE candidates).

A signaling server is a server that facilitates this exchange of metadata. It should be noted that the WebRTC specification doesn't define a specific signaling protocol, leaving it up to the developer to choose or design one. This flexibility allows developers to integrate WebRTC into existing services more easily.

Common methods for signaling include WebSockets, Server-Sent Events, and even traditional HTTP/AJAX polling. Once the signaling process is complete and both peers have the necessary metadata, a direct connection can be established between them, and the signaling server is no longer involved in the communication.

[greatest-ape]

FYI I’ve improved the documentation of the various aquatic_ws protocol structs, e.g, https://github.com/greatest-ape/aquatic/blob/master/crates/ws_protocol/src/incoming/announce.rs.

[fujinxiang]

This discussion is great! I'm waiting for supported webtorrent too.

I have try bittorrent-tracker, wt-trakcer, aquatic_ws, and I like more great projects.

[Tails]

All great of course, but not as awesome as when it would have been in Rust!

[greatest-ape]

aquatic is, though ;-)

[Tails]

I know! but it had some weird requirements that made it hard to build and I would have to run it in a container during development which was awkward. Torrust is way cleaner in that respect and would have my preference.

[greatest-ape]

Alright, fair enough :-)

[Power2All]

Some little update.
I've done a little initial work on my own side project of Torrust.
Right now, I see how it works, basically, you have WebSocket connections active (it could be a heavy duty work though, since the WebSockets are "live" connections and exchange updates between websocket clients) that exchange almost directly information between each other through the tracker about peers joining and leaving.
The issue I can see, is that there will be a hell of a lot of websocket connections active, which could burden the server with a lot of load though.
Though, I see the point with WebTorrent, as it exchanges basically a "signature" to authenticate WebRTC connections between browsers.