Alternatives to the Tracker REST API

[josecelano]

Today in our weekly meeting we had an interesting conversation about the tracker integration with other system in terms of exporting data. I will try to summary the points and conclusions.

Context

The Tracker runs a REST API (if enabled) that allow interaction with other external systems.

In the current version 1 we have 4 different contexts:

• Stats: get tracker statistics
• Torrents (get info about torrents)
  • Get a torrent
  • Get a torrent list
• Whitelist
  • Add a torrent to the whitelist
  • Remove a torrent from the whitelist
  • Reload the whitelist
• Authentication keys (only for HTTP trackers)
  • Generate a new authentication key
  • Delete an authentication key
  • Reload authentication keys

Proposal 1: GraphQL API

@da2ce7 proposed starting considering a new implementation in GraphQL.

Proposal 2: Emit events from core tracker

I proposed to first research about our needs: what was the goal of creating the REST API?

The main reason was to provide an integration point with the Torrust Index. One of the goal is to export data from the Tracker to external systems, being our our Index implementation or other indexes. However @da2ce7 pointed out that that was not the only reason because the API also accepts commands. In fact, in the first version the tracker configuration could be changed from the Index.

I think we can segregate responsibilities of the REST API into two responsibilities:

• Query
• Command

See https://martinfowler.com/bliki/CQRS.html

• Queries:
  • Stats
  • Torrents
    • Get a torrent
    • Get a torrent list
• Commands:
  • Whitelist
    • Add a torrent to the whitelist
    • Remove a torrent from the whitelist
    • Reload the whitelist
  • Authentication keys (only for HTTP trackers)
    • Generate a new authentication key
    • Delete an authentication key
    • Reload authentication keys

NOTE: maybe we could have to separate APIs because their requirements could be different. The query responsibility could be much more complex if we want to provide complex stats to private indexes. See #820.

What I proposed was at least to consider other alternatives to export data, for example: emitting events.

If the tracker core emited 4 basis events:

• Announce request received
• Announce response sent
• Scrape request received
• Scrape response sent

It would be possible to any external system to build any projection of the data (no data loss).
External system like an Index could subscribe to the tracker events and build their own views in their preferred DB systems.

Of course, this proposal does not excluded the GraphQL proposal. As I see it the GraphQL solution could be good for 80% of consumers while the "events" approach could fit for the most demanding systems (the other 20%), demanding data that we don't store directly in the tracker).

In fact, the current REST API has some scalability problems. See torrust/torrust-index#569. And we are only importing basic stats. It would be impossible to generate and import peer-level stats with this solution.

I'm not proposing any concrete implementation, only the pattern: event emitter/event subscriber.

This can be implemented in may different ways. For example, @da2ce7 proposed using the crate tracing which is already installed. Consumers can parse the logs an extract the "domain" events they are interested in. There could even be different driver implementations to send events in different ways to external services.

Pros:

• It should not affect tracker performance.
• The tracker can fulfill any data requirement from external services as long as the subscriber to events and build their own data structures.
• It should scale better.

Cons:

• It harder to implement for consumers.

My plan would be:

• To reflect on the types of integration we need: export data and receive commands.
• To decide if we want to resolve the problem one one API or two different solutions (read model and write model).
• To consider these and other alternatives that might work at different levels. For example events can be emitted from the tracker core and the GrapQL API can be implemented as a delivery layer on top of the tracker core.

[josecelano]

AI Research Prompt

I want to open a new disucssion in a Github repo I'm working on. It's a Bittorrent Trackr written in Rust. We are discussion different alternatives to build a new API for integration with external system. I want you to review the draft comment I want to start the discussion with. IT's the folowiing:

Today in our [weekly meeting](https://github.com/orgs/nautilus-cyberneering/discussions/135) we had an interesting conversation about the tracker integration with other system in terms of exporting data. I will try to summary the points and conclusions.

### Context

The Tracker runs a [REST API](https://docs.rs/torrust-tracker/latest/torrust_tracker/#api) (if enabled) that allow interaction with other external systems.

In the [current version 1](https://docs.rs/torrust-tracker/latest/torrust_tracker/servers/apis/v1/index.html) we have 4 different contexts:

- Stats: get tracker statistics
- Torrents (get info about torrents)
  - Get a torrent
  - Get a torrent list
- Whitelist
  - Add a torrent to the whitelist
  - Remove a torrent from the whitelist
  - Reload the whitelist
- Authentication keys (only for HTTP trackers)
  - Generate a new authentication key
  - Delete an authentication key
  - Reload authentication keys

### Proposal 1: GraphQL API

@da2ce7 proposed starting considering a new implementation in [GraphQL](https://graphql.org/).

### Proposal 2: Emit events from core tracker

I proposed to first research about our needs: what was the goal of creating the REST API?

The main reason was to provide an integration point with the Torrust Index. One of the goal is to export data from the Tracker to external systems, being our our Index implementation or other indexes. However @da2ce7 pointed out that that was not the only reason because the API also accepts commands. In fact, in the first version the tracker configuration could be changed from the Index.

I think we can segregate responsibilities of the REST API into two responsibilities:

- Query
- Command

See https://martinfowler.com/bliki/CQRS.html

- Queries:
  - Stats
  - Torrents
    - Get a torrent
    - Get a torrent list
- Commands:
  - Whitelist
    - Add a torrent to the whitelist
    - Remove a torrent from the whitelist
    - Reload the whitelist
  - Authentication keys (only for HTTP trackers)
    - Generate a new authentication key
    - Delete an authentication key
    - Reload authentication keys

**NOTE:** maybe we could have to separate APIs because their requirements could be different. The query responsibility could be much more complex if we want to provide complex stats to private indexes. See https://github.com/torrust/torrust-tracker/discussions/820.

What I proposed was at least to consider other alternatives to export data, for example: emitting events.

If the tracker core emitted 4 basis events:

- Announce request received
- Announce response sent
- Scrape request received
- Scrape response sent

It would be possible to any external system to build any projection of the data (no data loss).
External system like an Index could subscribe to the tracker events and build their own views in their preferred DB systems.

Of course, this proposal does not excluded the GraphQL proposal. As I see it the GraphQL solution could be good for 80% of consumers while the "events" approach could fit for the most demanding systems (the other 20%), demanding data that we don't store directly in the tracker).

In fact, the current REST API has some scalability problems. See https://github.com/torrust/torrust-index/issues/569. And we are only importing basic stats. It would be impossible to generate and import peer-level stats with this solution.

I'm not proposing any concrete implementation, only the pattern: event emitter/event subscriber.

This can be implemented in may different ways. For example, @da2ce7 proposed using the crate `tracing` which is already installed. Consumers can parse the logs an extract the "domain" events they are interested in. There could even be different driver implementations to send events in different ways to external services.

**Pros:**

- It should not affect tracker performance.
- The tracker can fulfill any data requirement from external services as long as the subscriber to events and build their own data structures.
- It should scale better.

**Cons:**

- It harder to implement for consumers.

My plan would be:

- To reflect on the types of integration we need: export data and receive commands.
- To decide if we want to resolve the problem one one API or two different solutions (read model and write model).
- To consider these and other alternatives that might work at different levels. For example events can be emitted from the tracker core and the GrapQL API can be implemented as a delivery layer on top of the tracker core.

[josecelano]

AI R1 (Friendli) response

Okay, the user wants me to review their draft comment for starting a discussion on GitHub about building a new API for their Rust-based Bittorrent Tracker. Let me read through it carefully to understand the context and their proposals.

First, the context: They have a REST API with four main areas: stats, torrents, whitelist, and authentication keys. They're discussing two proposals: moving to GraphQL or emitting events from the core tracker. The user has broken down the responsibilities into queries and commands, mentioned CQRS, and outlined pros and cons of each approach.

Now, I need to check the clarity and structure of the draft. Are the key points presented effectively? Is there any confusion between the proposals? Let's see.

The user starts by summarizing the meeting and context, which is good for readers who might not have been there. They explain the current REST API and its components. Then they present Proposal 1 (GraphQL) and Proposal 2 (event emission). The explanation of separating queries and commands using CQRS is clear, and the note about scalability issues with an example from their repo adds context.

Possible improvements: Maybe clarify how GraphQL and event-driven approaches can coexist. The user mentions that events are for 20% demanding cases, so emphasizing that these are complementary solutions might help. Also, the plan at the end could be formatted more clearly as action items or questions for the team.

Structure-wise, using bullet points and sections makes it readable. However, the sections could be marked more distinctly, like using headers for each proposal. Also, hyperlinking the references (like the CQRS article) is helpful. The "pros and cons" are listed, but maybe numbered lists would make them stand out more.

The user’s plan is to reflect on integration needs, decide on one or two APIs, and consider alternatives. They should maybe pose these steps as specific questions to the team to invite discussion. For example, asking which approach the team prefers, or if there are other alternatives.

Grammar and typos: "folowiing" in the initial message is a typo, but in the draft itself, it's okay. The phrase "It's the folowiing:" should be corrected in the user's query, but in the draft, the content seems fine.

Also, the user mentions @da2ce7's suggestions, which is good for attribution. However, ensuring all contributors are mentioned if they contributed ideas might be necessary.

In terms of technical accuracy: The user correctly differentiates between queries and commands, and the event-driven approach for real-time data needs. Pointing out the scalability issue with the current REST API and how events can offload that makes sense.

Suggested additions: Perhaps flowchart diagrams or examples of how each API would handle a sample request could help, but maybe that's too detailed for a discussion starter. Alternatively, posing open-ended questions to the team about their preferences, use cases, or potential pitfalls they foresee could foster more engagement.

Overall, the draft is well-structured and informative. Adding explicit questions to the team and slightly enhancing the formatting could make it more effective in prompting discussion.

Your draft is thorough and well-structured, but here are some suggestions to improve clarity, engagement, and decision-making:

---

Suggested Edits/Improvements:

1. Add a Clear Purpose & Call-to-Action

Start with a concise goal and direct ask to encourage participation:

"We need to design a scalable API for integrations. Let’s discuss two proposals and decide on next steps. What are your thoughts?"

---

2. Reorganize Proposals for Comparison

Use a table or bulleted list to contrast GraphQL vs. Events (or highlight complementary roles). For example:

Aspect	GraphQL API	Event Emission
Use Case	Ad-hoc queries, simple integrations	Real-time data, complex projections
Scalability	May struggle with heavy queries	Scales well (push-based model)
Implementation	Client-driven flexibility	Requires event consumers to process
Complementary?	✅ Could handle commands/queries	✅ Handles raw data for custom needs

---

3. Clarify Complementary Roles

Explicitly state how they could coexist:

"GraphQL could serve as the primary interface for common queries/commands, while event emission enables advanced use cases. This decouples core logic from integration demands."

---

4. Pose Specific Questions

Guide the discussion with actionable prompts:

• Should we separate the Query and Command APIs (CQRS)?
• Is GraphQL sufficient for current/future needs, or do we need event-driven architectures?
• Could we prototype event emission (e.g., via tracing) without blocking GraphQL work?

---

5. Simplify Technical Jargon

Replace niche terms (e.g., "projection") with plain language where possible to widen accessibility:

"Subscribers could build their own datasets from raw event streams instead of relying on pre-processed stats."

---

6. Highlight Risks 🔥

Call out potential downsides to spark debate:

• GraphQL Risk: Overfetching/complexity if clients abuse flexible queries.
• Event Risk: May require Kafka/Pulsar for reliable delivery, adding infra overhead.

---

7. Link to Prior Art

Reference similar projects/standards to anchor proposals:

• BitTorrent v2 Metadata
• OpenTelemetry Tracing (if using tracing crate)

---

8. Final Draft Structure

## Goal  
Design a scalable API for integration with external systems.  

## Context  
[Current REST API summary + limitations]  

## Proposals  
1. **GraphQL API** (@da2ce7)  
   - Pro: Flexible queries, simplifies client needs  
   - Con: May not scale for high-demand consumers  

2. **Event Emission**  
   - Pro: Decouples data consumers, scales infinitely  
   - Con: Requires subscribers to process raw streams  

## Open Questions  
- Can these approaches coexist?  
- Which should we prioritize?  
- What’s the MVP scope?  

---

This structure invites focused feedback while ensuring technical depth. Would you like me to draft a full revised version?

[da2ce7]

R1 (prompt)

We still need to use a query language for displaying information about the application, such as version, uptime, actual configuration, permissions, token validity, etc. the commands (mutations) are should change the state of the application and therefor it should be represented in the API.

For this we still should use a GraphQL to display the internal configuration and permission state of the application, and to display.

Even emissions from the rust-tracing libary are a good fit for GraphQL subscriptions feature.

If you agree, can you write a good summary of using GraphQL for mutations, application state, and subscribing to the traces?

Here’s a concise summary of using GraphQL to unify command execution, application state queries, and real-time event subscriptions via Rust’s tracing:

---

GraphQL as the Unified API Layer 🚀

Feature	GraphQL Use Case	Example
Mutations (Commands)	Modify tracker state (whitelist, auth keys, config)	graphql<br>mutation {<br> addToWhitelist(hashes: ["abc123"]) {<br> success<br> }<br>}<br>
Queries	Fetch static/cached state (config, permissions, torrent stats)	graphql<br>query {<br> config {<br> whitelist_enabled<br> }<br>}<br>
Subscriptions	Stream raw tracing events (announces, scrapes) for custom analytics/auditing	graphql<br>subscription {<br> trackerEvents {<br> type<br> peer_ip<br> timestamp<br> }<br>}<br>

---

Key Benefits

1. Single Endpoint

   • Clients interact with one API for all needs: control, monitoring, and auditing.
   • No need to juggle REST endpoints + WebSocket streams separately.

2. Real-Time Feedback

   • Subscriptions provide instant visibility into tracker activity (e.g., peer joins, scrapes).

3. Type Safety

   • Shared GraphQL schema ensures clients and servers agree on event structures.

4. Security

   • Centralize authZ/authN logic (e.g., JWT tokens for mutations).

---

Implementation Steps

1. Instrument Core Tracker with tracing

   #[tracing::instrument]
   async fn handle_announce(ctx: Context) {
       tracing::info!(event_type = "announce", peer = ctx.peer_id);
   }

2. Bridge tracing ⇨ GraphQL Subscriptions
   Use a channel (tokio::sync::broadcast) to forward events from tracing subscribers to GraphQL resolvers.

   struct EventStream {
       rx: Receiver<TrackerEvent>,
   }
   
   #[Subscription]
   impl EventStream {
       async fn tracker_events(&self) -> impl Stream<Item = TrackerEvent> {
           futures::stream::iter(&self.rx)
       }
   }

3. Define Schema

   type Mutation {
     addToWhitelist(hashes: [String!]!): CommandResult!
     revokeAuthKey(key: String!): CommandResult!
   }
   
   type Query {
     statistics: TrackerStats!
     config: TrackerConfig!
   }
   
   type Subscription {
     trackerEvents(filter: EventFilter): TrackerEvent!
   }
   
   enum EventType {
     ANNOUNCE
     SCRAPE
     WHITELIST_UPDATE
   }
   
   input EventFilter {
     types: [EventType!]
     peer_ip: String
   }

---

Caveats

• Performance: Subscriptions add overhead; rate-limiting may be needed.
• Filtering: Let clients narrow streams (e.g., filter: { types: [ANNOUNCE] }).
• Security: Audit mutations carefully (e.g., addToWhitelist should require admin rights).

---

Decision Needed

Should we adopt this hybrid model?
✅ Pros: Single interface, real-time capabilities.
⚠️ Cons: Adds GraphQL dependency; requires async event bridging.

---

Let me know if you’d like to refine specific aspects!

[josecelano]

In this week meeting @da2ce7 and I were discussing about this.

I would highlight two points:

1. @da2ce7's idea is to use the new API as a replacement (alternative) for all the current API:

• Tracker API
• HTTP tracker
• UDP tracker

That means not only the Index would use it but potentially even bittorrent clients or other apps in the future.

2. How can we expose integration events?

Apparently there can be two ways of conceptualize the tracker events using GraphQL.

The first one is:

• There are mutations for the peer lists
• You can subscribe to changes produced by those mutations

For example,

• A client mutate one swarm making an announce request. The peer list changes by adding the new peer to the list.
• Another client can subscribe to such changes to know when new peers are available.
• An "Index" client can subscribe to a different mutation, for example, when the number of seeders, leechers of downloads change after the announce request.

The bittorrent client would be interested in events like "peer added", "peer removed" and the index client would be interested in events like "seeder added", "seeder removed", "leecher added", "leecher removed", "donwload completed" or something more generic like "swarn stats changed".

The second one (although I don't know if it would be the same in the end):

• Events as a first-class citizen in the design.
• When a mutation happens, for example a new peer was added to a peer list, a new event is triggered and you can subscribe to new events directly.

I'm concern about this approach because we have been using only memory so far for the core data and we would have thousandths of events. Event if we ignore the announce requests at regular intervals the number of events in a subscriptions would be very high. In the torrust demo tracker we are receiving 2000 req/sec. I don't know how many of those request actually produce a mutation. I mean adding or removing a peer it's an interesting event you want to share with clients but a regular announce request it might not be.

I'm also concern about how can we let the index, for example, subscribe only the events for the info-hashes the index has. With a public tracker the index is normally only interested in the its torrents.

We might need to introduce a new concept like "torrent set" but that would imply the API clients need to provide to the server their sets. So the server has to store info about the clients. And I do not like that. Alternatively we can let the Index filter out torrents but that leads to the problem we have right now for public tracker. The Index has to pull info from one torrent at the time even if the info for those torrents hasn't changed.

[josecelano]

Relates to: torrust/torrust-index#569

Yesterday (in the weekly meeting) @da2ce7 and I've continued the discussion about the new GraphQL API.

I had a proposal. I think we new a way for the Index to communicate to the Tracker what are the torrents the index is interested in.

Context

There is a common scenario we need to handle efficiently. when you run a public Tracker the tracker migth contain millions of torrents but the Index might not. For example:

• Public tracker with 1M torrents.
• Semi-private Index with 1000 torrents.

The Index needs to import stats from the tracker regularly.

Current Solution

The Index is constantly importing tracker stats only for those 1000 torrents. It uses the REST API pulling only tens of torrents per request.

How can GraphQL help

In GraphQL you can setup a subscription so the Index can get updates.

The thing is ideally the Index should subscribe only to changes in its torrents stats. So basically we need to define a query like "give me updates on my stats".

@da2ce7 and I discussed how we can build that query.

Proposal 1: tags for indexes

This was my proposal.

• The Index uses a new API feature to tag a torrent.
• It could be implemented in both the REST API and the Graphql API.
• The Index subscribes to a query filtering torrent by its tag.

This is inspired in how cloud provides allow you to tag servers to for example collect metrics later on based on a given tag.

Proposal 2: torrents sets for indexes

This was @da2ce7's porposal.

• Each Index has a "key"
• There is a GraphQP mutation that Indexes can use to create their sets of torrents (add a torrent to their key).
• I guess there could also be a way to create new keys (or allow any).

Summary

I think both proposal are basically the same. I think the only difference is proposal 1 is more generic and it could be used for others things.