It seems persisted data about torrents is not loaded from database

[josecelano]

It seems persisted data about torrents (number of downloads) is not loaded from database when the tracker stars. The start function loads the authentication keys and the whitelist:

// Load peer keys
if config.core.private {
    app_container
        .keys_handler
        .load_peer_keys_from_database()
        .await
        .expect("Could not retrieve keys from database.");
}

// Load whitelisted torrents
if config.core.listed {
    app_container
        .whitelist_manager
        .load_whitelist_from_database()
        .await
        .expect("Could not load whitelist from database.");
}

However I don't see where we load torrent data. The method TorrentsManager::load_torrents_from_database is only used in test code.

Maybe that was the expected behavior. However that means the number of downloaded torrents in the database is reset every time the tracker is restarted because the persist_stats function only overwrite the in-memory counter. That in-memory counter is not preloaded with the database value when the tracker starts.

/// Persists torrent statistics to the database if persistence is enabled.
fn persist_stats(&self, info_hash: &InfoHash, swarm_metadata: &SwarmMetadata) {
    if self.config.tracker_policy.persistent_torrent_completed_stat {
        let completed = swarm_metadata.downloaded;
        let info_hash = *info_hash;

        drop(self.db_torrent_repository.save(&info_hash, completed));
    }
}

cc @da2ce7

[josecelano]

How to reproduce

1. Run the tracker with:

TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__TRACKER_POLICY__PERSISTENT_TORRENT_COMPLETED_STAT=true cargo run

2. Make sure the output show the configuration with

"persistent_torrent_completed_stat": true,

3. Run the tracker client to add a new torrent

cd console/tracker-client/   
cargo run --bin udp_tracker_client announce udp://127.0.0.1:6969 9c38422213e30bff212b30c360d26f9a02136422 | jq

You should see something like:

    Finished `dev` profile [optimized + debuginfo] target(s) in 0.10s
     Running `/home/josecelano/Documents/git/committer/me/github/torrust/torrust-tracker/target/debug/udp_tracker_client announce 'udp://127.0.0.1:6969' 9c38422213e30bff212b30c360d26f9a02136422`
{
  "AnnounceIpv4": {
    "transaction_id": -888840697,
    "announce_interval": 120,
    "leechers": 0,
    "seeders": 1,
    "peers": []
  }
}

4. Check with the sqlite3 client that the torrent was added

sqlite3 storage/tracker/lib/database/sqlite3.db 
SQLite version 3.46.1 2024-08-13 09:16:08
Enter ".help" for usage hints.
sqlite> select * from torrents;
1|9c38422213e30bff212b30c360d26f9a02136422|0
sqlite> 

Update the number of downloads manually:

sqlite> update `torrents` set completed = 1 where `info_hash` = "9c38422213e30bff212b30c360d26f9a02136422";
sqlite> select * from torrents;
1|9c38422213e30bff212b30c360d26f9a02136422|1
sqlite> 

5. Restart the tracker

6. Check with sqlite that the counter has been reset

You can also use the tracker API:

http://0.0.0.0:1212/api/v1/torrent/9c38422213e30bff212b30c360d26f9a02136422?token=MyAccessToken

{
  "info_hash": "9c38422213e30bff212b30c360d26f9a02136422",
  "seeders": 1,
  "completed": 2,
  "leechers": 0,
  "peers": [
    {
      "peer_id": {
        "id": "0x2d71423030303030303030303030303030303031",
        "client": null
      },
      "peer_addr": "0.0.0.0:17548",
      "updated": 1741179730781,
      "updated_milliseconds_ago": 1741179730781,
      "uploaded": 0,
      "downloaded": 0,
      "left": 0,
      "event": "Completed"
    }
  ]
}

The completed value should be the one the database.

[josecelano]

I think the naive implementation would be to load the persistent torrents initially when the tracker starts but that has some problems:

• Even if we load all torrents initially they would be peerless torrents. If removing peerless torrents is enabled in the configuration those entries will be removed from the torrent repository and we will loose the counter value again.
• After starting the tracker even if the counter value is right for a while the counter is going to be reset if the torrent is removed from the torrent repository because it does not have peers.

I have to think a good solution.

cc @da2ce7

[da2ce7]

Keep the "loaded" torrents in a separate structure, move them into the normal structure if they get announced.

The "loaded" structure can optionally expire after a configurable time. 😊

[josecelano]

Keep the "loaded" torrents in a separate structure, move them into the normal structure if they get announced.

The "loaded" structure can optionally expire after a configurable time. 😊

Hi @da2ce7 there are two problems with that approach:

• When a new download is completed I have to update:
  • The number of downloads inside the in-memory repository (what it's doing now).
  • The number of downloads in the database (it's also implemented in the current version).
  • The number of downloads in the new cache for the database data.

What I'm implementing in the PR is:

• Load counter from the database when the torrent is added to the repository (no matter if the tracker was restarted or the torrents was removed and added later again).
• Increase the counter immediately in the DB when increases in the in-memory repo.

All of that it's done only if persisting stats is enabled (config.tracker_policy.persistent_torrent_completed_stat).

I think I will merge the PR once that patch is implemented. However I think we can implement a better solution (I will open a new issue for that):

• Trigger an event from the in-memory repository with info about the announce event. Actually it's like re-triggering the event we have received from the peer (the announce event). The event can contain info like if the number of downloads was increased or not.
• We add a listener and increase the counter in the database wen needed.
• The announce handler does not have the responsibility to persist the stats (this counter).

[josecelano]

Hi @da2ce7 I've fixed the problem with a temporary path. I will open a new issue with a proposal for a better solution. In the mean time at least the bug it's fixed.

I have made two main changes:

1. Preset the counter for number of downloads every time the torrent is added to the in-memory torrent repository:

pub async fn announce(
    &self,
    info_hash: &InfoHash,
    peer: &mut peer::Peer,
    remote_client_ip: &IpAddr,
    peers_wanted: &PeersWanted,
) -> Result<AnnounceData, AnnounceError> {
    self.whitelist_authorization.authorize(info_hash).await?;

    let opt_persistent_torrent = if self.config.tracker_policy.persistent_torrent_completed_stat {
        self.db_torrent_repository.load(info_hash)?
    } else {
        None
    };

    peer.change_ip(&assign_ip_address_to_peer(remote_client_ip, self.config.net.external_ip));

    let number_of_downloads_increased =
        self.in_memory_torrent_repository
            .upsert_peer(info_hash, peer, opt_persistent_torrent);

    if self.config.tracker_policy.persistent_torrent_completed_stat && number_of_downloads_increased {
        self.db_torrent_repository.increase_number_of_downloads(info_hash)?;
    }

    Ok(self.build_announce_data(info_hash, peer, peers_wanted))
}

In the core announce handler we check if the torrent is new:

• If it's new and persistent torrents stats are enabled we load the counter value from the database and we pass it to the repository.
• If it's not new or persistent stats are disable we just ignore stats.

2. The in-memory repo has a new argument to preset the counter when it's provided.

Only if the torrent is new.

impl Repository<EntryMutexStd> for CrossbeamSkipList<EntryMutexStd>
where
    EntryMutexStd: EntrySync,
    EntrySingle: Entry,
{
    /// Upsert a peer into the swarm of a torrent.
    ///
    /// Optionally, it can also preset the number of downloads of the torrent
    /// only if it's the first time the torrent is being inserted.
    ///
    /// # Arguments
    ///
    /// * `info_hash` - The info hash of the torrent.
    /// * `peer` - The peer to upsert.
    /// * `opt_persistent_torrent` - The optional persisted data about a torrent
    ///   (number of downloads for the torrent).
    ///
    /// # Returns
    ///
    /// Returns `true` if the number of downloads was increased because the peer
    /// completed the download.
    fn upsert_peer(&self, info_hash: &InfoHash, peer: &peer::Peer, opt_persistent_torrent: Option<PersistentTorrent>) -> bool {
        if let Some(existing_entry) = self.torrents.get(info_hash) {
            existing_entry.value().upsert_peer(peer)
        } else {
            let new_entry = if let Some(number_of_downloads) = opt_persistent_torrent {
                EntryMutexStd::new(
                    EntrySingle {
                        swarm: PeerList::default(),
                        downloaded: number_of_downloads,
                    }
                    .into(),
                )
            } else {
                EntryMutexStd::default()
            };

            let inserted_entry = self.torrents.get_or_insert(*info_hash, new_entry);

            inserted_entry.value().upsert_peer(peer)
        }
    }
}

NOTES

1. In my first solution I tried to created the torrent in the AnnounceHadler if it does not exist with the preset value from the DB, but that does not work becuase race conditions migth happen with the job to clean peerless torrents.

• If the in-memory repo does not contain the torrent we add it with the preset counter value from the DB.
• If the clean job is executed before upserting the peer the torrent would be removed from the repo and added again later during the upsert but with a 0 value for the counter.

3. I have not implemented the patch for all the repository implementations because I'm considering a new solution described in point 3.2 I think the problems came from having the persisted counter in the in-memory repo. I think the code would be much easier if we remove that responsibility from the torrent repository.

• The repo trigger an event including what type of event the peer announce and whether the events affects the number of downloads or not (it we should increase the counter).
• There is a new listener that listens to that event and increases the counter in a different repo.
• The API takes the value of the counter from the new repo.
• The core AnnounceHandler does not have to deal with persisting stats.

With the new implementation the handler would be:

pub async fn announce(
    &self,
    info_hash: &InfoHash,
    peer: &mut peer::Peer,
    remote_client_ip: &IpAddr,
    peers_wanted: &PeersWanted,
) -> Result<AnnounceData, AnnounceError> {
    self.whitelist_authorization.authorize(info_hash).await?;

    peer.change_ip(&assign_ip_address_to_peer(remote_client_ip, self.config.net.external_ip));

    self.in_memory_torrent_repository.upsert_peer(info_hash, peer);

    Ok(self.build_announce_data(info_hash, peer, peers_wanted))
}

Notice there is no code related to stats at all and upsert_peer return unity.