Fix bug: number of downloads for a torrent is reset

[josecelano]

Problem

The number of downloads for a torrent is not persisted correctly.

Even when persistence is enabled, it does not work. There are two cases when the counter is reset:

• When you restart the tracker.
• When a torrent entry is removed from the torrents repository because the last peer was removed (peerless torrents). This only happens if "removing peerless torrents" is enabled in the configuration.

Check the issue to know how to reproduce the bug.

Solution

Every time a new torrent entry is added in the torrent repository we need to load stats (also called persistent torrent, which only includes number of downloads currently) from the database. If persistence for stats is enabled.

Other considerations

1. Unused code

There is a method to load the counters for all torrents but it's not used in production code.

fn import_persistent(&self, persistent_torrents: &PersistentTorrents) {
    for (info_hash, completed) in persistent_torrents {
        if self.torrents.contains_key(info_hash) {
            continue;
        }

        let entry = EntryMutexStd::new(
            EntrySingle {
                swarm: PeerList::default(),
                downloaded: *completed,
            }
            .into(),
        );

        self.torrents.insert(*info_hash, entry);
    }
}

I guess it was added to load the information from the database when the tracker is restarted. However that's not enough because torrents are removed from the repository when they do not have peers (peerless torrents). That would work only if removing peerless torrents is disabled.

2. Better solution

I think there is a better solution than the one this PR applies. This PR is a temporary patch to fix the bug asap.

I would differentiate to different counters:

1. number of downloads per torrent while it was active in the swarm (since the tracker was restarted)
2. historical number of downloads per torrent per web service (since the service was put into production)

The counter 1 is what we have before merging this PR. The counter is reset every time the torrent is removed from the swarm. We can keep like it's now before merging this PR. This counter is handled at the in-memory torrent repository.

The new counter (counter 2) would be handled by the announce handler. After processing the announce request if the number of downloads has increased the handler also increases the historical counter. This new value should be the one exposed in the API.

{
  "torrents": 379508,
  "seeders": 150440,
  "completed": 58992,  <- this should be the historical
  "leechers": 264865,
  "tcp4_connections_handled": 15,
  "tcp4_announces_handled": 15,
  "tcp4_scrapes_handled": 0,
  "tcp6_connections_handled": 0,
  "tcp6_announces_handled": 0,
  "tcp6_scrapes_handled": 0,
  "udp_requests_aborted": 556458,
  "udp_requests_banned": 24773536,
  "udp_banned_ips_total": 29868,
  "udp_avg_connect_processing_time_ns": 478212,
  "udp_avg_announce_processing_time_ns": 8724637,
  "udp_avg_scrape_processing_time_ns": 296950,
  "udp4_requests": 298728340,
  "udp4_connections_handled": 104369821,
  "udp4_announces_handled": 158354656,
  "udp4_scrapes_handled": 3664492,
  "udp4_responses": 273954123,
  "udp4_errors_handled": 7540716,
  "udp6_requests": 0,
  "udp6_connections_handled": 0,
  "udp6_announces_handled": 0,
  "udp6_scrapes_handled": 0,
  "udp6_responses": 0,
  "udp6_errors_handled": 0
}

This patches solves the problem but pre-loading the historical data every time the torrent is added to the repository.

[codecov]

Codecov Report

Attention: Patch coverage is 75.00000% with 63 lines in your changes missing coverage. Please review.

Project coverage is 81.43%. Comparing base (bf883da) to head (06ef1d5).
Report is 8 commits behind head on develop.

Files with missing lines	Patch %	Lines
...ackages/tracker-core/src/databases/driver/mysql.rs	0.00%	24 Missing ⚠️
...ckages/tracker-core/src/databases/driver/sqlite.rs	61.29%	4 Missing and 8 partials ⚠️
...nt-repository/src/repository/skip_map_mutex_std.rs	65.21%	7 Missing and 1 partial ⚠️
...pository/src/repository/rw_lock_tokio_mutex_std.rs	14.28%	5 Missing and 1 partial ⚠️
...sitory/src/repository/rw_lock_tokio_mutex_tokio.rs	14.28%	5 Missing and 1 partial ⚠️
packages/tracker-core/src/announce_handler.rs	88.23%	0 Missing and 2 partials ⚠️
...s/tracker-core/src/torrent/repository/persisted.rs	90.00%	1 Missing and 1 partial ⚠️
packages/udp-tracker-server/src/environment.rs	0.00%	2 Missing ⚠️
...nt-repository/src/repository/dash_map_mutex_std.rs	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1356      +/-   ##
===========================================
- Coverage    81.48%   81.43%   -0.06%     
===========================================
  Files          230      230              
  Lines        16366    16500     +134     
  Branches     16366    16500     +134     
===========================================
+ Hits         13336    13436     +100     
- Misses        2794     2825      +31     
- Partials       236      239       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[josecelano]

ACK 06ef1d5

[josecelano]

I've discarded the solution proposed in the PR description.

The new solution is described here. I will open a new issue for it.