POC: New API endpoint with extendable metrics

[josecelano]

Relates to:

• Overhaul stats: Segregated metrics for each tracker running on a different socket address #1263 (comment)
• Wrong statistics imported from the tracker (global stats) torrust-index#809

This is only a Proof of Concept.

It adds a new REST API endpoint with a new extendable format for metrics:

URL: http://0.0.0.0:1212/api/v1/metrics?token=MyAccessToken

Sample response:

{
  "labeled_metrics": [
    {
      "metric": {
        "name": "announce_requests_received_total",
        "kind": "counter",
        "value": 1
      },
      "labels": {
        "ip_version": "ipv4",
        "protocol": "http",
        "url": "http://0.0.0.0:7070"
      }
    },
    {
      "metric": {
        "name": "scrape_requests_received_total",
        "kind": "counter",
        "value": 1
      },
      "labels": {
        "ip_version": "ipv4",
        "protocol": "http",
        "url": "http://0.0.0.0:7070"
      }
    }
  ]
}

It only supports JSON format for now. It doesn't support Prometheus format like the current stats endpoint:

URL: http://0.0.0.0:1212/api/v1/stats?token=MyAccessToken&format=prometheus

The Prometheus format response should be like the following:

announce_requests_received_total{ ip_version="ipv4", protocol="http", url="http://0.0.0.0:7070"} 1

scrape_requests_received_total{ ip_version="ipv4", protocol="http", url="http://0.0.0.0:7070"} 1

TODO

• Replace primitive types in the new labeled metrics with new tpyes to enforce constraints. For example, metrics names, labels, metric types, etc.
• Inject the rigth URL scheme. It's hardcoded now.
• Implement for UDP metrics. It's only implemented for HTTP tracker metrics. This would require merging labeled metrics.
• Use a f64 for metric values instead of u64.
• The metric name and label set pair must be unique in the array of labeled metrics. Enforce constraint.
• Implement versioning. A given API version must contain a set of labeled metrics. Clients expects some labeled metrics to be included in a API version. We need to initialize the array of metrics with all the expected labeled metrics with the initial value.
• Review names for fields in the API response. For example rename url to server_url, ...
• Review what labels we include per metric. For example the "protocol" label can be derived from the URL but it requires parsing the URL and it makes harder to build graphs in Grafana. I guess we should include labels for the type of aggregate date we want to get.

IMPORTANT

• Each metric+label set values combination is a different time series in Prometheus. We can't use labels with high cardinality. See https://prometheus.io/docs/practices/naming/#labels. For example, we can't use client's IPs.

@da2ce7 it would be nice to have feedback before continuing working on this approach.

[codecov]

Codecov Report

Attention: Patch coverage is 75.43860% with 42 lines in your changes missing coverage. Please review.

Project coverage is 83.09%. Comparing base (9a10e3e) to head (c159351).

Files with missing lines	Patch %	Lines
...racker-api-server/src/v1/context/stats/handlers.rs	0.00%	16 Missing ⚠️
...s/rest-tracker-api-core/src/statistics/services.rs	0.00%	14 Missing ⚠️
...acker-api-server/src/v1/context/stats/resources.rs	0.00%	5 Missing ⚠️
...ckages/http-tracker-core/src/statistics/metrics.rs	94.20%	3 Missing and 1 partial ⚠️
...acker-api-server/src/v1/context/stats/responses.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1414      +/-   ##
===========================================
- Coverage    83.17%   83.09%   -0.08%     
===========================================
  Files          233      234       +1     
  Lines        17086    17242     +156     
  Branches     17086    17242     +156     
===========================================
+ Hits         14211    14328     +117     
- Misses        2608     2649      +41     
+ Partials       267      265       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.