[pull] master from netdata:master

[pull]

See Commits and Changes for more details.

---

Created by pull[bot] (v2.0.0-alpha.1)

Can you help keep this open source service alive? 💖 Please sponsor : )

Summary by Sourcery

Refactor the metric and dimension handling to use a UUID map for faster lookups and reduced memory usage. Use refcounts for managing object lifetimes and implement concurrent access protection.

Enhancements:

• Improve performance of metric lookups by using a UUID map.
• Reduce memory usage by deduplicating UUIDs.
• Improve thread safety by using refcounts and spinlocks.

Tests:

• Add unit tests for the UUID map implementation.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request refactors the metric management in the database engine by replacing the use of raw UUIDs with UUIDMAP_IDs. This change improves memory management and performance by reducing memory usage and lookups.

Class diagram showing the new UUID mapping structure

classDiagram
    class METRIC {
        -UUIDMAP_ID uuid
        -Word_t section
        -REFCOUNT refcount
        -uint8_t partition
        -uint32_t latest_update_every_s
        -time_t first_time_s
        -time_t latest_time_s_clean
        -time_t latest_time_s_hot
    }

    class RRDMETRIC {
        -UUIDMAP_ID uuid
        -RRD_FLAGS flags
        -STRING* id
        -STRING* name
        -RRDDIM* rrddim
    }

    class RRDINSTANCE {
        -UUIDMAP_ID uuid
        -int update_every_s
        -RRD_FLAGS flags
        -uint32_t priority
        -RRDSET_TYPE chart_type
        -STRING* id
        -STRING* name
        -STRING* title
        -STRING* units
        -STRING* family
    }

    METRIC --> "1" UUIDMAP_ID
    RRDMETRIC --> "1" UUIDMAP_ID
    RRDINSTANCE --> "1" UUIDMAP_ID

Loading

State diagram for reference counting

stateDiagram-v2
    [*] --> Active: acquire()
    Active --> Active: acquire()
    Active --> Active: release() > 0
    Active --> Available: release() == 0
    Available --> Deleted: acquire_for_deletion()
    Active --> WaitingDelete: acquire_for_deletion_and_wait()
    WaitingDelete --> Deleted: all references released
    Deleted --> [*]

    note right of Active: References > 0
    note right of Available: References = 0
    note right of WaitingDelete: References > 0
    note right of Deleted: References = REFCOUNT_DELETED

Loading

File-Level Changes

Change	Details	Files
Replaced direct UUID usage with UUIDMAP_IDs for metric management.	Introduced UUIDMAP_ID and related functions for mapping UUIDs to IDs. Modified metric structure to use UUIDMAP_ID instead of nd_uuid_t. Updated functions to use UUIDMAP_IDs for metric operations.	src/database/engine/metric.c
Refactored metric handling for improved memory management.	Simplified page acquisition and release logic. Removed unnecessary atomic operations. Improved handling of page deletion.	src/database/engine/cache.c
Simplified metric group handling.	Used JudyL instead of JudyHS for storing metric groups. Streamlined metric creation and retrieval. Improved handling of metric references.	src/database/ram/rrddim_mem.c
Updated instance handling to use UUIDMAP_IDs.	Modified instance structure to use UUIDMAP_ID. Updated instance loading and conflict resolution logic.	src/database/contexts/instance.c
Modified metric API to use UUIDMAP_IDs.	Added functions for getting and creating metrics by UUIDMAP_ID. Updated metric handling functions to use UUIDMAP_IDs.	src/database/engine/rrdengineapi.c
Updated rrd collector to use refcounts.	Replaced atomic operations with refcount functions for managing collector references.	src/database/rrdcollector.c
Refactored string handling to use refcounts.	Replaced atomic operations with refcount functions for managing string references. Simplified string duplication and freeing logic.	src/libnetdata/string/string.c
Updated metric structure to use UUIDMAP_ID.	Modified metric structure to use UUIDMAP_ID instead of nd_uuid_t. Updated metric conflict resolution logic.	src/database/contexts/metric.c
Added memory tracking for UUID map.	Added metrics for tracking UUID map memory usage.	src/daemon/pulse/pulse-daemon-memory.c
Removed unnecessary waste metrics.	Removed metrics related to unnecessary spinlocks.	src/daemon/pulse/pulse-db-dbengine.c
Updated dimension handling to use UUIDMAP_IDs.	Modified dimension structure to use UUIDMAP_ID. Updated dimension creation and deletion logic.	src/database/rrddim.c
Simplified stream sender locking.	Used waitq for locking instead of custom locking mechanism.	src/streaming/stream-sender.c
Added UUIDMAP_ID to metric and instance structures.	Added UUIDMAP_ID to metric and instance structures. Removed update_every_s from instance structure.	src/database/contexts/internal.h
Updated storage engine API to use UUIDMAP_IDs.	Added functions for getting metrics by UUIDMAP_ID. Added function for getting metric retention by UUIDMAP_ID.	src/database/storage_engine.c src/database/rrd.h src/database/ram/rrddim_mem.h src/database/engine/rrdengineapi.h
Updated ML dimension loading to use UUIDMAP_IDs.	Modified model loading and updating to use UUIDMAP_IDs.	src/ml/ml.cc
Added UUID map unit test.	Added UUID map unit test to the main test suite.	src/daemon/main.c
Updated context loading to use UUIDMAP_IDs.	Modified context loading to use UUIDMAP_IDs for dimensions and instances.	src/database/contexts/contexts-loading.c
Added API for retrieving contexts by UUID.	Added functionality to retrieve contexts by UUID.	src/database/contexts/api_v1_contexts.c
Updated context functions to use UUIDMAP_IDs.	Modified context functions to use UUIDMAP_IDs for dimensions and charts.	src/database/contexts/rrdcontext.c
Added UUID map implementation.	Implemented UUID map for managing UUID to ID mappings. Added functions for creating, retrieving, and freeing UUID mappings. Added unit tests for UUID map. Integrated UUID map into existing codebase.	src/libnetdata/uuid/uuidmap.c src/libnetdata/uuid/uuidmap.h src/libnetdata/common.h src/libnetdata/libnetdata.h CMakeLists.txt
Updated journal file handling to use UUIDMAP_IDs.	Modified journal file handling to use UUIDMAP_IDs for metrics.	src/database/engine/journalfile.c
Updated page cache handling to use UUIDMAP_IDs.	Modified page cache initialization to use UUIDMAP_IDs.	src/database/engine/pagecache.c
Updated page descriptor handling to use UUIDMAP_IDs.	Modified page descriptor handling to use UUIDMAP_IDs for metrics.	src/database/engine/pdc.c
Updated metric handling to use UUIDMAP_IDs.	Modified metric handling to use UUIDMAP_IDs for updating first time.	src/database/engine/rrdengine.c
Removed loop parameter from health sleep function.	Removed unused loop parameter from health_sleep function.	src/health/health_event_loop.c
Added UUID map to ARAL pulse.	Added UUID map statistics to ARAL pulse.	src/daemon/pulse/pulse-aral.c
Added refcount header file.	Added refcount header file for managing reference counts. Replaced custom refcount implementation with refcount functions.	src/libnetdata/atomics/atomics.h src/libnetdata/atomics/refcount.h src/libnetdata/dictionary/dictionary-internals.h src/database/engine/metric.h
Updated dictionary refcount handling.	Updated dictionary refcount handling to use refcount functions.	src/libnetdata/dictionary/dictionary-refcount.h
Updated log handling.	Added fatal logging functions. Added internal fatal assertion macro.	src/libnetdata/log/nd_log.h src/libnetdata/log/nd_log-fatal.h
Updated rrd engine collect handle options.	Updated rrd engine collect handle options to use enum.	src/database/engine/rrdengine.h

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.