[pull] master from netdata:master

[pull]

See Commits and Changes for more details.

---

Created by pull[bot] (v2.0.0-alpha.1)

Can you help keep this open source service alive? 💖 Please sponsor : )

Summary by Sourcery

Introduce detailed charts and logs for streaming connections, providing insights into connection status, events, and disconnections, categorized by reason. Improve ACLK error handling and logging, offering more specific error messages and context. Refactor streaming connection management to enhance clarity and robustness.

Enhancements:

• Improve logging and error reporting for ACLK and streaming components.

Tests:

• Add tests for ACLK and streaming components.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request introduces a new enum https_client_resp_t to represent the various responses from the HTTPS client. It also adds logging prefixes to the ACLK module and modifies the stream receiver to include a reason for disconnection.

Class diagram for HTTPS client response codes

classDiagram
    class https_client_resp_t {
        <<enumeration>>
        HTTPS_CLIENT_RESP_OK
        HTTPS_CLIENT_RESP_UNKNOWN_ERROR
        HTTPS_CLIENT_RESP_NO_MEM
        HTTPS_CLIENT_RESP_NONBLOCK_FAILED
        HTTPS_CLIENT_RESP_PROXY_NOT_200
        HTTPS_CLIENT_RESP_NO_SSL_CTX
        HTTPS_CLIENT_RESP_NO_SSL_VERIFY_PATHS
        HTTPS_CLIENT_RESP_NO_SSL_NEW
        HTTPS_CLIENT_RESP_NO_TLS_SNI
        HTTPS_CLIENT_RESP_SSL_CONNECT_FAILED
        HTTPS_CLIENT_RESP_SSL_START_FAILED
        HTTPS_CLIENT_RESP_UNKNOWN_REQUEST_TYPE
        HTTPS_CLIENT_RESP_HEADER_WRITE_FAILED
        HTTPS_CLIENT_RESP_PAYLOAD_WRITE_FAILED
        HTTPS_CLIENT_RESP_POLL_ERROR
        HTTPS_CLIENT_RESP_TIMEOUT
        HTTPS_CLIENT_RESP_READ_ERROR
        HTTPS_CLIENT_RESP_PARSE_ERROR
        HTTPS_CLIENT_RESP_ENV_AGENT_NOT_CLAIMED
        HTTPS_CLIENT_RESP_ENV_NOT_200
        HTTPS_CLIENT_RESP_ENV_EMPTY
        HTTPS_CLIENT_RESP_ENV_NOT_JSON
        HTTPS_CLIENT_RESP_OTP_CHALLENGE_NOT_200
        HTTPS_CLIENT_RESP_OTP_CHALLENGE_INVALID
        HTTPS_CLIENT_RESP_OTP_PASSWORD_NOT_201
        HTTPS_CLIENT_RESP_OTP_PASSWORD_EMPTY
        HTTPS_CLIENT_RESP_OTP_PASSWORD_NOT_JSON
        HTTPS_CLIENT_RESP_OTP_AGENT_NOT_CLAIMED
        HTTPS_CLIENT_RESP_OTP_CHALLENGE_DECRYPTION_FAILED
        HTTPS_CLIENT_RESP_MAX
    }
    note for https_client_resp_t "New enumeration for HTTPS client responses"

Loading

Class diagram for stream handshake error codes

classDiagram
    class STREAM_HANDSHAKE {
        <<enumeration>>
        STREAM_HANDSHAKE_OK_V3
        STREAM_HANDSHAKE_OK_V2
        STREAM_HANDSHAKE_OK_V1
        STREAM_HANDSHAKE_NEVER
        STREAM_HANDSHAKE_CONNECT_HANDSHAKE_FAILED
        STREAM_HANDSHAKE_PARENT_IS_LOCALHOST
        STREAM_HANDSHAKE_PARENT_NODE_ALREADY_CONNECTED
        STREAM_HANDSHAKE_PARENT_DENIED_ACCESS
        STREAM_HANDSHAKE_CONNECT_SEND_TIMEOUT
        STREAM_HANDSHAKE_CONNECT_RECEIVE_TIMEOUT
        STREAM_HANDSHAKE_CONNECT_INVALID_CERTIFICATE
        STREAM_HANDSHAKE_CONNECT_SSL_ERROR
        STREAM_HANDSHAKE_CONNECTION_FAILED
        STREAM_HANDSHAKE_PARENT_BUSY_TRY_LATER
        STREAM_HANDSHAKE_PARENT_INTERNAL_ERROR
        STREAM_HANDSHAKE_PARENT_IS_INITIALIZING
        STREAM_HANDSHAKE_RCV_DISCONNECT_PARSER_FAILED
        STREAM_HANDSHAKE_RCV_DISCONNECT_STALE_RECEIVER
        STREAM_HANDSHAKE_RCV_DECOMPRESSION_FAILED
        STREAM_HANDSHAKE_DISCONNECT_SIGNALED_TO_STOP
        STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN
        STREAM_HANDSHAKE_DISCONNECT_SOCKET_ERROR
        STREAM_HANDSHAKE_DISCONNECT_TIMEOUT
        STREAM_HANDSHAKE_DISCONNECT_SOCKET_CLOSED_BY_REMOTE
        STREAM_HANDSHAKE_DISCONNECT_BUFFER_OVERFLOW
        STREAM_HANDSHAKE_DISCONNECT_REPLICATION_STALLED
    }
    note for STREAM_HANDSHAKE "Updated stream handshake error codes with new states"

Loading

State diagram for ACLK connection status

stateDiagram-v2
    [*] --> OFFLINE
    OFFLINE --> CONNECTED: Connection successful
    CONNECTED --> OFFLINE_CLOUD_REQUESTED_DISCONNECT: Cloud requests disconnect
    CONNECTED --> OFFLINE_PING_TIMEOUT: Ping timeout
    CONNECTED --> OFFLINE_RELOADING_CONFIG: Config reload
    CONNECTED --> OFFLINE_POLL_ERROR: Poll error
    CONNECTED --> OFFLINE_CLOSED_BY_REMOTE: Remote closed
    CONNECTED --> OFFLINE_SOCKET_ERROR: Socket error
    CONNECTED --> OFFLINE_MQTT_PROTOCOL_ERROR: MQTT protocol error
    CONNECTED --> OFFLINE_WS_PROTOCOL_ERROR: WebSocket error
    CONNECTED --> OFFLINE_MESSAGE_TOO_BIG: Message too large
    OFFLINE_CLOUD_REQUESTED_DISCONNECT --> OFFLINE
    OFFLINE_PING_TIMEOUT --> OFFLINE
    OFFLINE_RELOADING_CONFIG --> OFFLINE
    OFFLINE_POLL_ERROR --> OFFLINE
    OFFLINE_CLOSED_BY_REMOTE --> OFFLINE
    OFFLINE_SOCKET_ERROR --> OFFLINE
    OFFLINE_MQTT_PROTOCOL_ERROR --> OFFLINE
    OFFLINE_WS_PROTOCOL_ERROR --> OFFLINE
    OFFLINE_MESSAGE_TOO_BIG --> OFFLINE

Loading

File-Level Changes

Change	Details	Files
Added https_client_resp_t enum to represent HTTPS client responses.	Defined the https_client_resp_t enum with various response types. Added string conversion functions for the https_client_resp_t enum. Modified the https_request function to return https_client_resp_t instead of an integer. Updated the read_parse_response function to return https_client_resp_t instead of an integer. Updated the handle_http_request function to return https_client_resp_t instead of an integer.	src/aclk/https_client.c src/aclk/https_client.h
Added logging prefixes to the ACLK module.	Added ACLK: prefix to log messages in https_client.c. Added ACLK: prefix to log messages in aclk.c. Added ACLK: prefix to log messages in aclk_otp.c.	src/aclk/https_client.c src/aclk/aclk.c src/aclk/aclk_otp.c
Modified stream receiver to include a reason for disconnection.	Modified the stream_receiver_remove function to accept a STREAM_HANDSHAKE reason. Modified the stream_receiver_log_status function to accept a STREAM_HANDSHAKE reason. Modified the stream_sender_move_running_to_connector_or_remove function to accept a STREAM_HANDSHAKE reason and a STREAM_HANDSHAKE receiver_reason. Modified the stream_sender_remove function to accept a STREAM_HANDSHAKE reason. Modified the stream_sender_log_disconnection function to accept a STREAM_HANDSHAKE reason and a STREAM_HANDSHAKE receiver_reason. Added new STREAM_HANDSHAKE enum values for various disconnection reasons. Modified the stream_receiver_send_opcode function to include a reason. Modified the stream_sender_send_opcode function to include a reason. Modified the stream_sender_signal_to_stop_and_wait function to include a reason. Modified the rrdhost_clear_receiver function to include a reason. Modified the stream_sender_move_running_to_connector_or_remove function to include a reason. Modified the stream_sender_remove function to include a reason. Modified the stream_receiver_remove function to include a reason. Modified the stream_receiver_log_status function to include a reason. Modified the stream_sender_log_disconnection function to include a reason. Modified the stream_sender_send_data function to include a reason. Modified the stream_receiver_send_data function to include a reason. Modified the stream_receiver_check_all_nodes_from_poll function to include a reason. Modified the stream_sender_check_all_nodes_from_poll function to include a reason. Modified the stream_receiver_replication_check_from_poll function to include a reason. Modified the stream_sender_replication_check_from_poll function to include a reason. Modified the stream_receiver_cleanup function to include a reason. Modified the stream_sender_cleanup function to include a reason. Modified the stream_receiver_move_to_running_unsafe function to include a reason. Modified the stream_sender_move_queue_to_running_unsafe function to include a reason. Modified the stream_connect_validate_first_response function to include a reason. Modified the stream_connect function to include a reason. Modified the stream_connector_remove function to include a reason. Modified the stream_parent_set_reconnect_delay function to include a reason. Modified the stream_parent_set_disconnect_reason function to include a reason. Modified the stream_parent_nd_sock_error_to_reason function to include a reason. Modified the stream_info_fetch function to include a reason. Modified the stream_parent_connect_to_one_unsafe function to include a reason. Modified the stream_sender_send_rrdset_definition function to include a reason. Modified the replication_response_execute_finalize_and_send function to include a reason. Modified the sender_buffer_commit function to include a reason. Modified the stream_receiver_send_opcode function to include a reason. Modified the stream_sender_send_opcode function to include a reason. Modified the stream_receiver_add_to_queue function to include a reason. Modified the stream_sender_add_to_queue function to include a reason. Modified the pulse_host_status function to include a reason. Modified the pulse_parents_do function to include a reason. Modified the pulse_sender_connection_failed function to include a reason. Modified the pulse_sender_stream_info_failed function to include a reason. Modified the pulse_parent_receiver_rejected function to include a reason. Modified the pulse_parent_stream_info_received_request function to include a reason. Modified the pulse_parent_receiver_request function to include a reason. Modified the stream_info_to_json_v1 function to include a reason. Modified the stream_connect_upgrade_prelude function to include a reason. Modified the stream_connect_validate_first_response function to include a reason. Modified the stream_connect function to include a reason. Modified the stream_connector_remove function to include a reason. Modified the stream_parent_set_reconnect_delay function to include a reason. Modified the stream_parent_set_disconnect_reason function to include a reason. Modified the stream_parent_nd_sock_error_to_reason function to include a reason. Modified the stream_info_fetch function to include a reason. Modified the stream_parent_connect_to_one_unsafe function to include a reason. Modified the stream_sender_send_rrdset_definition function to include a reason. Modified the replication_response_execute_finalize_and_send function to include a reason. Modified the sender_buffer_commit function to include a reason. Modified the stream_receiver_send_opcode function to include a reason. Modified the stream_sender_send_opcode function to include a reason. Modified the stream_receiver_add_to_queue function to include a reason. Modified the stream_sender_add_to_queue function to include a reason. Modified the pulse_host_status function to include a reason. Modified the pulse_parents_do function to include a reason. Modified the pulse_sender_connection_failed function to include a reason. Modified the pulse_sender_stream_info_failed function to include a reason. Modified the pulse_parent_receiver_rejected function to include a reason. Modified the pulse_parent_stream_info_received_request function to include a reason. Modified the pulse_parent_receiver_request function to include a reason. Modified the stream_info_to_json_v1 function to include a reason. Modified the stream_connect_upgrade_prelude function to include a reason. Modified the stream_connect_validate_first_response function to include a reason. Modified the stream_connect function to include a reason. Modified the stream_connector_remove function to include a reason. Modified the stream_parent_set_reconnect_delay function to include a reason. Modified the stream_parent_set_disconnect_reason function to include a reason. Modified the stream_parent_nd_sock_error_to_reason function to include a reason. Modified the stream_info_fetch function to include a reason. Modified the stream_parent_connect_to_one_unsafe function to include a reason. Modified the stream_sender_send_rrdset_definition function to include a reason. Modified the replication_response_execute_finalize_and_send function to include a reason. Modified the sender_buffer_commit function to include a reason. Modified the stream_receiver_send_opcode function to include a reason. Modified the stream_sender_send_opcode function to include a reason. Modified the stream_receiver_add_to_queue function to include a reason. Modified the stream_sender_add_to_queue function to include a reason. Modified the pulse_host_status function to include a reason. Modified the pulse_parents_do function to include a reason. Modified the pulse_sender_connection_failed function to include a reason. Modified the pulse_sender_stream_info_failed function to include a reason. Modified the pulse_parent_receiver_rejected function to include a reason. Modified the pulse_parent_stream_info_received_request function to include a reason. Modified the pulse_parent_receiver_request function to include a reason. Modified the stream_info_to_json_v1 function to include a reason. Modified the stream_connect_upgrade_prelude function to include a reason. Modified the stream_connect_validate_first_response function to include a reason. Modified the stream_connect function to include a reason. Modified the stream_connector_remove function to include a reason. Modified the stream_parent_set_reconnect_delay function to include a reason. Modified the stream_parent_set_disconnect_reason function to include a reason. Modified the stream_parent_nd_sock_error_to_reason function to include a reason. Modified the stream_info_fetch function to include a reason. Modified the stream_parent_connect_to_one_unsafe function to include a reason. Modified the stream_sender_send_rrdset_definition function to include a reason. Modified the replication_response_execute_finalize_and_send function to include a reason. Modified the sender_buffer_commit function to include a reason. Modified the stream_receiver_send_opcode function to include a reason. Modified the stream_sender_send_opcode function to include a reason. Modified the stream_receiver_add_to_queue function to include a reason. Modified the stream_sender_add_to_queue function to include a reason. Modified the pulse_host_status function to include a reason. Modified the pulse_parents_do function to include a reason. Modified the pulse_sender_connection_failed function to include a reason. Modified the pulse_sender_stream_info_failed function to include a reason. Modified the pulse_parent_receiver_rejected function to include a reason. Modified the pulse_parent_stream_info_received_request function to include a reason. Modified the pulse_parent_receiver_request function to include a reason.	src/streaming/stream-receiver.c src/streaming/stream-handshake.h src/streaming/stream-handshake.c src/streaming/stream-receiver-connection.c src/streaming/stream-sender.c src/streaming/stream-thread.h src/streaming/stream-connector.c src/streaming/stream-parents.c src/database/rrdhost.c src/streaming/stream-sender-api.c src/plugins.d/pluginsd_replication.c src/streaming/protocol/command-chart-definition.c src/plugins.d/pluginsd_parser.h src/streaming/stream-receiver-internals.h src/database/sqlite/sqlite_aclk.c src/libnetdata/socket/nd-sock.c src/libnetdata/socket/nd-sock.h src/aclk/mqtt_websockets/mqtt_wss_client.c src/aclk/mqtt_websockets/mqtt_wss_client.h src/aclk/aclk.h src/aclk/aclk_otp.h src/daemon/pulse/pulse.c src/daemon/pulse/pulse.h src/daemon/pulse/pulse-parents.c src/daemon/pulse/pulse-parents.h src/database/sqlite/sqlite_metadata.c src/libnetdata/uuid/uuid.h src/collectors/systemd-journal.plugin/systemd-journal-annotations.c src/streaming/stream-sender-commit.c src/plugins.d/pluginsd_parser.c src/streaming/stream-replication-sender.c src/packaging/version src/database/rrd.c src/libnetdata/uuid/uuid.h

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!
• Generate a plan of action for an issue: Comment @sourcery-ai plan on
  an issue to generate a plan of action for it.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.