Add syslog forwarding rules validation and thread exception handling

[ytzur1]

Add validation of rsyslog message from conf file like telemetry, bgp, gnmi, etc.
Implement thread exception handler for sub thread error capture

Description of PR

1. Adding validation of rsyslog messages from configuration files (telemetry, bgp, gnmi, etc.)
2. Implementing a thread exception handler for capturing subthread errors

Summary:
Fixes # (issue)

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• New Test case
• Skipped for non-supported platforms
• Test case improvement

Back port request

• 202012
• 202205
• 202305
• 202311
• 202405
• 202411

Approach

What is the motivation for this PR?

The current syslog testing framework lacks validation of rsyslog messages from various configuration files. Additionally, exception during subthreads were not being caught by the main thread.

How did you do it?

1. Added rsyslog message validation:

   • Implemented verification of logger messages from telemetry, bgp, gnmi,

2. Implemented thread-safe exception handling:

   • Created a global thread_exceptions list for capturing errors
   • Added ThreadWrapper class for safe exception capture
   • Implemented handle_thread_exceptions() for error reporting in main thread

How did you verify/test it?

Ran all test cases in test_syslog_source_ip.py

Any platform specific information?

No

Supported testbed topology if it's a new test case?

No

Documentation

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[bingwang-ms]

Related image change sonic-net/sonic-buildimage#21923

[liuh-80]

This command run in host.

However, the config causes this issue are:
'''
if re_match($programname, "bgp[0-9]*#(frr|zebra|staticd|watchfrr)") then {
/var/log/frr/zebra.log
stop
}
'''

Which means when reproduce the issue, the log send inside bgp docker container, then the 'programname' part can match.

I warry about this test case can't prevent regression. can you test with an image have this issue to make sure this test can catch issue?

[qiluo-msft]

This is good suggestion: test with an image have this issue to make sure this test can catch issue

[ytzur1]

Tested with an image that had the configuration issue, and the test failed and reporting that not all types of forwarded messages were successful.

[zbud-msft]

Hi @ytzur1, I manually ran your test changes from our test server and seeing following issues when running the testcase. Can you please double check your logic in capture_syslog_packets() function in syslog/syslog_utils.py

25/03/2025 23:13:49 base._run                                L0108 DEBUG  | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_Vrf-data.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_Vrf-data.pcap"}}, "_ansible_no_log": false, "changed": false}
25/03/2025 23:13:49 base._run                                L0108 DEBUG  | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_default.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_default.pcap"}}, "_ansible_no_log": false, "changed": false}
25/03/2025 23:13:49 base._run                                L0108 DEBUG  | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_mgmt.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_mgmt.pcap"}}, "_ansible_no_log": false, "changed": false}

[zbud-msft]

Check logic for capture_syslog_packets, seems to run into "file not found: /tmp/X.pcap" issue

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ytzur1]

Hi, @zbud-msft
The error was caused by a missing tab in tcpdump cmd command after pre-commit formatting of long lines.
It's fixed and latest version was tested and it's working.

Hi @ytzur1, I manually ran your test changes from our test server and seeing following issues when running the testcase. Can you please double check your logic in capture_syslog_packets() function in syslog/syslog_utils.py

25/03/2025 23:13:49 base._run                                L0108 DEBUG  | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_Vrf-data.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_Vrf-data.pcap"}}, "_ansible_no_log": false, "changed": false} 25/03/2025 23:13:49 base._run                                L0108 DEBUG  | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_default.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_default.pcap"}}, "_ansible_no_log": false, "changed": false} 25/03/2025 23:13:49 base._run                                L0108 DEBUG  | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_mgmt.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_mgmt.pcap"}}, "_ansible_no_log": false, "changed": false}

[zbud-msft]

@ytzur1 Would it be possible to post screenshot of testcases passing since kvm tests will skip?