Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ycabled][active-active] Fix in gRPC channel callback logic by creating swsscommon table within the context #509

Merged
merged 7 commits into from
Jul 17, 2024

Conversation

vdahiya12
Copy link
Contributor

@vdahiya12 vdahiya12 commented Jul 4, 2024

This PR Introduces a an initialization mechanism within the wait_for_state_change function to ensure that the YcableChannelStateTableHelper instance is created only once per port.
Also it Adds a check using hasattr to see if table_helper is already an attribute of wait_for_state_change.
YcableChannelStateTableHelper instances are keyed by port.

This helps ensure that channel object will have a seperate instance of swsscommon table and they will not interfere with normal functioning of state specific threads.

Description

MSFT ADO

28724244

Motivation and Context

How Has This Been Tested?

UTs and deploying changes on Libra Testbeds as well as pilot devices
This script was used to test and after each run the output was matched to be as expected.
The fwd_state table is continuously written when link goes down and comes up
unknown and active respectively.

admin@svcstr-7050-acs-1:~$ cat link_off.sh 
#!/bin/bash



declare -a runs=100

for ((i = 1; i <= 5000; i++));
do 
    sleep 1
    echo "*******shutting down the link*****************"
    sudo config interface shutdown Ethernet12
    sleep 1
    show mux status Ethernet12
    show mux hw mux Ethernet12
    redis-cli -n 0 hgetall "FORWARDING_STATE_RESPONSE:Ethernet12"
    sleep 1
    echo "***********starting up****************"
    sudo config interface start Ethernet12
    sleep 1
    show mux status Ethernet12
    show mux hw mux Ethernet12
    redis-cli -n 0 hgetall "FORWARDING_STATE_RESPONSE:Ethernet12"
    sleep 100
    echo "**********after long sleep************"
    show mux status Ethernet12
    show mux hw mux Ethernet12
    redis-cli -n 0 hgetall "FORWARDING_STATE_RESPONSE:Ethernet12"

done

example output for one iteration

*******shutting down the link*****************
PORT        STATUS    SERVER_STATUS    HEALTH     HWSTATUS      LAST_SWITCHOVER_TIME
----------  --------  ---------------  ---------  ------------  ----------------------
Ethernet12  standby   unknown          unhealthy  inconsistent
Port        Direction    Presence    PeerDirection    ConnectivityState
----------  -----------  ----------  ---------------  -------------------
Ethernet12  active       True        active           READY
response
unknown
response_peer
unknown
***********starting up****************
PORT        STATUS    SERVER_STATUS    HEALTH    HWSTATUS    LAST_SWITCHOVER_TIME
----------  --------  ---------------  --------  ----------  ---------------------------
Ethernet12  active    active           healthy   consistent  2024-Jul-15 23:51:32.477858
Port        Direction    Presence    PeerDirection    ConnectivityState
----------  -----------  ----------  ---------------  -------------------
Ethernet12  active       True        active           READY
response
active
response_peer
active

we can also check in syslog how the grpc connectivity state change syslog is posted with each link/up down

Jul 15 23:53:30.621827 svcstr-7050-acs-1 NOTICE pmon#CCmisApi: gRPC port Ethernet12 state changed to IDLE tid Thread-31
Jul 15 23:53:32.457226 svcstr-7050-acs-1 NOTICE pmon#CCmisApi: gRPC port Ethernet12 state changed to CONNECTING tid Thread-32
Jul 15 23:53:32.459680 svcstr-7050-acs-1 NOTICE pmon#CCmisApi: gRPC port Ethernet12 state changed to READY tid Thread-33

Additional Information (Optional)

another way to check thread function is correct is through logging, as seen fwd table is created in different threads

Jul  8 06:22:10.780902 svcstr-7050-acs-2 NOTICE pmon#CCmisApi:  created fwd table id Thread-2
Jul  8 06:22:10.806261 svcstr-7050-acs-2 NOTICE pmon#CCmisApi:  created fwd table id Thread-5
Jul  8 06:22:10.810397 svcstr-7050-acs-2 NOTICE pmon#CCmisApi: created fwd table id Thread-8

swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
@vdahiya12 vdahiya12 requested review from lolyu and prgeor July 4, 2024 02:15
Copy link

@zjswhhh zjswhhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have UTs for ycabled?

vdahiya12 added 6 commits July 8, 2024 06:48
Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Copy link

@zjswhhh zjswhhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

Please wait for @prgeor's signoff before merging.

@prgeor prgeor merged commit 8c89f6b into sonic-net:master Jul 17, 2024
5 checks passed
mssonicbld pushed a commit to mssonicbld/sonic-platform-daemons that referenced this pull request Jul 18, 2024
…ng swsscommon table within the context (sonic-net#509)

* [ycabled][active-active] Fix in gRPC channel callback logic by creating
swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* fix UT

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add more tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* typo

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add port

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add logging

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

---------

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
mssonicbld pushed a commit to mssonicbld/sonic-platform-daemons that referenced this pull request Jul 18, 2024
…ng swsscommon table within the context (sonic-net#509)

* [ycabled][active-active] Fix in gRPC channel callback logic by creating
swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* fix UT

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add more tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* typo

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add port

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add logging

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

---------

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
mssonicbld pushed a commit to mssonicbld/sonic-platform-daemons that referenced this pull request Jul 18, 2024
…ng swsscommon table within the context (sonic-net#509)

* [ycabled][active-active] Fix in gRPC channel callback logic by creating
swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* fix UT

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add more tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* typo

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add port

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add logging

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

---------

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202311: #516

@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202305: #517

@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202405: #518

mssonicbld pushed a commit that referenced this pull request Jul 19, 2024
…ng swsscommon table within the context (#509)

* [ycabled][active-active] Fix in gRPC channel callback logic by creating
swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* fix UT

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add more tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* typo

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add port

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add logging

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

---------

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
mssonicbld pushed a commit that referenced this pull request Jul 19, 2024
…ng swsscommon table within the context (#509)

* [ycabled][active-active] Fix in gRPC channel callback logic by creating
swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* fix UT

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add more tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* typo

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add port

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add logging

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

---------

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
mssonicbld pushed a commit that referenced this pull request Jul 19, 2024
…ng swsscommon table within the context (#509)

* [ycabled][active-active] Fix in gRPC channel callback logic by creating
swsscommon table within the context

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* fix UT

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add more tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* typo

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add port

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add logging

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

* add tests

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>

---------

Signed-off-by: Vaibhav Dahiya <vdahiya@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants