Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[202411][frr]: Local subnet routes are not propagated to the data plane #21900

Open
nazariig opened this issue Mar 3, 2025 · 5 comments
Open

[202411][frr]: Local subnet routes are not propagated to the data plane #21900

nazariig opened this issue Mar 3, 2025 · 5 comments
Labels
Bug 🐛 FRR 🚥 Triaged this issue has been triaged

Comments

@nazariig
Copy link
Collaborator

nazariig commented Mar 3, 2025

Description

Local subnet routes are not propagated to the data plane.
Only IP2ME routes are present, while no updates are received by fpmsyncd.

On attempt to configure fpm address, zebra crash is observed:

sonic(config)# fpm address 127.0.0.1

FRR version:

root@sonic:/# dpkg --list | grep frr
ii  frr                                10.0.1-sonic-0                 amd64        FRRouting suite of internet protocols (BGP, OSPF, IS-IS, ...)
ii  frr-snmp                           10.0.1-sonic-0                 amd64        FRRouting suite - SNMP support

root@sonic:/# /usr/lib/frr/zebra --version
zebra version 10.0.1
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
        '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--disable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-multipath=514' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

FRR config:

sonic# show running-config
Building configuration...

Current configuration:
!
frr version 10.0.1
frr defaults traditional
hostname sonic
log syslog informational
log facility local4
agentx
no service integrated-vtysh-config
!
password zebra
enable password zebra
!
vrf Vrf1
 vni 500200
 ip nht resolve-via-default
 ipv6 nht resolve-via-default
exit-vrf
!
router bgp 65000
 bgp router-id 10.1.0.32
 no bgp enforce-first-as
 bgp graceful-restart restart-time 300
 bgp graceful-restart
 bgp graceful-restart preserve-fw-state
 neighbor 30.0.0.2 remote-as 65000
 neighbor 40.0.0.3 remote-as 65000
 !
 address-family ipv4 unicast
  network 10.1.0.32/32
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor 30.0.0.2 activate
  neighbor 40.0.0.3 activate
  advertise-all-vni
  vni 500101
   rd 10.1.0.32:101
  exit-vni
  vni 500100
   rd 10.1.0.32:100
  exit-vni
  vni 50020
   rd 10.1.0.32:20
  exit-vni
  advertise-svi-ip
 exit-address-family
exit
!
router bgp 65000 vrf Vrf1
 no bgp enforce-first-as
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  advertise ipv6 unicast
  rd 10.1.0.32:200
 exit-address-family
exit
!
route-map RM_SET_SRC permit 10
 set src 10.1.0.32
exit
!
ip protocol bgp route-map RM_SET_SRC
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
end

SYSLOG:

2025 Feb 28 14:47:34.089026 sonic INFO bgp#fpmsyncd: :- accept: New connection accepted from: 127.0.0.1
2025 Feb 28 14:47:34.089391 sonic INFO bgp#supervisord: fpmsyncd Connected!
2025 Feb 28 14:47:34.089692 sonic NOTICE bgp#fpmsyncd: :- setWarmStartState: bgp warm start state changed to disabled
2025 Feb 28 14:47:34.090725 sonic CRIT bgp#ZEBRA[36]: Received signal 11 at 1740754054 (si_addr 0x0, PC 0x7f100bf6dee3); aborting...
2025 Feb 28 14:47:34.091376 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.091451 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 10.0.1.0/24
2025 Feb 28 14:47:34.091451 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.091511 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 10.1.0.32
2025 Feb 28 14:47:34.091511 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.091567 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 10.210.24.0/22
2025 Feb 28 14:47:34.091567 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.091623 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 30.0.0.0/24
2025 Feb 28 14:47:34.091623 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.091840 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 40.0.0.0/24
2025 Feb 28 14:47:34.091971 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.091971 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 50.0.0.0/24
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 69.0.0.0/24
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 69.0.1.0/24
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 100.0.0.0/24
2025 Feb 28 14:47:34.092059 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 101.0.0.0/24
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 120.0.0.0/24
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 121.0.0.0/24
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092194 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 122.0.0.0/24
2025 Feb 28 14:47:34.092288 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092288 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 123.0.0.0/24
2025 Feb 28 14:47:34.092288 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092499 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 124.0.0.0/24
2025 Feb 28 14:47:34.092499 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092499 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 125.0.0.0/24
2025 Feb 28 14:47:34.092499 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092499 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 100::/64
2025 Feb 28 14:47:34.092499 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092733 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 101::/64
2025 Feb 28 14:47:34.092733 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092733 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 3000::/64
2025 Feb 28 14:47:34.092812 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092812 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 4000::/64
2025 Feb 28 14:47:34.092812 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092829 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 6900::/64
2025 Feb 28 14:47:34.092829 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092841 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 6900:1::/64
2025 Feb 28 14:47:34.092852 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092948 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 7000:120::/64
2025 Feb 28 14:47:34.092948 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092948 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 7000:121::/64
2025 Feb 28 14:47:34.092948 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092968 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 7000:122::/64
2025 Feb 28 14:47:34.092968 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.092980 sonic CRIT bgp#ZEBRA[36]: zlog_signal+0xf5                   7f100c588e25     7ffe53108eb0 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f100c4c6000)
2025 Feb 28 14:47:34.092991 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 7000:123::/64
2025 Feb 28 14:47:34.092991 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093006 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 7000:124::/64
2025 Feb 28 14:47:34.093006 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for 7000:125::/64
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for fd00::/80
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for fe80::/64
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:20.0.0.0/24
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:20.0.0.2
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:200.0.0.0/24
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:20::/64
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:20::2
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:200::/64
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- isRawProcessing: Rx MsgType:24 Encap:0
2025 Feb 28 14:47:34.093369 sonic INFO bgp#fpmsyncd: :- onRouteMsg: Nexthop list is empty for Vrf1:fe80::/64
2025 Feb 28 14:47:34.094397 sonic CRIT bgp#ZEBRA[36]: PBKDF2_SHA256+0x4c1                7f100c5c2401     7ffe53108ff0 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f100c4c6000)
2025 Feb 28 14:47:34.095628 sonic CRIT bgp#ZEBRA[36]: __sigaction+0x40                   7f100c1d0050     7ffe53109140 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7f100c194000)
2025 Feb 28 14:47:34.096493 sonic CRIT bgp#ZEBRA[36]:     ---- signal ----
2025 Feb 28 14:47:34.096493 sonic CRIT bgp#ZEBRA[36]: ?                                  7f100bf6dee3     7ffe53109700 /usr/lib/x86_64-linux-gnu/frr/modules/dplane_fpm_sonic.so (mapped at 0x7f100bf64000)
2025 Feb 28 14:47:34.097578 sonic CRIT bgp#ZEBRA[36]: hash_iterate+0x4d                  7f100c574afd     7ffe53109750 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f100c4c6000)
2025 Feb 28 14:47:34.098667 sonic CRIT bgp#ZEBRA[36]: hash_iterate+0x4d                  7f100c574afd     7ffe53109790 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f100c4c6000)
2025 Feb 28 14:47:34.099446 sonic CRIT bgp#ZEBRA[36]: ?                                  7f100bf6ad39     7ffe531097d0 /usr/lib/x86_64-linux-gnu/frr/modules/dplane_fpm_sonic.so (mapped at 0x7f100bf64000)
2025 Feb 28 14:47:34.100570 sonic CRIT bgp#ZEBRA[36]: event_call+0x81                    7f100c5d4af1     7ffe53109820 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f100c4c6000)
2025 Feb 28 14:47:34.101617 sonic CRIT bgp#ZEBRA[36]: frr_run+0xc0                       7f100c580480     7ffe531098c0 /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0 (mapped at 0x7f100c4c6000)
2025 Feb 28 14:47:34.102184 sonic CRIT bgp#ZEBRA[36]: main+0x3e6                         5606795575c6     7ffe531099e0 /usr/lib/frr/zebra (mapped at 0x5606794d2000)
2025 Feb 28 14:47:34.103174 sonic CRIT bgp#ZEBRA[36]: __libc_init_first+0x8a             7f100c1bb24a     7ffe53109ae0 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7f100c194000)
2025 Feb 28 14:47:34.104208 sonic CRIT bgp#ZEBRA[36]: __libc_start_main+0x85             7f100c1bb305     7ffe53109b80 /lib/x86_64-linux-gnu/libc.so.6 (mapped at 0x7f100c194000)
2025 Feb 28 14:47:34.104767 sonic CRIT bgp#ZEBRA[36]: _start+0x21                        5606795585a1     7ffe53109bd0 /usr/lib/frr/zebra (mapped at 0x5606794d2000)
2025 Feb 28 14:47:34.104767 sonic CRIT bgp#ZEBRA[36]: in thread fpm_rmac_send scheduled from ../zebra/dplane_fpm_sonic.c:1967 fpm_rmac_reset()
2025 Feb 28 14:47:34.105059 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group libfrr
2025 Feb 28 14:47:34.105059 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Buffer                        :      6 *         24
2025 Feb 28 14:47:34.105110 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Host config                   :      8 * (variably sized)
2025 Feb 28 14:47:34.105110 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Command Tokens                :   3525 *         72
2025 Feb 28 14:47:34.105110 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Command Token Text            :   2433 * (variably sized)
2025 Feb 28 14:47:34.105173 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Command Token Help            :   2433 * (variably sized)
2025 Feb 28 14:47:34.105173 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Command Argument Name         :    697 * (variably sized)
2025 Feb 28 14:47:34.105173 sonic INFO bgp#supervisord: zebra core_handler: memstats:  RCU thread                    :      7 *        128
2025 Feb 28 14:47:34.105237 sonic INFO bgp#supervisord: zebra core_handler: memstats:  RCU sequence barrier          :      1 *         32
2025 Feb 28 14:47:34.105237 sonic INFO bgp#supervisord: zebra core_handler: memstats:  FRR POSIX Thread              :     14 * (variably sized)
2025 Feb 28 14:47:34.105237 sonic INFO bgp#supervisord: zebra core_handler: memstats:  POSIX sync primitives         :     14 * (variably sized)
2025 Feb 28 14:47:34.105237 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Graph                         :     30 *          8
2025 Feb 28 14:47:34.105996 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Graph Node                    :   4105 *         32
2025 Feb 28 14:47:34.106043 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Hash                          :    158 * (variably sized)
2025 Feb 28 14:47:34.106137 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Hash Bucket                   :    996 *         24
2025 Feb 28 14:47:34.106137 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Hash Index                    :     79 * (variably sized)
2025 Feb 28 14:47:34.106137 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Interface                     :     70 *        272
2025 Feb 28 14:47:34.106137 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Connected                     :     97 *         64
2025 Feb 28 14:47:34.106137 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Link List                     :    347 *         40
2025 Feb 28 14:47:34.106287 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Link Node                     :    200 *         24
2025 Feb 28 14:47:34.106287 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Temporary memory              :      8 * (variably sized)
2025 Feb 28 14:47:34.106287 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Bitfield memory               :      4 * (variably sized)
2025 Feb 28 14:47:34.106287 sonic INFO bgp#supervisord: zebra core_handler: memstats:  backend client                :      1 *        480
2025 Feb 28 14:47:34.106287 sonic INFO bgp#supervisord: zebra core_handler: memstats:  backend client name           :      1 *          6
2025 Feb 28 14:47:34.106287 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Module loading name           :      2 * (variably sized)
2025 Feb 28 14:47:34.106340 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Nexthop                       :     90 *        160
2025 Feb 28 14:47:34.106442 sonic INFO bgp#supervisord: zebra core_handler: memstats:  NetNS Context                 :      2 * (variably sized)
2025 Feb 28 14:47:34.106442 sonic INFO bgp#supervisord: zebra core_handler: memstats:  NetNS Name                    :      1 *         18
2025 Feb 28 14:47:34.106442 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Northbound Node               :    780 *       1208
2025 Feb 28 14:47:34.106442 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Northbound Configuration      :      2 *         16
2025 Feb 28 14:47:34.106442 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Northbound Configuration Entry:      5 *       1032
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Prefix                        :     97 *         56
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Privilege information         :      4 * (variably sized)
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route map                     :      1 *        120
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route map name                :      2 *         11
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route map index               :      1 *        152
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route map rule                :      1 *         40
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route map rule str            :      1 *         10
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route map compiled            :      1 *         16
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Stream                        :     11 * (variably sized)
2025 Feb 28 14:47:34.106733 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Stream FIFO                   :      9 *         72
2025 Feb 28 14:47:34.106828 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route table                   :     93 *         56
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route node                    :    156 * (variably sized)
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Thread                        :     37 *        160
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Thread master                 :     32 * (variably sized)
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Thread Poll Info              :     16 *     800000
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Thread stats                  :     54 *        112
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Typed-hash bucket             :     40 * (variably sized)
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Vector                        :   8271 *         24
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Vector index                  :   8271 * (variably sized)
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  VRF                           :      2 *        216
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  VRF bit-map                   :     10 *          8
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  VTY                           :      2 * (variably sized)
2025 Feb 28 14:47:34.107263 sonic INFO bgp#supervisord: zebra core_handler: memstats:  VTY server                    :      2 *         32
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Work queue                    :      2 * (variably sized)
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Work queue name string        :      1 *         22
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: memstats:  YANG module                   :      8 *         48
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: memstats:  log thread-local buffer       :      8 *      24608
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group Native message allocations
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group logging subsystem
2025 Feb 28 14:47:34.107365 sonic INFO bgp#supervisord: zebra core_handler: memstats:  syslog target                 :      1 *         56
2025 Feb 28 14:47:34.107422 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group Label Manager
2025 Feb 28 14:47:34.107422 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group Table Manager
2025 Feb 28 14:47:34.107422 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Table Manager Context         :      1 *         16
2025 Feb 28 14:47:34.107422 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group SRv6 Manager
2025 Feb 28 14:47:34.107422 sonic INFO bgp#supervisord: zebra core_handler: showing active allocations in memory group zebra
2025 Feb 28 14:47:34.107458 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra Interface Information   :     70 *        616
2025 Feb 28 14:47:34.107458 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra Netlink buffers         :      6 * (variably sized)
2025 Feb 28 14:47:34.107458 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Router Advertisement Prefix   :     13 *         80
2025 Feb 28 14:47:34.107481 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra DPlane Ctx              :      6 *       4608
2025 Feb 28 14:47:34.107481 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra DPlane Provider         :      2 *        240
2025 Feb 28 14:47:34.107499 sonic INFO bgp#supervisord: zebra core_handler: memstats:  DPlane NSes                   :      1 *         48
2025 Feb 28 14:47:34.107499 sonic INFO bgp#supervisord: zebra core_handler: memstats:  GR                            :      1 * (variably sized)
2025 Feb 28 14:47:34.107934 sonic INFO bgp#supervisord: zebra core_handler: memstats:  VNI hash                      :      4 *        104
2025 Feb 28 14:47:34.107934 sonic INFO bgp#supervisord: zebra core_handler: memstats:  VNI remote VTEP               :      4 *         24
2025 Feb 28 14:47:34.107934 sonic INFO bgp#supervisord: zebra core_handler: memstats:  EVPN MAC                      :     14 *        192
2025 Feb 28 14:47:34.107934 sonic INFO bgp#supervisord: zebra core_handler: memstats:  EVI Neighbor                  :     26 *        144
2025 Feb 28 14:47:34.107963 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Nexthop Group Entry           :     90 *        144
2025 Feb 28 14:47:34.107963 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Nexthop Group Connected       :     87 *         40
2025 Feb 28 14:47:34.107981 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra Name Space              :      1 *        592
2025 Feb 28 14:47:34.107981 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra NS Ifp                  :     70 *         56
2025 Feb 28 14:47:34.107997 sonic INFO bgp#supervisord: zebra core_handler: memstats:  PTM BFD process reg table     :      1 *         32
2025 Feb 28 14:47:34.107997 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Route Entry                   :    201 *        152
2025 Feb 28 14:47:34.108014 sonic INFO bgp#supervisord: zebra core_handler: memstats:  RIB destination               :     78 *         88
2025 Feb 28 14:47:34.108014 sonic INFO bgp#supervisord: zebra core_handler: memstats:  RIB table info                :      9 *         24
2025 Feb 28 14:47:34.108031 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra VRF table               :      9 *         56
2025 Feb 28 14:47:34.108052 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Nexthop tracking object       :      3 *        256
2025 Feb 28 14:47:34.108052 sonic INFO bgp#supervisord: zebra core_handler: memstats:  ZEBRA VRF                     :      2 *       5328
2025 Feb 28 14:47:34.108052 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Other Table                   :      1 *         56
2025 Feb 28 14:47:34.108074 sonic INFO bgp#supervisord: zebra core_handler: memstats:  host prefix                   :      5 *         88
2025 Feb 28 14:47:34.108074 sonic INFO bgp#supervisord: zebra core_handler: memstats:  L3 VNI hash                   :      1 *         80
2025 Feb 28 14:47:34.108103 sonic INFO bgp#supervisord: zebra core_handler: memstats:  EVPN L3VNI MAC                :      1 *        192
2025 Feb 28 14:47:34.108103 sonic INFO bgp#supervisord: zebra core_handler: memstats:  EVPN Neighbor                 :      2 *        144
2025 Feb 28 14:47:34.108103 sonic INFO bgp#supervisord: zebra core_handler: memstats:  zebra VxLAN VTEP IP           :      1 *         20
2025 Feb 28 14:47:34.108103 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Access Broadcast Domain       :      7 *         56
2025 Feb 28 14:47:34.108103 sonic INFO bgp#supervisord: zebra core_handler: memstats:  MH global info                :      1 *        128
2025 Feb 28 14:47:34.108178 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra neigh table             :      1 *          8
2025 Feb 28 14:47:34.108178 sonic INFO bgp#supervisord: zebra core_handler: memstats:  Zebra neigh entry             :     51 *         80
2025 Feb 28 14:47:34.108178 sonic INFO bgp#supervisord: zebra core_handler: memstats:  ZClients                      :      4 *       3784
2025 Feb 28 14:47:34.827647 sonic ERR bgp#mgmtd[35]: [X3G8F-PM93W] BE-adapter: mgmt_msg_read: got EOF/disconnect
2025 Feb 28 14:47:34.827647 sonic INFO bgp#supervisord: fpmsyncd Connection lost, reconnecting...
2025 Feb 28 14:47:34.827821 sonic INFO bgp#supervisord: fpmsyncd Waiting for fpm-client connection...
2025 Feb 28 14:47:34.828132 sonic INFO bgp#supervisord 2025-02-28 14:47:34,827 WARN exited: zebra (terminated by SIGSEGV (core dumped); not expected)
2025 Feb 28 14:47:35.832386 sonic INFO bgp#supervisor-proc-exit-listener: Process 'zebra' exited unexpectedly. Terminating supervisor 'bgp'
2025 Feb 28 14:47:35.832560 sonic NOTICE bgp#supervisor-proc-exit-listener: :- publish: EVENT_PUBLISHED: {"sonic-events-host:process-exited-unexpectedly":{"ctr_name":"bgp","process_name":"zebra","timestamp":"2025-02-28T14:47:35.832408Z"}}
2025 Feb 28 14:47:35.833170 sonic INFO bgp#supervisord 2025-02-28 14:47:35,832 WARN received SIGTERM indicating exit request
2025 Feb 28 14:47:35.833320 sonic INFO bgp#supervisord 2025-02-28 14:47:35,833 INFO waiting for supervisor-proc-exit-listener, rsyslogd, mgmtd, staticd, bgpd, bgpcfgd, bgpmon, fpmsyncd, staticroutebfd to die
2025 Feb 28 14:47:36.836217 sonic INFO bgp#supervisord 2025-02-28 14:47:36,835 INFO stopped: staticroutebfd (exit status 0)
2025 Feb 28 14:47:36.836966 sonic INFO bgp#supervisord 2025-02-28 14:47:36,836 WARN stopped: fpmsyncd (terminated by SIGTERM)
2025 Feb 28 14:47:36.838136 sonic INFO bgp#supervisord 2025-02-28 14:47:36,837 WARN stopped: bgpmon (terminated by SIGTERM)
2025 Feb 28 14:47:38.840455 sonic INFO bgp#supervisord 2025-02-28 14:47:38,840 INFO waiting for supervisor-proc-exit-listener, rsyslogd, mgmtd, staticd, bgpd, bgpcfgd to die
2025 Feb 28 14:47:42.619276 sonic INFO bgp#supervisord 2025-02-28 14:47:42,618 INFO waiting for supervisor-proc-exit-listener, rsyslogd, mgmtd, staticd, bgpd, bgpcfgd to die
2025 Feb 28 14:47:45.622144 sonic INFO bgp#supervisord 2025-02-28 14:47:45,621 INFO waiting for supervisor-proc-exit-listener, rsyslogd, mgmtd, staticd, bgpd, bgpcfgd to die
2025 Feb 28 14:47:47.624237 sonic INFO bgp#supervisord 2025-02-28 14:47:47,623 WARN killing 'bgpcfgd' (54) with SIGKILL
2025 Feb 28 14:47:47.626207 sonic INFO bgp#supervisord 2025-02-28 14:47:47,626 WARN stopped: bgpcfgd (terminated by SIGKILL)
2025 Feb 28 14:47:47.627538 sonic INFO bgp#supervisord 2025-02-28 14:47:47,627 WARN stopped: bgpd (terminated by SIGKILL)
2025 Feb 28 14:47:48.628646 sonic INFO bgp#supervisord 2025-02-28 14:47:48,628 INFO waiting for supervisor-proc-exit-listener, rsyslogd, mgmtd, staticd to die
2025 Feb 28 14:47:49.629999 sonic ERR bgp#mgmtd[35]: [X3G8F-PM93W] BE-adapter: mgmt_msg_read: got EOF/disconnect
2025 Feb 28 14:47:49.632806 sonic INFO bgp#supervisord: staticd staticd: showing active allocations in memory group libfrr
2025 Feb 28 14:47:49.632806 sonic INFO bgp#supervisord: staticd staticd: showing active allocations in memory group Native message allocations
2025 Feb 28 14:47:49.632806 sonic INFO bgp#supervisord: staticd staticd: showing active allocations in memory group logging subsystem
2025 Feb 28 14:47:49.632806 sonic INFO bgp#supervisord: staticd staticd: showing active allocations in memory group staticd
2025 Feb 28 14:47:49.633617 sonic INFO bgp#supervisord 2025-02-28 14:47:49,633 INFO stopped: staticd (exit status 0)
2025 Feb 28 14:47:49.633812 sonic NOTICE bgp#mgmtd[35]: [J2RAS-MZ95C] Terminating on signal
2025 Feb 28 14:47:49.641618 sonic INFO bgp#supervisord: mgmtd mgmtd: showing active allocations in memory group libfrr
2025 Feb 28 14:47:49.641654 sonic INFO bgp#supervisord: mgmtd mgmtd: showing active allocations in memory group Native message allocations
2025 Feb 28 14:47:49.641654 sonic INFO bgp#supervisord: mgmtd mgmtd: showing active allocations in memory group logging subsystem
2025 Feb 28 14:47:49.641668 sonic INFO bgp#supervisord: mgmtd mgmtd: memstats:  syslog target                 :      1 *         56
2025 Feb 28 14:47:49.641668 sonic INFO bgp#supervisord: mgmtd mgmtd: showing active allocations in memory group mgmt
2025 Feb 28 14:47:49.642460 sonic INFO bgp#supervisord 2025-02-28 14:47:49,642 INFO stopped: mgmtd (exit status 0)
2025 Feb 28 14:47:49.749131 sonic INFO containerd[1301]: time="2025-02-28T14:47:49.748566462Z" level=info msg="shim disconnected" id=417aca897ad5850df1e45037a562a747cc85c70ef437ee97fb449bffcfefa829
2025 Feb 28 14:47:49.749264 sonic INFO containerd[1301]: time="2025-02-28T14:47:49.748644154Z" level=warning msg="cleaning up after shim disconnected" id=417aca897ad5850df1e45037a562a747cc85c70ef437ee97fb449bffcfefa829 namespace=moby
2025 Feb 28 14:47:49.749323 sonic INFO containerd[1301]: time="2025-02-28T14:47:49.748666599Z" level=info msg="cleaning up dead shim"
2025 Feb 28 14:47:49.749758 sonic INFO dockerd[1377]: time="2025-02-28T14:47:49.748643509Z" level=info msg="ignoring event" container=417aca897ad5850df1e45037a562a747cc85c70ef437ee97fb449bffcfefa829 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025 Feb 28 14:47:49.759117 sonic INFO containerd[1301]: time="2025-02-28T14:47:49.758960872Z" level=warning msg="cleanup warnings time=\"2025-02-28T14:47:49Z\" level=info msg=\"starting signal loop\" namespace=moby pid=185394 runtime=io.containerd.runc.v2\n"
2025 Feb 28 14:47:49.772101 sonic INFO systemd[1]: var-lib-docker-overlay2-b8c39af3df1a28267e2719bd50a5e7bb69d5c9ebe5265bf4a49651a2def42dd2-merged.mount: Deactivated successfully.
2025 Feb 28 14:47:49.780744 sonic INFO container: docker cmd: wait for bgp
2025 Feb 28 14:47:49.874987 sonic NOTICE admin: Stopping bgp service...
2025 Feb 28 14:47:49.899816 sonic NOTICE admin: Warm boot flag: bgp false.
2025 Feb 28 14:47:49.905116 sonic NOTICE admin: Fast boot flag: bgp false.
2025 Feb 28 14:47:49.929537 sonic INFO bgp.sh[185420]: container stop bgp - No such container: bgp
2025 Feb 28 14:47:49.932009 sonic NOTICE admin: Stopped bgp service...

Steps to reproduce the issue:

  1. Install 202411 image
  2. Apply FRR config
  3. Set fpm address
  4. Observe zebra crash

Describe the results you received:

Local subnet routes are not propagated to the data plane.

Describe the results you expected:

Local subnet routes are installed to the data plane.

Output of show version:

  • N/A

Output of show techsupport:

  • N/A

Additional information you deem important (e.g. issue happens only occasionally):

  • N/A
@dgsudharsan
Copy link
Collaborator

@adyeung @hasan-brcm FYI. This is a high priority issue for us

@hasan-brcm
Copy link
Contributor

@dgsudharsan , the fpm address config missing issue is not seen with vtysh intergrated mode.

Here are a few workarounds to avoid the fpm address config missing issue in vtysh non-integrated mode:
(1) Place the fpm address config in the beginning of the zebra.conf , e.g.

hostname sonic
password zebra
enable password zebra
log syslog informational
log facility local4
!
fpm address 127.0.0.1
!
...

(2) After loading the 8.5.4 private config, configure fpm address, save, and reload frr.
The problem is not seen if the config was saved by frr. It is seen only when config is used from older frr version.

Regarding zebra crash, please collect and upload the techsupport. I am not able to hit the crash in my local setup.

@StormLiangMS
Copy link
Contributor

hi @dgsudharsan can this issue be repro?

@arlakshm
Copy link
Contributor

This issue is being triaged in the routing subgroup

@arlakshm arlakshm added the Triaged this issue has been triaged label Mar 12, 2025
@dgsudharsan
Copy link
Collaborator

@StormLiangMS Yes but this is with split mode. We already know another issue where the reason was "non integrated" mode in FRR. I am checking with the team internally to move to integrated mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug 🐛 FRR 🚥 Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dgsudharsan @nazariig @arlakshm @hasan-brcm @StormLiangMS