Routes between VLANs in different VRFs (route leaking) not working as expected

[davewj100]

Good day everyone, I hope someone out there can shed some light on this problem.

What I would like: Multiple VRFs each containing one or more VLANs, and route between them inside the L3 switch. I can do this successfully with an external router, but I have 100G interfaces, so I need the L3 speed. I need multiple VRFs because I want to route traffic differently depending on its VLAN.

According to this page, this configuration is called "Route Leeaking": https://github.com/sonic-net/SONiC/blob/master/doc/vrf/sonic-vrf-hld.md#VRF-route-leak-support

This is what my VLANs and their bound VRFs and subnets looks like:

# show ip int
Interface    Master        IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
-----------  ------------  -------------------  ------------  --------------  -------------
Loopback0                  10.1.0.1/32          up/up         N/A             N/A
Vlan102      Vrf_prod      192.168.2.4/32       up/up         N/A             N/A
Vlan1000     Vrf_prod      10.41.0.1/24         up/up         N/A             N/A
Vlan1001     Vrf_prod      10.41.1.1/24         up/up         N/A             N/A
Vlan1002     Vrf_prod      10.41.2.1/24         up/up         N/A             N/A
Vlan1003     Vrf_Vlan1003  10.41.3.1/24         up/up         N/A             N/A
docker0                    240.127.1.1/24       up/down       N/A             N/A
eth0         mgmt          192.168.2.3/24       up/up         N/A             N/A
lo                         127.0.0.1/16         up/up         N/A             N/A
lo-m         mgmt          127.0.0.1/16         up/up         N/A             N/A

Goal: I want to route between 10.41.3.0/24 in Vrf_Vlan1003 and 10.41.0.0/24 in Vrf_prod.

Here is how I attempted to set it up in vtysh:

# ip route 10.41.0.0/24 Vlan1000 vrf Vrf_Vlan1003 nexthop-vrf Vrf_prod
# ip route 10.41.3.0/24 Vlan1003 vrf Vrf_prod nexthop-vrf Vrf_Vlan1003

So I added routes for both directions, and after running those commands in vtysh, everything looks good:

# show ip route vrf all
:
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF Vrf_Vlan1003:
S>* 10.41.0.0/24 [1/0] is directly connected, Vlan1000 (vrf Vrf_prod), weight 1, 00:18:00
C>* 10.41.3.0/24 is directly connected, Vlan1003, 00:59:02

VRF Vrf_prod:
S>* 0.0.0.0/0 [1/0] via 10.41.0.2, Vlan1000, weight 1, 00:59:02
C>* 10.41.0.0/24 is directly connected, Vlan1000, 00:59:02
C>* 10.41.1.0/24 is directly connected, Vlan1001, 00:59:02
C>* 10.41.2.0/24 is directly connected, Vlan1002, 00:59:02
S>* 10.41.3.0/24 [1/0] is directly connected, Vlan1003 (vrf Vrf_Vlan1003), weight 1, 00:14:18
C>* 192.168.2.4/32 is directly connected, Vlan102, 00:59:02

VRF default:
C>* 10.1.0.1/32 is directly connected, Loopback0, 00:59:05

VRF mgmt:
K>* 0.0.0.0/0 [0/201] via 192.168.2.1, eth0, 00:59:15
C>* 192.168.2.0/24 is directly connected, eth0, 00:59:15

And yet, when I try to ping between them, it doesn't work:

# sudo ip vrf exec Vrf_Vlan1003 ping 10.41.0.1
PING 10.41.0.1 (10.41.0.1) 56(84) bytes of data.
^C
--- 10.41.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2046ms

Also, even when I save the config, those routes disappear on reboot. I also see this error in the syslog:

Jan  3 20:29:28.690102 dx010 WARNING route_check.py: Failure results: {{#012    "Unaccounted_ROUTE_ENTRY_TABLE_entries": [#012        "10.41.0.0/24",#012        "10.41.0.0/24",#012        "10.41.1.0/24",#012        "10.41.2.0/24",#012        "10.41.3.0/24",#012        "10.41.3.0/24"#012    ]#012}}
Jan  3 20:29:28.690343 dx010 WARNING route_check.py: Failed. Look at reported mismatches above
Jan  3 20:29:28.690525 dx010 WARNING route_check.py: add: []
Jan  3 20:29:28.690654 dx010 WARNING route_check.py: del: []

Another thing I notice, is there is a command in the config namespace (at regular cconsole) which has VRFs as inputs, but no matter what I try the same thing happens: A static route is added into config_db.conf, no error is returned, but no actual route is created:

#  sudo config route add -h
Usage: config route add [OPTIONS] prefix [vrf <vrf_name>] <A.B.C.D/M> nexthop
                        <[vrf <vrf_name>] <A.B.C.D>>|<dev <dev_name>>

  Add route command

Options:
  -h, -?, --help  Show this message and exit.

Anybody have any ideas? What is the correct way to do this route leaking thing? Any suggestions much appreciated. Thanks!

Below is my SONiC version info:

# show version

SONiC Software Version: SONiC.202205.196887-bca955f52
Distribution: Debian 11.6
Kernel: 5.10.0-18-2-amd64
Build commit: bca955f52
Build date: Sun Jan  1 18:29:17 UTC 2023
Built by: AzDevOps@vmss-soni0008F5

Platform: x86_64-cel_seastone-r0
HwSKU: Seastone-DX010
ASIC: broadcom
ASIC Count: 1
Serial Number: DX010B2F118C27LK100002
Model Number: R0872-F0010-01
Hardware Revision: N/A
Uptime: 20:49:35 up 17 min,  1 user,  load average: 0.77, 0.86, 0.66
Date: Tue 03 Jan 2023 20:49:35

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-gbsyncd-broncos        202205.196887-bca955f52   a61e11da5109   530MB
docker-gbsyncd-broncos        latest                    a61e11da5109   530MB
docker-gbsyncd-credo          202205.196887-bca955f52   cc897d8036e0   501MB
docker-gbsyncd-credo          latest                    cc897d8036e0   501MB
docker-syncd-brcm             202205.196887-bca955f52   e039524c4ebf   825MB
docker-syncd-brcm             latest                    e039524c4ebf   825MB
docker-orchagent              202205.196887-bca955f52   6f69b2f594fc   518MB
docker-orchagent              latest                    6f69b2f594fc   518MB
docker-fpm-frr                202205.196887-bca955f52   8406b98e93a1   528MB
docker-fpm-frr                latest                    8406b98e93a1   528MB
docker-teamd                  202205.196887-bca955f52   7230861f5be4   499MB
docker-teamd                  latest                    7230861f5be4   499MB
docker-macsec                 latest                    54064674ed11   501MB
docker-dhcp-relay             latest                    59aefc72c38f   492MB
docker-snmp                   202205.196887-bca955f52   5fd47466ca2d   528MB
docker-snmp                   latest                    5fd47466ca2d   528MB
docker-sonic-telemetry        202205.196887-bca955f52   69c0eb5e8901   563MB
docker-sonic-telemetry        latest                    69c0eb5e8901   563MB
docker-router-advertiser      202205.196887-bca955f52   2f50e07235ce   483MB
docker-router-advertiser      latest                    2f50e07235ce   483MB
docker-platform-monitor       202205.196887-bca955f52   311be3260c09   607MB
docker-platform-monitor       latest                    311be3260c09   607MB
docker-mux                    202205.196887-bca955f52   f6837694e531   531MB
docker-mux                    latest                    f6837694e531   531MB
docker-lldp                   202205.196887-bca955f52   014eb1688c4d   525MB
docker-lldp                   latest                    014eb1688c4d   525MB
docker-database               202205.196887-bca955f52   8c2eb4373c2c   483MB
docker-database               latest                    8c2eb4373c2c   483MB
docker-nat                    202205.196887-bca955f52   a7cf7b5dc450   470MB
docker-nat                    latest                    a7cf7b5dc450   470MB
docker-sflow                  202205.196887-bca955f52   0970d53e84af   469MB
docker-sflow                  latest                    0970d53e84af   469MB
docker-sonic-mgmt-framework   202205.196887-bca955f52   5cd43225baff   598MB
docker-sonic-mgmt-framework   latest                    5cd43225baff   598MB

[davewj100]

Just figured something out... The inter-VRF routes are actually working. I can ping other IPs across the VLANs, but I cannot ping the interface IPs on the switch between the VRFs. This must be something to do with the ACLs, but it doesn't matter in my case.

Now I just need to figure out why the routes disappear on reboot. If I can nail it down, I'll post what I did.

[davewj100]

OK I answered my own question. In order to make it persistent, you can modify the config_db.json manually and add lines like this:

  "STATIC_ROUTE": {
       "Vrf_prod|10.41.3.0/24": {
           "blackhole": "false",
           "distance": "0",
           "ifname": "Vlan1003",
           "nexthop-vrf": "Vrf_Vlan1003"
       }

The cmd doesn't seem to do it right, at least in my version; perhaps it's fixed in the later branches. I had a loud fan problem with one of the newer masters, maybe I'll try it again.

Thanks

[selvatechtalk]

Well, the ping works but is that routed in the CPU or data plane ? You may want to check. I have Host H1 in on VRF and H2 in the other VRF. I could ping across but the pkts are routed by the CPU.

[davewj100]

Thanks so much for the reply! I figured this out, and posted a question on ServeTheHome about it: https://forums.servethehome.com/index.php?threads/inter-vlan-routing-pegs-atom-cpu-to-100-in-100g-celestica-seastone-dx010-l3-switch.38707/ I ended up working around this issue by using a single VRF for my production network. Dave

…

On Sun, Jun 2, 2024 at 2:35 AM selvatechtalk ***@***.***> wrote: Well, the ping works but is that routed in the CPU or data plane ? You may want to check. I have Host H1 in on VRF and H2 in the other VRF. I could ping across but the pkts are routed by the CPU. — Reply to this email directly, view it on GitHub <#1204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIE4USJ5LEPM7OSBCDFEDJTZFK4LDAVCNFSM6AAAAAATQELXIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTG4YTSOJTGQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>