Skip to content

Zebra not deleting MAC/IP route #13973

@Sohn123

Description

@Sohn123
  • Did you check if this is a duplicate issue?
  • Did you test it on the latest FRRouting/frr master branch?

To Reproduce
We are using BGPEVPN with frr. We have the following setup:
We have three VTEPs two of them are Wi-Fi access points running OpenWRT and one Ubuntu 22.10 machine. The Ubuntu machine is also a route-reflector. They are all running master commit: dee79c33a425d264e53e0e5d0ad51b1bc13945d0. (The issue also occurred with frr 8.5.1).
The access points are running a daemon that creates a VXLAN iface and bridges it to an iface created by hostapd when a station connects. The interface that is created by hostapd receives all the L2 traffic from this station.
We noticed the following problem. When a station roams from one access point to another it looses L2 connectivity although vtysh with the command show evpn mac vni all reports that the address is learned locally on the new VTEP.
The Ubuntu machine properly forwards traffic to the new VTEP, but the new VTEP forwards it to the old VTEP instead of forwarding to the station.

We debugged it using a device with the MAC 34:cf:f6:b0:98:b6. On the VTEP we moved to, the bridge now shows the following output:

root@acs2:~# bridge fdb | grep 98
34:cf:f6:b0:98:b6 dev vlan4096 master bridge7
34:cf:f6:b0:98:b6 dev vxlan7 vlan 1 extern_learn master bridge7
34:cf:f6:b0:98:b6 dev vxlan7 dst 10.242.2.194 self extern_learn

The Mac address is learned on both ports, although it should only be learnt on the local port vlan4096. We observed that the correct bgp messages get exchanged and zebra gets notified about the deletion of the MAC/IP route. However zebra doesn't seem to apply the received information.

Expected behavior
The MAC address of the device should only be learnt by port vlan4096

Versions

  • OS Version: Ubuntu 22.10 (Route Reflector), openWrt 22.03 (VTEP)
  • Kernel: 5.15.0-75-generic (Route Reflector), 5.10.146 (VTEP)
  • FRR Version: 8.5.1 and built from git dee79c33a425d264e53e0e5d0ad51b1bc13945d0

Additional context
For the access points we used the following config:

debug zebra dplane
debug zebra events
debug zebra vxlan
log syslog debugging
ip nht resolve-via-default
ip6 nht resolve-via-default
router bgp 65000
  bgp router-id 10.242.2.170
  no bgp default ipv4-unicast
  neighbor fabric peer-group
  neighbor fabric remote-as 65000
  neighbor fabric capability extended-nexthop
  neighbor fabric ebgp-multihop 5
  ! BGP sessions with route reflectors
  neighbor 10.242.2.4 peer-group fabric
  !
  address-family l2vpn evpn
   neighbor fabric activate
   advertise-all-vni
  exit-address-family
  !
!

This is the log on the VTEP the device moved away from with IP 10.242.2.194:

Jul  3 09:52:35 acs1 zebra[7001]: [QX4D7-WJPES] Recv VTEP ADD 10.242.2.170 VNI 7 flood 0 from bgp
Jul  3 09:52:35 acs1 zebra[7001]: [XC8P3-66E56] Install 10.242.2.170 into flood list for VNI 7 intf vxlan7(38)
Jul  3 09:52:35 acs1 zebra[7001]: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 7 MAC 34:cf:f6:b0:98:b6 flags 0x0 seq 1 VTEP 10.242.2.170 ESI - from bgp
Jul  3 09:52:35 acs1 zebra[7001]: [RWQPR-6BEC9] Send MACIP Del f None  state 1 MAC 34:cf:f6:b0:98:b6 IP (null) seq 0 L2-VNI 7 ESI - to bgp
Jul  3 09:52:35 acs1 zebra[7001]: [S7Q3Q-N2C38] Processing neighbors on remote MAC 34:cf:f6:b0:98:b6 ADD, VNI 7
Jul  3 09:52:35 acs1 zebra[7001]: [QEDXC-E5122] dpAdd remote MAC 34:cf:f6:b0:98:b6 VID 1

This is the log on the VTEP the device moved to with IP 10.242.2.170:

Jul  3 09:52:34 acs2 zebra[9072]: [R0MNE-C165S] MESSAGE: ZEBRA_INTERFACE_ADD vlan4096 vrf default(0)
Jul  3 09:52:34 acs2 zebra[9072]: [SBFM4-2P25V] MESSAGE: ZEBRA_INTERFACE_DOWN vlan4096 vrf default(0)
Jul  3 09:52:34 acs2 zebra[9072]: [QQWTF-ZKJDD] rib_update: Scheduled VRF (ALL), event RIB_UPDATE_KERNEL
Jul  3 09:52:34 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP vlan4096 vrf default(0)
Jul  3 09:52:34 acs2 zebra[9072]: [M8JMF-J6PZV] rib_update_handle_vrf_all: Handling VRF (ALL) event RIB_UPDATE_KERNEL
Jul  3 09:52:34 acs2 zebra[9072]: [QR5K4-A1079] rib_update_table: IPv4 VRF default Table 254 event RIB_UPDATE_KERNEL Route type: wildcard
Jul  3 09:52:34 acs2 zebra[9072]: [QR5K4-A1079] rib_update_table: IPv4 VRF default Table 254 event RIB_UPDATE_KERNEL Route type: wildcard
Jul  3 09:52:34 acs2 zebra[9072]: [QR5K4-A1079] rib_update_table: IPv6 VRF default Table 254 event RIB_UPDATE_KERNEL Route type: wildcard
Jul  3 09:52:34 acs2 zebra[9072]: [QR5K4-A1079] rib_update_table: IPv6 VRF default Table 254 event RIB_UPDATE_KERNEL Route type: wildcard
Jul  3 09:52:34 acs2 hostapd: wlan0: STA 34:cf:f6:b0:98:b6 RADIUS: VLAN ID 4096
Jul  3 09:52:34 acs2 hostapd: nl80211: kernel reports: key addition failed
Jul  3 09:52:34 acs2 hostapd: wlan0: STA 34:cf:f6:b0:98:b6 IEEE 802.11: VLAN ID 4096
Jul  3 09:52:34 acs2 hostapd: wlan0: STA 34:cf:f6:b0:98:b6 IEEE 802.11: associated (aid 1)
Jul  3 09:52:34 acs2 hostapd: wlan0: AP-STA-CONNECTED 34:cf:f6:b0:98:b6
Jul  3 09:52:35 acs2 zebra[9072]: [R0MNE-C165S] MESSAGE: ZEBRA_INTERFACE_ADD vxlan7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [JT45G-495D1] Add L2-VNI 7 VRF default intf vxlan7(30) VLAN 0 local IP 0.0.0.0 mcast_grp 0.0.0.0 master 0
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP vxlan7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [K61WJ-XQQ3X] Intf vxlan7(30) L2-VNI 7 is UP
Jul  3 09:52:35 acs2 zebra[9072]: [H5BRT-63K8T] Cannot find bridge-vlan IF (31) for vlan update
Jul  3 09:52:35 acs2 zebra[9072]: [R0MNE-C165S] MESSAGE: ZEBRA_INTERFACE_ADD bridge7 vrf default(0)
Jul  3 09:52:35 acs2 kernel: [ 3874.041538] netlink: 'wimoved': attribute type 15 has an invalid length.
Jul  3 09:52:35 acs2 kernel: [ 3874.052885] bridge7: port 1(vxlan7) entered blocking state
Jul  3 09:52:35 acs2 kernel: [ 3874.058533] bridge7: port 1(vxlan7) entered disabled state
Jul  3 09:52:35 acs2 kernel: [ 3874.064342] device vxlan7 entered promiscuous mode
Jul  3 09:52:35 acs2 kernel: [ 3874.069382] bridge7: port 1(vxlan7) entered blocking state
Jul  3 09:52:35 acs2 kernel: [ 3874.075046] bridge7: port 1(vxlan7) entered forwarding state
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP vxlan7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [K61WJ-XQQ3X] Intf vxlan7(30) L2-VNI 7 is UP
Jul  3 09:52:35 acs2 zebra[9072]: [XT97D-1TD6P] Update L2-VNI 7 intf vxlan7(30) VLAN 0 local IP 0.0.0.0 master 31 chg 0x2
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [H5WMK-KQVJA] Reading MAC FDB and Neighbors for intf vxlan7(30) VNI 7 master 31
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP vxlan7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [K61WJ-XQQ3X] Intf vxlan7(30) L2-VNI 7 is UP
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [H5WMK-KQVJA] Reading MAC FDB and Neighbors for intf vxlan7(30) VNI 7 master 31
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP bridge7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [MS21T-EA6XZ] SVI bridge7(31) VNI 7 VRF default is UP, installing neighbors
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [QYXB9-6RNNK] RTM_NEWVLAN bridge IF vxlan7 NS 0
Jul  3 09:52:35 acs2 zebra[9072]: [TX4H9-TSE3T] VLANDB_ENTRY: VID (1) state=FORWARDING
Jul  3 09:52:35 acs2 zebra[9072]: [KKZGZ-8PCDW] Cannot find VNI for VID (1) IF vxlan7 for vlan state update
Jul  3 09:52:35 acs2 zebra[9072]: [XT97D-1TD6P] Update L2-VNI 7 intf vxlan7(30) VLAN 1 local IP 0.0.0.0 master 31 chg 0x4
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [H5WMK-KQVJA] Reading MAC FDB and Neighbors for intf vxlan7(30) VNI 7 master 31
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP bridge7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [MS21T-EA6XZ] SVI bridge7(31) VNI 7 VRF default is UP, installing neighbors
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [WBFP9-HYC3W] Intf bridge7(31) bridge changed MAC address
Jul  3 09:52:35 acs2 zebra[9072]: [XT97D-1TD6P] Update L2-VNI 7 intf vxlan7(30) VLAN 1 local IP 0.0.0.0 master 31 chg 0x10
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP bridge7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [MS21T-EA6XZ] SVI bridge7(31) VNI 7 VRF default is UP, installing neighbors
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [QX4D7-WJPES] Recv VTEP ADD 10.242.2.4 VNI 7 flood 0 from bgp
Jul  3 09:52:35 acs2 zebra[9072]: [QX4D7-WJPES] Recv VTEP ADD 10.242.2.194 VNI 7 flood 0 from bgp
Jul  3 09:52:35 acs2 zebra[9072]: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 7 MAC de:35:30:d8:21:b0 flags 0x0 seq 0 VTEP 10.242.2.4 ESI - from bgp
Jul  3 09:52:35 acs2 zebra[9072]: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 7 MAC de:35:30:d8:21:b0 IP 10.1.8.1 flags 0x0 seq 0 VTEP 10.242.2.4 ESI - from bgp
Jul  3 09:52:35 acs2 zebra[9072]: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 7 MAC de:35:30:d8:21:b0 IP fe80::dc35:30ff:fed8:21b0 flags 0x0 seq 0 VTEP 10.242.2.4 ESI - from bgp
Jul  3 09:52:35 acs2 zebra[9072]: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 7 MAC 34:cf:f6:b0:98:b6 flags 0x0 seq 0 VTEP 10.242.2.194 ESI - from bgp
Jul  3 09:52:35 acs2 zebra[9072]: [J7KAF-72ZD2] EVPN gateway macip Adv disabled on VNI 7, currently disabled
Jul  3 09:52:35 acs2 zebra[9072]: [KJP9G-QGA1D] EVPN SVI macip Adv disabled on VNI 7, currently disabled
Jul  3 09:52:35 acs2 kernel: [ 3874.088242] bridge7: port 2(vlan4096) entered blocking state
Jul  3 09:52:35 acs2 kernel: [ 3874.094016] bridge7: port 2(vlan4096) entered disabled state
Jul  3 09:52:35 acs2 kernel: [ 3874.099897] device vlan4096 entered promiscuous mode
Jul  3 09:52:35 acs2 kernel: [ 3874.105166] bridge7: port 2(vlan4096) entered blocking state
Jul  3 09:52:35 acs2 kernel: [ 3874.110870] bridge7: port 2(vlan4096) entered forwarding state
Jul  3 09:52:35 acs2 zebra[9072]: [XC8P3-66E56] Install 10.242.2.4 into flood list for VNI 7 intf vxlan7(30)
Jul  3 09:52:35 acs2 zebra[9072]: [XC8P3-66E56] Install 10.242.2.194 into flood list for VNI 7 intf vxlan7(30)
Jul  3 09:52:35 acs2 zebra[9072]: [JWQ3J-TKSAT] zebra_evpn_mac_add: MAC de:35:30:d8:21:b0 flags None 
Jul  3 09:52:35 acs2 zebra[9072]: [S7Q3Q-N2C38] Processing neighbors on remote MAC de:35:30:d8:21:b0 ADD, VNI 7
Jul  3 09:52:35 acs2 zebra[9072]: [JWQ3J-TKSAT] zebra_evpn_mac_add: MAC 34:cf:f6:b0:98:b6 flags None 
Jul  3 09:52:35 acs2 zebra[9072]: [S7Q3Q-N2C38] Processing neighbors on remote MAC 34:cf:f6:b0:98:b6 ADD, VNI 7
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP vlan4096 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP vlan4096 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [V6YM8-PV4KB] MESSAGE: ZEBRA_INTERFACE_UP bridge7 vrf default(0)
Jul  3 09:52:35 acs2 zebra[9072]: [MS21T-EA6XZ] SVI bridge7(31) VNI 7 VRF default is UP, installing neighbors
Jul  3 09:52:35 acs2 zebra[9072]: [V6240-A1P7N] Send EVPN_ADD 7 0.0.0.0 tenant vrf default(0) SVI index 31 to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [WBFP9-HYC3W] Intf bridge7(31) bridge changed MAC address
Jul  3 09:52:35 acs2 zebra[9072]: [XT97D-1TD6P] Update L2-VNI 7 intf vxlan7(30) VLAN 1 local IP 0.0.0.0 master 31 chg 0x10
Jul  3 09:52:35 acs2 zebra[9072]: [QEDXC-E5122] dpAdd remote MAC de:35:30:d8:21:b0 VID 1
Jul  3 09:52:35 acs2 zebra[9072]: [QEDXC-E5122] dpAdd remote MAC 34:cf:f6:b0:98:b6 VID 1
Jul  3 09:52:35 acs2 zebra[9072]: [TVHPV-4R7K8] UPD MAC 34:cf:f6:b0:98:b6 intf vlan4096(29) VID 0 -> VNI 7 curFlags REM 
Jul  3 09:52:35 acs2 zebra[9072]: [VEHV4-8T4MR] zebra_evpn_dup_addr_detect_for_mac: duplicate addr MAC 34:cf:f6:b0:98:b6 flags LOC detection time passed, reset learn count 0
Jul  3 09:52:35 acs2 zebra[9072]: [RWQPR-6BEC9] Send MACIP Add f None  state 1 MAC 34:cf:f6:b0:98:b6 IP (null) seq 1 L2-VNI 7 ESI - to bgp
Jul  3 09:52:35 acs2 zebra[9072]: [YZ7N7-23FTW] Processing neighbors on local MAC 34:cf:f6:b0:98:b6 ADD, VNI 7
Jul  3 09:52:35 acs2 zebra[9072]: [JWJY6-CBZ1H] Recv MACIP DEL VNI 7 MAC 34:cf:f6:b0:98:b6 Remote VTEP 10.242.2.194 from bgp
Jul  3 09:52:36 acs2 zebra[9072]: [NBGDS-5ZDQ9] MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD fe80::225:9cff:fe13:a29f/64 on vlan4096 vrf default(0)
Jul  3 09:52:36 acs2 zebra[9072]: [NBGDS-5ZDQ9] MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD fe80::aca1:48ff:fe43:d00f/64 on bridge7 vrf default(0)
Jul  3 09:52:36 acs2 zebra[9072]: [NBGDS-5ZDQ9] MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD fe80::1a:77ff:febc:cfb1/64 on vxlan7 vrf default(0)

The logs show that zebra on the new VTEP receives that it should delete the VNI entry on for the old VTEP but it seems like this is not happening. Do you have any idea what could be the reason for this or what we could do to mitigate this issue?

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageNeeds further investigation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions