ARP Timeout and MAC Address Table Consistency on an EVPN Data Center

In an EVPN/VXLAN fabric, the MAC address table is part of the control plane: when a MAC is learned or removed, the switch generates a BGP EVPN update so the rest of the fabric stays in sync. That tight coupling is exactly what makes EVPN work — but it also means that ordinary MAC aging on a quiet host can turn into a stream of BGP churn. This post is about a problem we chased down to the interaction between ARP timeout and MAC address-table timeout, and why the common assumption about “any packet refreshes the timer” is wrong.

Problem

We were seeing three symptoms that, at first, didn’t obviously belong together:

• Huge BGP updates across the fabric, even though we had no VM mobility (no vMotion) to explain the churn.
• Loss of connectivity for monitoring systems that rely on network scanning, and for server vendors’ ILO/iLO management tooling.
• Occasionally losing the first 1–3 packets destined to a target host when using fast-scanning applications.

What We Found

We narrowed the problem down to networks with very silent hosts — physical server interfaces and infrastructure networks where the host barely sends any traffic.

The root cause was the relationship between the default ARP timeout and the default MAC address-table timeout. On most vendors, the ARP timeout is significantly longer than the MAC address-table timeout. The sequence plays out like this:

1. A silent host stops sending frames.
2. Its entry ages out of the MAC address table first (shorter timer).
3. When the MAC is removed, the switch sends an EVPN withdraw/update.
4. Later, when traffic needs to reach that host, the entry has to be relearned — which is where the first few packets get lost during fast scans.

The fix that helped was to lower the ARP timeout below the MAC address-table timeout. With that ordering, as a host’s ARP entry approaches its timeout, the switch sends a unicast ARP request for the host (the exact timing is vendor-dependent — it may fire seconds before expiry, or for long ARP timeouts, minutes before). When the host replies, two things happen: its MAC aging timer is reset, so the entry never ages out of the MAC table, and therefore the switch never has to send a BGP update. The silent host is kept “alive” in both tables by the periodic ARP refresh.

Why This Matters More in EVPN

There’s an EVPN-specific twist that makes this worse than in a classic L2 network. In EVPN there is no separate route type for the MAC and for the ARP/IP binding — both are carried in the Type-2 (MAC/IP) route. So when the MAC ages out and the switch withdraws the Type-2 route, it withdraws both the MAC and the IP reachability, forcing every remote VTEP to unlearn the ARP entry too.

The reconvergence that follows is time-sensitive. If the next conversation to that host has to re-discover the MAC and re-advertise it over BGP, and that takes longer than about a second, the ingress switch can return ICMP unreachables to the sender — which in turn resets a TCP session that was perfectly healthy a moment ago. That’s how a silent-but-alive TCP application ends up with mysterious resets.

The design choice influences how badly this bites:

• Asymmetric IRB (every VTEP has every SVI; ingress routes, egress bridges) is the most exposed — a MAC timeout can directly cause the TCP-reset scenario above.
• Symmetric IRB (ingress routes into a transit VNI, egress routes into the destination VNI) largely eliminates the reset problem, but you’ll still see EVPN host-route churn every time a silent host’s MAC ages out.

Either way, the cure is the same: keep ARP entries refreshing before the MAC ages out.

When Does a Switch Actually Reset Its ARP Timer?

This is the part that’s widely misunderstood. ARP is a control-plane process — like in Linux, where ARP packets are handled by the kernel, switches handle ARP via the CPU / control plane. The ARP table lives in the control plane.

It is commonly believed that whenever the switch receives a packet from a host, it resets that host’s ARP timer. This is wrong. To reset a host’s ARP timer, the switch needs a packet destined for its own control plane. That could be an ICMP packet to the switch’s IP address (typically a host pinging its default gateway), or any control-plane process such as an ARP request — something that has to be punted to and processed by the control plane.

The host’s ordinary data traffic never touches the control plane or the CPU on modern switches; data packets are hardware-forwarded. So a host can be sending plenty of data and still let its ARP entry age out, because none of that data is seen by the control plane.

The Same Behavior in the Linux Kernel

This is also true in the Linux kernel: the ARP process itself is not responsible for keeping the ARP table fresh — the upper-layer protocols decide whether to refresh the timer.

• ICMP will not refresh the destination host’s ARP timer. That’s why you can see the kernel send ARP requests in the middle of a ping. ICMP doesn’t care whether the target’s MAC has changed; it only cares about Layer 3 information.
• TCP will refresh the timer. TCP is a stateful connection, so it has a stake in reachability: if the host’s MAC changed, that effectively means the host changed, and the connection should be dropped.

Default Timers by Platform

Half the battle is knowing what your platform ships with. The defaults are all over the place, and on several of them the ARP timer is longer than the MAC aging timer out of the box — which is exactly the wrong ordering:

Two takeaways. First, the classic IOS/EOS defaults (4-hour ARP vs 5-minute MAC) are the worst case and the most likely to bite you with silent hosts. Second, modern Linux hosts already use very short ARP timeouts, which is why a Linux box on the same VLAN tends to keep silent hosts refreshed for you — more on that below.

Configuration Examples

The goal is always the same: ARP/ND aging < MAC aging, with enough margin that the periodic ARP refresh reliably lands before the MAC entry expires. Validate exact syntax against your platform and release.

Cisco NX-OS (per-SVI ARP timer, global MAC aging):

1
2
3
4

interface Vlan100
  ip arp timeout 240
!
mac address-table aging-time 300

Cisco IOS XE (per-interface ARP timer):

1
2
3
4

interface Vlan100
  arp timeout 240
!
mac address-table aging-time 300

Arista EOS:

1
2
3
4

interface Vlan100
  arp aging timeout 240
!
mac address-table aging-time 300

Don’t forget IPv6: ND has its own cache-expiry timer (for example ipv6 nd cache expire on NX-OS), and the same ordering rule applies to the neighbor cache.

The Other Trap: ARP Suppression

While tuning timers, it’s worth knowing about a separate EVPN feature that interacts with all of this: ARP suppression (and its IPv6 sibling, ND suppression). The idea is that a leaf answers ARP requests locally on behalf of known remote hosts instead of flooding the request across the fabric — good for cutting BUM traffic. The defaults differ by vendor and will surprise you:

• Cisco NX-OS — disabled by default, enabled per-VNI with suppress-arp.
• Juniper QFX — enabled by default; disabled with the (hidden, and historically flaky) no-arp-suppression knob.
• Arista EOS — enabled by default only on VLANs that have an SVI.

The catch is that some applications use broadcast ARP as cluster keepalives, and ARP suppression can break them. A well-behaved implementation must still flood ARP probes (broadcast ARP with source 0.0.0.0, used for duplicate-address detection) and GARP — but not all of them get this right. On Arista you can exempt specific subnets from suppression:

1
2
3
4
5

router l2-vpn
   arp proxy prefix-list NO_SUPPRESS
!
ip prefix-list NO_SUPPRESS
   seq 10 deny 10.0.0.0/24

Treat ARP suppression as something to enable deliberately, not a default to leave unexamined — when one side suppresses and the other doesn’t, you get a mix of unnecessarily flooded and silently dropped ARP that is miserable to troubleshoot.

Detection and Verification

When you suspect this problem, the tell-tale signs are EVPN Type-2 churn that correlates with quiet hosts, and flooding in VLANs that contain appliances/servers that rarely transmit. Useful checks:

• Compare the two timers directly — confirm ARP aging is actually lower than MAC aging on the SVI/VLAN in question (show ip arp, show mac address-table aging-time on NX-OS; equivalents elsewhere).
• Watch the EVPN MAC/IP table and BGP for repeated advertise/withdraw of the same host (show l2route evpn mac-ip all / show bgp l2vpn evpn on NX-OS).
• Capture on the silent host’s port and confirm the switch is sending periodic unicast ARP requests and getting replies before the MAC ages out.

Mitigation Options

In order of preference:

1. Set ARP/ND aging below MAC aging on the affected SVIs/VLANs — the proper fix, and the one I’d make a fabric-wide standard.
2. Raise the MAC aging timer if you can’t lower ARP for some reason — it widens the window but doesn’t decouple the tables as cleanly.
3. Keep an active host on the VLAN. A Linux box (or any periodic ARP source) on the same VLAN naturally re-ARPs silent hosts; this is also why the old “scan the VLAN with nmap every few minutes” trick works — the scanning host’s short ARP cache keeps triggering requests and replies.
4. Rely on switch ARP/ND refresh/probing where the platform supports it, so the MAC and ARP tables are decoupled from application traffic patterns.
5. Limit unknown-unicast flooding as a blast-radius control, so that when a MAC does age out, the resulting flooding doesn’t saturate ingress-replication queues.

A Word of Caution on the Timers

Lowering the ARP timeout helps the silent-host problem, but don’t push it too low. The interval must still allow the switch control plane to see the host’s periodic ARP requests within the window, and very aggressive ARP refresh adds control-plane load at scale. Tuned correctly, this eliminates the unnecessary ARP requests for live hosts while keeping silent hosts from churning the EVPN control plane.

References

• Understanding Linux Network Internals — Ch. 28, §9 (ARP)
• Understanding Linux Network Internals — Ch. 28, §5 (ARP)

Further Reading

• Dmytro Shypovalov — ARP problems in EVPN (also on the APNIC blog): the asymmetric-vs-symmetric IRB analysis and the “ARP timeout < MAC age” best practice.
• Ivan Pepelnjak — Silent Hosts in EVPN Fabrics: platform default timers and the nmap-scan explanation.

The content of this post was created by the author; AI was used for editing and restructuring.