Philipp Lehmann @PhilippTheServer

How a VPN Client Silently Shrinks Your Pod Network MTU

When a lower-MTU hop appears after the CNI has already picked 1500.

kubernetesnetworking

The problem

A cluster node has a physical NIC with the usual Ethernet MTU of 1500. The CNI plugin starts, looks at the interface holding the default route, sees 1500, and configures the pod overlay to use it. Every pod on that node inherits an MTU of 1500 for its eth0.

Some time later, that node joins a WireGuard-based mesh VPN — for remote access, or to reach a resource on another network. A WireGuard tunnel does not carry 1500-byte packets end to end: the outer UDP/IP header and the encryption overhead eat into the payload, and the usable MTU across the tunnel typically settles around 1420, sometimes lower. Nothing tells the CNI. It configured the pod network once, when its daemon started, and does not watch the routing table for changes afterwards.

Traffic that stays inside 1420 bytes works. A curl to a small JSON endpoint works. A ping works — it sends 56-byte payloads by default, nowhere near either MTU. A health check doing a GET /healthz reports everything fine. Then someone pulls a large file or does a bulk database dump across that path, and it is slow — not failed, just slow, with retransmits and timeouts that look exactly like an overloaded backend.

This is what makes it hard to notice: the failure is size-dependent, showing up only under traffic no health check exercises, and silent rather than explicit. TCP has a mechanism for exactly this, Path MTU Discovery, and it depends on an ICMP message that VPN clients and firewalls routinely drop — so instead of one clean error, retransmissions run until they eventually time out.

Working through it

MTU auto-detection is a snapshot, not a policy

Flannel, Calico and most other CNI plugins offer to auto-detect the MTU for the pod overlay by inspecting whichever interface currently holds the node’s default route. That is a reasonable default: most nodes have one NIC, and matching its MTU avoids fragmentation on the common path. But “auto-detect” here means exactly one read, done when the CNI’s node agent starts. It is not a control loop, and it does not re-run when a wg0 or tun0 interface appears later and takes over the routing decision. Whatever value the agent read at start-up is what every pod gets, until the agent itself restarts — a node reboot or a manual daemonset restart, not a response to network changes.

The practical consequence: the MTU Kubernetes reports for pod interfaces reflects the state of the node’s networking at one moment in the past, with no signal anywhere in kubectl telling you whether that moment is still representative.

Path MTU Discovery is not a safety net here

The correct behaviour for a TCP sender that emits a too-large packet is to have it rejected by whichever router along the path cannot forward it at that size, with the router sending back an ICMP “Fragmentation Needed and DF set” (or ICMPv6 “Packet Too Big”). The sender’s stack uses that message to lower its idea of the path MTU and retransmits smaller. This is Path MTU Discovery, and in principle it makes MTU mismatches self-correcting.

In practice, it depends on that ICMP message surviving the trip back, and a lot of infrastructure discards ICMP by default: VPN clients that only route the tunnel’s own traffic, firewalls with blanket ICMP-deny rules, NAT that does not associate the error back to the connection that triggered it. When the message is dropped, the sender never finds out anything is wrong — it keeps sending the same oversized packet, it keeps getting silently dropped, and the connection stalls rather than failing. This is well-known enough to have a name, “black-hole” PMTUD, but it is easy to forget that your own infrastructure can cause it, not just some far-off network you do not control.

So PMTUD is not a fix for a foreseeable low-MTU hop; it is best-effort recovery for MTUs you did not predict. Where you know a VPN or other encapsulation is in the path, configure the correct MTU explicitly rather than hope discovery papers over the gap.

Pin the MTU instead of trusting detection

Every mainstream CNI that supports auto-detection also supports an explicit override, and the fix is to use it once you know a lower-MTU hop exists downstream of the pod network. The exact field differs by plugin and version — Flannel’s is an MTU key in its backend config, Calico’s a CALICO_IPV4POOL_MTU-style setting — so treat these as pointers, not gospel, and confirm the current key against your CNI version’s own docs before changing anything.

The number to set is not “slightly less than 1500” as a guess. It is the true minimum MTU across every hop the packet crosses, minus the overhead of the pod network’s own encapsulation if it has one (VXLAN, IPIP and similar add their own header). If you do not know the VPN’s effective MTU, measure it — the demonstration below is that measurement.

The solution

Reproducing the mismatch with network namespaces

This needs only root on a Linux machine — no cluster, no VPN, no VM. It needs three namespaces, not two: MTU is enforced on the sending interface only, so a bare veth pair with mismatched ends cannot show the ICMP behaviour PMTUD depends on. Something in the middle has to actually forward, as a real intermediate hop would. podside stands in for a pod whose interface still believes the MTU is 1500; router stands in for the node’s egress point, where the tunnel takes over; vpnside stands in for whatever sits beyond it.

# Namespaces
sudo ip netns add podside
sudo ip netns add router
sudo ip netns add vpnside

# podside <-> router
sudo ip link add veth-pod type veth peer name veth-r1
sudo ip link set veth-pod netns podside
sudo ip link set veth-r1 netns router

# router <-> vpnside
sudo ip link add veth-r2 type veth peer name veth-vpn
sudo ip link set veth-r2 netns router
sudo ip link set veth-vpn netns vpnside

# Addressing: two documentation-safe /30s either side of the router
sudo ip netns exec podside ip addr add 192.0.2.1/30 dev veth-pod
sudo ip netns exec router  ip addr add 192.0.2.2/30 dev veth-r1
sudo ip netns exec router  ip addr add 198.51.100.1/30 dev veth-r2
sudo ip netns exec vpnside ip addr add 198.51.100.2/30 dev veth-vpn

# Bring everything up
for ns in podside router vpnside; do
  sudo ip netns exec "$ns" ip link set lo up
done
sudo ip netns exec podside ip link set veth-pod up
sudo ip netns exec router  ip link set veth-r1 up
sudo ip netns exec router  ip link set veth-r2 up
sudo ip netns exec vpnside ip link set veth-vpn up

# Routing: podside and vpnside each only know the router
sudo ip netns exec podside ip route add 198.51.100.0/30 via 192.0.2.2
sudo ip netns exec vpnside ip route add 192.0.2.0/30 via 198.51.100.1

# The router actually forwards between the two links
sudo ip netns exec router sysctl -qw net.ipv4.ip_forward=1

# The stale CNI value: the pod-side link, and the router's near side, still believe 1500
sudo ip netns exec podside ip link set veth-pod mtu 1500
sudo ip netns exec router  ip link set veth-r1 mtu 1500

# The real ceiling: the router's far side, standing in for the VPN tunnel interface
sudo ip netns exec router  ip link set veth-r2 mtu 1420
sudo ip netns exec vpnside ip link set veth-vpn mtu 1420

Confirm connectivity works for small packets, exactly as a health check would see it:

sudo ip netns exec podside ping -c 2 198.51.100.2

2 packets transmitted, 2 received, 0% packet loss

Now send a full-size packet with the Don’t Fragment bit set, matching a large TCP segment. 1472 bytes of payload plus the 28-byte header is exactly 1500, the size podside’s own interface believes it can emit:

sudo ip netns exec podside ping -M do -s 1472 -c 2 198.51.100.2

From 192.0.2.2 icmp_seq=1 Frag needed and DF set (mtu = 1420)
From 192.0.2.2 icmp_seq=2 Frag needed and DF set (mtu = 1420)

This is PMTUD working correctly. Podside’s interface is large enough to emit the packet, so nothing stops it locally; the router is the first point that cannot forward it, and says so, with the true MTU, over ICMP. The failure this article opened with is not “PMTUD is broken” — it is “PMTUD’s message never arrived”.

Prove that by making the router drop the very message it just sent, which is what a VPN client or a firewall with a blanket ICMP-deny rule does in practice:

sudo ip netns exec router iptables -A OUTPUT -p icmp --icmp-type destination-unreachable -j DROP

sudo ip netns exec podside ping -M do -s 1472 -c 3 -W 2 198.51.100.2

3 packets transmitted, 0 received, 100% packet loss

Nothing local rejects the oversized packet, the router silently drops what does not fit and swallows its own warning, and podside gets nothing back — not an error, a hang. This is the black hole described earlier, reproduced on purpose. Revert the rule and align podside’s MTU to the router’s real ceiling, which fixes it without depending on any message arriving:

sudo ip netns exec router iptables -D OUTPUT -p icmp --icmp-type destination-unreachable -j DROP
sudo ip netns exec podside ip link set veth-pod mtu 1420

sudo ip netns exec podside ping -M do -s 1392 -c 2 198.51.100.2

2 packets transmitted, 2 received, 0% packet loss

1392 is 1420 minus the 28-byte header, the largest payload that now fits without fragmentation. Clean up:

sudo ip netns del podside
sudo ip netns del router
sudo ip netns del vpnside

Checking a real cluster

Against an actual cluster, probe between two pods with the same ping -M do -s <size> technique, sized just above your suspected true MTU minus 28:

apiVersion: v1
kind: Pod
metadata:
  name: mtu-probe
spec:
  containers:
    - name: probe
      image: busybox:1.36
      command: ["sleep", "3600"]

kubectl apply -f mtu-probe.yaml
kubectl exec -it mtu-probe -- ip link show eth0   # confirm the MTU Kubernetes reports
kubectl exec -it mtu-probe -- ping -M do -s 1472 -c 3 <other-pod-ip>

If that fails or times out while a smaller -s value succeeds, the pod network’s advertised MTU is larger than something in the real path can carry — the same problem, reproduced on your own infrastructure rather than in a namespace lab.

Conclusion

PMTUD is not a safety net for a hop you already know about. It exists to recover from mismatches nobody predicted, and depends on ICMP messages that VPNs and firewalls commonly filter. Where a lower-MTU hop is known, configure for it explicitly.

Auto-detected MTU is a value read once, not kept correct. The CNI agent reads the default-route interface’s MTU at start-up and never revisits it, so anything that changes the node’s effective path MTU afterwards — a VPN client chief among them — leaves a stale value in place indefinitely.

Size-dependent failures point the investigation the wrong way by default. A problem that only appears on large transfers and presents as slowness reads as an application or database issue. Treat “small requests fine, large ones degrade” as a recognisable network-layer symptom before profiling the application.

When you know the ceiling, set it. Measure the true minimum MTU across the whole path — the namespace lab above is that technique — and pin the CNI’s MTU to it rather than trust a mechanism that only ever runs once.