A second NIC on DHCP gives a Talos node a second default route

A macvlan parent interface left on DHCP installs a LAN default route at the same metric as the real one. Egress drifts onto the wrong network and replies to LAN clients take a different path than their requests. Why a /32 with no gateway is the fix, and the commands that show the problem.

Putting pods on the LAN with macvlan needs a parent interface that is up and on that LAN. My Talos nodes have a second NIC for it, and I’d left it on DHCP. DHCP gave it a lease and a default route via the LAN’s gateway. The first NIC already had one, with the same metric.

Two equal defaults mean the kernel installs only one of them, and you don’t get a say in which. Node and pod egress started leaving through the LAN. And replies to LAN clients that had called the cluster’s virtual IPs now went back out the LAN interface, bypassing the router that saw the request, so the router’s connection tracking never saw an answer.

Seeing it

Talos’ routing resources are the place to look. A few of the usual greps don’t work there:

talosctl -n <node> get routespecs        # shows both defaults
talosctl -n <node> get operatorspecs     # a dhcp4 operator for the second NIC
talosctl -n <node> get routes | grep -E 'RouteStatus +inet4/[0-9]'

grep default never matches because a default route has an empty destination column. Use the pattern above, and look for the route out of the first NIC. Check routespecs too: routes shows only what the kernel kept, routespecs shows what Talos was asked to install.

The asymmetric part shows up on the router. In the conntrack table, flows to the virtual IP sit at [UNREPLIED] with packets=0 on the reply side.

Fix

Give the interface a fixed address as a /32, turn DHCP off and don’t add a gateway:

machine:
  network:
    interfaces:
      - interface: ens19
        dhcp: false
        addresses:
          - 192.0.2.15/32

A /32 keeps the link up for macvlan without creating a connected route for the LAN, so nothing sends traffic through it by accident. A /22 would have put the connected route back and the replies with it. The macvlan pods carry their own LAN addresses.

Pick that address outside whatever your macvlan pods already use; I nearly took one that belonged to a pod.

Apply it live. talosctl patch machineconfig --patch @file.yaml --dry-run tells you whether it needs a reboot; this one didn’t. Right after the patch, the Talos API can time out for a few seconds, so retry before concluding it broke.

One more consequence to expect: this changes the address that flannel recorded for the node, which is how I found that page.