Putting a pod on the LAN for multicast discovery

mDNS and SSDP do not cross the pod network. A macvlan second interface through Multus gets around that, provided the node NIC, the CNI plugins and the attachment routes are all right. Each of them fails silently when it is not.

Some applications find devices by multicast: mDNS, SSDP, HomeKit. Home Assistant is the usual example. Multicast doesn’t leave the pod network, so in a normal pod discovery simply finds nothing, and nothing reports an error. hostNetwork: true would work, but it hands the pod the node’s entire network stack, which is more than I wanted to give it.

What I did instead was give the pod a second interface. Multus runs as a meta-plugin in front of the cluster CNI, and a macvlan attachment gives the pod its own address on the LAN. There were four layers to get right, and at every one of them the broken state looked like success.

1. A LAN interface on the node, with no gateway

macvlan needs a parent interface on the node that sits on the LAN. My nodes’ primary network is a separate VLAN, so that meant adding a second NIC.

I learned to configure that NIC explicitly, including switching DHCP off. Talos, for one, applies default DHCP to any interface the machine config doesn’t mention. So a NIC I’d added and “left alone” wasn’t inert at all. On a LAN with an ordinary DHCP server it picked up an address and a default route.

Two default routes at the same metric don’t produce an error. The kernel installs one and the other stays dormant, and you don’t get to choose which. In my case all node egress quietly moved from the cluster VLAN onto the household LAN, and nothing broke badly enough for me to notice. On Talos the evidence is in the gap between what was requested and what actually got installed:

talosctl -n <node> get routespecs   # both default routes requested
talosctl -n <node> get routes       # only one actually installed

grep default finds neither, by the way. talosctl prints an empty destination column for a default route and never the word default, so you have to match the empty destination:

talosctl -n <node> get routes | grep -E 'RouteStatus +inet4/[0-9]+\.[0-9.]+//'

The fix was a static address, dhcp: false, and no route. It’s dhcp: false that gets rid of the unwanted default route; adding a static address while DHCP stays on wouldn’t have. I select the NIC by MAC address (deviceSelector.hardwareAddr) because kernel interface names aren’t guaranteed. Keep in mind, though, that a selector matching nothing fails silently. If you apply the patch to the wrong node, that node just stays unconfigured.

2. The CNI plugins on the node

Most distributions ship the full set of reference CNI plugins. Talos ships bridge, firewall, flannel, host-local, loopback and portmap. That means no macvlan and no static IPAM, and its flannel DaemonSet doesn’t install any plugin binaries of its own. I’d check this before designing anything:

talosctl -n <node> ls /opt/cni/bin

Sidero publishes ghcr.io/siderolabs/install-cni:<version>, which has the full upstream set built for each Talos release. Don’t run its bundled install script. It copies everything, including over the flannel and bridge binaries the running cluster depends on. I used a DaemonSet with my own loop that copies only the missing plugins. Each copy goes to a temporary name first and is then moved into place with an atomic mv, because otherwise a CNI invocation running at the same moment on a live node could execute a half-written binary.

That DaemonSet needs privileged access to the host. On a cluster enforcing PodSecurity baseline it will be created and never run a pod unless you label its namespace privileged.

3. Multus in front of the cluster CNI

Multus installs itself as the CNI entrypoint and delegates the primary interface to the existing plugin. Get it wrong and pod networking breaks for the whole cluster. Two details from upstream’s thick-plugin manifest mattered.

With multusConfigFile: auto, Multus generates 00-multus.conf by wrapping the lexically first conflist in the CNI config directory. That’s how the existing CNI survives as the default delegate, and it’s also why the generated file has to sort before it: that ordering is what moves the entrypoint.

The CNI binary directory also has to be mounted at the same path inside the container as on the host. Upstream uses /opt/cni/bin for both. If you remap it to something tidier, delegation breaks.

Upstream also publishes moving snapshot-thick tags. In a GitOps repo I pin a release tag, and I check that the tag exists before committing it. A registry manifest request answers in about a second; waiting for ImagePullBackOff takes a good deal longer.

4. The attachment needs a multicast route and no gateway

The NetworkAttachmentDefinition is where discovery failed silently for me:

{
  "cniVersion": "0.3.1",
  "type": "macvlan",
  "master": "<lan-nic>",
  "mode": "bridge",
  "ipam": {
    "type": "static",
    "addresses": [{ "address": "<lan-address>/<prefix>" }],
    "routes": [{ "dst": "224.0.0.0/4" }]
  }
}

Without the 224.0.0.0/4 route, multicast follows the pod’s default route into the overlay and never reaches the LAN. Everything about the attachment looks right (the address is assigned and the LAN answers pings), yet discovery does nothing.

Adding a gateway causes the opposite problem. The macvlan default route then outranks the pod network’s, and cluster DNS breaks. The address and prefix on their own are enough to reach the LAN.

What two addresses do to anything that advertises one

Anything that advertises a single address now has to pick one of the two. Home Assistant’s HomeKit bridge can end up picking the pod-network address, which no Apple device can reach, so I had to set its “advertise IP address” option to the LAN address explicitly. Once a pod has a second interface, I’d check every service in it that announces itself in the same way.