A hostPath device mount does not grant device access

Why open() on a passed-through device node returns EPERM as root, how the cgroup v2 eBPF device filter causes it, and the short control test that tells you it is the cause.

I passed a USB serial device into a pod with a hostPath volume of type: CharDevice. The device node showed up in the container, the container ran as root, and opening the device failed with EPERM.

None of that fits together, which is why it took so long to debug. The file permissions were permissive, the uid was 0, and the kernel still said permission denied.

What was actually going on

hostPath with type: CharDevice puts the device node into the container’s filesystem. What it doesn’t do is add the device to the container’s cgroup device allowlist.

On cgroup v2, which every current distribution uses, runc doesn’t use the v1 devices controller. It attaches an eBPF device filter to the container’s cgroup instead, and the kernel consults that filter on open() regardless of file permissions or capabilities. My device wasn’t on runc’s default allowlist, so the open was denied before the file mode was even looked at.

That’s why the error made no sense to me at first. I was looking at the userspace permission model, and the denial was coming from a kernel policy I couldn’t see.

A control test with /dev/null

/dev/null is on runc’s default allowlist and your device isn’t, so compare the two:

kubectl -n <ns> exec deploy/<app> -- python3 -c \
  "import os; os.close(os.open('/dev/null', os.O_RDWR)); print('null OK')"

kubectl -n <ns> exec deploy/<app> -- python3 -c \
  "import os; os.close(os.open('/dev/yourdevice', os.O_RDWR|os.O_NOCTTY|os.O_NONBLOCK)); print('device OK')"

If /dev/null opens and your device doesn’t, you can stop looking at permissions, drivers and udev rules. It’s the device cgroup. That single comparison rules out a long list of plausible causes.

If you want to confirm the mechanism as well:

cat /proc/self/cgroup                     # "0::/" means cgroup v2
ls /sys/fs/cgroup/devices/devices.list    # absent under v2

Why I test with os.open instead of cat

My first reflex was cat /dev/yourdevice, and on a serial port that’s a bad test. It can’t distinguish two quite different outcomes. An EPERM means the kernel refused you. An empty read means you got in and the device simply had nothing to say.

os.open(..., O_RDWR|O_NOCTTY|O_NONBLOCK) reports the errno directly, and it doesn’t block waiting for data that may never arrive.

There are exactly two fixes

Kubernetes has no per-container devices: field, and Docker’s --device has no Kubernetes equivalent. That leaves exactly two ways to get an entry into runc’s allowlist:

There’s nothing in between, and no capability substitutes for either. Adding CAP_SYS_ADMIN or CAP_MKNOD doesn’t help, because the device filter is evaluated separately from the capability set. If you find yourself adding capabilities in the hope of opening a device, you’re working on a different problem from the one you actually have.

For a single dongle, I’m not convinced a DaemonSet that runs privileged is an improvement over one container that runs privileged. I’d decide that on blast radius, and not be put off by the word “privileged” alone.

Two things that look like failures but aren’t

privileged: true merges over an existing capabilities.add. A strategic-merge patch leaves both in the rendered spec. The leftover NET_ADMIN or NET_RAW list is harmless, since privileged supersedes it, but at first glance it looks as if the patch didn’t apply.

When a udev rule has renamed the node and the original name is gone, you can still identify the driver by its major number:

Prove the fix before deploying it

I tested with a throwaway probe pod: same image, same hostPath, pinned to the same node with nodeName, plus privileged: true. It answers the question in seconds and doesn’t touch the real workload’s Deployment. When it printed OPEN OK, I knew the patch was right before I’d written it.