Triaging an NFS mount failure: rpcinfo first

Why open ports do not prove an NFS server is alive, why a working mount on another host proves nothing, and how a failed automount quietly sends writes to local disk.

A host stopped mounting an NFS share. The error pointed at the client, the fix looked host-local, and believing that is what cost me the afternoon.

mount.nfs: Protocol not supported

These days, before I touch a single mount option, I check whether the server is actually serving:

rpcinfo -p <server>

A healthy NFS server registers these programs:

If the output shows only portmapper rows, the NFS daemons aren’t running, whatever anyone has told you, and nothing you do on the client will fix that.

showmount -e <server> returning RPC: Program not registered tells you the same thing from another angle.

Open ports don’t prove the service is alive

This is what made the problem look client-side to me. The NAS’s nfsd had died, yet ports 111, 2049 and the mountd port all still accepted TCP connections. A port scan looked perfectly healthy, but nothing was registered behind any of them.

A port that answers tells you the host is reachable. It tells you nothing about whether the service behind it is running, and the service was what I needed.

Two error strings with one cause

Debian’s mount.nfs negotiates downward: 4.2, then 4.1, 4.0, then 3. When the server answers none of them, the message you get depends on which version reported last:

• NFSv4 → Protocol not supported
• NFSv3 → requested NFS version or transport protocol is not supported

Both sound as if the client is asking for something unreasonable, but neither means the client is misconfigured. I tried every version explicitly, including mountport=/mountproto= to skip the rpcbind lookup entirely, and all that got me was the same failure four more ways.

“It works on the other host” proves nothing

This was the most misleading data point of the lot. A second host had the same share mounted and healthy, and that seemed to clear the server completely.

But an existing NFS mount survives the server’s nfsd dying. That mount had been set up months earlier and had simply kept working. It didn’t show that the server worked. It showed that the server had worked once, at boot, fifteen weeks ago.

The only comparison that tells you anything is a fresh mount to a scratch mountpoint:

mkdir -p /mnt/nfstest
mount -t nfs -o vers=3,soft,timeo=30,retrans=1 <server>:/<export> /mnt/nfstest

Run it on the host that “works”. When it fails the same way there, the fault is on the server and you can stop investigating the client.

There’s a warning in this as well. A host that’s running on a mount it can no longer re-establish will break on its next reboot, and until then nothing tells you.

The expensive part: what happens while it’s broken

An x-systemd.automount entry that can’t mount doesn’t stop anything. It just degrades quietly, and applications write straight into the underlying directory on local disk.

In my case, containers had been writing into the empty mountpoint ever since the automount gave up after boot. That came to 179 MB of application state on the root filesystem, under a path everyone believed was on the NAS.

When the server comes back, the NFS mount shadows that data. It isn’t deleted, you just can’t see it, and the service starts up again on whatever state the NAS holds. Rescue it before remounting:

systemctl stop docker.automount
mv /docker /docker.local-strays        # the shadowed data
mkdir /docker

The mount unit ends up failed

A container runtime will retrigger the automount in a tight loop:

Got automount request for /docker, triggered by <pid> (dockerd)

Six attempts in one second trips systemd’s start limit, and the unit ends up failed rather than just inactive. Starting it again does nothing until you clear the failure:

systemctl reset-failed docker.mount docker.automount
systemctl start docker.automount

The order I’d triage in now

rpcinfo -p <server>            # 1. is anything registered?  ← start here
showmount -e <server>          # 2. are exports visible?
mount -t nfs ... /mnt/nfstest  # 3. does a *fresh* mount work anywhere?
journalctl -u docker.mount     # 4. only now, the client
du -sh <mountpoint>            # 5. what got written while it was down?

Steps 1 and 3 tell you whether the problem is on the machine you’re logged into at all. If I’d run those two first, I’d have saved most of that afternoon.