A stale lock on NFS can keep a single-instance daemon down for good

MEGAcmd restarted every 14 seconds with "Unable to connect to service". The real reason was an fcntl lock on an NFSv3 mount that outlived the killed server. Giving the lock file a new inode fixed it.

I restarted the Docker daemon on an old host to fix a bind address, and one container didn’t come back. It was MEGAcmd, the command-line client that syncs a NAS share to MEGA’s cloud storage. It restarted every 14 seconds with exit code 254, and its log said only this:

Unable to connect to service: error=111

error=111 only means “the server isn’t there”

MEGAcmd is split in two. A long-running mega-cmd-server does the work, and the mega-* commands are clients that talk to it. Error 111 is ECONNREFUSED: the client found no server. Why the server didn’t start was invisible, because the container runs it with --do-not-log-to-stdout.

So I stopped the restart loop (docker update --restart=no <container>), and ran the server by hand in a one-off container against the real config directory:

mega-cmd-server --do-not-log-to-stdout
# Another instance of MEGAcmd Server is running

It exited with 254. No other instance was running anywhere.

Where the lock lives

The same config, copied to local disk, started fine, even through the image’s real entrypoint. So credentials and the image were fine, and the problem had to be the location. The config directory sat on the NAS over NFSv3.

MEGAcmd enforces a single instance with an fcntl lock on ~/.megaCmd/lockMCMD. On local disk the kernel drops that lock when the process dies. On NFSv3 the lock is held by the NAS’s lock manager on behalf of the client, and when dockerd killed the server hard, nobody released it. Every new server saw the lock, assumed another instance, and quit.

The fix: a new inode

The lock belongs to the file, not the name. Copy the file, move the old one aside, and put the copy in its place:

cd <config-dir>/.megaCmd
cp -p lockMCMD lockMCMD.new
mv lockMCMD lockMCMD.stale-$(date +%Y%m%d)
mv lockMCMD.new lockMCMD
docker update --restart=always <container>
docker start <container>

That was all. The server came up, mega-whoami showed the account, and the sync was running again with no restarts since. MEGAcmd also has a --skip-lock-check flag, but that just removes the protection against two servers fighting over one config.

The broader lesson is about where state lives. A config directory on NFS looks like ordinary disk until a lock outlives its process. When I later moved the sync to a new host, the config went on local disk and only the synced data stayed on NFS.