Taking a NAS down: record what you stopped, not what should run

Before a NAS reboots, everything using it has to stop. The inventory is wrong about what that is, umount does not release the mount, and the restore has to replay what was actually running.

Rebooting a NAS is easy. Doing it without corrupting anything that had a database open on it is harder, because everything mounting its shares has to stop first and start again afterwards. For me that meant Docker Compose stacks, Kubernetes workloads, and virtual machines whose disks live on the NAS.

I wrote it as an Ansible playbook with a down phase and an up phase, and what follows is what shaped it.

Ask the hosts, not the inventory

The inventory said which hosts ran which Compose stacks. It was wrong on every host I checked. One ran a Git server and its database off the share, another ran a CI runner, and neither of them was listed. I don’t think that’s unusual for an inventory. In my experience it’s the normal state.

So the down phase asks each host what’s actually running:

systemctl list-units 'docker-compose@*.service' --state=active --plain --no-legend

It does the same for VMs on the hypervisor and for NFS-backed workloads in the cluster.

Record what you stopped, and restore from that record

The down phase writes what it stopped into a state file per host. The up phase starts exactly that list and nothing else.

The obvious alternative is “start whatever is supposed to run”, and that turns a maintenance window into a surprise deployment. A service that someone had deliberately left down for weeks comes back because the playbook thought it should be up. I want a maintenance playbook to put the system back the way it found it, even where that differs from what the configuration describes.

There’s a flip side, and it belongs in the playbook’s own documentation: anything stopped by hand before the down phase isn’t in the record, so it won’t come back without an override. That’s the correct behaviour. It will still surprise someone.

Release the mount through the automount units

If the share is mounted with x-systemd.automount, running umount just fights the automounter, because the next access remounts it. Stop the units instead, automount first:

systemctl stop docker.automount docker.mount

I only reach for umount -l as a fallback when something is still holding the mount.

Suspend GitOps, then scale

In a Flux-managed cluster, the next reconcile reverts any Deployment you scaled to zero. Where each NFS-backed app has its own Kustomization, you can suspend it, and then a temporary scale-down holds without anything touching git:

flux suspend kustomization <app>
kubectl -n <ns> scale deploy/<app> --replicas=0
# ... maintenance ...
flux resume kustomization <app>   # reconcile restores the replica count

There’s no commit, no PR, and no drift left behind afterwards.

Check mounts with findmnt, not the hypervisor’s opinion

Proxmox reported the NFS storage as “inactive” the whole time, while df on the mount point answered without trouble. It turns out pvesm status checks liveness with showmount, and that fails against a NAS whose mount daemon is missing or listening on a non-standard port, whether or not the established mount works. When the question is “is it mounted, and is it in use”, I ask the kernel:

findmnt -T /mnt/pve/<storage>

The same distinction (probe the RPC services rather than the ports) is also the first step in triaging an NFS mount failure.

--check gives you a real dry run here

With --check, command tasks skip themselves and module tasks report what they would change. If you mark the read-only discovery tasks check_mode: false, the state files still get written, so the dry run shows you exactly which services and VMs would be stopped. I read that list before every real run.