A backup job that exits 0 and backs up nothing

The backup command I was asked to schedule did nothing on the current version: it printed usage text, exited 0 and wrote no file. So the job now checks the file it produces and refuses to upload anything that is not a real archive.

The request was specific: back up the Pi-hole configuration to the NAS every night with Pi-hole’s Teleporter export, pihole -a -t. It had become urgent that same afternoon. A migration had pruned an old volume, and the only copy of the household DNS configuration left was a single manual export on my workstation.

A CronJob that runs a command and copies the result somewhere is ten lines of YAML. Before writing any of them, I ran the command by hand in the running pod.

It printed Pi-hole’s usage text and exited 0.

The command that stopped being a command

pihole -a -t is Pi-hole v5 syntax. Version 6 doesn’t recognise it, so the call falls through to the generic help, which exits successfully and writes no file. If I’d built the job literally around that command, it would have reported success every night while backing up nothing, and nobody would have found out until the day we actually needed the backup.

In v6 you get the same zip archive either from pihole-FTL --teleporter inside the container or from GET /api/teleporter over HTTP.

None of this is really specific to Pi-hole. I verify a backup job by looking at the file it produced, because the exit code tells me very little. Upgrades rename commands, and shell wrappers, which is what most backup scripts are made of, are the tools most likely to quietly turn into “print help and exit 0”.

Why I went with HTTP instead of the CLI

The CLI has to run where the config volume is mounted. That volume is ReadWriteOnce and attached to whichever node runs Pi-hole, so a separate backup pod would have to be scheduled onto the same node. And the Pi-hole image has no ssh, scp or sshpass, so I couldn’t do the upload from inside it either. The API endpoint has neither problem: any pod on any node can fetch the archive through the Service.

I’d apply the same reasoning to other services on RWO volumes. If there’s a way to get a backup through the API or the network, I’ll take that over anything that needs the volume mounted.

Refusing to upload garbage

An HTTP endpoint fails in different ways from a CLI. A wrong password gets you a JSON error, a proxy hiccup gets you an HTML page, and a timeout gets you half a file. Often enough to matter, these come back with a success status, and any of them would have overwritten yesterday’s good backup.

So before uploading, the job checks what it downloaded:

head -c 2 teleporter.zip | grep -q '^PK' || { echo "not a zip"; exit 1; }
[ "$(wc -c < teleporter.zip)" -ge 10000 ] || { echo "too small"; exit 1; }

With the zip magic bytes and a minimum size checked, each of those failures shows up as a red job and the previous backup stays where it is. The first real run produced an archive of about 24 KB. I confirmed that by listing the directory on the NAS rather than trusting the job’s log.

An old SSH server, handled properly

The NAS runs OpenSSH 7.6, which only offers ssh-rsa host keys. OpenSSH 8.8 and later disable SHA-1 ssh-rsa by default, so scp fails with no matching host key type. Talking to it takes both options, and I added the legacy scp protocol on top:

scp -O \
  -o HostKeyAlgorithms=+ssh-rsa \
  -o PubkeyAcceptedAlgorithms=+ssh-rsa \
  -o UserKnownHostsFile=/etc/ssh-pin/known_hosts \
  -o StrictHostKeyChecking=yes \
  teleporter.zip "$NAS_USER@$NAS_HOST:$NAS_PATH/"

At this point most people reach for StrictHostKeyChecking=no. Pinning the host key costs one ConfigMap: run ssh-keyscan -t rsa <host> into it, mount it, and point UserKnownHostsFile at it. Host keys are public, so committing it is fine, and the job can’t be redirected to some other machine any more.

I also kept credentials off the command line. sshpass -e reads the password from $SSHPASS, and the API login is a JSON body piped into curl --data-binary @-. Anything in argv can be read from /proc by every process on the node.

Two things that cost me time while testing

The first was a pod that sat in ContainerCreating forever. It mounted a ConfigMap I hadn’t created yet, and the only place that shows up is the pod’s events (MountVolume.SetUp failed ... not found), not its status. From the outside it looks exactly like a slow image pull.

The second was backoffLimit: 0. If you delete the pod by hand with that setting, the Job ends for good with BackoffLimitExceeded. To retry a test you have to delete and recreate the Job itself, not its pod.

And then the whole cluster rebooted

Halfway through testing, all three nodes rebooted within seconds of each other. They were back after about three minutes, and the Pi-hole StatefulSet needed another ninety seconds to replace its pods. Household DNS was down for roughly six minutes.

The active/passive DNS design had already written this case off as out of scope: putting replicas on different nodes doesn’t help when every node goes down at once. To survive that you need a resolver outside the cluster. It also made the case for the backup rather better than I had that evening.