Automating an appliance Ansible cannot run modules on

An old NAS with Python 3.5 is a raw-only Ansible target. Everything then goes through a busybox shell that discards background jobs, reports success regardless, and has to be verified by its side effects.

Ansible needs Python on the target for everything except raw. An appliance that’s too old for current Ansible’s module payload (a NAS, a router, some embedded box) will still accept SSH and still authenticate, and then every module you point at it fails. What you have left is raw, which sends a command string to a remote shell. You can work with that, as long as you stop trusting exit codes.

In my case it was a QNAP with an ARMv5 CPU. The newest Python its package system offers is 3.5, and ansible-core’s module payload needs 3.6 or later because it uses f-strings. Pointing ansible_python_interpreter at the right binary can’t help with that, because it’s a hard version floor. What threw me at first was that connection and auth were healthy the whole time, so it looked like something I’d misconfigured.

A bare & is silently discarded

The first trap was starting a long-running process through raw. The shell on the other end is busybox, and a trailing & sent through Ansible does nothing at all:

touch /tmp/x &              # creates nothing, rc=0
setsid touch /tmp/x         # also nothing without a subshell
( setsid touch /tmp/x & )   # works

Only the last form, a subshell with the ampersand inside it, survives the session closing. All three return rc=0. So the first two fail completely and give no sign of it, and the only way I found to catch that is to check the side effect: the file exists, the process shows up in ps, the port answers.

Decide success by the thing you wanted

That turned out to be the rule for the whole appliance. The NAS’s package installer, started from a script, never returns. Its exit code tells you nothing, since it never exits. I treat the install as done when the thing it installs responds. Here that was an exporter’s /metrics endpoint, which I polled until it answered; from scratch on this hardware that took about 80 seconds.

Idempotency works the same way. Instead of tracking what the playbook did, I ask the appliance’s own state file what’s installed. On QNAP that’s qpkg.conf:

getcfg <Package> Version -f /etc/config/qpkg.conf
getcfg <Package> Install_Path -f /etc/config/qpkg.conf

Exporters that borrow node_exporter’s names

The exporter I was installing, qnapexporter, reuses node_exporter’s metric names: node_cpu_seconds_total, node_load1, node_memory_*. On a Prometheus that already scrapes real node-exporters, any query that doesn’t pin job="qnapexporter" quietly averages the NAS in with your Kubernetes nodes. The dashboard still looks plausible, and that’s why it’s easy to miss.

Two smaller things came out of the same install. A slow appliance may need its own scrape interval and timeout; this one went over a 10-second timeout now and then under I/O load. And a third-party Grafana dashboard exported from an old version may rely on panel plugins your Grafana doesn’t have, so check ls /var/lib/grafana/plugins in the pod before you import it.

And password auth no longer needs sshpass

ansible-core 2.20 handles ansible_password without the sshpass binary. A missing sshpass used to be a good sign that password auth was going to fail, but that’s no longer true. Test the connection instead of guessing from what’s installed.