Molecule skips prepare after an aborted run, and a destructive scenario then destroys nothing

A leftover state file makes Molecule report "instances already prepared" and skip the wipe. Run molecule reset first. Also: a cold start can outlast a 15 minute health gate.

I have a Molecule scenario that wipes a three-node Talos test cluster back to maintenance mode, rebuilds it from nothing, runs the build again for idempotence, and verifies. The first real run exited with code 2 and never wiped anything.

The clue was one line

The log said prepare: Skipping, instances already prepared. Molecule keeps the scenario’s state in a directory under ~/.ansible/tmp/, and an earlier aborted run had left it behind. So the wipe was skipped and converge ran against the old cluster. talosctl bootstrap then failed with etcd data directory is not empty and retried 60 times. Nothing destructive happened, but the run proved nothing either.

The fix is to clear the state before every run of a destructive scenario:

molecule reset -s lifecycle

For a scenario whose whole job is to start from nothing, a silent skip is the wrong default. The wrapper script should call reset, and ideally the scenario should fail instead of skipping.

The same run showed the role’s guard is too narrow: it looks for a listening etcd, not for an existing data directory, so it can try to bootstrap a node that already holds etcd data.

The second run: the health gate timed out

After the reset, the wipe, apply and bootstrap worked. Then the step “Wait for a healthy cluster” (talosctl health --wait-timeout 15m) failed after 14 minutes 59 seconds, with etcd still in Preparing: Running pre state on all three nodes. A few minutes later etcd was running and the checks passed.

The host was short of memory: the test VMs plus one other large VM added up to 22 GB on a 15.9 GB machine, with swap in use and the load at 6 to 8. A cold start of that cluster takes 10 to 15 minutes before the API answers. Either free memory first, or raise the timeout for the run, for example with -e talos_bootstrap_health_timeout=40m.

To read such a log without printing secrets, strip the colour codes (sed 's/\x1b\[[0-9;]*m//g') and grep for FAILED! => and the lines around it.