The cluster now builds from an Ansible collection, and the old roles are gone

Taking the Talos collection from a green test cluster to the live one: the drift a read-only check showed, three bugs only production could find, and a feature I moved and then deleted because Flux already did it.

For a while the Talos part of my homelab has been split across an old Ansible role and a newer collection that I’d only ever run against a test cluster. A few days ago I finished the move: the collection manages the live cluster, and the old roles are deleted. This is how that went, including the parts I got wrong.

Making the collection installable

The first steps were housekeeping. I wrote down how a change gets promoted from the test cluster, ran the collection’s integration workflow against the test nodes (green), added the MetalLB label patch to the collection’s test inventory, and tagged v0.1.0. The repository that uses it now pins the collection by tag in collections/requirements.yml, not by checkout. I’d worked out earlier how to test a collection without a cluster, and that setup is what made a tag feel safe.

What a read-only check said about production

Before applying anything I ran the playbook in --check --diff mode against the production group, with the output sent to a file because a diff may hold secrets, and read only the key names. It ran clean and showed the drift I’d expect. The rendered config added the node IP and the virtual IP to the cert SANs, moved the installer image from the stock Talos 1.13.4 to a factory image of 1.13.6, added --iface to the flannel arguments, and collapsed duplicated machine.disks partitions, which was the change that would trigger the reboot. It also removed two extraManifests (the kubelet serving-cert approver and metrics-server). That one made me stop: I had to confirm Flux delivers them before applying, or the cluster would lose its CSR approver and its metrics.

The check also said “etcd does not exist yet; bootstrap would run on” the first node. That’s a false positive. The talosconfig file for the cluster didn’t exist yet, because the first apply is what writes it, and the real run checks the etcd members first anyway.

Three bugs only production could find

The apply itself converged all three nodes, one at a time. I wrote up the node that came back on the wrong address separately. Around it, the collection needed three fixes, each tagged as a release, and none of them showed up on the test cluster.

• The upgrade role’s --check failed before the first apply had written a talosconfig.
• A failed final health gate had left no talosconfig behind, so the cluster role now keeps it before converging the nodes.
• The upgrade role refused to run because it thought my talosctl was a downgrade. talosctl version --client=false still prints a Client block, and my client (1.14.2) was newer than the nodes (1.13.6). The role now reads the Server block. On the test cluster the client and the server had the same version, so it never showed.

I also broke CI on the collection’s main for a few minutes by merging a pull request before its own CI run had finished. Lesson: wait for the PR’s run, not for the one on main.

The feature that wasn’t missing

With the collection pinned at v0.1.3, --tags upgrade --check against production reported Talos 1.13.6 and Kubernetes 1.36.1 up to date on every node, with changed=0. Time to delete the old roles.

The old role did one thing the collection doesn’t: it created the namespaces and Secrets the apps need. So I moved that into a small role of my own in the playbook repository, tagged so it never runs by default, and tested it on the test cluster. It worked: nine changes, then none on the second run.

Then I looked at the GitOps repository and found it already holds all eight of those Secrets as SOPS files, which Flux creates. The role was dead code, and worse, its default kubeconfig pointed at production. I removed it again. I should have grepped the GitOps repository for an owner before I wrote it.

While I was in there I deleted about 97 files of dead playbooks and a stale talosconfig. 19 of the 30 variable names in the encrypted vault had no reference in the repo. I dropped them, decrypting in memory and re-encrypting straight to the file with an assertion that the remaining values hadn’t changed. And I found a decrypted copy of the vault from September in an ignored tmp/ directory, which I shredded. After vault work, check tmp/.

Still open

The destructive scenario that rebuilds a test cluster from nothing hasn’t passed yet; its first runs ran into stale Molecule state. I haven’t decided whether the virtual IP should become the production endpoint, and the production machine.nodeLabels patch has to change shape before Talos 1.14.