GitOps for state that has no operator: a reconciling CronJob

When the thing you want declared lives in an application database behind an API, a CronJob that writes it every few minutes can do the operator's job. What went wrong for me was in the shell, in the API verbs, and in triggering the job by hand under Flux.

Some configuration isn’t a Kubernetes object and has no operator. Gitea’s Actions variables and secrets are rows in Gitea’s database, and so are its user accounts. There’s no CRD I could commit. If I wanted them declared in git, and put back whenever someone changed them in the web UI, something had to write them through the API on a schedule.

I used a CronJob for that. The non-secret values live in a ConfigMap where I can diff them, the secret values come from a Secret provisioned elsewhere, and every few minutes the job runs and makes the API match. It’s GitOps over an API that has no operator, and the same approach works for any external system with an admin API.

Setting it up is simple. Everything that went wrong went wrong quietly.

Write every value, every time, with the right verb

Most APIs won’t give you a secret back, so there’s nothing to compare against. I write every value on every run. A nice side effect is that rotation needs no extra plumbing: change the Secret and wait one tick.

You do need to know which verb creates and which updates. Gitea is a good example of an API where that differs per resource. A variable is created with POST (201) and updated with PUT (204), while a secret uses PUT for both. A PUT to a variable that doesn’t exist yet returns 404 variable not found, which reads like a permissions problem even though it isn’t one. My script tries PUT and falls back to POST on a 404.

Then there’s the question of which identity the endpoint actually wants. Gitea’s repository variable endpoints reject a collaborator with write access, with user should be the owner of the repo. A bot account that can push can’t necessarily configure the repository.

A loop at the end of a pipeline can’t fail

My first draft had this bug, and it’s the worst kind: a green job that had configured nothing.

rc=0
echo "$VARS" | while read -r name value; do
  put_variable "$name" "$value" || rc=1
done
exit $rc     # always 0

The right-hand side of a pipeline runs in a subshell, so rc=1 gets set in a process that exits when the loop ends, and the parent’s rc never changes. The fix is to redirect from a file so the loop runs in the current shell:

rc=0
printf '%s\n' "$VARS" > /tmp/vars
while read -r name value; do
  put_variable "$name" "$value" || rc=1
done < /tmp/vars

Telling “not provisioned yet” apart from “broken”

When the Secret comes from another repository or another tool, there’s a window where the CronJob exists and the Secret doesn’t. With a plain secretKeyRef, every tick in that window ends in CreateContainerConfigError, which looks like a fault. I mark it optional: true instead and have the script exit cleanly, with a message, when the value is empty.

Pin a numeric runAsUser

curlimages/curl is the obvious image for a job like this, and it declares its user by name (USER curl_user). With runAsNonRoot: true and no runAsUser, the kubelet has no way to prove that a named user isn’t root, so it refuses the pod:

CreateContainerConfigError: container has runAsNonRoot and image has non-numeric user

Since then I always set a numeric runAsUser next to runAsNonRoot. The kubelet won’t take the image metadata on trust.

Triggering it by hand gets the Job pruned

The obvious way to test the job without waiting for the schedule is this:

kubectl create job --from=cronjob/<name> <name>-manual

Under Flux, that Job can disappear in the middle of its run. --from=cronjob copies the CronJob’s own labels onto the new Job, and that includes kustomize.toolkit.fluxcd.io/name. Flux’s garbage collector then finds an object carrying its ownership label that isn’t in the source, and prunes it.

Jobs created by the cron controller are safe, because Flux labels the CronJob object and not .spec.jobTemplate.metadata. If you want to run one by hand, strip the Flux labels from the generated Job before applying it, or just wait for the schedule.

Read the defaults when the job creates accounts

If the job creates accounts, check what the admin API does by default. Gitea’s POST /api/v1/admin/users sets must_change_password to true unless you tell it otherwise, so an automation account created that way can never log in unattended. I pass "must_change_password": false for anything that isn’t a person.