A runner that takes exactly one job: "0" still turns the flag on

Gitea Actions jobs sat queued because act_runner was registering as ephemeral. The image only checks that GITEA_RUNNER_EPHEMERAL is non-empty, so setting it to "0" enables it, and the pod looks like a crash loop.

After I merged a run of changes to this site, none of the deploys happened. Gitea Actions showed some runs queued forever and others cancelled. The runner pod was up, but not in a way I liked: 1/2 CrashLoopBackOff, 25 restarts in three days. I already run a self-hosted runner as described in the runner notes, so I looked at the pod first.

A crash loop that wasn’t crashing

The runner container’s last state was Completed with exit code 0. No probes, no OOM kill. kubectl describe showed only BackOff, which is what Kubernetes says about any container that keeps exiting, however cleanly.

The runner’s own log was clearer. Each start printed that .runner was missing, registered again, picked up one task, and then logged shutdown initiated about 45 seconds later. The job it had picked up finished successfully at the same second. That timing is the test: if the runner shuts down right as its first job completes, it is running in ephemeral mode.

Why it was ephemeral

My Deployment had GITEA_RUNNER_EPHEMERAL: "0", and I had put it there meaning “off”. The image’s entrypoint script adds --ephemeral when the variable is non-empty. It never looks at the value, so the string "0" counts as set. The same goes for "false", and presumably for GITEA_RUNNER_ONCE, which is checked the same way.

The fix is to delete the variable. An ephemeral runner in a Deployment is almost always a mistake, because the Deployment restarts it with an ever-growing back-off, so throughput falls to one job per few minutes.

Why the deploys were cancelled, not just late

This part I only understood afterwards. The deploy workflow uses a concurrency group with cancel-in-progress: false, which sounds like “never cancel anything”. It still replaces older pending runs with the newest one. With a runner that took one job every five minutes, every new merge cancelled the deploy queued before it. For a “deploy the latest main” workflow that’s fine when the runner keeps up, and it’s exactly why four merges in a row produced no deploys when it didn’t.

Things I ran into while cleaning up

• With no state volume for .runner, every restart registers a new runner in Gitea. Expect a couple of dozen stale runner records to delete in the admin UI.
• Deleting a runner record while its pod is up leaves the pod logging unregistered runner every few seconds. Registration only happens in the entrypoint, so it never recovers on its own; delete the pod.
• I used rollout restart for that at first. Flux owns the Deployment and reverted the restart annotation within about a minute, which replaced the new pod and killed the job it had just picked up. That orphaned my re-run deploy a second time. Delete the pod instead.
• Jobs the removed runner was executing stay in_progress until Gitea’s zombie-task timeout. My bot token could neither cancel nor re-run them, so that was a job for the repository owner in the UI.
• Right after a pod restart the runner container often exits once or twice with could not read CA certificate. It races the Docker-in-Docker sidecar that writes the certificates, and settles after about fifteen seconds. It’s not the crash loop.