Graceful node shutdown leaves Completed pods that nothing cleans up

After each rolling reboot, Deployment pods stayed behind in phase Succeeded. The pod garbage collector only acts above 12,500 terminated pods, and lowering that would also eat CronJob history. A small CronJob that deletes only orphans owned by ReplicaSets, StatefulSets and DaemonSets does the job.

A health check of my cluster counted 61 pods in Completed state. Most of that is normal: a CronJob keeps the pods of its last few runs. But some of these belonged to Deployments, and a Deployment pod has no business being Completed.

Sorting them by owner

The quickest way to see what you’re looking at is to group terminated pods by the kind of their owner:

kubectl get pods -A --field-selector=status.phase==Succeeded -o json \
  | jq -r '.items[] | .metadata.ownerReferences[0].kind // "none"' | sort | uniq -c

48 were owned by Jobs, which means CronJob history, bounded by successfulJobsHistoryLimit and failedJobsHistoryLimit. 13 were owned by ReplicaSets. Their creation dates matched the days I’d rebooted the nodes, one after another.

Where they come from

Talos shuts nodes down gracefully, and the kubelet’s graceful node shutdown terminates the pods on the node before it goes. Those pods end in phase Succeeded or Failed. The ReplicaSet doesn’t care: it creates new pods elsewhere, and the old objects stay behind.

Kubernetes does have a garbage collector for terminated pods, in the kube-controller-manager. It only starts deleting when there are more than --terminated-pod-gc-threshold of them, and the default is 12,500. A small cluster never gets there, so these pods stay forever.

The options I looked at:

• Lowering the threshold is a control-plane flag (machine config on Talos, not something I can manage through GitOps). It would also delete Job pods, and with them the logs that are my CronJob history.
• ttlSecondsAfterFinished on the CronJobs is about Job pods, not these.
• The descheduler’s RemoveFailedPods handles Failed, not Succeeded.

A small CronJob

So I wrote one. Once a day it lists pods in phase Succeeded or Failed, keeps the ones owned by a ReplicaSet, StatefulSet or DaemonSet that finished more than an hour ago, and deletes them. Everything else is left alone. Job pods are the history. A pod with no owner was created by hand, and someone may still be reading it.

The script is plain Python with only the standard library, so the image is python:3.13-alpine, and it talks to the API with the ServiceAccount’s token and CA certificate from /var/run/secrets/kubernetes.io/serviceaccount. The official kubectl image has no shell, which makes filtering by owner awkward. The finish time is the latest finishedAt of the pod’s containers, falling back to the Ready condition’s transition time and then the creation time.

I tested the selection against a snapshot before giving it delete rights:

kubectl get pods -A -o json > pods.json
python3 -I pod_cleanup.py --from-file pods.json

It picked exactly the 13 orphans out of 173 pods. The first real run deleted them, and only Job-owned pods remained.