flux suspend does not hold a Kustomization that another Kustomization manages

A correction to my NAS maintenance notes: when a parent Kustomization applies the child, the next sync clears a manual suspend. Annotating the one object with reconcile=disabled holds, but it also stops Flux from pruning it.

In my notes on quiescing a cluster for NAS maintenance, I suggested flux suspend kustomization <app> and then a temporary scale-down. That works for a few minutes. In my cluster it doesn’t hold any longer than that, and this week it cost me a scale-up I thought I’d done.

What happened

I was moving a document management app into the cluster. Its scanner-ingest pod had to take over an address from an old host, so the manifests shipped it at replicas: 0. Starting it early would have put two machines answering ARP for the same address. On cutover day I suspended the app’s Kustomization and scaled the Deployment to one.

A few minutes later it was back at zero, and the Kustomization was no longer suspended.

Why

The app’s Kustomization object isn’t something I created by hand. It’s a YAML file in the repository, under the cluster directory, and the root Kustomization (flux-system) applies that directory. flux suspend sets spec.suspend: true on the live object. The file in git doesn’t have it, so on the root’s next sync, Flux applies the file and the suspend is gone. The child then reconciles, and puts replicas: 0 back.

So suspend is only durable on a Kustomization nothing else applies, which is the top one. Suspending the root would have held everything, and that was more than I wanted.

What holds

Flux checks an annotation on each object it applies:

kubectl -n <ns> annotate deploy/<name> \
  kustomize.toolkit.fluxcd.io/reconcile=disabled

With that, Flux leaves this one Deployment alone, and the rest of the app keeps reconciling. The scale to one stuck. I kept the annotation until the pull request that set replicas: 1 in git was merged, then removed it. Flux re-applied its templates and rolled the app’s web server once. That was harmless.

What it costs: no pruning

The same day I moved another app out of the cluster, onto a separate host, and used the same annotation to hold its old Deployment at zero while I copied its data out. The pull request that followed deleted that Deployment from git.

Flux didn’t delete it from the cluster. An object marked reconcile=disabled isn’t pruned either, so the Deployment stayed until I deleted it by hand. That’s consistent once you think about it. “Don’t touch this” includes “don’t delete this”. But it means every hold needs a matching cleanup. Either take the annotation off before the change that removes the object, or delete the object yourself afterwards.

The corrected recipe

For a temporary hold on a few objects:

kubectl -n <ns> annotate deploy/<name> kustomize.toolkit.fluxcd.io/reconcile=disabled
kubectl -n <ns> scale deploy/<name> --replicas=0
# ... maintenance ...
kubectl -n <ns> annotate deploy/<name> kustomize.toolkit.fluxcd.io/reconcile-
flux reconcile kustomization <app>   # restores the replica count from git

The old recipe is still right for a Kustomization that nothing else applies. To know which case you’re in, check whether the Kustomization’s own YAML lives in a path another Kustomization reconciles.