Flux's Ready condition can't tell you how long something has been broken

A Kustomization that keeps failing flips its Ready condition between False and Unknown on every reconcile, and lastTransitionTime moves with it. An alert for "not ready for ten minutes" built on that field almost never fires. Keep your own first-seen time.

I wrote a small health monitor for my cluster. It runs outside the cluster on purpose, on a Raspberry Pi with a read-only ServiceAccount, so it can still say “the API is unreachable” when everything else is down. It starts at the Flux Kustomizations and only drills into workloads, pods and events where something is wrong. One of its rules was simple: alert if a Kustomization has been not ready for more than ten minutes.

Built on the obvious field, that rule would almost never have fired, even for a Kustomization that was failing the whole time.

Why the timestamp keeps moving

Every condition in a Kubernetes status has a lastTransitionTime, and the obvious reading is “this has been the status since then”. The usual way to ask “how long has this been broken” is now minus lastTransitionTime on Ready=False.

Flux doesn’t leave that condition still while it retries. On each reconcile attempt, a failing Kustomization goes from Ready=False to Ready=Unknown (reconciliation in progress) and back to False when the attempt fails. Each flip is a transition, so the timestamp is reset every interval. Something broken for a day looks as if it went bad a minute ago.

kubectl -n flux-system get kustomization <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.lastTransitionTime}{"\n"}{end}'

Run that a few times against a failing Kustomization and watch the time move.

Keep your own clock

So the monitor keeps its own state file. The first time it sees a problem, it records the time and uses that from then on. The problem’s age is “now minus first seen”, whatever Flux’s timestamps do in between.

Two more things the state needed. The final code review caught the second:

• Keep the state as a map from problem to severity, not a list of problems. A problem that gets worse should alert again, and in a plain list it looks the same as before.
• Debounce carefully. A new problem has to be confirmed before it alerts. The first version treated an already-reported problem that escalated as a fresh, unconfirmed one, so it showed up as “resolved” for one run before it came back.

The read-only role is worth a check of its own. A monitoring ClusterRole should read Kustomizations, pods and events, and not Secrets:

kubectl auth can-i list secrets \
  --as=system:serviceaccount:<ns>:<sa>        # no
kubectl auth can-i list kustomizations.kustomize.toolkit.fluxcd.io \
  --as=system:serviceaccount:<ns>:<sa>        # yes