A Flux Kustomization fails on a fresh cluster when its objects need CRDs the same step installs

GitOpsKubernetes

Production applies these fine because the CRDs already exist. On a cluster rebuilt from nothing, a Kustomization with a HelmRelease and objects of the kinds that chart installs fails its dry run. Monitors for a Prometheus operator that is not there fail the same way. Split it into two stages.

My Flux manifests had only ever been applied to a cluster that already had every CRD, so I never saw this. When I rebuilt a test cluster from nothing and pointed Flux at the same manifests, three things broke in a row. All three come down to one fact: Flux dry-runs everything in a Kustomization before it applies anything in it.

Objects of a kind the release hasn’t installed yet

A Kustomization that holds a HelmRelease for cert-manager and also a ClusterIssuer and a Certificate fails with no matches for kind "Certificate". The CRDs come from the HelmRelease, which doesn’t exist yet because the dry run failed first. Same thing happened with a Traefik Middleware next to the Traefik release.

The fix is two stages. The second Kustomization lists the first in dependsOn and holds everything that needs the CRDs:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps-config
spec:
  dependsOn:
    - name: apps
  path: ./clusters/test/apps-config
  wait: true

Monitors for an operator that isn’t installed

The charts for MetalLB, Traefik and cert-manager all had a PodMonitor or ServiceMonitor switched on, which needs the Prometheus operator’s CRDs. Those only exist where the monitoring stack runs. The HelmRelease install failed with no matches for kind "PodMonitor" ... ensure CRDs are installed first, and with no install retries configured it stayed failed after the first attempt. Turn the monitors off in the overlay for a cluster that has no operator.

Things the base assumed

I also had to switch off persistence for Traefik, because its volume wanted a storage class the test cluster doesn’t have, and delete a DNS challenge webhook that had no use there.

The order of failures is the lesson. Each fix just let Flux get one step further and show the next problem, and every one was invisible on a cluster that already had the CRDs. A rebuild from nothing is the only real test of a base.