A SOPS secret deployed without decryption is just ciphertext

If the Flux Kustomization that applies a *.sops.yaml Secret has no decryption block, Flux applies the encrypted text as the value and reports success. The app then fails with a baffling auth error. How to spot it without reading the secret.

I encrypted an API key with SOPS and age, committed it as unifi-sync.sops.yaml, and merged. Flux reconciled without a complaint, the Secret appeared in the namespace, and the job that used it got 401 Unauthorized.

I knew the key was right, because I’d copied it from a file that worked. And the file in Git decrypted to the same value, so the secret wasn’t wrong in the repository. What the cluster held was.

What was in the Secret

The value was 151 characters long. The key is 32. A quick look, without printing anything, explains it:

kubectl get secret <name> -o jsonpath='{.data.api-key}' | base64 -d | grep -c '^ENC\['
# 1

The Secret contained the literal ENC[AES256_GCM,data:...] string. Flux had applied the encrypted manifest as it was. SOPS only encrypts the stringData values, so the result is a perfectly valid Secret whose content happens to be ciphertext.

Why nothing complained

Decryption in Flux belongs to the Kustomization that applies the file, and mine was a shared one with no decryption block. Only the per-app Kustomizations I’d created earlier had one, so the first secret to land in a shared one was applied raw. The Kustomization reported success, because from its point of view it applied what it was given. A server-side dry run of the file doesn’t help either: it rejects the sops: metadata block outright, which says nothing about what Flux will do with it.

Fix

Add the block to the Kustomization that applies the file:

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

sops-age is the Secret holding the age private key in flux-system. After the next reconcile, the same check prints 0 and the job worked on its first try.

If you add a first *.sops.yaml file to an existing app, check which Kustomization owns that directory. And when a freshly deployed secret misbehaves, count how many characters it has before you doubt the key.