k8wiz
Back to Articles

Ready to test your Flux CD knowledge? Try our Flux CD quiz!

Encrypting Flux secrets with SOPS and age, from first key to rotation

Set up SOPS and age with Flux step by step: create the key, encrypt Secrets in Git, configure decryption, catch mistakes in CI and rotate keys safely.

10 min read
By k8wiz Team
  • fluxcd
  • sops
  • secrets
  • security
  • gitops

GitOps wants everything in Git, and Secrets are where that idea gets uncomfortable. Base64 is an encoding, not encryption, so a plain Secret manifest in a repository is a leaked password waiting for someone to clone it. Flux solves this with SOPS. You encrypt the values of each Secret before you commit, and kustomize-controller decrypts them inside the cluster, right before it applies them.

SOPS can encrypt with OpenPGP, age, OpenBao/Vault or a cloud KMS. The Flux docs recommend age over OpenPGP where you can, and I agree: an age key is one short line of text, there is no keyring to manage, and nothing expires behind your back. This guide sets up age from scratch on a cluster bootstrapped with Flux, then covers the parts most tutorials skip, which are checking that nobody commits a plain Secret and rotating the key later.

You need the flux, kubectl, sops and age command line tools, and a cluster where Flux is already running in the flux-system namespace.

Step 1: generate an age key

An age key pair has a private identity and a public recipient. Anyone with the public key can encrypt, so it is safe to commit. Only the holder of the private key can decrypt, which in this setup means the cluster and whoever you trust to edit secrets.

Generate one key per cluster. If production and staging share a key, then anyone who can read staging secrets can read production secrets too.

age-keygen -o production.agekey

The command writes the private key to the file and prints the public key on stderr. It looks like Public key: age1.... You can print it again at any time:

age-keygen -y production.agekey

Store a copy of production.agekey in your password manager or another offline place now, before you do anything else. If the only copy lives in the cluster and the cluster is lost, every encrypted file in Git is lost with it.

Step 2: store the private key in the cluster

kustomize-controller reads decryption keys from a Kubernetes Secret. The controller decides what each entry holds by the end of its key name, so an age private key must sit under a key ending in .agekey. An entry called age.key or private is not detected as an age key, and decryption fails.

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=production.agekey

The namespace matters. spec.decryption.secretRef only takes a name, so the Secret must be in the same namespace as the Kustomization that refers to it. If your Kustomizations live in flux-system, put the Secret there.

Create this Secret by hand, or with your cluster provisioning tooling. Don't commit it to the repository, even encrypted, because Flux would need the key to decrypt the key.

Step 3: tell sops which key to use

You could pass the public key on every sops command, but someone will eventually forget the flag or paste the wrong key. A .sops.yaml file at the root of the repository fixes that by mapping file paths to keys and to the fields that get encrypted.

creation_rules:
  - path_regex: ^apps/production/.*\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1replacewithyourproductionpublickey00000000000000000000000
  - path_regex: ^apps/staging/.*\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1replacewithyourstagingpublickey000000000000000000000000000

Three details in this file trip people up. First, encrypted_regex limits encryption to data and stringData. kustomize-controller needs apiVersion, kind and metadata in plain text, and the Flux docs say outright that encrypting them is not supported.

Second, sops compares path_regex against the file path relative to the directory that holds .sops.yaml, and it uses the first rule that matches. A broad rule at the top of the list shadows every rule below it, so put the specific rules first. A rule with no path_regex matches every file.

Third, sops looks for .sops.yaml starting from your current working directory and walking up through the parent directories, not from the directory of the file you encrypt. Run sops from inside the repository. If you run it from your home directory with a path into the repository, it won't find the config and won't know which key to encrypt for.

Keep .sops.yaml out of any directory that a Flux Kustomization applies. When a path has no kustomization.yaml, kustomize-controller generates one from every YAML file under it, and a stray config file there breaks the build. The repository root is usually safe. Otherwise exclude it with the ignore field on the GitRepository.

Step 4: encrypt a Secret

Write the Secret manifest with kubectl in dry-run mode, so the values never touch the cluster in plain text, then encrypt the file in place.

mkdir -p apps/production/secrets
kubectl create secret generic db-credentials \
  --namespace=my-app \
  --from-literal=username=app \
  --from-literal=password='change-me' \
  --dry-run=client -o yaml > apps/production/secrets/db-credentials.yaml

sops encrypt --in-place apps/production/secrets/db-credentials.yaml

Open the file afterwards. The metadata block is still readable, every value under data now looks like ENC[AES256_GCM,data:...], and a sops section at the bottom records the recipients and a message authentication code. That MAC covers all the values, which is why you can't hand-edit an encrypted file and expect it to decrypt.

Don't run kubectl apply on this file. The API server would store the ciphertext as the Secret value. Encrypted files are meant to be read by kustomize-controller only.

Commit and push it.

git add .sops.yaml apps/production/secrets/db-credentials.yaml
git commit -m "Add encrypted database credentials"
git push

Step 5: turn on decryption in the Kustomization

Decryption is opt-in per Kustomization. Add a decryption block with the provider and the Secret from step 2. sops is the only provider the field accepts.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: production-secrets
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production/secrets
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age

Commit this manifest under the path your flux-system Kustomization already applies, such as clusters/production. If you prefer the CLI, flux create kustomization has --decryption-provider and --decryption-secret flags, and --export prints the YAML so you can commit it.

flux create kustomization production-secrets \
  --source=GitRepository/flux-system \
  --path=./apps/production/secrets \
  --prune=true \
  --interval=10m \
  --decryption-provider=sops \
  --decryption-secret=sops-age \
  --export > clusters/production/production-secrets.yaml

A common surprise is an encrypted file that ends up under the path of a Kustomization without decryption, often the bootstrap flux-system one. That Kustomization never decrypts, so the reconciliation fails with an error saying the Secret is SOPS encrypted and decryption must be configured. Keep encrypted files in directories that only decrypting Kustomizations read.

Step 6: check the result

Ask Flux to reconcile now instead of waiting for the interval, then confirm the Secret exists with real values.

flux reconcile kustomization production-secrets --with-source
kubectl -n my-app get secret db-credentials -o jsonpath='{.data.username}' | base64 -d

If the Kustomization reports a decryption failure, check three things in order. Is the Secret in the same namespace as the Kustomization? Does its entry name end in .agekey? Does the public key in the file's sops metadata match the private key in the cluster? age-keygen -y on your backup gives you the public key to compare.

Step 7: edit secrets without breaking them

Encrypting a new file only needs the public key. Changing an existing one needs the private key, because sops has to decrypt the values to recompute the MAC. For the people allowed to do that, sops reads age identities from the SOPS_AGE_KEY_FILE environment variable, or from sops/age/keys.txt in the user config directory (~/.config/sops/age/keys.txt on Linux).

export SOPS_AGE_KEY_FILE="$HOME/secure/production.agekey"
sops edit apps/production/secrets/db-credentials.yaml

sops edit decrypts into a temporary file, opens your editor and encrypts again when you save. Someone with only the public key who needs to change a value has to delete the file and write it again from scratch.

Step 8: stop plain Secrets in CI

The worst failure here is not a broken apply. It is somebody committing a Secret they forgot to encrypt, which Flux then applies without complaint. A short CI check catches it. sops filestatus prints JSON with an encrypted field, so the check doesn't depend on grepping for ENC[.

#!/usr/bin/env bash
set -euo pipefail

status=0
for f in $(find apps -path '*/secrets/*' -name '*.yaml'); do
  if [ "$(sops filestatus "$f" | jq -r .encrypted)" != "true" ]; then
    echo "not encrypted: $f"
    status=1
  fi
done
exit $status

The check needs no keys at all, so it is safe to run on pull requests from forks.

Step 9: rotate the age key

Rotate when someone who held the private key leaves, when a laptop goes missing, or on a schedule. The trick is to change the recipients in Git and the key in the cluster in an order that never leaves Flux unable to decrypt.

Start with a new key, and add it to the cluster Secret next to the old one. kustomize-controller accepts several .agekey entries in one Secret, so during the change it can decrypt files encrypted for either key.

age-keygen -o production-2.agekey

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=production.agekey \
  --from-file=age-2.agekey=production-2.agekey \
  --dry-run=client -o yaml | kubectl apply -f -

Replace the old public key with the new one in .sops.yaml, then run sops updatekeys on every encrypted file. It re-encrypts each file's data key for the recipients now listed in the config, and leaves the values alone. You need the old private key in SOPS_AGE_KEY_FILE for this, because sops must decrypt the data key first.

export SOPS_AGE_KEY_FILE="$HOME/secure/production.agekey"
for f in $(find apps/production -path '*/secrets/*' -name '*.yaml'); do
  sops updatekeys --yes "$f"
done

updatekeys only changes who can open the data key. Someone who copied the old key and an old version of a file can still read the values in that version, and Git history keeps every version. If a key leaked, also run sops rotate --in-place on each file, which generates a new data key and re-encrypts every value, and above all change the real passwords. Encryption can't take back a secret that was already exposed.

Commit, push, and wait until every decrypting Kustomization is ready again.

flux get kustomizations --all-namespaces

When everything reports ready, remove the old entry from the cluster Secret and delete the old key from your password manager.

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age-2.agekey=production-2.agekey \
  --dry-run=client -o yaml | kubectl apply -f -

One key Secret for many namespaces

If you run Kustomizations in many tenant namespaces, copying the key Secret into each one gets tedious. Since v1.7.0, kustomize-controller can take a global age key Secret through the --sops-age-secret flag. That Secret has to be in the controller's own namespace. Each Kustomization still needs decryption.provider: sops, but it can leave out secretRef, and when a Kustomization does set secretRef, that Secret wins over the global one. Turning it on is a patch to the kustomize-controller Deployment, which you add in the kustomization.yaml that bootstrap created under clusters/production/flux-system.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - target:
      kind: Deployment
      name: kustomize-controller
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --sops-age-secret=sops-age

Think before you do this on a shared cluster. A global key lets any tenant's Kustomization decrypt any file encrypted for that key, which undoes part of the isolation that per-namespace Secrets give you. For a single team it is a convenience. On a multi-tenant cluster, give each tenant its own key and its own Secret.

Test yourself

Ready to test your Flux CD knowledge? Try our Flux CD quiz!