About this scenario
During a cleanup someone deleted the db StatefulSet and its claim, data-db-0. The StatefulSet was applied again an hour later and Postgres started, with an empty database: kubectl logs db-0 shows it creating a new cluster.
The StorageClass, gp3-retain, has reclaimPolicy: Retain. The old volume, labelled team=orders, still exists with every order on it. Kubernetes won't hand it to a new claim by itself.
Put db-0 back on the original volume. The empty replacement volume can go.
Objectives
- Find the volume that was left behind
- Make the original volume claimable again
- data-db-0 is bound to the original volume
- db-0 runs on it and Postgres finds its database
What you'll practice
- With
reclaimPolicy: Retain, deleting a claim leaves its PersistentVolume as Released with the data intact. With Delete, the default for most dynamic classes, the disk is gone. - A Released volume is not reused, because
spec.claimRef still names the old claim by UID. Removing the UID makes it Available and reserved for a claim of that name; removing the whole claimRef makes it free for any claim that fits. - A claim binds for life. To move a workload to another volume you delete the claim and create one that binds to the volume you want.
- A StatefulSet recreates missing claims from its template, and deleting the StatefulSet never deletes them. The claim is what you must protect.
- Postgres initializes an empty data directory without complaint. A database that starts cleanly on the wrong volume looks healthy.
How the playground works
The cluster runs entirely in your browser: a simulated API server, scheduler, kubelets and controllers, with virtual nodes in three zones. kubectl output follows the real formats, errors use the real messages, and the add-ons (CSI storage, Traefik, Envoy Gateway, cert-manager, Helm, Flux, Argo CD, Kyverno, Cilium, Istio) behave like their real counterparts for the common paths. It's a simulation, so some behavior is simplified. Your progress is saved in this browser.