Backup and restore

What needs backing up

Component	Back up	Why
PostgreSQL	Yes	All durable state
APP_KEK	Yes	Without it the backup is unreadable
INDEX_PEPPER	Yes	Without it blind indexes stop matching
AUTH_TOKEN_PEPPER	Yes	Without it stored token hashes are useless
Object storage	Optional	Artifacts regenerate from the database
Cluster state	No	Reinstall the chart

The second row is the one people miss.

warning

A database backup without APP_KEK is not a backup. The sensitive columns in it are encrypted with a key that lives in a Kubernetes Secret, and a namespace deletion takes that key with it.

Store the crypto material in a cloud secret manager or password manager, in a different failure domain from the cluster, before you need it.

Database backups

The chart does not manage them. Use your provider's, and set retention to whatever your policy says.

gcloud sql backups create --instance presponsieve-pg
aws rds create-db-snapshot --db-instance-identifier presponsieve-pg \
  --db-snapshot-identifier presponsieve-$(date +%Y%m%d)
az postgres flexible-server backup create \
  --name presponsieve-pg --resource-group presponsieve-rg

Take one before every upgrade.

Logical backups

Provider snapshots are primary. A pg_dump is useful for moving between providers:

kubectl run pgdump --rm -it --image=postgres:16 -n presponsieve -- \
  pg_dump "$DATABASE_URL" -Fc > backup.dump

The dump contains ciphertext for encrypted columns. Restoring it somewhere without the KEK gives you a schema and unreadable payloads — which is either a problem or a feature, depending on why you took it.

Restoring

# 1. Scale to zero so nothing writes during the restore
kubectl scale deploy/presponsieve --replicas=0 -n presponsieve

# 2. Restore through your provider

# 3. Point the app at the restored instance if the endpoint changed,
#    then scale back up
kubectl scale deploy/presponsieve --replicas=2 -n presponsieve

Verify the restore

Do not stop at "pods are running." Run one analysis end to end and open an existing report from before the restore.

The second part is what actually tests decryption. A restore that succeeds structurally but cannot decrypt looks healthy until someone opens an old profile.

Rehearse it

A backup you have never restored is a hypothesis.

Restore into a non-production environment periodically, and write down two numbers: how long it takes, and how much data it loses. Neither can be derived from documentation.

Disaster recovery

Rebuilding the deployment is helm upgrade --install against a new cluster. Rebuilding the data is a snapshot restore.

The binding constraint is key material. If APP_KEK exists only in the cluster you lost, there is no recovery — so replicate it, or move to KMS_BACKEND with a multi-region key and remove the problem.