Deployment order
Use the same sequence for Vercel, Docker, ECS, and Kubernetes:
- Run CI with the target commit: format, lint, unit/integration tests, build, and Chromium E2E.
- Build one immutable artifact and record its commit SHA. Promote that artifact; do not rebuild separately for production.
- Take or confirm the database backup required by the provider's retention policy.
- Apply database migrations with
pnpm db:migrate. - Deploy the application artifact.
- Verify
GET /api/health, logs, error tracking, and the enterprise E2E evidence. - Monitor the first migration and traffic window before removing the previous artifact.
Database migrations must be forward-compatible with the application currently serving traffic. For breaking schema changes, use expand/contract: add the new shape, deploy code that can read both shapes, backfill, then remove the old shape in a later release.
Database backup matrix
Codapult does not copy production database credentials or data into the application container. Backups belong to the database provider and must be encrypted, retained, and access-controlled there.
| Provider | Backup procedure | Restore target |
|---|---|---|
| Turso/libSQL | Enable and verify the provider's scheduled backup or point-in-time retention for the database. Record the database name, group, and retention policy with the release. | Restore to a separate database/branch first, validate it, then switch TURSO_DATABASE_URL and TURSO_AUTH_TOKEN. |
| PostgreSQL | Create an encrypted custom-format dump with pg_dump --format=custom --file=codapult-$(date +%Y%m%d-%H%M%S).dump "$DATABASE_URL" and retain provider snapshots/PITR as the primary recovery path. | Restore into an empty or isolated database with pg_restore --clean --if-exists --dbname="$DATABASE_URL" backup.dump after confirming the target explicitly. |
| Local SQLite | Stop the application before copying local.db; this is a development convenience, not a production backup strategy. | Copy the database to a new local path, set TURSO_DATABASE_URL=file:..., then run migrations. |
Never place dumps, database URLs, auth tokens, or .env.local in the repository, Docker image, CI artifacts, or issue tracker.
Restore procedure
-
Declare the incident owner and freeze destructive maintenance jobs.
-
Identify the recovery point and create an isolated restore target. Do not overwrite the primary database for the first validation.
-
Restore the provider snapshot or dump into that target.
-
Set the target application's database variables and run
pnpm db:migrate. -
Check schema parity and application health:
npx @codapult/cli db schema-diff curl -fsS "$APP_URL/api/health" pnpm exec playwright test --project=chromium e2e/security-boundaries.spec.ts -
Verify authentication, organization membership, a representative read-only dashboard flow, and payment/webhook records.
-
Switch the deployment to the validated target using the hosting provider's secret/configuration mechanism.
-
Re-run health checks and monitor error rate, database latency, background jobs, and webhook delivery.
-
Preserve the original database and restore evidence until the incident is closed.
For PostgreSQL, run restore commands only against an explicitly confirmed target. pg_restore --clean is destructive for objects in that target.
Application rollback
Application rollback and database rollback are separate decisions:
- If the schema is backward-compatible, redeploy the previous immutable image and keep the migrated database.
- If a migration changed data or removed a column, do not blindly roll back the application. Ship a forward fix or restore an isolated database after confirming the recovery point.
- Keep migration files append-only. Do not edit an applied migration to repair production state.
- After rollback or forward fix, repeat
/api/health, security-boundary E2E, and the relevant domain integration tests.
Object storage recovery
When STORAGE_PROVIDER is s3 or r2, enable bucket versioning and a retention/lifecycle policy in the provider account. The application database stores object references; a database restore without the corresponding bucket version is incomplete. Record the bucket, region/endpoint, retention policy, and restore owner with the release.
Release evidence
For each production deployment retain:
- commit SHA and image digest;
- migration command output and database provider;
- backup/recovery-point identifier;
- health-check and E2E result;
- deployment URL, operator, and rollback/forward-fix decision.
The runbook is complete only when the restored environment has been tested and the recovery path is recorded. A configured backup without a verified restore is not recovery evidence.