Runbook: Migration failed
Symptom: The app refuses to start. Logs show a database migration error. The container exits immediately or enters a crash loop.Diagnose
1. Check the logs for the error:migration failed: pq: column "X" of relation "Y" already exists— migration was partially applied (crashed midway)migration failed: pq: relation "X" does not exist— earlier migration was skipped or rolled backmigration failed: dirty database version N; fix and force version— the migration table is marked dirty from a previous crashpermission denied for schema public— database user lacks DDL privileges
dirty = true row means the migration at that version was interrupted.
Fix: dirty migration state
This is the most common failure. The migration tool marks the version as dirty when a migration crashes partway through. Step 1 — Take a database backup first:.up.sql file for that version to understand what state the database may be in.
Step 3 — Force the migration version:
If the migration was partially applied and the schema is now in an unknown state, you may need to manually clean up the partial change and then force the version:
<version-1> is the version number before the failed migration (i.e., the last known-good migration).
Then restart the app normally — it will re-apply the failed migration from scratch.
Step 4 — If the partial migration left schema artifacts:
Connect directly to PostgreSQL and manually clean up. For example, if a migration to add a column failed halfway:
migrate force and restart.
Fix: permission denied
The database user needs DDL privileges to run migrations. Grant them:Fix: migration already applied (column exists)
If a migration tries to add a column that already exists, it’s because a previous partial run added it before crashing. The fix is the same as the dirty state fix above — force the version back and let it retry cleanly.Resolve
- Confirm the app starts successfully and logs show
running database migrations... doneorno migrations to run - Check
GET /readyreturns"database":"ok" - Do a quick smoke test: open the UI, check the incidents list
Prevention
- Always take a database backup before deploying a new version (
pg_dumpbeforedocker pull/helm upgrade) - Test migrations on a staging database with a copy of production data before deploying to production
- Regen migrations are written to be idempotent where possible — but complex schema changes (type changes, constraint additions) are not always safely re-runnable
- In Kubernetes, the Helm chart runs migrations as a Job that must complete before the Deployment starts — this ensures the app never runs against an un-migrated schema
