​

Runbook: Database connection lost

​

Diagnose

curl -s https://your-regen-host/ready | jq .

# Docker Compose
docker logs fluidify-regen --since 10m 2>&1 | grep -i "database\|postgres\|dial\|connect\|sql"

# Kubernetes
kubectl logs -n fluidify deploy/fluidify-regen --since=10m | grep -i "database\|postgres\|dial\|connect"

• dial tcp: connection refused — PostgreSQL is not running or not reachable on the configured host/port
• password authentication failed — wrong credentials in DATABASE_URL
• too many connections — connection pool exhausted; PostgreSQL max_connections reached
• SSL connection required — PostgreSQL requires TLS but DATABASE_URL has sslmode=disable

# Docker Compose
docker ps | grep regen-db
docker exec fluidify-regen-db pg_isready -U regen

# Kubernetes (if self-hosted)
kubectl get pods -n fluidify | grep postgres
kubectl exec -n fluidify deploy/postgres -- pg_isready -U regen

docker exec fluidify-regen-db psql -U regen -c \
  "SELECT count(*) FROM pg_stat_activity WHERE datname = 'regen';"

​

Mitigate

# Docker Compose
docker compose restart db

# Kubernetes
kubectl rollout restart statefulset/postgres -n fluidify

1. Update DATABASE_URL in .env or the Kubernetes secret
2. Restart the app

make start  # or kubectl rollout restart

• If PostgreSQL requires SSL: remove sslmode=disable from DATABASE_URL or set sslmode=require
• If PostgreSQL doesn’t have SSL: add ?sslmode=disable to DATABASE_URL

​

Resolve

1. Confirm GET /ready returns "database":"ok"
2. Spot-check the UI — incidents list should load
3. Check logs for any delayed background job errors (background workers retry on reconnect automatically)

​

Prevention

• Run PostgreSQL with a persistent volume — data survives container restarts
• Set DB_MAX_OPEN_CONNS to no more than 20% of PostgreSQL’s max_connections
• Monitor pg_stat_activity connection count via Prometheus + postgres_exporter
• Configure /ready as the health check endpoint (not /health) — it verifies the database connection on every call