Skip to main content

Runbook — Twenty CRM Recovery

Recovery procedures for EchoHuman Twenty CRM when it becomes unresponsive or fails to start.

Symptoms​

  • CRM not accessible at https://crm.echohuman.org
  • Container stuck in unhealthy state
  • node dist/main process not binding port 3000
  • Database migration errors
  • Empty core schema in Postgres

Quick Checks​

# Check container status
docker ps --filter name=echohuman-twenty

# Check logs
docker logs echohuman-twenty-server-1 -f --tail=100

# Check port binding
docker exec echohuman-twenty-server-1 ss -ltnp | grep 3000

# Check database
docker exec echohuman-twenty-db-1 psql -U twenty -d default \
-c "SELECT count(*) FROM information_schema.tables WHERE table_schema='core';"

Recovery Procedure 1: Restart Container​

cd /opt/stacks/echohuman-twenty
docker compose restart echohuman-twenty-server-1
docker logs echohuman-twenty-server-1 -f --tail=50

Wait 5–10 minutes for startup. If still unhealthy, proceed to Procedure 2.

Recovery Procedure 2: Rebuild Database​

If database schema is corrupted (0 tables in core schema):

# Stop the stack
cd /opt/stacks/echohuman-twenty
docker compose down

# Drop corrupted schema
docker run --rm \
--network echohuman-twenty_default \
postgres:16 \
psql -h echohuman-twenty-db-1 -U twenty -d default \
-c "DROP SCHEMA core CASCADE;"

# Restart stack
docker compose up -d

# Watch startup
docker logs echohuman-twenty-server-1 -f --tail=100

Expected output after ~10 min:

Successfully registered all background sync jobs!

Then check port:

docker exec echohuman-twenty-server-1 ss -ltnp | grep 3000

Should show:

tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN

Recovery Procedure 3: Full Reset​

If Procedure 2 fails, reset the entire stack:

cd /opt/stacks/echohuman-twenty

# Backup current state (optional)
docker exec echohuman-twenty-db-1 pg_dumpall -U postgres > /tmp/twenty_backup_$(date +%Y%m%d_%H%M%S).sql

# Stop and remove all containers and volumes
docker compose down -v

# Restart from scratch
docker compose up -d

# Watch startup
docker logs echohuman-twenty-server-1 -f --tail=100

Warning: This destroys all data. Only use if Procedure 2 fails.

Verification​

After recovery, verify:

# 1. Container is healthy
docker ps --filter name=echohuman-twenty

# 2. Port is bound
docker exec echohuman-twenty-server-1 ss -ltnp | grep 3000

# 3. Database has schema
docker exec echohuman-twenty-db-1 psql -U twenty -d default \
-c "SELECT count(*) FROM information_schema.tables WHERE table_schema='core';"

# 4. Public URL responds
curl -I https://crm.echohuman.org

Known Issues​

IssueCauseFix
APP_SECRET emptyShell env overrides .envUse env_file: in compose, not ${VAR}
0 tables in core schemaPartial migrationDrop schema and restart (Procedure 2)
Port 3000 not boundStartup failureCheck logs for specific error
Database connection refusedDB container not readyWait longer, check DB logs

Escalation​

If all procedures fail:

  1. Check /opt/stacks/echohuman-twenty/.env for correct credentials
  2. Verify network: docker network ls
  3. Check disk space: df -h /var/lib/docker
  4. Review full logs: docker logs echohuman-twenty-server-1 | tail -200
  5. Contact infrastructure team with logs

Prevention​

  • Monitor container health regularly
  • Keep backups of Postgres database
  • Document any custom configuration changes
  • Test recovery procedures quarterly