Runbook — Storage Expansion

Procedures for expanding storage on Helsinki2 containers and VMs when running low on disk space.

Monitoring Storage

Check current usage:

# Proxmox host
df -h /

# All containers
for ct in $(pct list | awk 'NR>1{print $1}'); do
  echo "CT$ct: $(pct config $ct | grep rootfs)"
done

# VM700
ssh [email protected] "df -h /"

# Docker volumes
ssh [email protected] "docker system df -v"

Container Rootfs Expansion

When a container is running low on space:

# Check current size
pct config <CTID> | grep rootfs

# Expand by 5GB
pct resize <CTID> rootfs +5G

# Verify
pct config <CTID> | grep rootfs

Note: Rootfs can only be expanded, not shrunk.

VM700 Docker Volume Cleanup

Before expanding, try cleanup:

ssh [email protected] << 'EOF'
# Remove stopped containers
docker container prune -f

# Remove unused images
docker image prune -f

# Check usage
docker system df -v
EOF

Never run: docker system prune -a --volumes — this destroys persistent data.

VM700 Disk Expansion

If /var/lib/docker is full:

# Check usage
ssh [email protected] "df -h /var/lib/docker"

# Identify large volumes
ssh [email protected] "du -h /var/lib/docker/volumes | sort -h | tail -20"

# Clean up old logs
ssh [email protected] << 'EOF'
find /var/lib/docker/containers -name "*.log" -mtime +30 -delete
EOF

If still full, expand VM700 disk via Proxmox:

# Check current size
qm config 700 | grep scsi0

# Expand disk (Proxmox GUI or CLI)
# Then on VM700:
ssh [email protected] << 'EOF'
# Resize partition
sudo growpart /dev/sda 1

# Resize filesystem
sudo resize2fs /dev/sda1

# Verify
df -h /
EOF

CT320 Postgres Storage

If CT320 (Postgres) is running low:

# Check size
pct config 320 | grep rootfs

# Expand
pct resize 320 rootfs +10G

# Verify inside container
pct exec 320 -- df -h /

Backup Before Expansion

Always backup before expanding:

# Container snapshot
pct snapshot <CTID> pre-expansion-$(date +%Y%m%d)

# VM snapshot
qm snapshot 700 pre-expansion-$(date +%Y%m%d)

# Database backup
ssh [email protected] << 'EOF'
docker exec echohuman-twenty-db-1 pg_dump -U twenty -d default > /tmp/twenty_backup_$(date +%Y%m%d).sql
EOF

Monitoring Headroom

Establish minimum headroom policy:

• Containers: keep 20% free space
• VMs: keep 15% free space
• Docker volumes: keep 10% free space

Alert when usage exceeds:

• Container: 80%
• VM: 85%
• Docker: 90%

Long-Term Solutions

If storage is constantly full:

1. Archive old data: Move logs, backups, archives to external storage
2. Increase host storage: Add new disk to Proxmox host
3. Optimize containers: Reduce log retention, clean caches
4. Cleanup Docker: Remove old images, prune unused volumes
5. Review quotas: Set per-container limits to prevent runaway growth

Troubleshooting

Issue	Cause	Fix
Expansion fails	Container running	Stop container first: pct stop <CTID>
Filesystem not resized	Partition not grown	Use growpart or parted on VM
Docker still full	Volumes not cleaned	Check docker system df -v for large volumes
Postgres slow	Disk full	Expand CT320 and restart Postgres

Notes

• Expansion is non-destructive — safe to perform
• Always backup before expanding
• Monitor storage regularly to avoid emergencies
• Document all expansions in infrastructure logs