ERP/docs/OPERATIONS.md

Operations runbook

Day-2 operations for the ERP stack: deploys, health, backups, monitoring. Secrets and rotation live in SECRET-MANAGEMENT.md; Casdoor/auth details in casdoor-auth.md.

Deploy flow (push-to-deploy)

1. Work lands on production only via merge from a feature branch.
2. Every push to production hits the webhook receiver (host port 9000), which verifies the Gitea HMAC signature, rebuilds docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build and then polls /health/ready (defaults to https://$API_HOST/health/ready from .env).
3. A failed build or a failed readiness check is a failed deploy: journalctl -u erp-deploy-webhook -n 200 on the VPS. Rollback: docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build after git revert + rebuild — or re-push the previous commit.

Health endpoints (unauthenticated, for probes)

Endpoint	Meaning
GET /health	liveness — process is up
GET /health/ready	readiness — Postgres + MinIO + Casdoor all reachable (503 + per-check booleans when degraded)
GET /metrics	Prometheus exposition (scraped by the monitoring stack)

Backups

The db-backup sidecar dumps BOTH databases (procurement, casdoor) with pg_dump -Fc every BACKUP_INTERVAL_SECONDS (default daily) into BACKUP_DIR (default ./backups → set to /srv/erp/backups on the VPS), keeping BACKUP_KEEP_DAYS (default 14) days.

Restore (⚠️ replaces all data — stop api + casdoor first):

cd /srv/erp && docker compose -f docker-compose.yml -f docker-compose.prod.yml stop api casdoor
docker cp backups/procurement-<stamp>.dump procurement-postgres:/tmp/
docker cp backups/casdoor-<stamp>.dump procurement-postgres:/tmp/
docker exec procurement-postgres pg_restore -U postgres -d procurement --clean --if-exists /tmp/procurement-<stamp>.dump
docker exec procurement-postgres pg_restore -U postgres -d casdoor --clean --if-exists /tmp/casdoor-<stamp>.dump
docker compose -f docker-compose.yml -f docker-compose.prod.yml start casdoor api

Test the restore quarterly on a scratch volume — an untested backup is a hope, not a backup. Copy archives off-box periodically (the bind mount is still on the same disk as the volumes).

Audit trail

audit_log (in the procurement DB) records who changed what for role assignments, user creation, and inventory transactions. Query recent activity:

docker exec procurement-postgres psql -U postgres -d procurement \
  -c "SELECT occurred_at, actor, action, entity, entity_id FROM audit_log ORDER BY occurred_at DESC LIMIT 50;"

quote_events continues to hold the quote lifecycle history (its actor is now the verified user's display name, not a raw token).

Monitoring stack

Start/update (separate compose project — app rebuilds don't touch it):

cd /srv/erp
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d

• Grafana — 127.0.0.1:3001 on the VPS (SSH tunnel), admin password in .env; the «ERP — Stack overview» dashboard auto-provisions.
• GlitchTip — public on GLITCHTIP_HOST (browser DSN) + loopback :8001 for administration. Create backend/frontend projects there and put the DSNs in .env (SENTRY_DSN_BACKEND / SENTRY_DSN_FRONTEND).
• OpenPanel — see monitoring/README.md for the self-hosting setup and the event list the app emits.
• Alerts — Prometheus rules surface in Grafana alerting (API down, 5xx rate, p95 latency, disk, RAM, DB connections). Attach a notification channel in the Grafana UI.

Logs: every container's stdout ships to Loki via Promtail — in Grafana: {compose_project="erp", compose_service="api"}; JSON lines carry request_id, http_status, duration_ms.

Routine checks

docker compose -f docker-compose.yml -f docker-compose.prod.yml ps   # all Up/healthy?
docker compose logs --since 10m api | grep -i error
ls -lh /srv/erp/backups | tail                                       # dumps fresh?
curl -fsS https://$API_HOST/health/ready | jq                        # all checks true?
df -h /                                                              # ≥20% free (builds need headroom)