# Operations runbook

Day-2 operations for the ERP stack: deploys, health, backups, monitoring.
Secrets and rotation live in [`SECRET-MANAGEMENT.md`](SECRET-MANAGEMENT.md);
Casdoor/auth details in [`casdoor-auth.md`](casdoor-auth.md).

## Deploy flow (push-to-deploy)

1. Work lands on `production` only via merge from a feature branch.
2. Every push to `production` hits the webhook receiver (host port 9000),
   which verifies the Gitea HMAC signature, rebuilds
   `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build`
   and then polls `/health/ready` (defaults to
   `https://$API_HOST/health/ready` from `.env`).
3. A failed build or a failed readiness check is a **failed deploy**:
   `journalctl -u erp-deploy-webhook -n 200` on the VPS.
   Rollback: `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build`
   after `git revert` + rebuild — or re-push the previous commit.

## Health endpoints (unauthenticated, for probes)

| Endpoint | Meaning |
|---|---|
| `GET /health` | liveness — process is up |
| `GET /health/ready` | readiness — Postgres + MinIO + Casdoor all reachable (503 + per-check booleans when degraded) |
| `GET /metrics` | Prometheus exposition (scraped by the monitoring stack) |

## Backups

The `db-backup` sidecar dumps BOTH databases (`procurement`, `casdoor`)
with `pg_dump -Fc` every `BACKUP_INTERVAL_SECONDS` (default daily) into
`BACKUP_DIR` (default `./backups` → set to `/srv/erp/backups` on the VPS),
keeping `BACKUP_KEEP_DAYS` (default 14) days.

Restore (⚠️ replaces all data — stop `api` + `casdoor` first):

```bash
cd /srv/erp && docker compose -f docker-compose.yml -f docker-compose.prod.yml stop api casdoor
docker cp backups/procurement-<stamp>.dump procurement-postgres:/tmp/
docker cp backups/casdoor-<stamp>.dump procurement-postgres:/tmp/
docker exec procurement-postgres pg_restore -U postgres -d procurement --clean --if-exists /tmp/procurement-<stamp>.dump
docker exec procurement-postgres pg_restore -U postgres -d casdoor --clean --if-exists /tmp/casdoor-<stamp>.dump
docker compose -f docker-compose.yml -f docker-compose.prod.yml start casdoor api
```

**Test the restore quarterly** on a scratch volume — an untested backup is
a hope, not a backup. Copy archives off-box periodically (the bind mount is
still on the same disk as the volumes).

## Audit trail

`audit_log` (in the `procurement` DB) records who changed what for
role assignments, user creation, and inventory transactions. Query recent
activity:

```bash
docker exec procurement-postgres psql -U postgres -d procurement \
  -c "SELECT occurred_at, actor, action, entity, entity_id FROM audit_log ORDER BY occurred_at DESC LIMIT 50;"
```

`quote_events` continues to hold the quote lifecycle history (its `actor`
is now the verified user's display name, not a raw token).

## Monitoring stack

Start/update (separate compose project — app rebuilds don't touch it):

```bash
cd /srv/erp
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
```

- **Grafana** — `127.0.0.1:3001` on the VPS (SSH tunnel), admin password in
  `.env`; the «ERP — Stack overview» dashboard auto-provisions.
- **GlitchTip** — public on `GLITCHTIP_HOST` (browser DSN) + loopback
  `:8001` for administration. Create backend/frontend projects there and put
  the DSNs in `.env` (`SENTRY_DSN_BACKEND` / `SENTRY_DSN_FRONTEND`).
- **OpenPanel** — see `monitoring/README.md` for the self-hosting setup and
  the event list the app emits.
- **Alerts** — Prometheus rules surface in Grafana alerting (API down,
  5xx rate, p95 latency, disk, RAM, DB connections). Attach a notification
  channel in the Grafana UI.

Logs: every container's stdout ships to Loki via Promtail — in Grafana:
`{compose_project="erp", compose_service="api"}`; JSON lines carry
`request_id`, `http_status`, `duration_ms`.

## Routine checks

```bash
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps   # all Up/healthy?
docker compose logs --since 10m api | grep -i error
ls -lh /srv/erp/backups | tail                                       # dumps fresh?
curl -fsS https://$API_HOST/health/ready | jq                        # all checks true?
df -h /                                                              # ≥20% free (builds need headroom)
```