From 9364915be0680f36790ca9348b868ce0cc21a8de Mon Sep 17 00:00:00 2001 From: MMD Date: Fri, 18 Sep 2026 22:30:16 +0330 Subject: [PATCH] docs: operations runbook, README verify section, frontend env example - docs/OPERATIONS.md: deploy flow + rollback, health endpoints, backup/ restore procedure, audit queries, monitoring stack routine - README: verification now points at readiness/metrics endpoints and the authenticated smoke tests (bare quotes curl is 401 by design) - frontend/.env.example: observability vars documented --- README.md | 12 +++++- docs/OPERATIONS.md | 96 +++++++++++++++++++++++++++++++++++++++++++ frontend/.env.example | 13 ++++-- 3 files changed, 116 insertions(+), 5 deletions(-) create mode 100644 docs/OPERATIONS.md diff --git a/README.md b/README.md index 77c54f3..09e9cf0 100644 --- a/README.md +++ b/README.md @@ -214,15 +214,23 @@ All should show `Up` (postgres also `(healthy)`). | Frontend UI | http://localhost | | API docs (Swagger) | http://localhost:8000/docs | | API health | http://localhost:8000/health → `{"status":"ok","service":"procurement-api"}` | +| API readiness (DB + MinIO + Casdoor) | http://localhost:8000/health/ready | +| Metrics | http://localhost:8000/metrics | | MinIO console | http://localhost:9001 (user/pass `minioadmin`/`minioadmin`) | | Database | `localhost:5433`, db `procurement`, user/pass `postgres`/`postgres` (Casdoor uses the same instance, db `casdoor`) | -Quick end-to-end check (Traefik → frontend proxy → API → DB): +Automated end-to-end checks (authenticates via `SMOKE_USER` / +`SMOKE_PASSWORD` — the seeder users): ```powershell -curl http://localhost/api/v1/quotes?expand=supplier,project,line_items,termsheets,events,decision,receipt +backend\tools\smoke_test.py # read paths +backend\tools\smoke_test_write.py / smoke_test_misc.py ``` +> Every API route now requires a Casdoor JWT — a bare +> `curl http://localhost/api/v1/quotes` returns **401** by design. +> Day-2 operations (deploys, backups, monitoring): [`docs/OPERATIONS.md`](docs/OPERATIONS.md). + --- ## MinIO (object storage for invoice / proforma files) diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md new file mode 100644 index 0000000..a07aae6 --- /dev/null +++ b/docs/OPERATIONS.md @@ -0,0 +1,96 @@ +# Operations runbook + +Day-2 operations for the ERP stack: deploys, health, backups, monitoring. +Secrets and rotation live in [`SECRET-MANAGEMENT.md`](SECRET-MANAGEMENT.md); +Casdoor/auth details in [`casdoor-auth.md`](casdoor-auth.md). + +## Deploy flow (push-to-deploy) + +1. Work lands on `production` only via merge from a feature branch. +2. Every push to `production` hits the webhook receiver (host port 9000), + which verifies the Gitea HMAC signature, rebuilds + `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build` + and then polls `/health/ready` (defaults to + `https://$API_HOST/health/ready` from `.env`). +3. A failed build or a failed readiness check is a **failed deploy**: + `journalctl -u erp-deploy-webhook -n 200` on the VPS. + Rollback: `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build` + after `git revert` + rebuild — or re-push the previous commit. + +## Health endpoints (unauthenticated, for probes) + +| Endpoint | Meaning | +|---|---| +| `GET /health` | liveness — process is up | +| `GET /health/ready` | readiness — Postgres + MinIO + Casdoor all reachable (503 + per-check booleans when degraded) | +| `GET /metrics` | Prometheus exposition (scraped by the monitoring stack) | + +## Backups + +The `db-backup` sidecar dumps BOTH databases (`procurement`, `casdoor`) +with `pg_dump -Fc` every `BACKUP_INTERVAL_SECONDS` (default daily) into +`BACKUP_DIR` (default `./backups` → set to `/srv/erp/backups` on the VPS), +keeping `BACKUP_KEEP_DAYS` (default 14) days. + +Restore (⚠️ replaces all data — stop `api` + `casdoor` first): + +```bash +cd /srv/erp && docker compose -f docker-compose.yml -f docker-compose.prod.yml stop api casdoor +docker cp backups/procurement-.dump procurement-postgres:/tmp/ +docker cp backups/casdoor-.dump procurement-postgres:/tmp/ +docker exec procurement-postgres pg_restore -U postgres -d procurement --clean --if-exists /tmp/procurement-.dump +docker exec procurement-postgres pg_restore -U postgres -d casdoor --clean --if-exists /tmp/casdoor-.dump +docker compose -f docker-compose.yml -f docker-compose.prod.yml start casdoor api +``` + +**Test the restore quarterly** on a scratch volume — an untested backup is +a hope, not a backup. Copy archives off-box periodically (the bind mount is +still on the same disk as the volumes). + +## Audit trail + +`audit_log` (in the `procurement` DB) records who changed what for +role assignments, user creation, and inventory transactions. Query recent +activity: + +```bash +docker exec procurement-postgres psql -U postgres -d procurement \ + -c "SELECT occurred_at, actor, action, entity, entity_id FROM audit_log ORDER BY occurred_at DESC LIMIT 50;" +``` + +`quote_events` continues to hold the quote lifecycle history (its `actor` +is now the verified user's display name, not a raw token). + +## Monitoring stack + +Start/update (separate compose project — app rebuilds don't touch it): + +```bash +cd /srv/erp +docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d +``` + +- **Grafana** — `127.0.0.1:3001` on the VPS (SSH tunnel), admin password in + `.env`; the «ERP — Stack overview» dashboard auto-provisions. +- **GlitchTip** — public on `GLITCHTIP_HOST` (browser DSN) + loopback + `:8001` for administration. Create backend/frontend projects there and put + the DSNs in `.env` (`SENTRY_DSN_BACKEND` / `SENTRY_DSN_FRONTEND`). +- **OpenPanel** — see `monitoring/README.md` for the self-hosting setup and + the event list the app emits. +- **Alerts** — Prometheus rules surface in Grafana alerting (API down, + 5xx rate, p95 latency, disk, RAM, DB connections). Attach a notification + channel in the Grafana UI. + +Logs: every container's stdout ships to Loki via Promtail — in Grafana: +`{compose_project="erp", compose_service="api"}`; JSON lines carry +`request_id`, `http_status`, `duration_ms`. + +## Routine checks + +```bash +docker compose -f docker-compose.yml -f docker-compose.prod.yml ps # all Up/healthy? +docker compose logs --since 10m api | grep -i error +ls -lh /srv/erp/backups | tail # dumps fresh? +curl -fsS https://$API_HOST/health/ready | jq # all checks true? +df -h / # ≥20% free (builds need headroom) +``` diff --git a/frontend/.env.example b/frontend/.env.example index cd17e26..88b7bde 100644 --- a/frontend/.env.example +++ b/frontend/.env.example @@ -5,6 +5,13 @@ VITE_API_BASE_URL= # Show the "Developer Mode" panel on the login page. -# true (default) / false — in Docker builds this is controlled by the -# DEV_LOGIN_ENABLED build arg (see docker-compose.yml), not this file. -VITE_ENABLE_DEV_LOGIN=true +# true (default in local dev) / false — in Docker builds this is controlled +# by the DEV_LOGIN_ENABLED build arg (see docker-compose.yml), not this file. +VITE_ENABLE_DEV_LOGIN= + +# ---- Observability (all optional — empty = disabled, no network calls) ---- +# OpenPanel: browser origin of the self-hosted dashboard/API + project client id. +VITE_OPENPANEL_URL= +VITE_OPENPANEL_CLIENT_ID= +# GlitchTip/Sentry DSN for frontend exception capture. +VITE_SENTRY_DSN=