docs: operations runbook, README verify section, frontend env example
- docs/OPERATIONS.md: deploy flow + rollback, health endpoints, backup/ restore procedure, audit queries, monitoring stack routine - README: verification now points at readiness/metrics endpoints and the authenticated smoke tests (bare quotes curl is 401 by design) - frontend/.env.example: observability vars documented
This commit is contained in:
@@ -214,15 +214,23 @@ All should show `Up` (postgres also `(healthy)`).
|
|||||||
| Frontend UI | http://localhost |
|
| Frontend UI | http://localhost |
|
||||||
| API docs (Swagger) | http://localhost:8000/docs |
|
| API docs (Swagger) | http://localhost:8000/docs |
|
||||||
| API health | http://localhost:8000/health → `{"status":"ok","service":"procurement-api"}` |
|
| API health | http://localhost:8000/health → `{"status":"ok","service":"procurement-api"}` |
|
||||||
|
| API readiness (DB + MinIO + Casdoor) | http://localhost:8000/health/ready |
|
||||||
|
| Metrics | http://localhost:8000/metrics |
|
||||||
| MinIO console | http://localhost:9001 (user/pass `minioadmin`/`minioadmin`) |
|
| MinIO console | http://localhost:9001 (user/pass `minioadmin`/`minioadmin`) |
|
||||||
| Database | `localhost:5433`, db `procurement`, user/pass `postgres`/`postgres` (Casdoor uses the same instance, db `casdoor`) |
|
| Database | `localhost:5433`, db `procurement`, user/pass `postgres`/`postgres` (Casdoor uses the same instance, db `casdoor`) |
|
||||||
|
|
||||||
Quick end-to-end check (Traefik → frontend proxy → API → DB):
|
Automated end-to-end checks (authenticates via `SMOKE_USER` /
|
||||||
|
`SMOKE_PASSWORD` — the seeder users):
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
curl http://localhost/api/v1/quotes?expand=supplier,project,line_items,termsheets,events,decision,receipt
|
backend\tools\smoke_test.py # read paths
|
||||||
|
backend\tools\smoke_test_write.py / smoke_test_misc.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> Every API route now requires a Casdoor JWT — a bare
|
||||||
|
> `curl http://localhost/api/v1/quotes` returns **401** by design.
|
||||||
|
> Day-2 operations (deploys, backups, monitoring): [`docs/OPERATIONS.md`](docs/OPERATIONS.md).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## MinIO (object storage for invoice / proforma files)
|
## MinIO (object storage for invoice / proforma files)
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Operations runbook
|
||||||
|
|
||||||
|
Day-2 operations for the ERP stack: deploys, health, backups, monitoring.
|
||||||
|
Secrets and rotation live in [`SECRET-MANAGEMENT.md`](SECRET-MANAGEMENT.md);
|
||||||
|
Casdoor/auth details in [`casdoor-auth.md`](casdoor-auth.md).
|
||||||
|
|
||||||
|
## Deploy flow (push-to-deploy)
|
||||||
|
|
||||||
|
1. Work lands on `production` only via merge from a feature branch.
|
||||||
|
2. Every push to `production` hits the webhook receiver (host port 9000),
|
||||||
|
which verifies the Gitea HMAC signature, rebuilds
|
||||||
|
`docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build`
|
||||||
|
and then polls `/health/ready` (defaults to
|
||||||
|
`https://$API_HOST/health/ready` from `.env`).
|
||||||
|
3. A failed build or a failed readiness check is a **failed deploy**:
|
||||||
|
`journalctl -u erp-deploy-webhook -n 200` on the VPS.
|
||||||
|
Rollback: `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-build`
|
||||||
|
after `git revert` + rebuild — or re-push the previous commit.
|
||||||
|
|
||||||
|
## Health endpoints (unauthenticated, for probes)
|
||||||
|
|
||||||
|
| Endpoint | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `GET /health` | liveness — process is up |
|
||||||
|
| `GET /health/ready` | readiness — Postgres + MinIO + Casdoor all reachable (503 + per-check booleans when degraded) |
|
||||||
|
| `GET /metrics` | Prometheus exposition (scraped by the monitoring stack) |
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
The `db-backup` sidecar dumps BOTH databases (`procurement`, `casdoor`)
|
||||||
|
with `pg_dump -Fc` every `BACKUP_INTERVAL_SECONDS` (default daily) into
|
||||||
|
`BACKUP_DIR` (default `./backups` → set to `/srv/erp/backups` on the VPS),
|
||||||
|
keeping `BACKUP_KEEP_DAYS` (default 14) days.
|
||||||
|
|
||||||
|
Restore (⚠️ replaces all data — stop `api` + `casdoor` first):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /srv/erp && docker compose -f docker-compose.yml -f docker-compose.prod.yml stop api casdoor
|
||||||
|
docker cp backups/procurement-<stamp>.dump procurement-postgres:/tmp/
|
||||||
|
docker cp backups/casdoor-<stamp>.dump procurement-postgres:/tmp/
|
||||||
|
docker exec procurement-postgres pg_restore -U postgres -d procurement --clean --if-exists /tmp/procurement-<stamp>.dump
|
||||||
|
docker exec procurement-postgres pg_restore -U postgres -d casdoor --clean --if-exists /tmp/casdoor-<stamp>.dump
|
||||||
|
docker compose -f docker-compose.yml -f docker-compose.prod.yml start casdoor api
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test the restore quarterly** on a scratch volume — an untested backup is
|
||||||
|
a hope, not a backup. Copy archives off-box periodically (the bind mount is
|
||||||
|
still on the same disk as the volumes).
|
||||||
|
|
||||||
|
## Audit trail
|
||||||
|
|
||||||
|
`audit_log` (in the `procurement` DB) records who changed what for
|
||||||
|
role assignments, user creation, and inventory transactions. Query recent
|
||||||
|
activity:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec procurement-postgres psql -U postgres -d procurement \
|
||||||
|
-c "SELECT occurred_at, actor, action, entity, entity_id FROM audit_log ORDER BY occurred_at DESC LIMIT 50;"
|
||||||
|
```
|
||||||
|
|
||||||
|
`quote_events` continues to hold the quote lifecycle history (its `actor`
|
||||||
|
is now the verified user's display name, not a raw token).
|
||||||
|
|
||||||
|
## Monitoring stack
|
||||||
|
|
||||||
|
Start/update (separate compose project — app rebuilds don't touch it):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /srv/erp
|
||||||
|
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Grafana** — `127.0.0.1:3001` on the VPS (SSH tunnel), admin password in
|
||||||
|
`.env`; the «ERP — Stack overview» dashboard auto-provisions.
|
||||||
|
- **GlitchTip** — public on `GLITCHTIP_HOST` (browser DSN) + loopback
|
||||||
|
`:8001` for administration. Create backend/frontend projects there and put
|
||||||
|
the DSNs in `.env` (`SENTRY_DSN_BACKEND` / `SENTRY_DSN_FRONTEND`).
|
||||||
|
- **OpenPanel** — see `monitoring/README.md` for the self-hosting setup and
|
||||||
|
the event list the app emits.
|
||||||
|
- **Alerts** — Prometheus rules surface in Grafana alerting (API down,
|
||||||
|
5xx rate, p95 latency, disk, RAM, DB connections). Attach a notification
|
||||||
|
channel in the Grafana UI.
|
||||||
|
|
||||||
|
Logs: every container's stdout ships to Loki via Promtail — in Grafana:
|
||||||
|
`{compose_project="erp", compose_service="api"}`; JSON lines carry
|
||||||
|
`request_id`, `http_status`, `duration_ms`.
|
||||||
|
|
||||||
|
## Routine checks
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps # all Up/healthy?
|
||||||
|
docker compose logs --since 10m api | grep -i error
|
||||||
|
ls -lh /srv/erp/backups | tail # dumps fresh?
|
||||||
|
curl -fsS https://$API_HOST/health/ready | jq # all checks true?
|
||||||
|
df -h / # ≥20% free (builds need headroom)
|
||||||
|
```
|
||||||
+10
-3
@@ -5,6 +5,13 @@
|
|||||||
VITE_API_BASE_URL=
|
VITE_API_BASE_URL=
|
||||||
|
|
||||||
# Show the "Developer Mode" panel on the login page.
|
# Show the "Developer Mode" panel on the login page.
|
||||||
# true (default) / false — in Docker builds this is controlled by the
|
# true (default in local dev) / false — in Docker builds this is controlled
|
||||||
# DEV_LOGIN_ENABLED build arg (see docker-compose.yml), not this file.
|
# by the DEV_LOGIN_ENABLED build arg (see docker-compose.yml), not this file.
|
||||||
VITE_ENABLE_DEV_LOGIN=true
|
VITE_ENABLE_DEV_LOGIN=
|
||||||
|
|
||||||
|
# ---- Observability (all optional — empty = disabled, no network calls) ----
|
||||||
|
# OpenPanel: browser origin of the self-hosted dashboard/API + project client id.
|
||||||
|
VITE_OPENPANEL_URL=
|
||||||
|
VITE_OPENPANEL_CLIENT_ID=
|
||||||
|
# GlitchTip/Sentry DSN for frontend exception capture.
|
||||||
|
VITE_SENTRY_DSN=
|
||||||
|
|||||||
Reference in New Issue
Block a user