feat(monitoring): observability stack + app instrumentation
- /health/ready (DB+MinIO+Casdoor checks, webhook post-deploy gate), /metrics via prometheus-fastapi-instrumentator, Sentry/GlitchTip capture, X-Request-ID middleware + JSON access logging (PROCUREMENT_LOG_FORMAT) - server-side analytics (OpenPanel Track API): login, quote lifecycle, receipts — fire-and-forget, disabled without env config - monitoring/ compose project: GlitchTip(+pg), Prometheus(+alerts), Grafana(provisioned ERP dashboard), Loki+Promtail, node/postgres exporters, cAdvisor — mem-limited (~2.2GB), loopback admin UIs, separate from app deploys; erp_shared network for scraping - frontend: @openpanel/web + @sentry/react (build-arg gated, no-op when unset), funnel events (login, wizard steps, quote submitted) - compose: api healthcheck, observability env wiring, .env.example keys
This commit is contained in:
MMD
@@ -0,0 +1,111 @@
|
||||
# Monitoring & analytics stack
|
||||
|
||||
What runs where, and why — the ERP gets the three missing observability
|
||||
layers on the same 8 GB VPS:
|
||||
|
||||
| Layer | Tool | What it answers | Budget |
|
||||
|---|---|---|---|
|
||||
| Product analytics | **OpenPanel** (self-hosted) | Which features are used? Where do users drop out of the quote wizard? Funnel: login → wizard steps → quote submitted → accepted → received. Session replay **off** by design (heavy; revisit with sampling later) | ~1–1.5 GB |
|
||||
| Error tracking | **GlitchTip** (Sentry-compatible) | What broke, for whom, with which stack trace — frontend (ErrorBoundary + fetch errors) and backend (all unhandled exceptions) | ~750 MB |
|
||||
| Metrics + logs | **Prometheus + Grafana + Loki/Promtail + node/postgres-exporters + cAdvisor** | Request rate/latency/5xx, container RAM, Postgres connections, disk pressure; full container logs searchable in Grafana | ~1.4 GB |
|
||||
|
||||
The monitoring compose is a **separate compose project** — push-to-deploy
|
||||
rebuilds of the app never restart monitoring containers.
|
||||
|
||||
## First-time deployment (VPS)
|
||||
|
||||
```bash
|
||||
cd /srv/erp # the repo checkout used by the deploy webhook
|
||||
|
||||
# 1) monitoring section of .env (see .env.example) — fill in real values:
|
||||
# GLITCHTIP_SECRET_KEY, GLITCHTIP_DOMAIN/HOST, GLITCHTIP_DB_PASSWORD,
|
||||
# GRAFANA_ADMIN_PASSWORD, POSTGRES_PASSWORD (already set), LOG_FORMAT=json
|
||||
|
||||
# 2) the main compose gained a shared network — re-up once so it exists:
|
||||
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
|
||||
|
||||
# 3) start monitoring:
|
||||
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
|
||||
|
||||
# 4) GlitchTip is routed publicly through Traefik for the browser SDK:
|
||||
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env \
|
||||
--profile public up -d glitchtip
|
||||
```
|
||||
|
||||
Admin UIs stay **loopback-only** (same posture as the MinIO console):
|
||||
|
||||
```powershell
|
||||
ssh -L 3001:127.0.0.1:3001 -L 8001:127.0.0.1:8001 <user>@<server-host>
|
||||
# Grafana → http://localhost:3001 (dashboards auto-provisioned)
|
||||
# GlitchTip → http://localhost:8001 (create the first admin on first open)
|
||||
```
|
||||
|
||||
In GlitchTip: create an **organization → project** for the backend and one
|
||||
for the frontend, copy the DSNs into `/srv/erp/.env`
|
||||
(`SENTRY_DSN_BACKEND`, `SENTRY_DSN_FRONTEND`) and redeploy the app once.
|
||||
|
||||
## Product analytics (OpenPanel)
|
||||
|
||||
Deployed from the upstream self-hosting branch (kept outside this repo —
|
||||
it ships its own compose + update tooling):
|
||||
|
||||
```bash
|
||||
cd /opt
|
||||
git clone -b self-hosting https://github.com/Openpanel-dev/openpanel.git
|
||||
cd openpanel/self-hosting
|
||||
./setup # interactive
|
||||
./start
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
1. Open the OpenPanel dashboard (expose it via Traefik on
|
||||
`stats.bennu-api-bahrestan.ir` following its docs — add the container to
|
||||
`traefik_network` and give the dashboard + its `/api` path labels),
|
||||
create a project → copy the **client id / client secret**.
|
||||
2. Put them in `/srv/erp/.env`:
|
||||
`OPENPANEL_API_URL` (in-network URL of its api service),
|
||||
`OPENPANEL_CLIENT_ID`, `OPENPANEL_CLIENT_SECRET`, and
|
||||
`OPENPANEL_PUBLIC_URL` (browser-facing origin) — then rebuild the app:
|
||||
`docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build api frontend`.
|
||||
|
||||
What gets tracked automatically once configured:
|
||||
|
||||
- **Frontend**: page views, `login_success`, `wizard_step` (funnel across
|
||||
the 5 quote-wizard steps), `quote_submitted`, `user_identified`.
|
||||
- **Backend** (server-side, cannot be blocked by the browser):
|
||||
`login_success`, `quote_offered/edited/accepted/arrived/expired/rejected`,
|
||||
`receipt_confirmed` — see `backend/app/services/analytics.py`.
|
||||
|
||||
Without the env vars everything is a no-op — the app never waits on
|
||||
analytics (3 s timeout, swallow-all).
|
||||
|
||||
## Session replay
|
||||
|
||||
Deliberately **off** (records full DOM of every session → storage + privacy
|
||||
costs; low value for a small internal team at launch). If wanted later:
|
||||
enable OpenPanel replay with **10–25% sampling** and review retention —
|
||||
budget +1 GB storage and re-check RAM headroom first.
|
||||
|
||||
## Alerting
|
||||
|
||||
`monitoring/prometheus/alerts.yml` defines: API down, 5xx > 5%, p95 > 2 s,
|
||||
root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in
|
||||
Grafana alerting; wire a notification channel (email/Telegram) from the
|
||||
Grafana UI when ready.
|
||||
|
||||
## Lean mode (if RAM gets tight)
|
||||
|
||||
Drop prometheus+grafana+loki+promtail+cadvisor+node-exporter (~1.1 GB) and
|
||||
keep GlitchTip + OpenPanel only; add `uptime-kuma` (~100 MB) for
|
||||
up/down pings. The app keeps `/health/ready` either way — the deploy
|
||||
webhook uses it post-rebuild regardless of the metrics stack.
|
||||
|
||||
## What the app exposes for this stack
|
||||
|
||||
- `GET /health` — liveness (process up).
|
||||
- `GET /health/ready` — readiness: Postgres + MinIO + Casdoor checks.
|
||||
Used by the deploy webhook after every rebuild.
|
||||
- `GET /metrics` — Prometheus format (unauthenticated, for scraping).
|
||||
- JSON request logs with `X-Request-ID` (set `LOG_FORMAT=json` in prod)
|
||||
→ shipped to Loki by Promtail.
|
||||
Reference in New Issue
Block a user