ERP/monitoring

Monitoring & analytics stack

What runs where, and why — the ERP gets the three missing observability layers on the same 8 GB VPS:

Layer	Tool	What it answers	Budget
Product analytics	OpenPanel (self-hosted)	Which features are used? Where do users drop out of the quote wizard? Funnel: login → wizard steps → quote submitted → accepted → received. Session replay off by design (heavy; revisit with sampling later)	~1–1.5 GB
Error tracking	GlitchTip (Sentry-compatible)	What broke, for whom, with which stack trace — frontend (ErrorBoundary + fetch errors) and backend (all unhandled exceptions)	~750 MB
Metrics + logs	Prometheus + Grafana + Loki/Promtail + node/postgres-exporters + cAdvisor	Request rate/latency/5xx, container RAM, Postgres connections, disk pressure; full container logs searchable in Grafana	~1.4 GB

The monitoring compose is a separate compose project — push-to-deploy rebuilds of the app never restart monitoring containers.

First-time deployment (VPS)

cd /srv/erp   # the repo checkout used by the deploy webhook

# 1) monitoring section of .env (see .env.example) — fill in real values:
#    GLITCHTIP_SECRET_KEY, GLITCHTIP_DOMAIN/HOST, GLITCHTIP_DB_PASSWORD,
#    GRAFANA_ADMIN_PASSWORD, POSTGRES_PASSWORD (already set), LOG_FORMAT=json

# 2) the main compose gained a shared network — re-up once so it exists.
#    NOTE (2026-10-25): GRAFANA MOVED into the main docker-compose.yml —
#    this one command now brings the app AND grafana up together:
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# 3) start the REST of monitoring (glitchtip/prometheus/loki — no grafana):
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d

# 4) GlitchTip is routed publicly through Traefik for the browser SDK:
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env \
    --profile public up -d glitchtip

Admin UIs stay loopback-only (same posture as the MinIO console):

ssh -L 3001:127.0.0.1:3001 -L 8001:127.0.0.1:8001 <user>@<server-host>
# Grafana  → http://localhost:3001  (runs in the MAIN app stack since
#            2026-10-25; dashboards auto-provisioned; restarts on app deploys
#            are harmless — settings live in the grafana_data volume)
# GlitchTip → http://localhost:8001 (create the first admin on first open)

In GlitchTip: create an organization → project for the backend and one for the frontend, copy the DSNs into /srv/erp/.env (SENTRY_DSN_BACKEND, SENTRY_DSN_FRONTEND) and redeploy the app once.

Product analytics (OpenPanel)

Deployed from the upstream self-hosting branch (kept outside this repo — it ships its own compose + update tooling):

cd /opt
git clone -b self-hosting https://github.com/Openpanel-dev/openpanel.git
cd openpanel/self-hosting
./setup     # interactive
./start

Then:

1. Open the OpenPanel dashboard (expose it via Traefik on stats.bennu-api-bahrestan.ir following its docs — add the container to traefik_network and give the dashboard + its /api path labels), create a project → copy the client id / client secret.
2. Put them in /srv/erp/.env: OPENPANEL_API_URL (in-network URL of its api service), OPENPANEL_CLIENT_ID, OPENPANEL_CLIENT_SECRET, and OPENPANEL_PUBLIC_URL (browser-facing origin) — then rebuild the app: docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build api frontend.

What gets tracked automatically once configured:

• Frontend: page views, login_success, wizard_step (funnel across the 5 quote-wizard steps), quote_submitted, user_identified.
• Backend (server-side, cannot be blocked by the browser): login_success, quote_offered/edited/accepted/arrived/expired/rejected, receipt_confirmed — see backend/app/services/analytics.py.

Without the env vars everything is a no-op — the app never waits on analytics (3 s timeout, swallow-all).

Session replay

Deliberately off (records full DOM of every session → storage + privacy costs; low value for a small internal team at launch). If wanted later: enable OpenPanel replay with 10–25% sampling and review retention — budget +1 GB storage and re-check RAM headroom first.

Bot observability dashboards (Bale bot)

Two Grafana dashboards read the bot's own tables (bot_traces / bot_trace_events) directly — no metrics pipeline, no data duplication:

• بات — Observability خط لوله (bot-observability.json): KPI row (real traffic, events, Jev/fallback errors, Jev p95, tokens, commits, monthly archive guard), events/day, per-stage p50/p95 latency, final mention statuses (exact/ambiguity/new — arbiter + crisis-fallback aggregates plus the legacy per-mention rows), DeepSeek arbiter actions on Jev suggestions, tokens/day, model drift vs the pinned jev-1.13.0, user-choice A/B proxy, STT + audio-enhancement health, archive size guard (>5 GB).
• بات — اکسپلورر ترس‌ها (bot-trace-explorer.json): latest traces table
  • full event list of one trace (paste its trace_id in the dashboard variable).

Setup (one-time):

1. PROCUREMENT_BOT_OBS_READER_PASSWORD=<secret> in .env — the bot provisions the read-only role obs_reader at startup (USAGE on schema public + SELECT on the two tables only; empty value = role skipped).
2. Start/restart grafana after the bot ran once — the BotPostgres datasource (uid bot-postgres) is provisioned from datasources.yml via the grafana service env; grafana reaches postgres over the shared (erp_shared) network.

Aggregate panels count real traffic only (source not in calibration/demo and chat_id ≠ 0). To fill every panel on a fresh install, seed synthetic traces: docker compose exec bot python tools/seed_observability_demo.py --days 7 (marked source='demo').

The same data is also available inside the ERP itself (admin-only, Persian RTL): /bot-monitor — served by GET /api/v1/bot-metrics/* (backend/app/routers/bot_metrics.py). The page is tabbed: نمای کلی (overview) / مدل‌ها (a record per model — Jev, DeepSeek-NLU, DeepSeek arbiter, DeepSeek crisis fallback, every STT provider, audio-enhancer — with per-model daily charts) / بخش‌ها (pipeline stages grouped by section) / پیام‌ها (message records with inline voice playback, download and every STT transcript side-by-side; media is streamed from the bot's MinIO bucket bot-media with the on-disk archive as fallback, via short-lived signed URLs). Grafana's bot-observability dashboard carries a matching «کارکرد مدل‌ها» table.

Alerting

monitoring/prometheus/alerts.yml defines: API down, 5xx > 5%, p95 > 2 s, root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in Grafana alerting; wire a notification channel (email/Telegram) from the Grafana UI when ready. (Bot observability alerts are deliberately NOT part of this update — display only, per owner decision 2026-10-24.)

Lean mode (if RAM gets tight)

Drop prometheus+grafana+loki+promtail+cadvisor+node-exporter (~1.1 GB) and keep GlitchTip + OpenPanel only; add uptime-kuma (~100 MB) for up/down pings. The app keeps /health/ready either way — the deploy webhook uses it post-rebuild regardless of the metrics stack.

What the app exposes for this stack

• GET /health — liveness (process up).
• GET /health/ready — readiness: Postgres + MinIO + Casdoor checks. Used by the deploy webhook after every rebuild.
• GET /metrics — Prometheus format (unauthenticated, for scraping).
• JSON request logs with X-Request-ID (set LOG_FORMAT=json in prod) → shipped to Loki by Promtail.

Local runs (Docker Desktop)

Grafana starts with the app stack — plain docker compose up -d from the repo root brings api/frontend/bot/postgres/… and grafana (loopback :3001). For the rest of the monitoring stack start WITHOUT --profile host-metrics — node-exporter and cAdvisor need Linux rootfs mounts and will fail to start on Windows/macOS hosts:

docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d

Per-container and host metrics are absent locally; API metrics, Postgres metrics, logs, GlitchTip and Grafana all work the same. On the Linux VPS always include --profile host-metrics.