whqreza
cb621ca5d0
Problem: the dashboard was Jev-centric; NLU/arbiter/fallback DeepSeek, STT
providers and the audio-enhancer were barely visible, and stored texts/voices
had no real access path.
Bot (additive; reply path untouched):
- nlu.result now records latency_ms + the DeepSeek usage tokens
- matching.arbitration/fallback.result fill the model + latency_ms columns
(payload unchanged; SQL coalesces column/payload so old rows still count)
- storage.py: MinIO dual-write layer — every voice (raw + denoised) and user
attachment is mirrored fire-and-forget into bucket bot-media; the blocking
minio SDK runs via asyncio.to_thread (senior-engineer review) so the event
loop is never stalled; empty MINIO_* env = layer off; bucket auto-created
- requirements: minio>=7.2
Backend (admin-gated, read-only):
- GET /bot-metrics/models — one record per model (jev, deepseek-nlu,
deepseek-arbiter, deepseek-fallback, audio-enhancer + dynamic per-provider
STT) with calls/ok-rate/p50/p95/tokens/last-seen + version-drift vs the pin
- GET /bot-metrics/models/{id}/daily — per-model daily calls/fails/tokens/p50
- GET /bot-metrics/stages — every stage tagged with its Persian pipeline
section (صوت/درک پیام/تطبیق/ابهام و انتخاب/پیشنویس)
- /traces now carries input_kind + a 120-char text excerpt (consensus →
input.text → first STT transcript)
- media serving: POST /media-url mints a short-lived HMAC URL (audio/a tags
cannot send Authorization); GET /media streams from MinIO bot-media with
the read-only ./bot/data mount as fallback for pre-MinIO files; hard ref
validation (voices|attachments only, no .., extension whitelist)
- router exempted from the global Bearer dependency (files.py pattern) —
every endpoint keeps its own require_role('admin')
Frontend /bot-monitor (RTL, recharts):
- four tabs: نمای کلی / مدلها (records table + per-model expandable daily
charts: calls/fails, tokens, p50) / بخشها (sections grouped) / پیامها
- message drawer rebuilt: inline <audio controls> for raw + denoised voice,
download links, every STT transcript + consensus side-by-side, user
attachments, then the event timeline
- terminology: «ترس» → «پروندهی پیام» with tooltips + a one-line legend
Grafana: bot-observability gains a «کارکرد مدلها» table (stage→model
mapping for legacy rows); the explorer gains a message text column.
Seed: real (tiny) ogg files written + mirrored to MinIO, NLU usage tokens,
column-level model/latency on matching events.
Verified locally: /models returns all 7 records incl. per-STT and NLU tokens;
media 200 via signed URL (MinIO) and 401 without; ../ and .py rejected (400);
disk fallback serves a legacy file; pm-role 403 on every new endpoint;
bucket auto-created with 14 mirrored objects; tsc + vite build clean;
Grafana re-provisions 24 panels.
169 lines
7.8 KiB
Markdown
169 lines
7.8 KiB
Markdown
# Monitoring & analytics stack
|
||
|
||
What runs where, and why — the ERP gets the three missing observability
|
||
layers on the same 8 GB VPS:
|
||
|
||
| Layer | Tool | What it answers | Budget |
|
||
|---|---|---|---|
|
||
| Product analytics | **OpenPanel** (self-hosted) | Which features are used? Where do users drop out of the quote wizard? Funnel: login → wizard steps → quote submitted → accepted → received. Session replay **off** by design (heavy; revisit with sampling later) | ~1–1.5 GB |
|
||
| Error tracking | **GlitchTip** (Sentry-compatible) | What broke, for whom, with which stack trace — frontend (ErrorBoundary + fetch errors) and backend (all unhandled exceptions) | ~750 MB |
|
||
| Metrics + logs | **Prometheus + Grafana + Loki/Promtail + node/postgres-exporters + cAdvisor** | Request rate/latency/5xx, container RAM, Postgres connections, disk pressure; full container logs searchable in Grafana | ~1.4 GB |
|
||
|
||
The monitoring compose is a **separate compose project** — push-to-deploy
|
||
rebuilds of the app never restart monitoring containers.
|
||
|
||
## First-time deployment (VPS)
|
||
|
||
```bash
|
||
cd /srv/erp # the repo checkout used by the deploy webhook
|
||
|
||
# 1) monitoring section of .env (see .env.example) — fill in real values:
|
||
# GLITCHTIP_SECRET_KEY, GLITCHTIP_DOMAIN/HOST, GLITCHTIP_DB_PASSWORD,
|
||
# GRAFANA_ADMIN_PASSWORD, POSTGRES_PASSWORD (already set), LOG_FORMAT=json
|
||
|
||
# 2) the main compose gained a shared network — re-up once so it exists:
|
||
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
|
||
|
||
# 3) start monitoring:
|
||
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
|
||
|
||
# 4) GlitchTip is routed publicly through Traefik for the browser SDK:
|
||
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env \
|
||
--profile public up -d glitchtip
|
||
```
|
||
|
||
Admin UIs stay **loopback-only** (same posture as the MinIO console):
|
||
|
||
```powershell
|
||
ssh -L 3001:127.0.0.1:3001 -L 8001:127.0.0.1:8001 <user>@<server-host>
|
||
# Grafana → http://localhost:3001 (dashboards auto-provisioned)
|
||
# GlitchTip → http://localhost:8001 (create the first admin on first open)
|
||
```
|
||
|
||
In GlitchTip: create an **organization → project** for the backend and one
|
||
for the frontend, copy the DSNs into `/srv/erp/.env`
|
||
(`SENTRY_DSN_BACKEND`, `SENTRY_DSN_FRONTEND`) and redeploy the app once.
|
||
|
||
## Product analytics (OpenPanel)
|
||
|
||
Deployed from the upstream self-hosting branch (kept outside this repo —
|
||
it ships its own compose + update tooling):
|
||
|
||
```bash
|
||
cd /opt
|
||
git clone -b self-hosting https://github.com/Openpanel-dev/openpanel.git
|
||
cd openpanel/self-hosting
|
||
./setup # interactive
|
||
./start
|
||
```
|
||
|
||
Then:
|
||
|
||
1. Open the OpenPanel dashboard (expose it via Traefik on
|
||
`stats.bennu-api-bahrestan.ir` following its docs — add the container to
|
||
`traefik_network` and give the dashboard + its `/api` path labels),
|
||
create a project → copy the **client id / client secret**.
|
||
2. Put them in `/srv/erp/.env`:
|
||
`OPENPANEL_API_URL` (in-network URL of its api service),
|
||
`OPENPANEL_CLIENT_ID`, `OPENPANEL_CLIENT_SECRET`, and
|
||
`OPENPANEL_PUBLIC_URL` (browser-facing origin) — then rebuild the app:
|
||
`docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build api frontend`.
|
||
|
||
What gets tracked automatically once configured:
|
||
|
||
- **Frontend**: page views, `login_success`, `wizard_step` (funnel across
|
||
the 5 quote-wizard steps), `quote_submitted`, `user_identified`.
|
||
- **Backend** (server-side, cannot be blocked by the browser):
|
||
`login_success`, `quote_offered/edited/accepted/arrived/expired/rejected`,
|
||
`receipt_confirmed` — see `backend/app/services/analytics.py`.
|
||
|
||
Without the env vars everything is a no-op — the app never waits on
|
||
analytics (3 s timeout, swallow-all).
|
||
|
||
## Session replay
|
||
|
||
Deliberately **off** (records full DOM of every session → storage + privacy
|
||
costs; low value for a small internal team at launch). If wanted later:
|
||
enable OpenPanel replay with **10–25% sampling** and review retention —
|
||
budget +1 GB storage and re-check RAM headroom first.
|
||
|
||
## Bot observability dashboards (Bale bot)
|
||
|
||
Two Grafana dashboards read the bot's own tables (`bot_traces` /
|
||
`bot_trace_events`) directly — no metrics pipeline, no data duplication:
|
||
|
||
- **بات — Observability خط لوله** (`bot-observability.json`): KPI row (real
|
||
traffic, events, Jev/fallback errors, Jev p95, tokens, commits, monthly
|
||
archive guard), events/day, per-stage p50/p95 latency, final mention
|
||
statuses (exact/ambiguity/new — arbiter + crisis-fallback aggregates plus
|
||
the legacy per-mention rows), DeepSeek arbiter actions on Jev suggestions,
|
||
tokens/day, model drift vs the pinned `jev-1.13.0`, user-choice A/B proxy,
|
||
STT + audio-enhancement health, archive size guard (>5 GB).
|
||
- **بات — اکسپلورر ترسها** (`bot-trace-explorer.json`): latest traces table
|
||
+ full event list of one trace (paste its `trace_id` in the dashboard
|
||
variable).
|
||
|
||
Setup (one-time):
|
||
|
||
1. `PROCUREMENT_BOT_OBS_READER_PASSWORD=<secret>` in `.env` — the **bot**
|
||
provisions the read-only role `obs_reader` at startup (`USAGE` on schema
|
||
`public` + `SELECT` on the two tables only; empty value = role skipped).
|
||
2. Start/restart grafana after the bot ran once — the `BotPostgres` datasource
|
||
(uid `bot-postgres`) is provisioned from `datasources.yml` via the grafana
|
||
service env; grafana reaches postgres over the `shared` (erp_shared)
|
||
network.
|
||
|
||
Aggregate panels count **real traffic only** (`source` not in
|
||
calibration/demo and `chat_id ≠ 0`). To fill every panel on a fresh install,
|
||
seed synthetic traces: `docker compose exec bot python
|
||
tools/seed_observability_demo.py --days 7` (marked `source='demo'`).
|
||
|
||
The same data is also available inside the ERP itself (admin-only, Persian
|
||
RTL): **/bot-monitor** — served by `GET /api/v1/bot-metrics/*`
|
||
(`backend/app/routers/bot_metrics.py`). The page is tabbed: نمای کلی
|
||
(overview) / **مدلها** (a record per model — Jev, DeepSeek-NLU, DeepSeek
|
||
arbiter, DeepSeek crisis fallback, every STT provider, audio-enhancer — with
|
||
per-model daily charts) / **بخشها** (pipeline stages grouped by section) /
|
||
**پیامها** (message records with inline voice playback, download and every
|
||
STT transcript side-by-side; media is streamed from the bot's MinIO bucket
|
||
`bot-media` with the on-disk archive as fallback, via short-lived signed
|
||
URLs). Grafana's bot-observability dashboard carries a matching
|
||
«کارکرد مدلها» table.
|
||
|
||
## Alerting
|
||
|
||
`monitoring/prometheus/alerts.yml` defines: API down, 5xx > 5%, p95 > 2 s,
|
||
root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in
|
||
Grafana alerting; wire a notification channel (email/Telegram) from the
|
||
Grafana UI when ready. (Bot observability alerts are deliberately NOT part
|
||
of this update — display only, per owner decision 2026-10-24.)
|
||
|
||
## Lean mode (if RAM gets tight)
|
||
|
||
Drop prometheus+grafana+loki+promtail+cadvisor+node-exporter (~1.1 GB) and
|
||
keep GlitchTip + OpenPanel only; add `uptime-kuma` (~100 MB) for
|
||
up/down pings. The app keeps `/health/ready` either way — the deploy
|
||
webhook uses it post-rebuild regardless of the metrics stack.
|
||
|
||
## What the app exposes for this stack
|
||
|
||
- `GET /health` — liveness (process up).
|
||
- `GET /health/ready` — readiness: Postgres + MinIO + Casdoor checks.
|
||
Used by the deploy webhook after every rebuild.
|
||
- `GET /metrics` — Prometheus format (unauthenticated, for scraping).
|
||
- JSON request logs with `X-Request-ID` (set `LOG_FORMAT=json` in prod)
|
||
→ shipped to Loki by Promtail.
|
||
|
||
## Local runs (Docker Desktop)
|
||
|
||
Start WITHOUT `--profile host-metrics` — node-exporter and cAdvisor need
|
||
Linux rootfs mounts and will fail to start on Windows/macOS hosts:
|
||
|
||
```bash
|
||
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
|
||
```
|
||
|
||
Per-container and host metrics are absent locally; API metrics, Postgres
|
||
metrics, logs, GlitchTip and Grafana all work the same. On the Linux VPS
|
||
always include `--profile host-metrics`.
|