feat(monitoring): bot observability dashboards — Grafana + Persian RTL ERP page

Grafana (full use of the existing monitoring/ stack, per OSS policy):
- datasources.yml: BotPostgres (uid bot-postgres) over the shared network;
  grafana service joins 'shared' + carries the datasource env
- db_init.py: idempotent read-only role obs_reader (USAGE on schema public +
  SELECT on bot_traces/bot_trace_events; skipped when
  OBS_READER_PASSWORD empty; ALTER ROLE for rotation) + composite index
  ix_bot_trace_events_stage_time (stage, created_at DESC) — senior-engineer
  review applied with the column-name fix (event_type -> stage); (trace_id)
  already covered by the existing (trace_id, created_at) index
- dashboards/bot-observability.json: KPI row (real traffic, events, Jev/
  fallback errors, Jev p95, tokens, commits, archive guard), events/day,
  per-stage p50/p95, mention statuses (arbiter+fallback aggregates + legacy
  rows), DeepSeek arbiter actions, tokens/day, model-drift vs jev-1.13.0,
  user-choice A/B, STT + audio.enhance health, archive size guard
- dashboards/bot-trace-explorer.json: trace table + full event drill-down
  via the trace_id variable

ERP page (admin-only, Persian RTL, recharts@^3 for React 19):
- backend bot_metrics.py: read-only require_role('admin') endpoints
  (summary/routing/quality/jev/stt/traces + trace detail); aggregates count
  real traffic only (calibration/demo/chat_id=0 excluded)
- frontend BotMonitorPage at /bot-monitor (KPI cards, daily events,
  status donut, arbiter actions, user-choice A/B, STT/enhance tables,
  stage latency, tokens chart, traces table + detail drawer with payload
  viewer); entry link from the admin panel; MobileTabBar hidden there
- api.ts: typed botMetrics client section

Tooling/docs:
- tools/seed_observability_demo.py: deterministic demo traces (source='demo',
  every current pipeline stage incl. audio.enhance/jev.failure/arbitration/
  fallback/user choices) — excluded from real-traffic aggregates
- monitoring/README.md + bot/docs/DEPLOY-AND-MONITORING.md: setup, seed and
  rollback notes; alerts deliberately deferred (owner decision 2026-10-24)

Verified locally: obs_reader SELECT ok / INSERT denied; panels' SQL run as
obs_reader with data; datasource health OK; both dashboards survive a grafana
restart; endpoints 200 as admin and 403 without role; tsc + vite build clean;
page renders in-browser (admin session); bot startup provisions role+index.
This commit is contained in:
2026-10-24 15:09:01 +03:30
parent 787824a19e
commit 0adadd2cf8
19 changed files with 1912 additions and 11 deletions
+37 -1
View File
@@ -87,12 +87,48 @@ costs; low value for a small internal team at launch). If wanted later:
enable OpenPanel replay with **10–25% sampling** and review retention —
budget +1 GB storage and re-check RAM headroom first.
## Bot observability dashboards (Bale bot)
Two Grafana dashboards read the bot's own tables (`bot_traces` /
`bot_trace_events`) directly — no metrics pipeline, no data duplication:
- **بات — Observability خط لوله** (`bot-observability.json`): KPI row (real
traffic, events, Jev/fallback errors, Jev p95, tokens, commits, monthly
archive guard), events/day, per-stage p50/p95 latency, final mention
statuses (exact/ambiguity/new — arbiter + crisis-fallback aggregates plus
the legacy per-mention rows), DeepSeek arbiter actions on Jev suggestions,
tokens/day, model drift vs the pinned `jev-1.13.0`, user-choice A/B proxy,
STT + audio-enhancement health, archive size guard (>5 GB).
- **بات — اکسپلورر ترس‌ها** (`bot-trace-explorer.json`): latest traces table
+ full event list of one trace (paste its `trace_id` in the dashboard
variable).
Setup (one-time):
1. `PROCUREMENT_BOT_OBS_READER_PASSWORD=<secret>` in `.env` — the **bot**
provisions the read-only role `obs_reader` at startup (`USAGE` on schema
`public` + `SELECT` on the two tables only; empty value = role skipped).
2. Start/restart grafana after the bot ran once — the `BotPostgres` datasource
(uid `bot-postgres`) is provisioned from `datasources.yml` via the grafana
service env; grafana reaches postgres over the `shared` (erp_shared)
network.
Aggregate panels count **real traffic only** (`source` not in
calibration/demo and `chat_id ≠ 0`). To fill every panel on a fresh install,
seed synthetic traces: `docker compose exec bot python
tools/seed_observability_demo.py --days 7` (marked `source='demo'`).
The same data is also available inside the ERP itself (admin-only, Persian
RTL): **/bot-monitor** — served by `GET /api/v1/bot-metrics/*`
(`backend/app/routers/bot_metrics.py`).
## Alerting
`monitoring/prometheus/alerts.yml` defines: API down, 5xx > 5%, p95 > 2 s,
root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in
Grafana alerting; wire a notification channel (email/Telegram) from the
Grafana UI when ready.
Grafana UI when ready. (Bot observability alerts are deliberately NOT part
of this update — display only, per owner decision 2026-10-24.)
## Lean mode (if RAM gets tight)