feat(monitoring): bot observability dashboards — Grafana + Persian RTL ERP page
Grafana (full use of the existing monitoring/ stack, per OSS policy):
- datasources.yml: BotPostgres (uid bot-postgres) over the shared network;
grafana service joins 'shared' + carries the datasource env
- db_init.py: idempotent read-only role obs_reader (USAGE on schema public +
SELECT on bot_traces/bot_trace_events; skipped when
OBS_READER_PASSWORD empty; ALTER ROLE for rotation) + composite index
ix_bot_trace_events_stage_time (stage, created_at DESC) — senior-engineer
review applied with the column-name fix (event_type -> stage); (trace_id)
already covered by the existing (trace_id, created_at) index
- dashboards/bot-observability.json: KPI row (real traffic, events, Jev/
fallback errors, Jev p95, tokens, commits, archive guard), events/day,
per-stage p50/p95, mention statuses (arbiter+fallback aggregates + legacy
rows), DeepSeek arbiter actions, tokens/day, model-drift vs jev-1.13.0,
user-choice A/B, STT + audio.enhance health, archive size guard
- dashboards/bot-trace-explorer.json: trace table + full event drill-down
via the trace_id variable
ERP page (admin-only, Persian RTL, recharts@^3 for React 19):
- backend bot_metrics.py: read-only require_role('admin') endpoints
(summary/routing/quality/jev/stt/traces + trace detail); aggregates count
real traffic only (calibration/demo/chat_id=0 excluded)
- frontend BotMonitorPage at /bot-monitor (KPI cards, daily events,
status donut, arbiter actions, user-choice A/B, STT/enhance tables,
stage latency, tokens chart, traces table + detail drawer with payload
viewer); entry link from the admin panel; MobileTabBar hidden there
- api.ts: typed botMetrics client section
Tooling/docs:
- tools/seed_observability_demo.py: deterministic demo traces (source='demo',
every current pipeline stage incl. audio.enhance/jev.failure/arbitration/
fallback/user choices) — excluded from real-traffic aggregates
- monitoring/README.md + bot/docs/DEPLOY-AND-MONITORING.md: setup, seed and
rollback notes; alerts deliberately deferred (owner decision 2026-10-24)
Verified locally: obs_reader SELECT ok / INSERT denied; panels' SQL run as
obs_reader with data; datasource health OK; both dashboards survive a grafana
restart; endpoints 200 as admin and 403 without role; tsc + vite build clean;
page renders in-browser (admin session); bot startup provisions role+index.
This commit is contained in:
+37
-1
@@ -87,12 +87,48 @@ costs; low value for a small internal team at launch). If wanted later:
|
||||
enable OpenPanel replay with **10–25% sampling** and review retention —
|
||||
budget +1 GB storage and re-check RAM headroom first.
|
||||
|
||||
## Bot observability dashboards (Bale bot)
|
||||
|
||||
Two Grafana dashboards read the bot's own tables (`bot_traces` /
|
||||
`bot_trace_events`) directly — no metrics pipeline, no data duplication:
|
||||
|
||||
- **بات — Observability خط لوله** (`bot-observability.json`): KPI row (real
|
||||
traffic, events, Jev/fallback errors, Jev p95, tokens, commits, monthly
|
||||
archive guard), events/day, per-stage p50/p95 latency, final mention
|
||||
statuses (exact/ambiguity/new — arbiter + crisis-fallback aggregates plus
|
||||
the legacy per-mention rows), DeepSeek arbiter actions on Jev suggestions,
|
||||
tokens/day, model drift vs the pinned `jev-1.13.0`, user-choice A/B proxy,
|
||||
STT + audio-enhancement health, archive size guard (>5 GB).
|
||||
- **بات — اکسپلورر ترسها** (`bot-trace-explorer.json`): latest traces table
|
||||
+ full event list of one trace (paste its `trace_id` in the dashboard
|
||||
variable).
|
||||
|
||||
Setup (one-time):
|
||||
|
||||
1. `PROCUREMENT_BOT_OBS_READER_PASSWORD=<secret>` in `.env` — the **bot**
|
||||
provisions the read-only role `obs_reader` at startup (`USAGE` on schema
|
||||
`public` + `SELECT` on the two tables only; empty value = role skipped).
|
||||
2. Start/restart grafana after the bot ran once — the `BotPostgres` datasource
|
||||
(uid `bot-postgres`) is provisioned from `datasources.yml` via the grafana
|
||||
service env; grafana reaches postgres over the `shared` (erp_shared)
|
||||
network.
|
||||
|
||||
Aggregate panels count **real traffic only** (`source` not in
|
||||
calibration/demo and `chat_id ≠ 0`). To fill every panel on a fresh install,
|
||||
seed synthetic traces: `docker compose exec bot python
|
||||
tools/seed_observability_demo.py --days 7` (marked `source='demo'`).
|
||||
|
||||
The same data is also available inside the ERP itself (admin-only, Persian
|
||||
RTL): **/bot-monitor** — served by `GET /api/v1/bot-metrics/*`
|
||||
(`backend/app/routers/bot_metrics.py`).
|
||||
|
||||
## Alerting
|
||||
|
||||
`monitoring/prometheus/alerts.yml` defines: API down, 5xx > 5%, p95 > 2 s,
|
||||
root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in
|
||||
Grafana alerting; wire a notification channel (email/Telegram) from the
|
||||
Grafana UI when ready.
|
||||
Grafana UI when ready. (Bot observability alerts are deliberately NOT part
|
||||
of this update — display only, per owner decision 2026-10-24.)
|
||||
|
||||
## Lean mode (if RAM gets tight)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user