feat(monitoring): bot observability dashboards — Grafana + Persian RTL ERP page

Grafana (full use of the existing monitoring/ stack, per OSS policy):
- datasources.yml: BotPostgres (uid bot-postgres) over the shared network;
  grafana service joins 'shared' + carries the datasource env
- db_init.py: idempotent read-only role obs_reader (USAGE on schema public +
  SELECT on bot_traces/bot_trace_events; skipped when
  OBS_READER_PASSWORD empty; ALTER ROLE for rotation) + composite index
  ix_bot_trace_events_stage_time (stage, created_at DESC) — senior-engineer
  review applied with the column-name fix (event_type -> stage); (trace_id)
  already covered by the existing (trace_id, created_at) index
- dashboards/bot-observability.json: KPI row (real traffic, events, Jev/
  fallback errors, Jev p95, tokens, commits, archive guard), events/day,
  per-stage p50/p95, mention statuses (arbiter+fallback aggregates + legacy
  rows), DeepSeek arbiter actions, tokens/day, model-drift vs jev-1.13.0,
  user-choice A/B, STT + audio.enhance health, archive size guard
- dashboards/bot-trace-explorer.json: trace table + full event drill-down
  via the trace_id variable

ERP page (admin-only, Persian RTL, recharts@^3 for React 19):
- backend bot_metrics.py: read-only require_role('admin') endpoints
  (summary/routing/quality/jev/stt/traces + trace detail); aggregates count
  real traffic only (calibration/demo/chat_id=0 excluded)
- frontend BotMonitorPage at /bot-monitor (KPI cards, daily events,
  status donut, arbiter actions, user-choice A/B, STT/enhance tables,
  stage latency, tokens chart, traces table + detail drawer with payload
  viewer); entry link from the admin panel; MobileTabBar hidden there
- api.ts: typed botMetrics client section

Tooling/docs:
- tools/seed_observability_demo.py: deterministic demo traces (source='demo',
  every current pipeline stage incl. audio.enhance/jev.failure/arbitration/
  fallback/user choices) — excluded from real-traffic aggregates
- monitoring/README.md + bot/docs/DEPLOY-AND-MONITORING.md: setup, seed and
  rollback notes; alerts deliberately deferred (owner decision 2026-10-24)

Verified locally: obs_reader SELECT ok / INSERT denied; panels' SQL run as
obs_reader with data; datasource health OK; both dashboards survive a grafana
restart; endpoints 200 as admin and 403 without role; tsc + vite build clean;
page renders in-browser (admin session); bot startup provisions role+index.
This commit is contained in:
2026-10-24 15:09:01 +03:30
parent 787824a19e
commit 0adadd2cf8
19 changed files with 1912 additions and 11 deletions
+39
View File
@@ -97,10 +97,48 @@ DDL = [
""",
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_trace ON bot_trace_events (trace_id, created_at)",
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_stage ON bot_trace_events (stage)",
# dashboard queries filter stage + time range together (senior-engineer review
# 2026-10-24; column is `stage` — the review's `event_type` doesn't exist here)
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_stage_time ON bot_trace_events (stage, created_at DESC)",
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_payload ON bot_trace_events USING GIN (payload jsonb_path_ops)",
]
async def ensure_obs_reader(session: AsyncSession) -> None:
"""Read-only role for the Grafana Postgres datasource (dashboard plan phase 1).
Idempotent; entirely skipped when PROCUREMENT_BOT_OBS_READER_PASSWORD is empty
(local dev may keep using the postgres superuser for the datasource instead).
Password comes from env so rotation never needs a code change.
"""
from app.config import settings
pwd = (settings.OBS_READER_PASSWORD or "").strip()
if not pwd:
logger.info("obs_reader role skipped (no OBS_READER_PASSWORD configured)")
return
# CREATE ROLE's PASSWORD clause takes a string literal only (no function
# expressions), so the env value is embedded with standard '' escaping —
# bound parameters cannot cross into a DO block.
lit = pwd.replace("'", "''")
await session.execute(text(f"""
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'obs_reader') THEN
CREATE ROLE obs_reader LOGIN PASSWORD '{lit}';
ELSE
ALTER ROLE obs_reader WITH LOGIN PASSWORD '{lit}';
END IF;
END
$$
"""))
await session.execute(text("GRANT USAGE ON SCHEMA public TO obs_reader"))
await session.execute(text(
"GRANT SELECT ON TABLE public.bot_traces, public.bot_trace_events TO obs_reader"))
await session.commit()
logger.info("obs_reader role ready (USAGE on public + SELECT on bot_traces/bot_trace_events)")
async def ensure_bot_tables(session: AsyncSession) -> None:
for stmt in DDL:
await session.execute(text(stmt))
@@ -113,3 +151,4 @@ async def ensure_schema() -> None:
async with SessionLocal() as session:
await ensure_bot_tables(session)
await ensure_obs_reader(session)