feat(monitoring): bot observability dashboards — Grafana + Persian RTL ERP page
Grafana (full use of the existing monitoring/ stack, per OSS policy):
- datasources.yml: BotPostgres (uid bot-postgres) over the shared network;
grafana service joins 'shared' + carries the datasource env
- db_init.py: idempotent read-only role obs_reader (USAGE on schema public +
SELECT on bot_traces/bot_trace_events; skipped when
OBS_READER_PASSWORD empty; ALTER ROLE for rotation) + composite index
ix_bot_trace_events_stage_time (stage, created_at DESC) — senior-engineer
review applied with the column-name fix (event_type -> stage); (trace_id)
already covered by the existing (trace_id, created_at) index
- dashboards/bot-observability.json: KPI row (real traffic, events, Jev/
fallback errors, Jev p95, tokens, commits, archive guard), events/day,
per-stage p50/p95, mention statuses (arbiter+fallback aggregates + legacy
rows), DeepSeek arbiter actions, tokens/day, model-drift vs jev-1.13.0,
user-choice A/B, STT + audio.enhance health, archive size guard
- dashboards/bot-trace-explorer.json: trace table + full event drill-down
via the trace_id variable
ERP page (admin-only, Persian RTL, recharts@^3 for React 19):
- backend bot_metrics.py: read-only require_role('admin') endpoints
(summary/routing/quality/jev/stt/traces + trace detail); aggregates count
real traffic only (calibration/demo/chat_id=0 excluded)
- frontend BotMonitorPage at /bot-monitor (KPI cards, daily events,
status donut, arbiter actions, user-choice A/B, STT/enhance tables,
stage latency, tokens chart, traces table + detail drawer with payload
viewer); entry link from the admin panel; MobileTabBar hidden there
- api.ts: typed botMetrics client section
Tooling/docs:
- tools/seed_observability_demo.py: deterministic demo traces (source='demo',
every current pipeline stage incl. audio.enhance/jev.failure/arbitration/
fallback/user choices) — excluded from real-traffic aggregates
- monitoring/README.md + bot/docs/DEPLOY-AND-MONITORING.md: setup, seed and
rollback notes; alerts deliberately deferred (owner decision 2026-10-24)
Verified locally: obs_reader SELECT ok / INSERT denied; panels' SQL run as
obs_reader with data; datasource health OK; both dashboards survive a grafana
restart; endpoints 200 as admin and 403 without role; tsc + vite build clean;
page renders in-browser (admin session); bot startup provisions role+index.
This commit is contained in:
@@ -97,10 +97,48 @@ DDL = [
|
||||
""",
|
||||
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_trace ON bot_trace_events (trace_id, created_at)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_stage ON bot_trace_events (stage)",
|
||||
# dashboard queries filter stage + time range together (senior-engineer review
|
||||
# 2026-10-24; column is `stage` — the review's `event_type` doesn't exist here)
|
||||
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_stage_time ON bot_trace_events (stage, created_at DESC)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_bot_trace_events_payload ON bot_trace_events USING GIN (payload jsonb_path_ops)",
|
||||
]
|
||||
|
||||
|
||||
async def ensure_obs_reader(session: AsyncSession) -> None:
|
||||
"""Read-only role for the Grafana Postgres datasource (dashboard plan phase 1).
|
||||
|
||||
Idempotent; entirely skipped when PROCUREMENT_BOT_OBS_READER_PASSWORD is empty
|
||||
(local dev may keep using the postgres superuser for the datasource instead).
|
||||
Password comes from env so rotation never needs a code change.
|
||||
"""
|
||||
from app.config import settings
|
||||
|
||||
pwd = (settings.OBS_READER_PASSWORD or "").strip()
|
||||
if not pwd:
|
||||
logger.info("obs_reader role skipped (no OBS_READER_PASSWORD configured)")
|
||||
return
|
||||
# CREATE ROLE's PASSWORD clause takes a string literal only (no function
|
||||
# expressions), so the env value is embedded with standard '' escaping —
|
||||
# bound parameters cannot cross into a DO block.
|
||||
lit = pwd.replace("'", "''")
|
||||
await session.execute(text(f"""
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'obs_reader') THEN
|
||||
CREATE ROLE obs_reader LOGIN PASSWORD '{lit}';
|
||||
ELSE
|
||||
ALTER ROLE obs_reader WITH LOGIN PASSWORD '{lit}';
|
||||
END IF;
|
||||
END
|
||||
$$
|
||||
"""))
|
||||
await session.execute(text("GRANT USAGE ON SCHEMA public TO obs_reader"))
|
||||
await session.execute(text(
|
||||
"GRANT SELECT ON TABLE public.bot_traces, public.bot_trace_events TO obs_reader"))
|
||||
await session.commit()
|
||||
logger.info("obs_reader role ready (USAGE on public + SELECT on bot_traces/bot_trace_events)")
|
||||
|
||||
|
||||
async def ensure_bot_tables(session: AsyncSession) -> None:
|
||||
for stmt in DDL:
|
||||
await session.execute(text(stmt))
|
||||
@@ -113,3 +151,4 @@ async def ensure_schema() -> None:
|
||||
|
||||
async with SessionLocal() as session:
|
||||
await ensure_bot_tables(session)
|
||||
await ensure_obs_reader(session)
|
||||
|
||||
Reference in New Issue
Block a user