whqreza
0adadd2cf8
Grafana (full use of the existing monitoring/ stack, per OSS policy):
- datasources.yml: BotPostgres (uid bot-postgres) over the shared network;
grafana service joins 'shared' + carries the datasource env
- db_init.py: idempotent read-only role obs_reader (USAGE on schema public +
SELECT on bot_traces/bot_trace_events; skipped when
OBS_READER_PASSWORD empty; ALTER ROLE for rotation) + composite index
ix_bot_trace_events_stage_time (stage, created_at DESC) — senior-engineer
review applied with the column-name fix (event_type -> stage); (trace_id)
already covered by the existing (trace_id, created_at) index
- dashboards/bot-observability.json: KPI row (real traffic, events, Jev/
fallback errors, Jev p95, tokens, commits, archive guard), events/day,
per-stage p50/p95, mention statuses (arbiter+fallback aggregates + legacy
rows), DeepSeek arbiter actions, tokens/day, model-drift vs jev-1.13.0,
user-choice A/B, STT + audio.enhance health, archive size guard
- dashboards/bot-trace-explorer.json: trace table + full event drill-down
via the trace_id variable
ERP page (admin-only, Persian RTL, recharts@^3 for React 19):
- backend bot_metrics.py: read-only require_role('admin') endpoints
(summary/routing/quality/jev/stt/traces + trace detail); aggregates count
real traffic only (calibration/demo/chat_id=0 excluded)
- frontend BotMonitorPage at /bot-monitor (KPI cards, daily events,
status donut, arbiter actions, user-choice A/B, STT/enhance tables,
stage latency, tokens chart, traces table + detail drawer with payload
viewer); entry link from the admin panel; MobileTabBar hidden there
- api.ts: typed botMetrics client section
Tooling/docs:
- tools/seed_observability_demo.py: deterministic demo traces (source='demo',
every current pipeline stage incl. audio.enhance/jev.failure/arbitration/
fallback/user choices) — excluded from real-traffic aggregates
- monitoring/README.md + bot/docs/DEPLOY-AND-MONITORING.md: setup, seed and
rollback notes; alerts deliberately deferred (owner decision 2026-10-24)
Verified locally: obs_reader SELECT ok / INSERT denied; panels' SQL run as
obs_reader with data; datasource health OK; both dashboards survive a grafana
restart; endpoints 200 as admin and 403 without role; tsc + vite build clean;
page renders in-browser (admin session); bot startup provisions role+index.
145 lines
15 KiB
JSON
145 lines
15 KiB
JSON
{
|
||
"annotations": { "list": [] },
|
||
"editable": true,
|
||
"graphTooltip": 1,
|
||
"refresh": "5m",
|
||
"schemaVersion": 41,
|
||
"tags": ["erp", "bot"],
|
||
"time": { "from": "now-7d", "to": "now" },
|
||
"timezone": "browser",
|
||
"title": "بات — Observability خط لوله",
|
||
"uid": "bot-observability",
|
||
"version": 1,
|
||
"panels": [
|
||
{ "type": "row", "title": "نمای کلی", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 100 },
|
||
{
|
||
"type": "stat", "title": "روندها (ترافیک واقعی)", "id": 1,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 4, "w": 4, "x": 0, "y": 1 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT count(*) AS value FROM bot_traces WHERE coalesce(source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(created_at)" }],
|
||
"fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null } ] } }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "stat", "title": "رخدادها", "id": 2,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 4, "w": 4, "x": 4, "y": 1 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT count(*) AS value FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at)" }]
|
||
},
|
||
{
|
||
"type": "stat", "title": "خطای Jev / فالبک", "id": 3,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 4, "w": 4, "x": 8, "y": 1 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT count(*) AS value FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE (e.stage='jev.failure' OR (e.stage='jev.result' AND NOT e.ok) OR e.stage LIKE 'matching.%.failure') AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at)" }],
|
||
"fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "stat", "title": "p95 تأخیر Jev (ms)", "id": 4,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 4, "w": 4, "x": 12, "y": 1 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT coalesce(round(percentile_cont(0.95) WITHIN GROUP (ORDER BY e.latency_ms)),0) AS value FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='jev.result' AND e.latency_ms IS NOT NULL AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at)" }],
|
||
"fieldConfig": { "defaults": { "unit": "ms" }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "stat", "title": "ثبت نهایی پیشنویس", "id": 5,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 4, "w": 4, "x": 16, "y": 1 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT count(*) AS value FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='draft.commit' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at)" }]
|
||
},
|
||
{
|
||
"type": "stat", "title": "رخداد این ماه (گارد آرشیو ۱۰۰k)", "id": 6,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 4, "w": 4, "x": 20, "y": 1 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT count(*) AS value FROM bot_trace_events WHERE created_at >= date_trunc('month', now())" }],
|
||
"fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 50000 }, { "color": "red", "value": 100000 } ] } }, "overrides": [] }
|
||
},
|
||
|
||
{ "type": "row", "title": "حجم و تأخیر", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, "id": 101 },
|
||
{
|
||
"type": "timeseries", "title": "رخداد در روز (ترافیک واقعی)", "id": 10,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 6 },
|
||
"targets": [{ "refId": "A", "format": "time_series", "rawSql": "SELECT $__timeGroupAlias(e.created_at,'1d'), count(*) AS value FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1 ORDER BY 1" }],
|
||
"fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 60, "lineWidth": 1 } }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "timeseries", "title": "تأخیر p50/p95 به تفکیک مرحله", "id": 11,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 6 },
|
||
"targets": [{ "refId": "A", "format": "time_series", "rawSql": "SELECT $__timeGroupAlias(e.created_at,'1h'), round(percentile_cont(0.5) WITHIN GROUP (ORDER BY e.latency_ms)) AS p50, round(percentile_cont(0.95) WITHIN GROUP (ORDER BY e.latency_ms)) AS p95, e.stage AS metric FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('jev.result','nlu.result','stt.result','audio.enhance.result','matching.arbitration.result') AND e.latency_ms IS NOT NULL AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1, e.stage ORDER BY 1" }],
|
||
"fieldConfig": { "defaults": { "unit": "ms" }, "overrides": [] }
|
||
},
|
||
|
||
{ "type": "row", "title": "تطبیق موجودیت (Jev + داور DeepSeek)", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 }, "id": 102 },
|
||
{
|
||
"type": "piechart", "title": "وضعیت نهایی mentionها (exact/ambiguity/new)", "id": 20,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 15 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT 'exact' AS metric, sum(coalesce((e.payload->'statuses'->>'exact')::int,0)) AS value FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('matching.arbitration.result','matching.fallback.result') AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) UNION ALL SELECT 'ambiguity', sum(coalesce((e.payload->'statuses'->>'ambiguity')::int,0)) FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('matching.arbitration.result','matching.fallback.result') AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) UNION ALL SELECT 'new', sum(coalesce((e.payload->'statuses'->>'new')::int,0)) FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('matching.arbitration.result','matching.fallback.result') AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) UNION ALL SELECT m->>'status', count(*) FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id, jsonb_array_elements(e.payload->'mentions') m WHERE e.stage='routing.decision' AND e.payload->>'source'='jev' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1" }],
|
||
"fieldConfig": { "defaults": {}, "overrides": [] },
|
||
"options": { "pieType": "donut", "legend": { "displayMode": "table", "placement": "right" } }
|
||
},
|
||
{
|
||
"type": "table", "title": "اقدامات داور DeepSeek روی پیشنهاد Jev", "id": 21,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 15 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT coalesce(e.payload->'actions'->>'endorsed_jev','0')::int AS \"تأیید Jev\", coalesce(e.payload->'actions'->>'vetoed_to_new','0')::int AS \"وتو → جدید\", coalesce(e.payload->'actions'->>'demoted_to_ambiguity','0')::int AS \"تنزل → ابهام\", coalesce(e.payload->'actions'->>'other','0')::int AS \"سایر\", count(*) OVER () AS \"فراخوانیها\" FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='matching.arbitration.result' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at)" }],
|
||
"fieldConfig": { "defaults": {}, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "timeseries", "title": "توکن مصرفی در زمان (ورودی/خروجی)", "id": 22,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 15 },
|
||
"targets": [{ "refId": "A", "format": "time_series", "rawSql": "SELECT $__timeGroupAlias(e.created_at,'1d'), sum(coalesce((e.payload->'usage'->>'input_tokens')::bigint,0)) AS input, sum(coalesce((e.payload->'usage'->>'output_tokens')::bigint,0)) AS output FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('jev.result','matching.arbitration.result','matching.fallback.result') AND e.payload->'usage' IS NOT NULL AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1 ORDER BY 1" }],
|
||
"fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 50 } }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "table", "title": "نسخه مدل Jev دیدهشده (پین: jev-1.13.0)", "id": 23,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 6, "w": 8, "x": 0, "y": 23 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT coalesce(e.model,'?') AS \"مدل\", count(*) AS \"تعداد\", max(e.created_at)::text AS \"آخرین بار\" FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='jev.result' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1 ORDER BY 2 DESC" }]
|
||
},
|
||
{
|
||
"type": "timeseries", "title": "فراخوانی مراحل تطبیق (Jev/داور/فالبک)", "id": 24,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 6, "w": 8, "x": 8, "y": 23 },
|
||
"targets": [{ "refId": "A", "format": "time_series", "rawSql": "SELECT $__timeGroupAlias(e.created_at,'1d'), count(*) AS value, e.stage AS metric FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('jev.request','matching.arbitration.result','matching.fallback.result') AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1, e.stage ORDER BY 1" }],
|
||
"fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 40, "stacking": { "mode": "normal" } } }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "stat", "title": "حجم Observability (گارد آرشیو ۵GB)", "id": 25,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 6, "w": 8, "x": 16, "y": 23 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT (pg_total_relation_size('bot_trace_events') + pg_total_relation_size('bot_traces')) AS value" }],
|
||
"fieldConfig": { "defaults": { "unit": "bytes", "color": { "mode": "thresholds" }, "thresholds": { "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 3000000000 }, { "color": "red", "value": 5000000000 } ] } }, "overrides": [] }
|
||
},
|
||
|
||
{ "type": "row", "title": "کیفیت و ورودی", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 29 }, "id": 103 },
|
||
{
|
||
"type": "table", "title": "انتخاب کاربر (A/B واقعی)", "id": 30,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 30 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT payload->>'action' AS \"اقدام\", coalesce(payload->>'arg','-') AS \"آرگومان\", count(*) AS \"تعداد\" FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='ambiguity.user_choice' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1,2 UNION ALL SELECT CASE WHEN (payload->>'idx')::int = 0 THEN 'انتخاب: گزینه اول (پیشنهاد مدل)' ELSE 'انتخاب: گزینه غیراول' END, '-', count(*) FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='user_choice.pick' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1" }]
|
||
},
|
||
{
|
||
"type": "table", "title": "سلامت STT به تفکیک پرووایدر", "id": 31,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 30 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT coalesce(e.payload->>'provider','?') AS \"پرووایدر\", count(*) FILTER (WHERE e.ok) AS \"موفق\", count(*) AS \"کل\", round(100.0*count(*) FILTER (WHERE e.ok)/greatest(count(*),1),1)::text || '%' AS \"نرخ موفقیت\", coalesce(round(percentile_cont(0.5) WITHIN GROUP (ORDER BY e.latency_ms))::text,'—') AS \"p50 (ms)\" FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='stt.result' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1 ORDER BY 2 DESC" }]
|
||
},
|
||
{
|
||
"type": "timeseries", "title": "پیشپردازش صدا (audio.enhance)", "id": 32,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 30 },
|
||
"targets": [{ "refId": "A", "format": "time_series", "rawSql": "SELECT $__timeGroupAlias(e.created_at,'1d'), count(*) FILTER (WHERE e.ok) AS value, 'موفق' AS metric FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='audio.enhance.result' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1 UNION ALL SELECT $__timeGroupAlias(e.created_at,'1d'), count(*) FILTER (WHERE NOT e.ok), 'ناموفق (بازگشت به صدای خام)' FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage='audio.enhance.result' AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1" }],
|
||
"fieldConfig": { "defaults": { "custom": { "drawStyle": "bars", "fillOpacity": 50, "stacking": { "mode": "normal" } } }, "overrides": [] }
|
||
},
|
||
{
|
||
"type": "table", "title": "چرخهی پیشنویس", "id": 33,
|
||
"datasource": { "type": "postgres", "uid": "bot-postgres" },
|
||
"gridPos": { "h": 5, "w": 24, "x": 0, "y": 38 },
|
||
"targets": [{ "refId": "A", "format": "table", "rawSql": "SELECT e.stage AS \"مرحله\", count(*) AS \"تعداد\" FROM bot_trace_events e JOIN bot_traces t ON e.trace_id=t.id WHERE e.stage IN ('draft.render','draft.commit','draft.cancel','ambiguity.sent') AND coalesce(t.source,'bale') NOT IN ('calibration','demo') AND $__timeFilter(e.created_at) GROUP BY 1 ORDER BY 2 DESC" }]
|
||
}
|
||
],
|
||
"templating": { "list": [] }
|
||
}
|