feat(monitoring): multi-model dashboard v2.1 — tabs, per-model records, voice playback + MinIO dual-write

Problem: the dashboard was Jev-centric; NLU/arbiter/fallback DeepSeek, STT providers and the audio-enhancer were barely visible, and stored texts/voices had no real access path. Bot (additive; reply path untouched): - nlu.result now records latency_ms + the DeepSeek usage tokens - matching.arbitration/fallback.result fill the model + latency_ms columns (payload unchanged; SQL coalesces column/payload so old rows still count) - storage.py: MinIO dual-write layer — every voice (raw + denoised) and user attachment is mirrored fire-and-forget into bucket bot-media; the blocking minio SDK runs via asyncio.to_thread (senior-engineer review) so the event loop is never stalled; empty MINIO_* env = layer off; bucket auto-created - requirements: minio>=7.2 Backend (admin-gated, read-only): - GET /bot-metrics/models — one record per model (jev, deepseek-nlu, deepseek-arbiter, deepseek-fallback, audio-enhancer + dynamic per-provider STT) with calls/ok-rate/p50/p95/tokens/last-seen + version-drift vs the pin - GET /bot-metrics/models/{id}/daily — per-model daily calls/fails/tokens/p50 - GET /bot-metrics/stages — every stage tagged with its Persian pipeline section (صوت/درک پیام/تطبیق/ابهام و انتخاب/پیش‌نویس) - /traces now carries input_kind + a 120-char text excerpt (consensus → input.text → first STT transcript) - media serving: POST /media-url mints a short-lived HMAC URL (audio/a tags cannot send Authorization); GET /media streams from MinIO bot-media with the read-only ./bot/data mount as fallback for pre-MinIO files; hard ref validation (voices|attachments only, no .., extension whitelist) - router exempted from the global Bearer dependency (files.py pattern) — every endpoint keeps its own require_role('admin') Frontend /bot-monitor (RTL, recharts): - four tabs: نمای کلی / مدل‌ها (records table + per-model expandable daily charts: calls/fails, tokens, p50) / بخش‌ها (sections grouped) / پیام‌ها - message drawer rebuilt: inline <audio controls> for raw + denoised voice, download links, every STT transcript + consensus side-by-side, user attachments, then the event timeline - terminology: «ترس» → «پرونده‌ی پیام» with tooltips + a one-line legend Grafana: bot-observability gains a «کارکرد مدل‌ها» table (stage→model mapping for legacy rows); the explorer gains a message text column. Seed: real (tiny) ogg files written + mirrored to MinIO, NLU usage tokens, column-level model/latency on matching events. Verified locally: /models returns all 7 records incl. per-STT and NLU tokens; media 200 via signed URL (MinIO) and 401 without; ../ and .py rejected (400); disk fallback serves a legacy file; pm-role 403 on every new endpoint; bucket auto-created with 14 mirrored objects; tsc + vite build clean; Grafana re-provisions 24 panels.
2026-10-25 22:02:40 +03:30
parent 0adadd2cf8
commit cb621ca5d0
18 changed files with 2028 additions and 543 deletions
@@ -152,6 +152,14 @@ class Settings(BaseSettings):
    # postgres user instead). Rotating = change env + restart the bot.
    OBS_READER_PASSWORD: str = ""

+    # Second copy of every bot media file in the ERP's existing MinIO
+    # (dual-write with the on-disk archive; empty endpoint = mirroring off).
+    MINIO_ENDPOINT: str = ""          # e.g. "minio:9000" inside the compose network
+    MINIO_ACCESS_KEY: str = ""
+    MINIO_SECRET_KEY: str = ""
+    MINIO_BUCKET: str = "bot-media"
+    MINIO_SECURE: bool = False
+
    # Jev (TypeSafe) entity matching — phases 2-5. Model version is PINNED
    # (phase 0 resolved jev-1.13.0) so upstream updates cannot silently change
    # decisions; every response's actual model is recorded for drift detection.
@@ -333,8 +333,10 @@ async def match_with_deepseek(
        for v in variants:
            results_map[normalize_text(v)] = entry

-    # Observability: matching.fallback.result
-    await obs.event(trace_id, "matching.fallback.result", ok=True, payload={
+    # Observability: matching.fallback.result (model/latency also as columns so
+    # per-model dashboards aggregate like jev.result's do; payload unchanged)
+    await obs.event(trace_id, "matching.fallback.result", ok=True,
+                    model=settings.LLM_MODEL, latency_ms=latency_ms, payload={
        "latency_ms": latency_ms,
        "usage": usage,
        "statuses": statuses_count,
@@ -540,7 +542,8 @@ async def adjudicate_with_deepseek(
            results_map[normalize_text(v)] = entry

    # Observability: matching.arbitration.result
-    await obs.event(trace_id, "matching.arbitration.result", ok=True, payload={
+    await obs.event(trace_id, "matching.arbitration.result", ok=True,
+                    model=settings.LLM_MODEL, latency_ms=latency_ms, payload={
        "latency_ms": latency_ms,
        "usage": usage,
        "actions": actions_count,
@@ -8,6 +8,7 @@ import hashlib
 import json
 import logging
 import re
+import time

 import httpx

@@ -220,6 +221,7 @@ async def understand(settings: Settings, text: str | list[dict], draft_summary:
    })

    try:
+        t0 = time.perf_counter()
        async with httpx.AsyncClient(timeout=settings.LLM_TIMEOUT_SECONDS) as client:
            resp = await client.post(
                f"{settings.AI_BASE_URL.rstrip('/')}/chat/completions",
@@ -234,29 +236,35 @@ async def understand(settings: Settings, text: str | list[dict], draft_summary:
                    "response_format": {"type": "json_object"},
                },
            )
+        latency_ms = int((time.perf_counter() - t0) * 1000)
        if resp.status_code != 200:
            logger.error("LLM HTTP %s: %s", resp.status_code, resp.text[:300])
            await obs.event(trace_id, "nlu.result", ok=False, model=settings.LLM_MODEL,
+                            latency_ms=latency_ms,
                            payload={"error": f"HTTP {resp.status_code}", "body": resp.text[:300]})
            raise NLUFailureError(f"HTTP {resp.status_code}")
-        content = ((resp.json() or {}).get("choices") or [{}])[0].get("message", {}).get("content", "")
+        resp_json = resp.json() or {}
+        content = ((resp_json.get("choices") or [{}])[0].get("message", {}) or {}).get("content", "")
        parsed = _extract_json(content)
        if not parsed or parsed.get("intent") not in INTENTS:
            logger.warning("unparseable LLM output: %r", content[:200])
            await obs.event(trace_id, "nlu.result", ok=False, model=settings.LLM_MODEL,
+                            latency_ms=latency_ms,
                            payload={"error": "unparseable", "raw": content[:500]})
            raise NLUFailureError("unparseable LLM output")
        parsed.setdefault("patch", {})
        parsed["entity_mentions"] = sanitize_mentions(parsed.get("entity_mentions"))
        if not parsed.get("consensus_text"):
            parsed["consensus_text"] = first_text
-        await obs.event(trace_id, "nlu.result", ok=True, model=settings.LLM_MODEL, payload={
-            "intent": parsed.get("intent"),
-            "confidence": parsed.get("confidence"),
-            "patch": parsed.get("patch"),
-            "consensus_text": parsed.get("consensus_text"),
-            "entity_mentions": parsed.get("entity_mentions"),
-        })
+        await obs.event(trace_id, "nlu.result", ok=True, model=settings.LLM_MODEL,
+                        latency_ms=latency_ms, payload={
+                            "intent": parsed.get("intent"),
+                            "confidence": parsed.get("confidence"),
+                            "patch": parsed.get("patch"),
+                            "consensus_text": parsed.get("consensus_text"),
+                            "entity_mentions": parsed.get("entity_mentions"),
+                            "usage": resp_json.get("usage"),
+                        })
        return parsed
    except Exception as exc:
        if isinstance(exc, NLUFailureError):
@@ -200,6 +200,7 @@ def save_voice(trace_id: str | None, message_id: int, content: bytes,
        base.mkdir(parents=True, exist_ok=True)     # الزام سخت‌سازی: مسیر همیشه ساخته می‌شود
        path = base / f"{trace_id}_{message_id}{ext}"
        path.write_bytes(content)
+        _mirror_bg(path, "audio/ogg")
        return str(path)
    except Exception:
        logger.warning("obs: save_voice failed for trace %s", trace_id, exc_info=True)
@@ -217,7 +218,27 @@ def save_enhanced_voice(trace_id: str | None, message_id: int, content: bytes,
            ext = f".{ext}"
        path = base / f"{trace_id}_{message_id}_enhanced{ext or '.ogg'}"
        path.write_bytes(content)
+        _mirror_bg(path, "audio/ogg")
        return str(path)
    except Exception:
        logger.warning("obs: save_enhanced_voice failed for trace %s", trace_id, exc_info=True)
        return None
+
+
+async def _mirror(path: Path, content_type: str) -> None:
+    """Second copy in MinIO (fire-and-forget; see app/storage.py)."""
+    try:
+        from app import storage
+        ref = storage.media_ref(path)
+        if ref:
+            await storage.mirror_to_minio(ref, path, content_type)
+    except Exception:
+        pass
+
+
+def _mirror_bg(path: Path, content_type: str) -> None:
+    """Sync-context wrapper around the MinIO mirror (never raises)."""
+    try:
+        asyncio.get_running_loop().create_task(_mirror(path, content_type))
+    except RuntimeError:
+        pass
@@ -1,11 +1,24 @@
-"""Persistent storage for draft attachments (proforma files)."""
+"""Persistent storage for draft attachments (proforma files) + MinIO mirroring.
+
+Dual-write policy (senior-engineer review 2026-10-25): every file the bot
+persists (voices, enhanced voices, proforma/user attachments) stays on disk
+under ./bot/data (primary copy, "archive — never delete") and is ALSO
+uploaded fire-and-forget to the ERP's existing MinIO under bucket `bot-media`
+with the SAME key as its path relative to the data dir, e.g.
+`voices/20261025/<trace>_<msg>.ogg` — the dashboard's media endpoint derives
+the object key from that relative path, so nothing extra is stored in payloads.
+"""
 from datetime import datetime, timezone
+import asyncio
 import logging
 import os
 from pathlib import Path
 import re
+import threading
 from uuid import UUID, uuid4

+from app.config import settings
+
 logger = logging.getLogger("bot.storage")


@@ -42,12 +55,102 @@ async def save_proforma_file(
    target_path.write_bytes(content)
    logger.info("Persisted proforma (%d bytes) to %s", len(content), target_path)

+    # Second copy in MinIO (never blocks/fails the bot; disabled when unset)
+    rel = media_ref(target_path)
+    if rel:
+        await mirror_to_minio(rel, target_path,
+                              content_type=mime_type or "application/octet-stream")
+
    return {
        "file_name": clean_name,
        "saved_filename": saved_filename,
        "saved_path": str(target_path.resolve()),
        "relative_path": str(target_path.as_posix()),
+        "media_ref": rel,
        "size_bytes": len(content),
        "mime_type": mime_type or "application/octet-stream",
        "saved_at": datetime.now(timezone.utc).isoformat(),
    }
+
+
+# ---------------------------------------------------------------------------
+# MinIO mirror — second copy of every bot media file
+# ---------------------------------------------------------------------------
+# The official `minio` Python client is synchronous/blocking; every SDK call
+# therefore runs in a worker thread (asyncio.to_thread) and the whole upload
+# is fire-and-forget so a MinIO outage can never stall the reply path.
+
+DATA_DIR = Path("data")  # ./bot/data inside the container (compose volume)
+
+_client = None
+_client_lock = threading.Lock()
+_bg: set = set()
+
+
+def media_ref(path: str | Path) -> str | None:
+    """Relative ref of a file under ./bot/data ('voices/.../x.ogg'), else None.
+
+    This ref doubles as the MinIO object key and as the dashboard's media
+    identifier — files outside the data dir are simply not mirrored/servable.
+    """
+    try:
+        rel = Path(path).resolve().relative_to((DATA_DIR).resolve()).as_posix()
+        return rel if rel and ".." not in rel else None
+    except ValueError:
+        return None
+
+
+def _get_client():
+    """Lazily build the blocking MinIO client (None when unconfigured)."""
+    global _client
+    if not (settings.MINIO_ENDPOINT and settings.MINIO_ACCESS_KEY and settings.MINIO_SECRET_KEY):
+        return None
+    if _client is not None:
+        return _client
+    with _client_lock:
+        if _client is not None:
+            return _client
+        try:
+            from minio import Minio  # imported only when configured
+
+            _client = Minio(settings.MINIO_ENDPOINT,
+                            access_key=settings.MINIO_ACCESS_KEY,
+                            secret_key=settings.MINIO_SECRET_KEY,
+                            secure=settings.MINIO_SECURE)
+        except Exception:
+            logger.exception("storage: minio client init failed — mirroring disabled")
+            _client = None
+    return _client
+
+
+def _upload_sync(key: str, path: Path, content_type: str) -> None:
+    client = _get_client()
+    if client is None:
+        return
+    if not client.bucket_exists(settings.MINIO_BUCKET):
+        client.make_bucket(settings.MINIO_BUCKET)
+        logger.info("storage: created bucket %s", settings.MINIO_BUCKET)
+    client.fput_object(settings.MINIO_BUCKET, key, str(path), content_type=content_type)
+
+
+async def mirror_to_minio(key: str, path: str | Path,
+                          content_type: str = "audio/ogg") -> None:
+    """Fire-and-forget upload of an already-saved disk file (never raises)."""
+    try:
+        p = Path(path)
+        if not p.is_file():
+            logger.warning("storage: mirror skipped — missing file %s", p)
+            return
+
+        async def _run() -> None:
+            try:
+                await asyncio.to_thread(_upload_sync, key, p, content_type)
+                logger.info("storage: mirrored %s -> %s/%s", p.name, settings.MINIO_BUCKET, key)
+            except Exception:
+                logger.warning("storage: minio mirror failed for %s", p.name, exc_info=True)
+
+        task = asyncio.get_running_loop().create_task(_run())
+        _bg.add(task)
+        task.add_done_callback(_bg.discard)
+    except RuntimeError:
+        pass  # no running loop (sync unit tests) — skip silently
@@ -59,13 +59,20 @@
 4. راستی‌آزمایی پایش: `docker compose exec bot python tools/monitor.py --days 1`
   → قیف باید پر شود و رخدادهای همان پیام‌ها در `bot_trace_events` باشد.
 5. چک سهمیه: `usage` در `jev.result` (توکن ورودی/خروجی) + latency p95.
-6. **داشبوردها (۲۰۲۶-۱۰-۲۴):** `PROCUREMENT_BOT_OBS_READER_PASSWORD` را در `.env` ست کنید و
+6. **داشبوردها (به‌روز ۲۰۲۶-۱۰-۲۵، v2.1):** `PROCUREMENT_BOT_OBS_READER_PASSWORD` را در `.env` ست کنید و
   یک بار بات را restart کنید تا نقش فقط‌خواندنی `obs_reader` ساخته شود؛ سپس گرفانا را
   (ری)استارت کنید — دیتاسورس `BotPostgres` و دو داشبورد «بات — Observability خط لوله» و
-   «بات — اکسپلورر ترس‌ها» خودکار provision می‌شوند (جزئیات در `monitoring/README.md`).
-   صفحه‌ی فارسی داخل ERP هم روی مسیر `/bot-monitor` (فقط ادمین) است.
+   «بات — اکسپلورر پیام‌ها» خودکار provision می‌شوند (جزئیات در `monitoring/README.md`).
+   صفحه‌ی فارسی داخل ERP روی `/bot-monitor` (فقط ادمین) چهار تب دارد: نمای کلی،
+   **مدل‌ها** (رکورد و روند زمانی هر مدل: Jev، DeepSeek-NLU، داور، فالبک، هر STT، بهبود صدا)،
+   **بخش‌ها** (مراحل به تفکیک بخش خط‌لوله) و **پیام‌ها** (پرونده‌ی هر پیام با پخش/دانلود ویس و
+   ترنسکریپت همه‌ی موتورها کنار هم).
+   **رسانه (v2.1):** فایل‌های بات دو-نویسی می‌شوند — دیسک (آرشیو) + باکت MinIO `bot-media`
+   (env: `BOT_MINIO_ENDPOINT/BOT_MINIO_BUCKET`؛ خالی = خاموش). سرو پخش/دانلود از MinIO با
+   fallback به دیسک از `/api/v1/bot-metrics/media` با لینک امضاشده‌ی کوتاه‌مدت انجام می‌شود؛
+   سرویس api ولوم فقط‌خواندنی `./bot/data` دارد (ولیم جدید = `up -d --build api` برای recreate).
   برای دیدن همه‌ی پنل‌ها روی نصب تازه: `docker compose exec bot python tools/seed_observability_demo.py --days 7`
-   (داده‌ها با `source='demo'` از آمار ترافیک واقعی جدا می‌مانند).
+   (داده‌ها با `source='demo'` از آمار ترافیک واقعی جدا می‌مانند و در MinIO هم آینه می‌شوند).

 ## د) Rollback (بدون دیپلوی — فقط env + restart)

@@ -3,3 +3,4 @@ sqlalchemy[asyncio]>=2.0.30
 asyncpg>=0.29.0
 pydantic>=2.7.0
 pydantic-settings>=2.2.0
+minio>=7.2.0
@@ -82,8 +82,19 @@ async def one_trace(s, chat_id: int, t0: datetime, kind: str) -> None:
    m = 0.0
    # ---- input
    if kind == "voice":
+        # a real (tiny, silent) ogg so the media player/download path has bytes
+        day = t0.strftime("%Y%m%d")
+        vdir = Path("data/voices") / day
+        vdir.mkdir(parents=True, exist_ok=True)
+        vpath = vdir / f"{tid}_{rng.randrange(1000, 99999)}.ogg"
+        vpath.write_bytes(bytes.fromhex("4f6767530002000000000000000000000001") + b"\x00" * 3000)
        await ev("input.voice", minutes=m, payload={
-            "path": f"data/voices/demo/{tid}.ogg", "size": 21400, "mime": "audio/ogg"})
+            "path": vpath.as_posix(), "size": vpath.stat().st_size, "mime": "audio/ogg"})
+        try:  # second copy in MinIO exactly like the live bot does
+            from app import storage
+            await storage.mirror_to_minio(storage.media_ref(vpath) or "", vpath, "audio/ogg")
+        except Exception:
+            pass
        # audio enhancement (new in 787824a)
        ok_enh = rng.random() < 0.85
        await ev("audio.enhance.result", minutes=(m := m + 0.1), ok=ok_enh,
@@ -111,7 +122,9 @@ async def one_trace(s, chat_id: int, t0: datetime, kind: str) -> None:
    mentions = _mentions(rng.randint(3, 6))
    await ev("nlu.result", minutes=(m := m + 0.9), latency=rng.randrange(1500, 3600),
             model="deepseek-chat", payload={
-                 "intent": "register_quote", "consensus_text": payload_text})
+                 "intent": "register_quote", "consensus_text": payload_text,
+                 "usage": {"input_tokens": rng.randint(600, 2200),
+                           "output_tokens": rng.randint(150, 900)}})
    await ev("entities.extracted", minutes=(m := m + 0.02), payload={"entity_mentions": mentions})

    # ---- Jev scoring + arbitration/fallback (two-stage, 787824a)