feat(monitoring): multi-model dashboard v2.1 — tabs, per-model records, voice playback + MinIO dual-write
Problem: the dashboard was Jev-centric; NLU/arbiter/fallback DeepSeek, STT
providers and the audio-enhancer were barely visible, and stored texts/voices
had no real access path.
Bot (additive; reply path untouched):
- nlu.result now records latency_ms + the DeepSeek usage tokens
- matching.arbitration/fallback.result fill the model + latency_ms columns
(payload unchanged; SQL coalesces column/payload so old rows still count)
- storage.py: MinIO dual-write layer — every voice (raw + denoised) and user
attachment is mirrored fire-and-forget into bucket bot-media; the blocking
minio SDK runs via asyncio.to_thread (senior-engineer review) so the event
loop is never stalled; empty MINIO_* env = layer off; bucket auto-created
- requirements: minio>=7.2
Backend (admin-gated, read-only):
- GET /bot-metrics/models — one record per model (jev, deepseek-nlu,
deepseek-arbiter, deepseek-fallback, audio-enhancer + dynamic per-provider
STT) with calls/ok-rate/p50/p95/tokens/last-seen + version-drift vs the pin
- GET /bot-metrics/models/{id}/daily — per-model daily calls/fails/tokens/p50
- GET /bot-metrics/stages — every stage tagged with its Persian pipeline
section (صوت/درک پیام/تطبیق/ابهام و انتخاب/پیشنویس)
- /traces now carries input_kind + a 120-char text excerpt (consensus →
input.text → first STT transcript)
- media serving: POST /media-url mints a short-lived HMAC URL (audio/a tags
cannot send Authorization); GET /media streams from MinIO bot-media with
the read-only ./bot/data mount as fallback for pre-MinIO files; hard ref
validation (voices|attachments only, no .., extension whitelist)
- router exempted from the global Bearer dependency (files.py pattern) —
every endpoint keeps its own require_role('admin')
Frontend /bot-monitor (RTL, recharts):
- four tabs: نمای کلی / مدلها (records table + per-model expandable daily
charts: calls/fails, tokens, p50) / بخشها (sections grouped) / پیامها
- message drawer rebuilt: inline <audio controls> for raw + denoised voice,
download links, every STT transcript + consensus side-by-side, user
attachments, then the event timeline
- terminology: «ترس» → «پروندهی پیام» with tooltips + a one-line legend
Grafana: bot-observability gains a «کارکرد مدلها» table (stage→model
mapping for legacy rows); the explorer gains a message text column.
Seed: real (tiny) ogg files written + mirrored to MinIO, NLU usage tokens,
column-level model/latency on matching events.
Verified locally: /models returns all 7 records incl. per-STT and NLU tokens;
media 200 via signed URL (MinIO) and 401 without; ../ and .py rejected (400);
disk fallback serves a legacy file; pm-role 403 on every new endpoint;
bucket auto-created with 14 mirrored objects; tsc + vite build clean;
Grafana re-provisions 24 panels.
This commit is contained in:
+104
-1
@@ -1,11 +1,24 @@
|
||||
"""Persistent storage for draft attachments (proforma files)."""
|
||||
"""Persistent storage for draft attachments (proforma files) + MinIO mirroring.
|
||||
|
||||
Dual-write policy (senior-engineer review 2026-10-25): every file the bot
|
||||
persists (voices, enhanced voices, proforma/user attachments) stays on disk
|
||||
under ./bot/data (primary copy, "archive — never delete") and is ALSO
|
||||
uploaded fire-and-forget to the ERP's existing MinIO under bucket `bot-media`
|
||||
with the SAME key as its path relative to the data dir, e.g.
|
||||
`voices/20261025/<trace>_<msg>.ogg` — the dashboard's media endpoint derives
|
||||
the object key from that relative path, so nothing extra is stored in payloads.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import threading
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from app.config import settings
|
||||
|
||||
logger = logging.getLogger("bot.storage")
|
||||
|
||||
|
||||
@@ -42,12 +55,102 @@ async def save_proforma_file(
|
||||
target_path.write_bytes(content)
|
||||
logger.info("Persisted proforma (%d bytes) to %s", len(content), target_path)
|
||||
|
||||
# Second copy in MinIO (never blocks/fails the bot; disabled when unset)
|
||||
rel = media_ref(target_path)
|
||||
if rel:
|
||||
await mirror_to_minio(rel, target_path,
|
||||
content_type=mime_type or "application/octet-stream")
|
||||
|
||||
return {
|
||||
"file_name": clean_name,
|
||||
"saved_filename": saved_filename,
|
||||
"saved_path": str(target_path.resolve()),
|
||||
"relative_path": str(target_path.as_posix()),
|
||||
"media_ref": rel,
|
||||
"size_bytes": len(content),
|
||||
"mime_type": mime_type or "application/octet-stream",
|
||||
"saved_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MinIO mirror — second copy of every bot media file
|
||||
# ---------------------------------------------------------------------------
|
||||
# The official `minio` Python client is synchronous/blocking; every SDK call
|
||||
# therefore runs in a worker thread (asyncio.to_thread) and the whole upload
|
||||
# is fire-and-forget so a MinIO outage can never stall the reply path.
|
||||
|
||||
DATA_DIR = Path("data") # ./bot/data inside the container (compose volume)
|
||||
|
||||
_client = None
|
||||
_client_lock = threading.Lock()
|
||||
_bg: set = set()
|
||||
|
||||
|
||||
def media_ref(path: str | Path) -> str | None:
|
||||
"""Relative ref of a file under ./bot/data ('voices/.../x.ogg'), else None.
|
||||
|
||||
This ref doubles as the MinIO object key and as the dashboard's media
|
||||
identifier — files outside the data dir are simply not mirrored/servable.
|
||||
"""
|
||||
try:
|
||||
rel = Path(path).resolve().relative_to((DATA_DIR).resolve()).as_posix()
|
||||
return rel if rel and ".." not in rel else None
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _get_client():
|
||||
"""Lazily build the blocking MinIO client (None when unconfigured)."""
|
||||
global _client
|
||||
if not (settings.MINIO_ENDPOINT and settings.MINIO_ACCESS_KEY and settings.MINIO_SECRET_KEY):
|
||||
return None
|
||||
if _client is not None:
|
||||
return _client
|
||||
with _client_lock:
|
||||
if _client is not None:
|
||||
return _client
|
||||
try:
|
||||
from minio import Minio # imported only when configured
|
||||
|
||||
_client = Minio(settings.MINIO_ENDPOINT,
|
||||
access_key=settings.MINIO_ACCESS_KEY,
|
||||
secret_key=settings.MINIO_SECRET_KEY,
|
||||
secure=settings.MINIO_SECURE)
|
||||
except Exception:
|
||||
logger.exception("storage: minio client init failed — mirroring disabled")
|
||||
_client = None
|
||||
return _client
|
||||
|
||||
|
||||
def _upload_sync(key: str, path: Path, content_type: str) -> None:
|
||||
client = _get_client()
|
||||
if client is None:
|
||||
return
|
||||
if not client.bucket_exists(settings.MINIO_BUCKET):
|
||||
client.make_bucket(settings.MINIO_BUCKET)
|
||||
logger.info("storage: created bucket %s", settings.MINIO_BUCKET)
|
||||
client.fput_object(settings.MINIO_BUCKET, key, str(path), content_type=content_type)
|
||||
|
||||
|
||||
async def mirror_to_minio(key: str, path: str | Path,
|
||||
content_type: str = "audio/ogg") -> None:
|
||||
"""Fire-and-forget upload of an already-saved disk file (never raises)."""
|
||||
try:
|
||||
p = Path(path)
|
||||
if not p.is_file():
|
||||
logger.warning("storage: mirror skipped — missing file %s", p)
|
||||
return
|
||||
|
||||
async def _run() -> None:
|
||||
try:
|
||||
await asyncio.to_thread(_upload_sync, key, p, content_type)
|
||||
logger.info("storage: mirrored %s -> %s/%s", p.name, settings.MINIO_BUCKET, key)
|
||||
except Exception:
|
||||
logger.warning("storage: minio mirror failed for %s", p.name, exc_info=True)
|
||||
|
||||
task = asyncio.get_running_loop().create_task(_run())
|
||||
_bg.add(task)
|
||||
task.add_done_callback(_bg.discard)
|
||||
except RuntimeError:
|
||||
pass # no running loop (sync unit tests) — skip silently
|
||||
|
||||
Reference in New Issue
Block a user