cb621ca5d0957117c036c32ba3952b3fc959417a
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 whqreza
|
cb621ca5d0 |
feat(monitoring): multi-model dashboard v2.1 — tabs, per-model records, voice playback + MinIO dual-write
Problem: the dashboard was Jev-centric; NLU/arbiter/fallback DeepSeek, STT
providers and the audio-enhancer were barely visible, and stored texts/voices
had no real access path.
Bot (additive; reply path untouched):
- nlu.result now records latency_ms + the DeepSeek usage tokens
- matching.arbitration/fallback.result fill the model + latency_ms columns
(payload unchanged; SQL coalesces column/payload so old rows still count)
- storage.py: MinIO dual-write layer — every voice (raw + denoised) and user
attachment is mirrored fire-and-forget into bucket bot-media; the blocking
minio SDK runs via asyncio.to_thread (senior-engineer review) so the event
loop is never stalled; empty MINIO_* env = layer off; bucket auto-created
- requirements: minio>=7.2
Backend (admin-gated, read-only):
- GET /bot-metrics/models — one record per model (jev, deepseek-nlu,
deepseek-arbiter, deepseek-fallback, audio-enhancer + dynamic per-provider
STT) with calls/ok-rate/p50/p95/tokens/last-seen + version-drift vs the pin
- GET /bot-metrics/models/{id}/daily — per-model daily calls/fails/tokens/p50
- GET /bot-metrics/stages — every stage tagged with its Persian pipeline
section (صوت/درک پیام/تطبیق/ابهام و انتخاب/پیشنویس)
- /traces now carries input_kind + a 120-char text excerpt (consensus →
input.text → first STT transcript)
- media serving: POST /media-url mints a short-lived HMAC URL (audio/a tags
cannot send Authorization); GET /media streams from MinIO bot-media with
the read-only ./bot/data mount as fallback for pre-MinIO files; hard ref
validation (voices|attachments only, no .., extension whitelist)
- router exempted from the global Bearer dependency (files.py pattern) —
every endpoint keeps its own require_role('admin')
Frontend /bot-monitor (RTL, recharts):
- four tabs: نمای کلی / مدلها (records table + per-model expandable daily
charts: calls/fails, tokens, p50) / بخشها (sections grouped) / پیامها
- message drawer rebuilt: inline <audio controls> for raw + denoised voice,
download links, every STT transcript + consensus side-by-side, user
attachments, then the event timeline
- terminology: «ترس» → «پروندهی پیام» with tooltips + a one-line legend
Grafana: bot-observability gains a «کارکرد مدلها» table (stage→model
mapping for legacy rows); the explorer gains a message text column.
Seed: real (tiny) ogg files written + mirrored to MinIO, NLU usage tokens,
column-level model/latency on matching events.
Verified locally: /models returns all 7 records incl. per-STT and NLU tokens;
media 200 via signed URL (MinIO) and 401 without; ../ and .py rejected (400);
disk fallback serves a legacy file; pm-role 403 on every new endpoint;
bucket auto-created with 14 mirrored objects; tsc + vite build clean;
Grafana re-provisions 24 panels.
|
||
|
whqreza
|
0adadd2cf8 |
feat(monitoring): bot observability dashboards — Grafana + Persian RTL ERP page
Grafana (full use of the existing monitoring/ stack, per OSS policy):
- datasources.yml: BotPostgres (uid bot-postgres) over the shared network;
grafana service joins 'shared' + carries the datasource env
- db_init.py: idempotent read-only role obs_reader (USAGE on schema public +
SELECT on bot_traces/bot_trace_events; skipped when
OBS_READER_PASSWORD empty; ALTER ROLE for rotation) + composite index
ix_bot_trace_events_stage_time (stage, created_at DESC) — senior-engineer
review applied with the column-name fix (event_type -> stage); (trace_id)
already covered by the existing (trace_id, created_at) index
- dashboards/bot-observability.json: KPI row (real traffic, events, Jev/
fallback errors, Jev p95, tokens, commits, archive guard), events/day,
per-stage p50/p95, mention statuses (arbiter+fallback aggregates + legacy
rows), DeepSeek arbiter actions, tokens/day, model-drift vs jev-1.13.0,
user-choice A/B, STT + audio.enhance health, archive size guard
- dashboards/bot-trace-explorer.json: trace table + full event drill-down
via the trace_id variable
ERP page (admin-only, Persian RTL, recharts@^3 for React 19):
- backend bot_metrics.py: read-only require_role('admin') endpoints
(summary/routing/quality/jev/stt/traces + trace detail); aggregates count
real traffic only (calibration/demo/chat_id=0 excluded)
- frontend BotMonitorPage at /bot-monitor (KPI cards, daily events,
status donut, arbiter actions, user-choice A/B, STT/enhance tables,
stage latency, tokens chart, traces table + detail drawer with payload
viewer); entry link from the admin panel; MobileTabBar hidden there
- api.ts: typed botMetrics client section
Tooling/docs:
- tools/seed_observability_demo.py: deterministic demo traces (source='demo',
every current pipeline stage incl. audio.enhance/jev.failure/arbitration/
fallback/user choices) — excluded from real-traffic aggregates
- monitoring/README.md + bot/docs/DEPLOY-AND-MONITORING.md: setup, seed and
rollback notes; alerts deliberately deferred (owner decision 2026-10-24)
Verified locally: obs_reader SELECT ok / INSERT denied; panels' SQL run as
obs_reader with data; datasource health OK; both dashboards survive a grafana
restart; endpoints 200 as admin and 403 without role; tsc + vite build clean;
page renders in-browser (admin session); bot startup provisions role+index.
|
||
|
whqreza
|
76b297f823 |
feat(audio): add pluggable audio preprocessing and denoising layer before STT
- Add audio_enhancer module with 60s timeout, zero-transcoding default, and fail-soft fallback - Persist enhanced audio in daily folders with _enhanced suffix and emit audio.enhance.result event - Connect audio enhancer to transcribe_multi and transcribe in stt.py - Forward message_id and trace_id from handlers - Ensure consensus_text fallback persistence and remove scenario leakage in nlu.py - Add test_enhancer.py unit suite and update smoke_test.py |
||
|
whqreza
|
a61da4a378 |
feat(bot): phase-6 calibration — mock seed, A/B harness, low-trust fallback
- tools/seed_mock.py: idempotent additive mock data (uuid5 ids) covering all tables (units, brands, specs, suppliers, materials + EAV/brand/unit links, projects, warehouses, inventory, quotes with termsheets/line items/decisions, receipts) so phases 6-7 can be exercised end-to-end - tools/calibrate_matching.py: A/B difflib vs Jev on 35 scenarios (31 positive + 4 negative); collects RAW picks with widened bands, classifies against the configured thresholds, sweeps EXACT 0.55-0.95 (negatives count as binds), recommends FLOOR strictly above the highest wrong pick with margin for ±0.05 run-to-run confidence jitter; markdown report + real traces - matching.py: below-FLOOR picks no longer suppress difflib — they log a routing.decision event (source=jev-lowtrust) and leave no result so the existing difflib path keeps its chance (0.54 «پردیس» was lost to «new») - config.py: JEV_CONF_FLOOR 0.60 -> 0.65 (removes the جعبه→بسته @0.61 wrong candidate offer; correct sub-floor picks باق اسپارت/کیلو are difflib-covered) - test_matching.py: +2 low-trust cases (13/13 mock, 16/16 live) - docker-compose.yml: mount bot/tools into the bot container |
||
|
whqreza
|
1c853e50a9 |
feat(matching): Jev entity matching phases 2-5 (per-field fan-out, variants, candidate ambiguity buttons)
Phase 2 — jev.py client: POST /v1/systemone, PINNED model jev-1.13.0 (drift warning when the response model differs), one retry on 429/5xx, fail-soft (None on any error), usage/latency returned for observability. Config adds TYPESAFE_API_KEY / JEV_* (thresholds 0.85/0.60, 15 questions per field); no daily cap by owner decision. bot/.env is now mounted read-only into the container so the key never enters compose/git.
Phase 3 — matching.py: برونگروهی (one request per field, parallel), درونگروهی (one choice question per mention sharing the same full candidate list + «_none»), درونواریانسی (all STT spellings in one question). Routing: conf>=exact → bind; floor..exact → ambiguity; else/_none → new; missing/failed field → difflib fallback (bug fixed during testing: a failed request used to record 'new' instead of falling back). Resolver gains full cached lists for suppliers/materials (paged) plus brands (API+DB) and specs (EAV values + draft specs). NLU prompt now emits entity_mentions (6 fields) with the voice-variant grouping rule; sanitize_mentions guards the shape. handle_message runs matching only for register_quote/update_draft/add_material behind JEV_ENABLED; jev_map threads into quote/material resolve_payload (Jev hit skips difflib; exact writes ids; ambiguity skips difflib). Spec effect per owner: spelling normalization only (normalize_brand_spec).
Phase 4 — entity-match ambiguity sub-messages: create_ambiguities() builds deduped entries {kind: entity_match, field, variants, candidates, details with Persian percentages}; new keyboards.ambiguity_candidates_keyboard renders one button per candidate + «هیچکدام — ایجاد جدید»; callbacks handle amb⛏️{prefix}:{amb_id}:{idx} (applies the candidate via matching.apply_choice then re-resolves/renders) and amb:new (rejected → auto-create path at commit). Every user choice lands in observability (ambiguity.user_choice with arg+candidates).
Phase 5 — voice variants: NLU groups different STT spellings of one entity into one variants[] (verified live: مصطفوی/مستفدی → one supplier mention); tools/test_matching.py live section (RUN_LIVE=1) exercises real DeepSeek + real Jev without touching Bale.
Tests: test_jev.py 5/5 (mock HTTP: no-key, happy path, 429-retry, 400, exception), test_matching.py 11/11 mock + 3/3 live (real NLU produced 7 mentions with correct fields; real Jev matched «50 کیلویی»→«۵۰ کیلویی» 0.98 and «کیسه» 1.0 and correctly returned new against the emptied catalog); observability regression 8/8.
|
||
|
whqreza
|
d58b619107 |
feat(observability): phase-1 silent pipeline recorder + phase-0 Jev validation
Phase 0 — Jev validated with the real key (5 Persian scenarios, api.typesafe.ai): stale project name «منشیانه»->«منشانه» conf 0.73 (ambiguity band), STT spelling variants «مصطفوی/مستفدی/مسطفی»->«بازرگانی مصطفوی» conf 0.93, exact «دفتر» 0.88, proper _none rejection for «کاغذ A4» 0.92, and two mentions in ONE request (0.77/0.64). Exact model version to pin: jev-1.13.0. Latency 0.7-1.1s per field request; ~450-650 in / 64-133 out tokens. Phase 1 — Observability (DB-only, zero extra LLM tokens, nothing is ever deleted): - new tables bot_traces + bot_trace_events (idempotent DDL in db_init.py; ORM in models.py) - new app/observability.py: _jsonable (UUID/datetime/Decimal/set/enum/dataclass-safe), trace_start, event, log_resolution, link_draft, latest_trace_for_draft, set_trace_status, span() context manager, save_voice (Path.mkdir parents+exist_ok, day folders) - capture points wired: input.text; input.voice (+voice persisted under OBS_VOICE_DIR); stt.run + per-provider stt.result (success AND failure, latency, model); nlu.request + nlu.result with prompt_hash/prompt_version; entities.extracted; routing.decision (difflib baseline for the phase-6 A/B); ambiguity.sent; ambiguity.user_choice (buttons + free-text reply path); user_choice.pick; draft.render; draft.edit (wizard-side edits detected by the sync loop); draft.commit (quote/supplier/material/project + trace status/quote id); draft.cancel - compose: persistent ./bot/data:/srv/bot/data volume; bot/.gitignore: data/ (runtime data never committed, never deleted) - tests: bot/tools/test_observability.py -> 8/8 pass (jsonable cases, trace lifecycle, JSONB round-trip, draft linking, status update, save_voice day-folder mkdir, span task). Verified live: voice file lands on the host volume and is gitignored; rows + files persist across bot restart. |
||
|
whqreza
|
638076dbaa |
refactor(bot)!: rename 'ai procurement' -> 'bot/' so the compose build context is committed
The bot service in docker-compose.yml builds from ./bot, but the agent code lived in 'ai procurement/' and ./bot was only a Windows junction on reza.g's machine (and was gitignored) — exactly as the team reported: 'bot/ was never committed — prod-v7's bot is un-buildable from the repo.' - junction removed; 'ai procurement/' renamed to 'bot/' via git mv (all 41 files keep their history as pure renames) - 'bot/' dropped from .gitignore (voice/ stays ignored — runtime voice notes; bot/.env stays ignored by bot/.gitignore) - compose needs no change: context: ./bot and ./bot/app now resolve to committed files - bot/INTEGRATION-REPORT.md added: full story (problem, root cause, git state, security note on the committed BALE_BOT_TOKEN / AI_API_KEY compose defaults, and usage instructions for the team) Verified: 41 renames staged with zero secret/cache files; voice/ and bot/.env still ignored. |