feat(matching): Jev entity matching phases 2-5 (per-field fan-out, variants, candidate ambiguity buttons)

Phase 2 — jev.py client: POST /v1/systemone, PINNED model jev-1.13.0 (drift warning when the response model differs), one retry on 429/5xx, fail-soft (None on any error), usage/latency returned for observability. Config adds TYPESAFE_API_KEY / JEV_* (thresholds 0.85/0.60, 15 questions per field); no daily cap by owner decision. bot/.env is now mounted read-only into the container so the key never enters compose/git.

Phase 3 — matching.py: برون‌گروهی (one request per field, parallel), درون‌گروهی (one choice question per mention sharing the same full candidate list + «_none»), درون‌واریانسی (all STT spellings in one question). Routing: conf>=exact → bind; floor..exact → ambiguity; else/_none → new; missing/failed field → difflib fallback (bug fixed during testing: a failed request used to record 'new' instead of falling back). Resolver gains full cached lists for suppliers/materials (paged) plus brands (API+DB) and specs (EAV values + draft specs). NLU prompt now emits entity_mentions (6 fields) with the voice-variant grouping rule; sanitize_mentions guards the shape. handle_message runs matching only for register_quote/update_draft/add_material behind JEV_ENABLED; jev_map threads into quote/material resolve_payload (Jev hit skips difflib; exact writes ids; ambiguity skips difflib). Spec effect per owner: spelling normalization only (normalize_brand_spec).

Phase 4 — entity-match ambiguity sub-messages: create_ambiguities() builds deduped entries {kind: entity_match, field, variants, candidates, details with Persian percentages}; new keyboards.ambiguity_candidates_keyboard renders one button per candidate + «هیچ‌کدام — ایجاد جدید»; callbacks handle amb⛏️{prefix}:{amb_id}:{idx} (applies the candidate via matching.apply_choice then re-resolves/renders) and amb:new (rejected → auto-create path at commit). Every user choice lands in observability (ambiguity.user_choice with arg+candidates).

Phase 5 — voice variants: NLU groups different STT spellings of one entity into one variants[] (verified live: مصطفوی/مستفدی → one supplier mention); tools/test_matching.py live section (RUN_LIVE=1) exercises real DeepSeek + real Jev without touching Bale.

Tests: test_jev.py 5/5 (mock HTTP: no-key, happy path, 429-retry, 400, exception), test_matching.py 11/11 mock + 3/3 live (real NLU produced 7 mentions with correct fields; real Jev matched «50 کیلویی»→«۵۰ کیلویی» 0.98 and «کیسه» 1.0 and correctly returned new against the emptied catalog); observability regression 8/8.
This commit is contained in:
2026-10-12 21:24:03 +03:30
parent d58b619107
commit 1c853e50a9
16 changed files with 1029 additions and 70 deletions
+31 -1
View File
@@ -25,6 +25,23 @@ def normalize_digits(text: str) -> str:
ZWNJ = "\u200c"
MENTION_FIELDS = {"project", "supplier", "material", "brand", "spec", "unit"}
def sanitize_mentions(raw) -> list[dict]:
"""Keep only valid {field, variants} mention objects from the LLM output."""
out: list[dict] = []
if not isinstance(raw, list):
return out
for m in raw:
if not isinstance(m, dict):
continue
field = str(m.get("field") or "").strip()
variants = [str(v).strip() for v in (m.get("variants") or []) if str(v or "").strip()]
if field in MENTION_FIELDS and variants:
out.append({"field": field, "variants": variants[:6]})
return out
def normalize_text(text: str) -> str:
"""Normalize Persian text for fuzzy matching (ZWNJ, arabic ya/ka, spaces)."""
@@ -60,6 +77,7 @@ SYSTEM_PROMPT = """تو دستیار هوشمند تدارکات یک سیستم
"consensus_text": "بیانیه رسمی، پالایش‌شده، فشرده و بدون تکیه‌کلام از تقاضای تدارکات کاربر",
"intent": یکی از {intents},
"patch": {شکل partial هر فعالیتی که کاربر داده},
"entity_mentions": [{"field": "project|supplier|material|brand|spec|unit", "variants": ["همه‌ی نگارش‌ها"]}],
"confidence": 0.0..1.0
}
@@ -95,6 +113,15 @@ SYSTEM_PROMPT = """تو دستیار هوشمند تدارکات یک سیستم
- اگر درباره پذیرش استعلام است → intent="accept_quote" و در patch.target نام تأمین‌کننده/استعلام.
- اگر درباره اعلام رسیدن اقلام است → intent="arrive_items" و در patch.target نام تأمین‌کننده/استعلام و در patch.arrivals مقادیر.
۶. استخراج موجودیت‌های قابل تطبیق (entity_mentions):
- فیلدهای مجاز: "project" | "supplier" | "material" | "brand" | "spec" | "unit"
- هر نام کالا/تأمین‌کننده/پروژه/برند/واحد و هر مشخصه که در پیام آمده، یک عضو جداگانه است.
- اگر در نسخه‌های مختلف STT همان نام با املاهای متفاوت پیاده شده (مثل «مصطفوی»، «مستفدی»، «مسطفی»)،
همه را در variants همان یک عضو بگذار — نه چند عضو جداگانه.
- برای پیام متنی معمولی، variants یک عضو تک‌عنصر است.
- مشخصه (spec) و برند (brand) جدا از متریال و جدا از هم استخراج شوند.
- وظیفه‌ی تطبیق با موجودیت‌های سیستم بر عهده‌ی تو نیست؛ هیچ نامی را به نزدیک‌ترین نام موجود «حدس» نزن.
شکل patch برای intent="register_quote"/"update_draft":
{"supplier_name": str, "project_name": str, "project_names": [str], "is_official": bool, "notes": str, "expiry_days": int, "expiry_date": "YYYY-MM-DD",
"items": [{"material_name": str, "brand_name": str, "specs": [str], "quantity": number, "unit_name": str, "unit_price": number, "project_name": str, "notes": str}],
@@ -148,7 +175,8 @@ def _extract_json(raw: str) -> dict | None:
async def understand(settings: Settings, text: str | list[dict], draft_summary: str | None = None,
trace_id: str | None = None) -> dict:
"""Return {"consensus_text": ..., "intent": ..., "patch": {...}, "confidence": ...}; never raises."""
fallback = {"consensus_text": "", "intent": "unknown", "patch": {}, "confidence": 0.0}
fallback = {"consensus_text": "", "intent": "unknown", "patch": {},
"entity_mentions": [], "confidence": 0.0}
if not settings.AI_BASE_URL or not settings.AI_API_KEY:
logger.warning("AI provider not configured; cannot understand message")
return fallback
@@ -214,6 +242,7 @@ async def understand(settings: Settings, text: str | list[dict], draft_summary:
payload={"error": "unparseable", "raw": content[:500]})
return fallback
parsed.setdefault("patch", {})
parsed["entity_mentions"] = sanitize_mentions(parsed.get("entity_mentions"))
if not parsed.get("consensus_text"):
parsed["consensus_text"] = first_text
await obs.event(trace_id, "nlu.result", ok=True, model=settings.LLM_MODEL, payload={
@@ -221,6 +250,7 @@ async def understand(settings: Settings, text: str | list[dict], draft_summary:
"confidence": parsed.get("confidence"),
"patch": parsed.get("patch"),
"consensus_text": parsed.get("consensus_text"),
"entity_mentions": parsed.get("entity_mentions"),
})
return parsed
except (httpx.HTTPError, json.JSONDecodeError, KeyError, TypeError):