feat(matching): Jev entity matching phases 2-5 (per-field fan-out, variants, candidate ambiguity buttons)
Phase 2 — jev.py client: POST /v1/systemone, PINNED model jev-1.13.0 (drift warning when the response model differs), one retry on 429/5xx, fail-soft (None on any error), usage/latency returned for observability. Config adds TYPESAFE_API_KEY / JEV_* (thresholds 0.85/0.60, 15 questions per field); no daily cap by owner decision. bot/.env is now mounted read-only into the container so the key never enters compose/git.
Phase 3 — matching.py: برون‌گروهی (one request per field, parallel), درون‌گروهی (one choice question per mention sharing the same full candidate list + «_none»), درون‌واریانسی (all STT spellings in one question). Routing: conf>=exact → bind; floor..exact → ambiguity; else/_none → new; missing/failed field → difflib fallback (bug fixed during testing: a failed request used to record 'new' instead of falling back). Resolver gains full cached lists for suppliers/materials (paged) plus brands (API+DB) and specs (EAV values + draft specs). NLU prompt now emits entity_mentions (6 fields) with the voice-variant grouping rule; sanitize_mentions guards the shape. handle_message runs matching only for register_quote/update_draft/add_material behind JEV_ENABLED; jev_map threads into quote/material resolve_payload (Jev hit skips difflib; exact writes ids; ambiguity skips difflib). Spec effect per owner: spelling normalization only (normalize_brand_spec).
Phase 4 — entity-match ambiguity sub-messages: create_ambiguities() builds deduped entries {kind: entity_match, field, variants, candidates, details with Persian percentages}; new keyboards.ambiguity_candidates_keyboard renders one button per candidate + «هیچ‌کدام — ایجاد جدید»; callbacks handle amb⛏️{prefix}:{amb_id}:{idx} (applies the candidate via matching.apply_choice then re-resolves/renders) and amb:new (rejected → auto-create path at commit). Every user choice lands in observability (ambiguity.user_choice with arg+candidates).
Phase 5 — voice variants: NLU groups different STT spellings of one entity into one variants[] (verified live: مصطفوی/مستفدی → one supplier mention); tools/test_matching.py live section (RUN_LIVE=1) exercises real DeepSeek + real Jev without touching Bale.
Tests: test_jev.py 5/5 (mock HTTP: no-key, happy path, 429-retry, 400, exception), test_matching.py 11/11 mock + 3/3 live (real NLU produced 7 mentions with correct fields; real Jev matched «50 کیلویی»→«۵۰ کیلویی» 0.98 and «کیسه» 1.0 and correctly returned new against the emptied catalog); observability regression 8/8.
This commit is contained in:
@@ -0,0 +1,321 @@
|
||||
"""Entity matching with Jev (TypeSafe) — phases 3-5 of the final plan.
|
||||
|
||||
Mechanics (final plan v5 §2):
|
||||
- برون‌گروهی: ONE Jev request per FIELD that has mentions (requests run in
|
||||
parallel; latency ≈ one round-trip regardless of field count).
|
||||
- درون‌گروهی: inside a field request, ONE `choice` question per mention; every
|
||||
question rides with the SAME full candidate list (+ a `_none` option).
|
||||
- درون‌واریانسی: all STT spellings of one entity travel in ONE question, and
|
||||
the instructions ask the model to treat them as one name.
|
||||
|
||||
Routing (thresholds from env; calibrated in phase 6):
|
||||
conf >= JEV_CONF_EXACT → exact (bind id, no ambiguity sub-message)
|
||||
JEV_CONF_FLOOR..JEV_CONF_EXACT → ambiguity (candidate buttons sub-message)
|
||||
below floor / _none → new (existing auto-create path)
|
||||
|
||||
Fail-soft: a failed field simply yields no results for its mentions; callers
|
||||
fall back to the difflib path (`safe_resolve`). With JEV_ENABLED=false nothing
|
||||
in this module runs (zero regression).
|
||||
"""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
|
||||
from app import observability as obs
|
||||
from app.config import settings
|
||||
from app.jev import ask as jev_ask
|
||||
from app.nlu import normalize_text
|
||||
|
||||
logger = logging.getLogger("bot.matching")
|
||||
|
||||
FIELD_ORDER = ["project", "supplier", "material", "brand", "spec", "unit"]
|
||||
|
||||
FIELD_FA = {
|
||||
"project": "پروژه",
|
||||
"supplier": "تأمین‌کننده",
|
||||
"material": "متریال",
|
||||
"brand": "برند",
|
||||
"spec": "مشخصه",
|
||||
"unit": "واحد",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# candidate lists (full lists — no prefilter, per owner decision)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _candidates(resolver, field: str) -> list:
|
||||
if field == "project":
|
||||
return await resolver.projects()
|
||||
if field == "supplier":
|
||||
return await resolver.all_suppliers()
|
||||
if field == "material":
|
||||
return await resolver.all_materials()
|
||||
if field == "brand":
|
||||
return await resolver.brands()
|
||||
if field == "spec":
|
||||
return await resolver.specs()
|
||||
if field == "unit":
|
||||
return await resolver.units()
|
||||
return []
|
||||
|
||||
|
||||
def _hint(field: str, cand) -> str:
|
||||
"""Short, token-cheap description shown next to a candidate name."""
|
||||
if field == "supplier":
|
||||
extra = cand.extra or {}
|
||||
bits = [b for b in (extra.get("category_name"), extra.get("location")) if b]
|
||||
return "، ".join(str(b) for b in bits) if bits else "تأمین‌کننده"
|
||||
return ""
|
||||
|
||||
|
||||
def _instructions(field: str, variants: list[str]) -> str:
|
||||
fa = FIELD_FA.get(field, field)
|
||||
vlist = "»، «".join(v for v in variants if v)
|
||||
if len(variants) > 1:
|
||||
return (
|
||||
f"کاربر در پیام (صوتی) خود این {fa} را ذکر کرده و مدل‌های مختلف STT آن را به شکل‌های "
|
||||
f"«{vlist}» پیاده کرده‌اند؛ این‌ها چند نگارش از یک نام واحدند. با توجه به بافت جمله و "
|
||||
f"تشابه آوایی میان این واریانت‌ها، مشخص کن به کدام گزینه از فهرست اشاره دارد. "
|
||||
f"اگر هیچ‌کدام منطبق نیست یا تردید جدی داری «_none» را انتخاب کن."
|
||||
)
|
||||
return (
|
||||
f"کاربر «{vlist}» را به‌عنوان {fa} ذکر کرده است. این عبارت به کدام گزینه از فهرست اشاره دارد؟ "
|
||||
f"اگر هیچ‌کدام منطبق نیست یا تردید جدی داری «_none» را انتخاب کن."
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the per-field fan-out
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def resolve_mentions(ctx, patch: dict, mentions: list[dict], trace_id) -> dict:
|
||||
"""Run Jev per field (parallel); return a map keyed by normalized variant.
|
||||
|
||||
Each value: {"field", "status": exact|ambiguity|new", "variants",
|
||||
"id"?, "name"?, "extra"?, "confidence"?, "candidates": [...]}
|
||||
"""
|
||||
by_field: dict[str, list[dict]] = {}
|
||||
for m in mentions or []:
|
||||
field = str((m or {}).get("field") or "").strip()
|
||||
variants = [str(v).strip() for v in ((m or {}).get("variants") or []) if str(v or "").strip()]
|
||||
if field in FIELD_ORDER and variants:
|
||||
by_field.setdefault(field, []).append({"field": field, "variants": variants[:6]})
|
||||
|
||||
results: dict[str, dict] = {}
|
||||
if not by_field:
|
||||
return results
|
||||
|
||||
async def one_field(field: str, ms: list[dict]) -> None:
|
||||
try:
|
||||
cands = await _candidates(ctx.resolver, field)
|
||||
except Exception:
|
||||
logger.exception("matching: candidate fetch failed for %s", field)
|
||||
return
|
||||
if not cands:
|
||||
return
|
||||
|
||||
by_name: dict[str, object] = {}
|
||||
criteria: dict[str, str] = {}
|
||||
for c in cands:
|
||||
if c.name in criteria:
|
||||
continue
|
||||
criteria[c.name] = _hint(field, c)
|
||||
by_name[c.name] = c
|
||||
criteria["_none"] = "هیچ‌کدام از موارد موجود نیست"
|
||||
|
||||
capped = ms[: settings.JEV_MAX_QUESTIONS_PER_FIELD]
|
||||
questions = {
|
||||
f"mention_{i}": {
|
||||
"type": "choice",
|
||||
"instructions": _instructions(field, m["variants"]),
|
||||
"criteria": criteria,
|
||||
}
|
||||
for i, m in enumerate(capped)
|
||||
}
|
||||
state = {
|
||||
"field": field,
|
||||
"patch_context": json.dumps(patch, ensure_ascii=False, default=str)[:1500],
|
||||
}
|
||||
await obs.event(trace_id, "jev.request", field=field, payload={
|
||||
"n_mentions": len(capped), "n_candidates": len(cands),
|
||||
})
|
||||
resp = await jev_ask(state, questions)
|
||||
await obs.event(trace_id, "jev.result", field=field, ok=resp is not None,
|
||||
model=(resp or {}).get("model"),
|
||||
latency_ms=(resp or {}).get("latency_ms"),
|
||||
payload={"answers": (resp or {}).get("answers"),
|
||||
"usage": (resp or {}).get("usage")})
|
||||
if resp is None:
|
||||
# Field-wide failure → leave no results so callers fall back to difflib.
|
||||
return
|
||||
answers = resp.get("answers") or {}
|
||||
|
||||
decisions = []
|
||||
for i, m in enumerate(capped):
|
||||
ans = answers.get(f"mention_{i}") or {}
|
||||
if not ans:
|
||||
# No answer for this mention → difflib fallback for it.
|
||||
continue
|
||||
choice = ans.get("choice")
|
||||
conf = float(ans.get("confidence") or 0.0)
|
||||
probs = ans.get("probabilities") or {}
|
||||
cand = by_name.get(choice) if choice else None
|
||||
ranked = sorted(
|
||||
({"id": by_name[n].id, "name": n, "prob": float(p or 0.0)}
|
||||
for n, p in probs.items() if n != "_none" and n in by_name),
|
||||
key=lambda x: -x["prob"],
|
||||
)[:6]
|
||||
|
||||
if cand is not None and conf >= settings.JEV_CONF_EXACT:
|
||||
res = {"field": field, "status": "exact", "variants": m["variants"],
|
||||
"id": cand.id, "name": cand.name, "extra": cand.extra or {},
|
||||
"confidence": conf}
|
||||
elif cand is not None and conf >= settings.JEV_CONF_FLOOR:
|
||||
res = {"field": field, "status": "ambiguity", "variants": m["variants"],
|
||||
"id": cand.id, "name": cand.name, "extra": cand.extra or {},
|
||||
"confidence": conf,
|
||||
"candidates": ranked or [{"id": cand.id, "name": cand.name, "prob": conf}]}
|
||||
else:
|
||||
res = {"field": field, "status": "new", "variants": m["variants"],
|
||||
"confidence": conf, "candidates": ranked}
|
||||
for v in m["variants"]:
|
||||
results[normalize_text(v)] = res
|
||||
decisions.append({"variants": m["variants"], "status": res["status"],
|
||||
"choice": res.get("name"), "confidence": conf,
|
||||
"probs_top": [{k: round(v, 3)} for k, v in
|
||||
sorted(probs.items(), key=lambda kv: -kv[1])[:3]]})
|
||||
await obs.event(trace_id, "routing.decision", field=field,
|
||||
payload={"source": "jev", "mentions": decisions})
|
||||
|
||||
await asyncio.gather(*(one_field(f, ms) for f, ms in by_field.items()))
|
||||
return results
|
||||
|
||||
|
||||
def lookup(jev_map: dict | None, name: str | None) -> dict | None:
|
||||
"""Find the Jev result for a raw name (by normalized variant)."""
|
||||
if not jev_map or not name:
|
||||
return None
|
||||
return jev_map.get(normalize_text(name))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# writing results into drafts
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def apply_choice(payload: dict, field: str, cand: dict, variants: list[str]) -> None:
|
||||
"""Bind one chosen candidate (exact match or a user pick) into the payload."""
|
||||
name, cid = cand.get("name"), cand.get("id")
|
||||
extra = cand.get("extra") or {}
|
||||
variants_norm = {normalize_text(v) for v in (variants or []) if v}
|
||||
|
||||
if field == "supplier":
|
||||
payload["supplier_id"] = cid
|
||||
payload["supplier_name"] = name
|
||||
for k in ("location", "phone", "contact_person", "category_name"):
|
||||
if extra.get(k):
|
||||
payload[f"supplier_{k}"] = extra[k]
|
||||
elif field == "project":
|
||||
resolved = payload.setdefault("resolved_projects", {})
|
||||
for v in (variants or []):
|
||||
resolved[v] = cid
|
||||
resolved[name] = cid
|
||||
if cid not in payload.get("project_ids", []):
|
||||
payload.setdefault("project_ids", []).append(cid)
|
||||
if not payload.get("project_id"):
|
||||
payload["project_id"] = cid
|
||||
elif field == "material":
|
||||
for item in payload.get("items") or []:
|
||||
if normalize_text(item.get("material_name") or "") in variants_norm:
|
||||
item["material_name"] = name
|
||||
item["material_id"] = cid
|
||||
item["material_row"] = {"id": cid,
|
||||
"canonical_unit_id": extra.get("canonical_unit_id")}
|
||||
elif field == "unit":
|
||||
for item in payload.get("items") or []:
|
||||
if normalize_text(item.get("unit_name") or "") in variants_norm:
|
||||
item["unit_id"] = cid
|
||||
item["unit_explicit"] = True
|
||||
item["unit_name"] = name
|
||||
if not payload.get("items"):
|
||||
payload["canonical_unit_id"] = cid
|
||||
payload["unit_name"] = name
|
||||
elif field == "brand":
|
||||
for item in payload.get("items") or []:
|
||||
if normalize_text(item.get("brand_name") or "") in variants_norm:
|
||||
item["brand_name"] = name
|
||||
elif field == "spec":
|
||||
for item in payload.get("items") or []:
|
||||
specs = item.get("specs") or []
|
||||
for i, s in enumerate(specs):
|
||||
if normalize_text(s) in variants_norm:
|
||||
specs[i] = name
|
||||
|
||||
|
||||
def apply_exact(payload: dict, res: dict) -> None:
|
||||
"""Bind one exact Jev match into a draft payload."""
|
||||
apply_choice(payload, res.get("field"), {"id": res.get("id"), "name": res.get("name"),
|
||||
"extra": res.get("extra") or {}},
|
||||
res.get("variants") or [])
|
||||
|
||||
|
||||
def normalize_brand_spec(payload: dict, jev_map: dict | None) -> None:
|
||||
"""Canonicalize brand/spec strings via exact matches (spec effect per owner:
|
||||
spelling normalization only)."""
|
||||
if not jev_map:
|
||||
return
|
||||
for item in payload.get("items") or []:
|
||||
brand = item.get("brand_name")
|
||||
if brand:
|
||||
hit = lookup(jev_map, brand)
|
||||
if hit and hit.get("status") == "exact":
|
||||
item["brand_name"] = hit["name"]
|
||||
specs = item.get("specs") or []
|
||||
for i, s in enumerate(specs):
|
||||
hit = lookup(jev_map, s)
|
||||
if hit and hit.get("status") == "exact":
|
||||
specs[i] = hit["name"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# phase 4 — ambiguity sub-messages with candidate buttons
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _amb_id(field: str, variants: list[str]) -> str:
|
||||
digest = hashlib.sha1("|".join(sorted(variants)).encode()).hexdigest()[:4]
|
||||
return f"em_{field[:4]}_{digest}"
|
||||
|
||||
|
||||
def _fa_num(n: int) -> str:
|
||||
return str(n).translate(str.maketrans("0123456789", "۰۱۲۳۴۵۶۷۸۹"))
|
||||
|
||||
|
||||
def create_ambiguities(payload: dict, jev_map: dict | None) -> None:
|
||||
"""One `entity_match` ambiguity entry per ambiguous mention (deduped)."""
|
||||
if not jev_map:
|
||||
return
|
||||
ambs = payload.setdefault("ambiguities", [])
|
||||
seen = {a.get("id") for a in ambs}
|
||||
unique: dict[int, dict] = {}
|
||||
for res in jev_map.values():
|
||||
unique[id(res)] = res # dedupe identical result objects
|
||||
for res in unique.values():
|
||||
if res.get("status") != "ambiguity" or not res.get("candidates"):
|
||||
continue
|
||||
amb_id = _amb_id(res["field"], res.get("variants") or [])
|
||||
if amb_id in seen:
|
||||
continue
|
||||
cands = res["candidates"][:6]
|
||||
lines = [f"• {c['name']} — {_fa_num(int(round(c['prob'] * 100)))}٪" for c in cands]
|
||||
ambs.append({
|
||||
"id": amb_id,
|
||||
"kind": "entity_match",
|
||||
"field": res["field"],
|
||||
"variants": res.get("variants") or [],
|
||||
"candidates": [{"id": c["id"], "name": c["name"]} for c in cands],
|
||||
"title": f"تطبیق {FIELD_FA.get(res['field'], res['field'])}: «{' / '.join(res.get('variants') or [])}»",
|
||||
"details": "\n".join(lines),
|
||||
"status": "pending",
|
||||
})
|
||||
seen.add(amb_id)
|
||||
Reference in New Issue
Block a user