# Monitoring & analytics stack

What runs where, and why — the ERP gets the three missing observability
layers on the same 8 GB VPS:

| Layer | Tool | What it answers | Budget |
|---|---|---|---|
| Product analytics | **OpenPanel** (self-hosted) | Which features are used? Where do users drop out of the quote wizard? Funnel: login → wizard steps → quote submitted → accepted → received. Session replay **off** by design (heavy; revisit with sampling later) | ~1–1.5 GB |
| Error tracking | **GlitchTip** (Sentry-compatible) | What broke, for whom, with which stack trace — frontend (ErrorBoundary + fetch errors) and backend (all unhandled exceptions) | ~750 MB |
| Metrics + logs | **Prometheus + Grafana + Loki/Promtail + node/postgres-exporters + cAdvisor** | Request rate/latency/5xx, container RAM, Postgres connections, disk pressure; full container logs searchable in Grafana | ~1.4 GB |

The monitoring compose is a **separate compose project** — push-to-deploy
rebuilds of the app never restart monitoring containers.

## First-time deployment (VPS)

```bash
cd /srv/erp   # the repo checkout used by the deploy webhook

# 1) monitoring section of .env (see .env.example) — fill in real values:
#    GLITCHTIP_SECRET_KEY, GLITCHTIP_DOMAIN/HOST, GLITCHTIP_DB_PASSWORD,
#    GRAFANA_ADMIN_PASSWORD, POSTGRES_PASSWORD (already set), LOG_FORMAT=json

# 2) the main compose gained a shared network — re-up once so it exists:
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# 3) start monitoring:
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d

# 4) GlitchTip is routed publicly through Traefik for the browser SDK:
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env \
    --profile public up -d glitchtip
```

Admin UIs stay **loopback-only** (same posture as the MinIO console):

```powershell
ssh -L 3001:127.0.0.1:3001 -L 8001:127.0.0.1:8001 <user>@<server-host>
# Grafana  → http://localhost:3001  (dashboards auto-provisioned)
# GlitchTip → http://localhost:8001 (create the first admin on first open)
```

In GlitchTip: create an **organization → project** for the backend and one
for the frontend, copy the DSNs into `/srv/erp/.env`
(`SENTRY_DSN_BACKEND`, `SENTRY_DSN_FRONTEND`) and redeploy the app once.

## Product analytics (OpenPanel)

Deployed from the upstream self-hosting branch (kept outside this repo —
it ships its own compose + update tooling):

```bash
cd /opt
git clone -b self-hosting https://github.com/Openpanel-dev/openpanel.git
cd openpanel/self-hosting
./setup     # interactive
./start
```

Then:

1. Open the OpenPanel dashboard (expose it via Traefik on
   `stats.bennu-api-bahrestan.ir` following its docs — add the container to
   `traefik_network` and give the dashboard + its `/api` path labels),
   create a project → copy the **client id / client secret**.
2. Put them in `/srv/erp/.env`:
   `OPENPANEL_API_URL` (in-network URL of its api service),
   `OPENPANEL_CLIENT_ID`, `OPENPANEL_CLIENT_SECRET`, and
   `OPENPANEL_PUBLIC_URL` (browser-facing origin) — then rebuild the app:
   `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build api frontend`.

What gets tracked automatically once configured:

- **Frontend**: page views, `login_success`, `wizard_step` (funnel across
  the 5 quote-wizard steps), `quote_submitted`, `user_identified`.
- **Backend** (server-side, cannot be blocked by the browser):
  `login_success`, `quote_offered/edited/accepted/arrived/expired/rejected`,
  `receipt_confirmed` — see `backend/app/services/analytics.py`.

Without the env vars everything is a no-op — the app never waits on
analytics (3 s timeout, swallow-all).

## Session replay

Deliberately **off** (records full DOM of every session → storage + privacy
costs; low value for a small internal team at launch). If wanted later:
enable OpenPanel replay with **10–25% sampling** and review retention —
budget +1 GB storage and re-check RAM headroom first.

## Bot observability dashboards (Bale bot)

Two Grafana dashboards read the bot's own tables (`bot_traces` /
`bot_trace_events`) directly — no metrics pipeline, no data duplication:

- **بات — Observability خط لوله** (`bot-observability.json`): KPI row (real
  traffic, events, Jev/fallback errors, Jev p95, tokens, commits, monthly
  archive guard), events/day, per-stage p50/p95 latency, final mention
  statuses (exact/ambiguity/new — arbiter + crisis-fallback aggregates plus
  the legacy per-mention rows), DeepSeek arbiter actions on Jev suggestions,
  tokens/day, model drift vs the pinned `jev-1.13.0`, user-choice A/B proxy,
  STT + audio-enhancement health, archive size guard (>5 GB).
- **بات — اکسپلورر ترس‌ها** (`bot-trace-explorer.json`): latest traces table
  + full event list of one trace (paste its `trace_id` in the dashboard
  variable).

Setup (one-time):

1. `PROCUREMENT_BOT_OBS_READER_PASSWORD=<secret>` in `.env` — the **bot**
   provisions the read-only role `obs_reader` at startup (`USAGE` on schema
   `public` + `SELECT` on the two tables only; empty value = role skipped).
2. Start/restart grafana after the bot ran once — the `BotPostgres` datasource
   (uid `bot-postgres`) is provisioned from `datasources.yml` via the grafana
   service env; grafana reaches postgres over the `shared` (erp_shared)
   network.

Aggregate panels count **real traffic only** (`source` not in
calibration/demo and `chat_id ≠ 0`). To fill every panel on a fresh install,
seed synthetic traces: `docker compose exec bot python
tools/seed_observability_demo.py --days 7` (marked `source='demo'`).

The same data is also available inside the ERP itself (admin-only, Persian
RTL): **/bot-monitor** — served by `GET /api/v1/bot-metrics/*`
(`backend/app/routers/bot_metrics.py`).

## Alerting

`monitoring/prometheus/alerts.yml` defines: API down, 5xx > 5%, p95 > 2 s,
root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in
Grafana alerting; wire a notification channel (email/Telegram) from the
Grafana UI when ready. (Bot observability alerts are deliberately NOT part
of this update — display only, per owner decision 2026-10-24.)

## Lean mode (if RAM gets tight)

Drop prometheus+grafana+loki+promtail+cadvisor+node-exporter (~1.1 GB) and
keep GlitchTip + OpenPanel only; add `uptime-kuma` (~100 MB) for
up/down pings. The app keeps `/health/ready` either way — the deploy
webhook uses it post-rebuild regardless of the metrics stack.

## What the app exposes for this stack

- `GET /health` — liveness (process up).
- `GET /health/ready` — readiness: Postgres + MinIO + Casdoor checks.
  Used by the deploy webhook after every rebuild.
- `GET /metrics` — Prometheus format (unauthenticated, for scraping).
- JSON request logs with `X-Request-ID` (set `LOG_FORMAT=json` in prod)
  → shipped to Loki by Promtail.

## Local runs (Docker Desktop)

Start WITHOUT `--profile host-metrics` — node-exporter and cAdvisor need
Linux rootfs mounts and will fail to start on Windows/macOS hosts:

```bash
docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
```

Per-container and host metrics are absent locally; API metrics, Postgres
metrics, logs, GlitchTip and Grafana all work the same. On the Linux VPS
always include `--profile host-metrics`.