feat(monitoring): observability stack + app instrumentation

- /health/ready (DB+MinIO+Casdoor checks, webhook post-deploy gate), /metrics via prometheus-fastapi-instrumentator, Sentry/GlitchTip capture, X-Request-ID middleware + JSON access logging (PROCUREMENT_LOG_FORMAT) - server-side analytics (OpenPanel Track API): login, quote lifecycle, receipts — fire-and-forget, disabled without env config - monitoring/ compose project: GlitchTip(+pg), Prometheus(+alerts), Grafana(provisioned ERP dashboard), Loki+Promtail, node/postgres exporters, cAdvisor — mem-limited (~2.2GB), loopback admin UIs, separate from app deploys; erp_shared network for scraping - frontend: @openpanel/web + @sentry/react (build-arg gated, no-op when unset), funnel events (login, wizard steps, quote submitted) - compose: api healthcheck, observability env wiring, .env.example keys
2026-09-18 22:18:02 +03:30
parent f7b1f7f751
commit e17d1f0883
26 changed files with 1106 additions and 3 deletions
@@ -0,0 +1,111 @@
+# Monitoring & analytics stack
+
+What runs where, and why — the ERP gets the three missing observability
+layers on the same 8 GB VPS:
+
+| Layer | Tool | What it answers | Budget |
+|---|---|---|---|
+| Product analytics | **OpenPanel** (self-hosted) | Which features are used? Where do users drop out of the quote wizard? Funnel: login → wizard steps → quote submitted → accepted → received. Session replay **off** by design (heavy; revisit with sampling later) | ~1–1.5 GB |
+| Error tracking | **GlitchTip** (Sentry-compatible) | What broke, for whom, with which stack trace — frontend (ErrorBoundary + fetch errors) and backend (all unhandled exceptions) | ~750 MB |
+| Metrics + logs | **Prometheus + Grafana + Loki/Promtail + node/postgres-exporters + cAdvisor** | Request rate/latency/5xx, container RAM, Postgres connections, disk pressure; full container logs searchable in Grafana | ~1.4 GB |
+
+The monitoring compose is a **separate compose project** — push-to-deploy
+rebuilds of the app never restart monitoring containers.
+
+## First-time deployment (VPS)
+
+```bash
+cd /srv/erp   # the repo checkout used by the deploy webhook
+
+# 1) monitoring section of .env (see .env.example) — fill in real values:
+#    GLITCHTIP_SECRET_KEY, GLITCHTIP_DOMAIN/HOST, GLITCHTIP_DB_PASSWORD,
+#    GRAFANA_ADMIN_PASSWORD, POSTGRES_PASSWORD (already set), LOG_FORMAT=json
+
+# 2) the main compose gained a shared network — re-up once so it exists:
+docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
+
+# 3) start monitoring:
+docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
+
+# 4) GlitchTip is routed publicly through Traefik for the browser SDK:
+docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env \
+    --profile public up -d glitchtip
+```
+
+Admin UIs stay **loopback-only** (same posture as the MinIO console):
+
+```powershell
+ssh -L 3001:127.0.0.1:3001 -L 8001:127.0.0.1:8001 <user>@<server-host>
+# Grafana  → http://localhost:3001  (dashboards auto-provisioned)
+# GlitchTip → http://localhost:8001 (create the first admin on first open)
+```
+
+In GlitchTip: create an **organization → project** for the backend and one
+for the frontend, copy the DSNs into `/srv/erp/.env`
+(`SENTRY_DSN_BACKEND`, `SENTRY_DSN_FRONTEND`) and redeploy the app once.
+
+## Product analytics (OpenPanel)
+
+Deployed from the upstream self-hosting branch (kept outside this repo —
+it ships its own compose + update tooling):
+
+```bash
+cd /opt
+git clone -b self-hosting https://github.com/Openpanel-dev/openpanel.git
+cd openpanel/self-hosting
+./setup     # interactive
+./start
+```
+
+Then:
+
+1. Open the OpenPanel dashboard (expose it via Traefik on
+   `stats.bennu-api-bahrestan.ir` following its docs — add the container to
+   `traefik_network` and give the dashboard + its `/api` path labels),
+   create a project → copy the **client id / client secret**.
+2. Put them in `/srv/erp/.env`:
+   `OPENPANEL_API_URL` (in-network URL of its api service),
+   `OPENPANEL_CLIENT_ID`, `OPENPANEL_CLIENT_SECRET`, and
+   `OPENPANEL_PUBLIC_URL` (browser-facing origin) — then rebuild the app:
+   `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build api frontend`.
+
+What gets tracked automatically once configured:
+
+- **Frontend**: page views, `login_success`, `wizard_step` (funnel across
+  the 5 quote-wizard steps), `quote_submitted`, `user_identified`.
+- **Backend** (server-side, cannot be blocked by the browser):
+  `login_success`, `quote_offered/edited/accepted/arrived/expired/rejected`,
+  `receipt_confirmed` — see `backend/app/services/analytics.py`.
+
+Without the env vars everything is a no-op — the app never waits on
+analytics (3 s timeout, swallow-all).
+
+## Session replay
+
+Deliberately **off** (records full DOM of every session → storage + privacy
+costs; low value for a small internal team at launch). If wanted later:
+enable OpenPanel replay with **10–25% sampling** and review retention —
+budget +1 GB storage and re-check RAM headroom first.
+
+## Alerting
+
+`monitoring/prometheus/alerts.yml` defines: API down, 5xx > 5%, p95 > 2 s,
+root disk < 10%, host RAM < 10%, Postgres > 80 connections. They appear in
+Grafana alerting; wire a notification channel (email/Telegram) from the
+Grafana UI when ready.
+
+## Lean mode (if RAM gets tight)
+
+Drop prometheus+grafana+loki+promtail+cadvisor+node-exporter (~1.1 GB) and
+keep GlitchTip + OpenPanel only; add `uptime-kuma` (~100 MB) for
+up/down pings. The app keeps `/health/ready` either way — the deploy
+webhook uses it post-rebuild regardless of the metrics stack.
+
+## What the app exposes for this stack
+
+- `GET /health` — liveness (process up).
+- `GET /health/ready` — readiness: Postgres + MinIO + Casdoor checks.
+  Used by the deploy webhook after every rebuild.
+- `GET /metrics` — Prometheus format (unauthenticated, for scraping).
+- JSON request logs with `X-Request-ID` (set `LOG_FORMAT=json` in prod)
+  → shipped to Loki by Promtail.
@@ -0,0 +1,191 @@
+# ============================================================================
+# Monitoring stack — SEPARATE compose project from the ERP app on purpose:
+# app rebuilds (push-to-deploy) must never restart monitoring containers.
+#
+#   cd /srv/erp   # repo root, so --env-file picks up the app's .env
+#   docker compose -f monitoring/docker-compose.monitoring.yml --env-file .env up -d
+#
+# Included (≈2.2 GB RAM budgeted):
+#   GlitchTip (errors, Sentry-compatible) + its own small Postgres
+#   Prometheus (metrics) + Grafana (dashboards) + Loki/Promtail (logs)
+#   node-exporter (host) + postgres-exporter (app DB) + cAdvisor (containers)
+#
+# Admin UIs (Grafana :3001, GlitchTip :8001) bind to 127.0.0.1 only —
+# reach them via SSH tunnel exactly like the MinIO console. GlitchTip ALSO
+# needs a public origin for the browser SDK: set GLITCHTIP_HOST and enable
+# the "public" profile (Traefik route + TLS) — see monitoring/README.md.
+# ============================================================================
+
+services:
+  # ---------------------------------------------------------------- errors --
+  glitchtip:
+    image: glitchtip/glitchtip:latest
+    container_name: monitoring-glitchtip
+    mem_limit: 640m
+    environment:
+      SECRET_KEY: ${GLITCHTIP_SECRET_KEY:?set GLITCHTIP_SECRET_KEY in .env}
+      GLITCHTIP_DOMAIN: ${GLITCHTIP_DOMAIN:?set GLITCHTIP_DOMAIN in .env}
+      DATABASE_URL: postgres://glitchtip:${GLITCHTIP_DB_PASSWORD:-glitchtip}@glitchtip-postgres:5432/glitchtip
+      # Single-container mode (embedded worker) — no redis needed at this scale.
+      GLITCHTIP_EMBED_WORKER: "true"
+      VALKEY_URL: ""
+      ENABLE_USER_REGISTRATION: "false"
+      ENABLE_ORGANIZATION_CREATION: "false"
+      DEFAULT_FROM_EMAIL: ${GLITCHTIP_FROM_EMAIL:-noreply@bennu-api-bahrestan.ir}
+      EMAIL_URL: ${GLITCHTIP_EMAIL_URL:-consolemail://}
+    ports:
+      - "127.0.0.1:8001:8000"
+    depends_on:
+      glitchtip-postgres:
+        condition: service_healthy
+    networks:
+      - monitoring
+      - shared
+      - traefik # public DSN route (profile "public")
+    labels:
+      - traefik.enable=true
+      - traefik.http.routers.glitchtip.rule=Host(`${GLITCHTIP_HOST:-errors.localhost}`)
+      - traefik.http.routers.glitchtip.entrypoints=web
+      - traefik.http.services.glitchtip.loadbalancer.server.port=8000
+    restart: unless-stopped
+
+  glitchtip-postgres:
+    image: postgres:16-alpine
+    container_name: monitoring-glitchtip-postgres
+    mem_limit: 128m
+    environment:
+      POSTGRES_DB: glitchtip
+      POSTGRES_USER: glitchtip
+      POSTGRES_PASSWORD: ${GLITCHTIP_DB_PASSWORD:-glitchtip}
+    volumes:
+      - glitchtip_postgres_data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U glitchtip -d glitchtip"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+  # --------------------------------------------------------------- metrics --
+  prometheus:
+    image: prom/prometheus:latest
+    container_name: monitoring-prometheus
+    mem_limit: 512m
+    command:
+      - --config.file=/etc/prometheus/prometheus.yml
+      - --storage.tsdb.retention.time=14d
+      - --storage.tsdb.retention.size=2GB
+    volumes:
+      - ./prometheus:/etc/prometheus:ro
+      - prometheus_data:/prometheus
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+  grafana:
+    image: grafana/grafana:latest
+    container_name: monitoring-grafana
+    mem_limit: 256m
+    environment:
+      GF_SECURITY_ADMIN_USER: admin
+      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD in .env}
+      GF_USERS_ALLOW_SIGN_UP: "false"
+      GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3001/}
+    volumes:
+      - grafana_data:/var/lib/grafana
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
+    ports:
+      - "127.0.0.1:3001:3000"
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+  # ------------------------------------------------------------------ logs --
+  loki:
+    image: grafana/loki:latest
+    container_name: monitoring-loki
+    mem_limit: 384m
+    command: -config.file=/etc/loki/loki-config.yml
+    volumes:
+      - ./loki/loki-config.yml:/etc/loki/loki-config.yml:ro
+      - loki_data:/loki
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+  promtail:
+    image: grafana/promtail:latest
+    container_name: monitoring-promtail
+    mem_limit: 64m
+    command: -config.file=/etc/promtail/promtail-config.yml
+    volumes:
+      - ./promtail/promtail-config.yml:/etc/promtail/promtail-config.yml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+  # -------------------------------------------------------------- exporters --
+  node-exporter:
+    image: prom/node-exporter:latest
+    container_name: monitoring-node-exporter
+    mem_limit: 32m
+    command:
+      - --path.rootfs=/host
+    volumes:
+      - /:/host:ro,rslave
+    pid: host
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+  postgres-exporter:
+    image: prometheuscommunity/postgres-postgres-exporter:latest
+    container_name: monitoring-postgres-exporter
+    mem_limit: 48m
+    environment:
+      # App database on the main stack's postgres (reaches it via erp_shared).
+      DATA_SOURCE_NAME: postgres://postgres:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/procurement?sslmode=disable
+    networks:
+      - monitoring
+      - shared
+    restart: unless-stopped
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    container_name: monitoring-cadvisor
+    mem_limit: 128m
+    privileged: true
+    devices:
+      - /dev/kmsg
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+    networks:
+      - monitoring
+    restart: unless-stopped
+
+networks:
+  monitoring:
+    driver: bridge
+  # Reaches the ERP containers (api:8000/metrics, postgres:5432) — the main
+  # compose declares the same fixed-name network as "shared".
+  shared:
+    name: erp_shared
+    external: true
+  # Created by the main compose; required only for the public GlitchTip route.
+  traefik:
+    name: traefik_network
+    external: true
+
+volumes:
+  glitchtip_postgres_data:
+  prometheus_data:
+  grafana_data:
+  loki_data:
@@ -0,0 +1,159 @@
+{
+  "annotations": { "list": [] },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+      "fieldConfig": {
+        "defaults": {
+          "custom": { "fillOpacity": 15, "lineWidth": 2, "showPoints": "never" },
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
+      "id": 1,
+      "options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" } },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+          "expr": "sum by (status) (rate(http_requests_total{job=\"erp-api\"}[5m]))",
+          "legendFormat": "HTTP {{status}}"
+        }
+      ],
+      "title": "API — request rate by status",
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+      "fieldConfig": {
+        "defaults": {
+          "custom": { "fillOpacity": 10, "lineWidth": 2, "showPoints": "never" },
+          "unit": "s",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "orange", "value": 1 },
+              { "color": "red", "value": 2 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
+      "id": 2,
+      "options": { "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" } },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"erp-api\"}[5m])) by (le))",
+          "legendFormat": "p95"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+          "expr": "histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket{job=\"erp-api\"}[5m])) by (le))",
+          "legendFormat": "p50"
+        }
+      ],
+      "title": "API — latency (p50 / p95)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+      "fieldConfig": {
+        "defaults": {
+          "custom": { "fillOpacity": 15, "lineWidth": 2, "showPoints": "never" },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
+      "id": 3,
+      "options": { "legend": { "calcs": ["lastNotNull", "max"], "displayMode": "table", "placement": "bottom" } },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+          "expr": "sum by (name) (container_memory_working_set_bytes{job=\"cadvisor\", name!=\"\", image!=\"\"})",
+          "legendFormat": "{{name}}"
+        }
+      ],
+      "title": "Containers — memory in use",
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
+      "id": 4,
+      "options": { "legend": { "calcs": ["lastNotNull"], "displayMode": "table", "placement": "bottom" } },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+          "expr": "sum(pg_stat_activity_count)",
+          "legendFormat": "postgres connections"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
+          "expr": "node_memory_MemAvailable_bytes",
+          "legendFormat": "host RAM available"
+        }
+      ],
+      "title": "Database connections / host RAM",
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
+      "gridPos": { "h": 10, "w": 24, "x": 0, "y": 16 },
+      "id": 5,
+      "options": {
+        "sortOrder": "Descending",
+        "wrapLogMessage": true,
+        "showTime": true,
+        "enableLogDetails": true
+      },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${DS_LOKI}" },
+          "expr": "{compose_project=\"erp\"}",
+          "refId": "A"
+        }
+      ],
+      "title": "All ERP container logs (Loki)",
+      "type": "logs"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": ["erp"],
+  "templating": {
+    "list": [
+      {
+        "name": "DS_PROMETHEUS",
+        "label": "Prometheus",
+        "type": "datasource",
+        "query": "prometheus",
+        "hide": 2,
+        "current": { "text": "Prometheus", "value": "Prometheus" }
+      },
+      {
+        "name": "DS_LOKI",
+        "label": "Loki",
+        "type": "datasource",
+        "query": "loki",
+        "hide": 2,
+        "current": { "text": "Loki", "value": "Loki" }
+      }
+    ]
+  },
+  "time": { "from": "now-6h", "to": "now" },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "ERP — Stack overview",
+  "uid": "erp-overview",
+  "version": 1
+}
@@ -0,0 +1,11 @@
+apiVersion: 1
+
+providers:
+  - name: erp-dashboards
+    orgId: 1
+    folder: ERP
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 60
+    options:
+      path: /var/lib/grafana/dashboards
@@ -0,0 +1,15 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    editable: false
@@ -0,0 +1,43 @@
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+
+common:
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+schema_config:
+  configs:
+    - from: 2024-01-01
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+limits_config:
+  # Keep the log store bounded — this box shares 8 GB RAM / limited disk.
+  retention_period: 14d
+  ingestion_rate_mb: 4
+  ingestion_burstsize_mb: 8
+  max_query_series: 500
+
+compactor:
+  working_directory: /loki/compactor
+  retention_enabled: true
+  delete_request_store: filesystem
+
+ruler:
+  storage:
+    type: local
+    local:
+      directory: /loki/rules
@@ -0,0 +1,58 @@
+# Alert rules surfaced in Grafana's alerting UI (no Alertmanager needed at
+# this scale — Grafana can route them later via email/Telegram).
+groups:
+  - name: erp-availability
+    rules:
+      - alert: ErpApiDown
+        expr: up{job="erp-api"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "ERP API is not scraping (down or /metrics failing)"
+
+      - alert: ErpApiHighErrorRate
+        expr: |
+          sum(rate(http_requests_total{job="erp-api", status=~"5.."}[5m]))
+            / clamp_min(sum(rate(http_requests_total{job="erp-api"}[5m])), 0.001) > 0.05
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "ERP API 5xx rate above 5% for 5 minutes"
+
+      - alert: ErpApiSlowRequests
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket{job="erp-api"}[5m])) by (le)) > 2
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "ERP API p95 latency above 2s for 10 minutes"
+
+  - name: host
+    rules:
+      - alert: HostDiskAlmostFull
+        expr: (node_filesystem_avail_bytes{mountpoint="/", fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{mountpoint="/", fstype!~"tmpfs|overlay"}) < 0.1
+        for: 15m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Root filesystem below 10% free (deploy builds will fail)"
+
+      - alert: HostMemoryPressure
+        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Host RAM below 10% available (check monitoring stack limits)"
+
+      - alert: PostgresTooManyConnections
+        expr: sum(pg_stat_activity_count) > 80
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "PostgreSQL connection count above 80"
@@ -0,0 +1,34 @@
+global:
+  scrape_interval: 30s
+  evaluation_interval: 30s
+
+rule_files:
+  - /etc/prometheus/alerts.yml
+
+scrape_configs:
+  # The ERP API — /metrics is exposed by prometheus-fastapi-instrumentator
+  # (request rate, latency histogram, error counts). Reached over the
+  # erp_shared network the main compose also joins.
+  - job_name: erp-api
+    metrics_path: /metrics
+    static_configs:
+      - targets: ["api:8000"]
+
+  # Host-level metrics (CPU, RAM, disk, network).
+  - job_name: node
+    static_configs:
+      - targets: ["node-exporter:9100"]
+
+  # Per-container metrics (CPU, memory, restarts) for every container.
+  - job_name: cadvisor
+    static_configs:
+      - targets: ["cadvisor:8080"]
+
+  # App database metrics (connections, cache hit ratio, WAL, ...).
+  - job_name: postgres
+    static_configs:
+      - targets: ["postgres-exporter:9187"]
+
+  - job_name: prometheus
+    static_configs:
+      - targets: ["localhost:9090"]
@@ -0,0 +1,40 @@
+# Ships every container's JSON logs to Loki (docker service discovery),
+# tagging streams with the compose project / service / container name so
+# logs are filterable in Grafana, e.g. {compose_service="api"}.
+server:
+  http_listen_port: 9080
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://loki:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 30s
+        filters:
+          - name: label
+            values: ["com.docker.compose.service"]
+    relabel_configs:
+      - source_labels: ["__meta_docker_container_label_com_docker_compose_project"]
+        target_label: compose_project
+      - source_labels: ["__meta_docker_container_label_com_docker_compose_service"]
+        target_label: compose_service
+      - source_labels: ["__meta_docker_container_name"]
+        target_label: container
+    pipeline_stages:
+      - json:
+          expressions:
+            ts: time
+            stream: stream
+            log: log
+      - labels:
+          stream:
+      - timestamp:
+          source: ts
+          format: RFC3339Nano
+      - output:
+          source: log