feat(stack): full overhaul — hardening, ops tooling, monitoring, analytics, chatbot

Security: deep /api/health (DB+MinIO probes, 503 on failure) + /api/live; rate-limited public forms with honeypots; real client IP behind Traefik (--proxy-headers); TipTap URL sanitization on write (nh3); non-root containers + slim .dockerignore; prod MinIO console bound to 127.0.0.1; HSTS/nosniff on TLS routers; JWT 8h; no default credentials in code. Reliability: Alembic migrations (create_all baseline); graceful shutdown (exec uvicorn); hardened backups (checksums, sanity check, freshness gauge, optional rclone off-site) + restore.sh + hourly db-backup sidecar (ops profile); stack.sh ops tooling (deploy/logs/health/backup/restore/monitoring); .gitattributes LF for shell scripts. Orchestration: json-file log rotation, pinned images, per-service memory limits, edge/internal networks, Traefik ping/accesslog/metrics + flood limiter, conditional minio-setup, pgvector db image, one-shot db-init (umami + aramesh_chatbot databases). Chatbot: unified from site-chatbot branch (RAG + pgvector, SSE, ChatWidget). Monitoring: Prometheus/Alertmanager/Loki/Promtail/Grafana + cAdvisor, node/postgres/blackbox exporters, provisioned dashboards and 10 alert rules; alerts delivered to Bale (tapi.bale.ai — Telegram-compatible). Analytics: Umami on shared Postgres, build-time tracker injection, custom events (contact/newsletter/phone/chat/lang) via frontend/src/analytics.ts. Hygiene: CSV export for contact messages and subscribers; upload image optimization (Pillow); CI script + GitHub-compatible workflow; audit.sh; gen-env.sh for strong credentials.
2026-09-19 00:24:57 +03:30
parent 805f4594d4
commit ce0735d775
75 changed files with 4287 additions and 153 deletions
@@ -0,0 +1,118 @@
+# قواعد هشدار — از Prometheus به Alertmanager (تلگرام) می‌روند.
+# آستانه‌ها محافظه‌کارانه‌اند؛ پس از یک هفته داده، دقیق‌تر تنظیم کنید.
+
+groups:
+  - name: aramesh-availability
+    rules:
+      - alert: ServiceDown
+        expr: up == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "{{ $labels.job }} پاسخ نمی‌دهد"
+          description: "سرویس {{ $labels.job }} ({{ $labels.instance }}) حداقل ۲ دقیقه پایین است."
+
+      - alert: BackendHealthFailing
+        expr: probe_success{job="blackbox-internal-health"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "بررسی عمق سلامت بک‌اند شکست می‌خورد"
+          description: "‏/api/health اتصال DB یا MinIO را آزموده و 503 برگردانده — وابستگی‌ها را ببینید."
+
+      - alert: PublicSiteDown
+        expr: probe_success{job="blackbox-public"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "سایت از مسیر عمومی در دسترس نیست"
+          description: "پروب {{ $labels.instance }} ناموفق است — Traefik/TLS/DNS را بررسی کنید."
+
+  - name: aramesh-resources
+    rules:
+      - alert: HostDiskAlmostFull
+        expr: |
+          1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) > 0.85
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "دیسک میزبان پر می‌شود"
+          description: "پرشدن بیش از ۸۵٪ روی {{ $labels.mountpoint }} — پشتیبان‌ها و لاگ‌ها را بررسی کنید."
+
+      - alert: HostMemoryPressure
+        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "RAM میزبان رو به تمام‌شدن است"
+          description: "کمتر از ۱۰٪ حافظه باقی مانده است."
+
+      - alert: ContainerMemoryNearLimit
+        expr: |
+          container_memory_usage_bytes{image!="", container!="", container_spec_memory_limit_bytes > 0}
+            / container_spec_memory_limit_bytes{container!="", container_spec_memory_limit_bytes > 0} > 0.9
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "کانتینر {{ $labels.container }} به سقف حافظه نزدیک است"
+
+  - name: aramesh-app
+    rules:
+      - alert: Http5xxRateHigh
+        expr: |
+          sum(rate(http_request_duration_seconds_count{status=~"5.."}[5m])) by (instance)
+            / sum(rate(http_request_duration_seconds_count[5m])) by (instance) > 0.05
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "نرخ خطای 5xx بک‌اند بالاست"
+
+      - alert: Proxy5xxRateHigh
+        expr: |
+          sum(rate(traefik_service_requests_total{code=~"5.."}[5m])) by (service)
+            / sum(rate(traefik_service_requests_total[5m])) by (service) > 0.05
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "نرخ خطای 5xx روی سرویس {{ $labels.service }} پشت Traefik بالاست"
+
+      - alert: HighBackendLatency
+        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 2
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "p95 تأخیر بک‌اند بالای ۲ ثانیه است"
+
+      - alert: PostgresTooManyConnections
+        expr: sum(pg_stat_activity_count) > 80
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "تعداد اتصال‌های Postgres نزدیک سقف است"
+
+      - alert: TlsCertExpiringSoon
+        expr: traefik_tls_certs_not_after - time() < 1209600
+        labels:
+          severity: warning
+        annotations:
+          summary: "گواهی TLS کمتر از ۱۴ روز اعتبار دارد"
+          description: "تمدید خودکار Let's Encrypt کار نکرده — ACME/TLS را بررسی کنید."
+
+      - alert: BackupStale
+        expr: time() - backup_success_time > 93600
+        for: 30m
+        labels:
+          severity: critical
+        annotations:
+          summary: "آخرین پشتیبان موفق بیش از ۲۶ ساعت پیش است"
+          description: "کرون/اسکریپت پشتیبان‌گیری یا سرویس db-backup خفته — فوراً بررسی کنید."