feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

Architecture:
- ja4_logs: raw log ingestion (http_logs_raw, http_logs, mv_http_logs)
- ja4_processing: analytics, aggregation, ML, dictionaries, audit

Configuration (env vars):
- CLICKHOUSE_DB_LOGS (default: ja4_logs)
- CLICKHOUSE_DB_PROCESSING (default: ja4_processing)

Changes:
- SQL migrations (10 files): all mabase_prod refs → ja4_logs or ja4_processing
  with correct cross-database references (MVs, views, dicts)
- deploy_schema.sh: substitutes DB names from env vars at deploy time
- Python shared settings: added CLICKHOUSE_DB_LOGS + CLICKHOUSE_DB_PROCESSING
- Dashboard routes (19 files): replaced ~80 hardcoded mabase_prod refs
  with settings.CLICKHOUSE_DB_LOGS / settings.CLICKHOUSE_DB_PROCESSING
- Bot-detector: DB → CLICKHOUSE_DB_PROCESSING, fetch_rules.py configurable
- Correlator: DSN example updated to ja4_logs
- Docker-compose + .env files: new env vars with defaults
- All documentation updated (14 markdown files)

All tests pass: sentinel 10/10, correlator 67.1%, bot-detector 11, dashboard 20, ja4_common 18

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

services/dashboard/.env.example

@@ -1,7 +1,9 @@
 # dashboard configuration — DO NOT COMMIT real values
 CLICKHOUSE_HOST=clickhouse
 CLICKHOUSE_PORT=8123
-CLICKHOUSE_DB=mabase_prod
+CLICKHOUSE_DB=ja4_processing
+CLICKHOUSE_DB_LOGS=ja4_logs
+CLICKHOUSE_DB_PROCESSING=ja4_processing
 CLICKHOUSE_USER=analyst
 CLICKHOUSE_PASSWORD=
 API_HOST=0.0.0.0

services/dashboard/.github/copilot-instructions.md

@@ -4,7 +4,7 @@
 
 This is a **SOC (Security Operations Center) dashboard** for visualizing bot detections from an upstream `bot_detector_ai` service. It is a **single-service, full-stack app**: the FastAPI backend serves the built React frontend as static files *and* exposes a REST API, all on port 8000. There is no separate frontend server in production and **no authentication**.
 
-**Data source:** ClickHouse database (`mabase_prod`), primarily the `ml_detected_anomalies` table and the `view_dashboard_entities` view.
+**Data source:** ClickHouse database (`ja4_processing`), primarily the `ml_detected_anomalies` table and the `view_dashboard_entities` view.
 
 ```
 dashboard/
@@ -95,7 +95,7 @@ curl "http://localhost:8000/api/detections?page=1&page_size=5" | jq '.items | le
 |---|---|---|
 | `CLICKHOUSE_HOST` | `clickhouse` | ClickHouse hostname |
 | `CLICKHOUSE_PORT` | `8123` | ClickHouse HTTP port (set in code) |
-| `CLICKHOUSE_DB` | `mabase_prod` | Database name |
+| `CLICKHOUSE_DB` | `ja4_processing` | Database name |
 | `CLICKHOUSE_USER` | `admin` | |
 | `CLICKHOUSE_PASSWORD` | `` | |
 | `API_HOST` | `0.0.0.0` | Uvicorn bind host |
@@ -111,4 +111,4 @@ curl "http://localhost:8000/api/detections?page=1&page_size=5" | jq '.items | le
 | `ml_detected_anomalies` | Primary source for detections, metrics, variability, analysis |
 | `view_dashboard_entities` | User agents, client headers, paths, query params (entities routes) |
 | `classifications` | SOC analyst classifications (created by `analysis.py`) |
-| `mabase_prod.audit_logs` | Audit trail (optional — missing table is handled silently) |
+| `ja4_processing.audit_logs` | Audit trail (optional — missing table is handled silently) |

services/dashboard/README.md

@@ -241,7 +241,7 @@ curl http://localhost:8000/api/reputation/ip/162.55.94.175 | jq
 |----------|--------|-------------|
 | `CLICKHOUSE_HOST` | `clickhouse` | Hôte ClickHouse |
 | `CLICKHOUSE_PORT` | `8123` | Port HTTP ClickHouse |
-| `CLICKHOUSE_DB` | `mabase_prod` | Base de données |
+| `CLICKHOUSE_DB` | `ja4_processing` | Base de données |
 | `CLICKHOUSE_USER` | `admin` | Utilisateur |
 | `CLICKHOUSE_PASSWORD` | `` | Mot de passe |
 | `API_HOST` | `0.0.0.0` | Bind Uvicorn |
@@ -339,11 +339,11 @@ Déflation Hotelling : retire PC1 de X avant de calculer PC2
 
 | Table / Vue | Routes |
 |---|---|
-| `mabase_prod.ml_detected_anomalies` | metrics, detections, variability, analysis, clustering |
-| `mabase_prod.agg_host_ip_ja4_1h` | tcp_spoofing, clustering, investigation_summary |
-| `mabase_prod.view_dashboard_entities` | entities (UA, JA4, paths, query params) |
-| `mabase_prod.classifications` | analysis (classifications SOC manuelles) |
-| `mabase_prod.audit_logs` | audit (optionnel — silencieux si absent) |
+| `ja4_processing.ml_detected_anomalies` | metrics, detections, variability, analysis, clustering |
+| `ja4_processing.agg_host_ip_ja4_1h` | tcp_spoofing, clustering, investigation_summary |
+| `ja4_processing.view_dashboard_entities` | entities (UA, JA4, paths, query params) |
+| `ja4_processing.classifications` | analysis (classifications SOC manuelles) |
+| `ja4_processing.audit_logs` | audit (optionnel — silencieux si absent) |
 
 **Conventions SQL :**
 - IPs stockées en IPv6-mappé : `replaceRegexpAll(toString(src_ip), '^::ffff:', '')`
@@ -477,15 +477,15 @@ curl -s http://localhost:3000 | head -20
 
 ```bash
 # Compter les détections (24h)
-docker compose exec clickhouse clickhouse-client -d mabase_prod -q \
+docker compose exec clickhouse clickhouse-client -d ja4_processing -q \
   "SELECT count() FROM ml_detected_anomalies WHERE detected_at >= now() - INTERVAL 24 HOUR"
 
 # Voir un échantillon
-docker compose exec clickhouse clickhouse-client -d mabase_prod -q \
+docker compose exec clickhouse clickhouse-client -d ja4_processing -q \
   "SELECT src_ip, threat_level, model_name, detected_at FROM ml_detected_anomalies ORDER BY detected_at DESC LIMIT 5"
 
 # Vérifier les vues du dashboard
-docker compose exec clickhouse clickhouse-client -d mabase_prod -q \
+docker compose exec clickhouse clickhouse-client -d ja4_processing -q \
   "SELECT * FROM view_dashboard_summary"
 ```
 
@@ -524,7 +524,7 @@ docker compose up -d dashboard_web
 
 ```bash
 # 1. Vérifier qu'il y a des données dans ClickHouse
-docker compose exec clickhouse clickhouse-client -d mabase_prod -q \
+docker compose exec clickhouse clickhouse-client -d ja4_processing -q \
   "SELECT count() FROM ml_detected_anomalies WHERE detected_at >= now() - INTERVAL 24 HOUR"
 
 # Si le résultat est 0:

services/dashboard/backend/config.py

@@ -8,7 +8,9 @@ class Settings(BaseSettings):
     # ClickHouse
     CLICKHOUSE_HOST: str = "clickhouse"
     CLICKHOUSE_PORT: int = 8123
-    CLICKHOUSE_DB: str = "mabase_prod"
+    CLICKHOUSE_DB: str = "ja4_processing"  # default connection database
+    CLICKHOUSE_DB_LOGS: str = "ja4_logs"
+    CLICKHOUSE_DB_PROCESSING: str = "ja4_processing"
     CLICKHOUSE_USER: str = "admin"
     CLICKHOUSE_PASSWORD: str = ""
 

services/dashboard/backend/main.py

@@ -139,7 +139,7 @@ app = FastAPI(
     title="Bot Detector Dashboard API",
     description=(
         "API REST du **Bot Detector SOC Dashboard**.\n\n"
-        "Permet d'interroger la base ClickHouse (`mabase_prod`) pour visualiser et analyser "
+        "Permet d'interroger les bases ClickHouse (`ja4_logs` / `ja4_processing`) pour visualiser et analyser "
         "les détections de bots générées par le service `bot_detector_ai`.\n\n"
         "**Endpoint clé :** `GET /api/investigation/{ip}/summary` — synthèse complète en un appel.\n\n"
         "Documentation interactive : `/docs` (Swagger UI) · `/redoc` (ReDoc)"

services/dashboard/docker-compose.yaml

@@ -16,7 +16,9 @@ services:
     environment:
       # ClickHouse
       CLICKHOUSE_HOST: ${CLICKHOUSE_HOST:-clickhouse}
-      CLICKHOUSE_DB: ${CLICKHOUSE_DB:-mabase_prod}
+      CLICKHOUSE_DB: ${CLICKHOUSE_DB:-ja4_processing}
+      CLICKHOUSE_DB_LOGS: ${CLICKHOUSE_DB_LOGS:-ja4_logs}
+      CLICKHOUSE_DB_PROCESSING: ${CLICKHOUSE_DB_PROCESSING:-ja4_processing}
       CLICKHOUSE_USER: ${CLICKHOUSE_USER:-admin}
       CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-}