docs: guide de déploiement complet (DB, users, services, vérification)
Couvre les 7 étapes de mise en place : 1. Installation ClickHouse 2. Déploiement du schéma (deploy_schema.sh + migrations) 3. Configuration des utilisateurs (data_writer, analyst, bot_writer) 4. Fichiers CSV externes pour les dictionnaires 5. Installation des services Go via RPM (sentinel, correlator, mod-reqin-log) 6. Installation des services Python via Docker (bot-detector, dashboard) 7. Vérification bout-en-bout (ingestion, agrégation, ML, dashboard) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
452
docs/deployment.md
Normal file
452
docs/deployment.md
Normal file
@ -0,0 +1,452 @@
|
|||||||
|
# Guide de déploiement — ja4-platform
|
||||||
|
|
||||||
|
## Prérequis
|
||||||
|
|
||||||
|
| Composant | Version minimale | Notes |
|
||||||
|
|-----------|-----------------|-------|
|
||||||
|
| ClickHouse | 23.8+ | Requis pour `REGEXP_TREE`, `IP_TRIE` dictionaries |
|
||||||
|
| Docker | 20.10+ | Pour build des images et RPMs |
|
||||||
|
| Make | 3.81+ | Orchestration des builds |
|
||||||
|
| OS cible (RPM) | Rocky/RHEL 8, 9, ou 10 | RPMs générés pour les 3 versions |
|
||||||
|
|
||||||
|
### Services et leur runtime
|
||||||
|
|
||||||
|
| Service | Runtime | Port par défaut |
|
||||||
|
|---------|---------|-----------------|
|
||||||
|
| sentinel | Go binary + libpcap (RPM) | — (capture réseau passive) |
|
||||||
|
| correlator | Go binary (RPM) | 8080 (metrics, optionnel) |
|
||||||
|
| mod-reqin-log | Apache module .so (RPM) | — (intégré à httpd) |
|
||||||
|
| bot-detector | Python 3.11 (Docker) | 8080 (health check) |
|
||||||
|
| dashboard | Python 3.11 / FastAPI (Docker) | 8000 (API) + 3000 (frontend) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 1 — Installation de ClickHouse
|
||||||
|
|
||||||
|
### Installation (Rocky Linux / RHEL)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo yum install -y yum-utils
|
||||||
|
sudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo
|
||||||
|
sudo yum install -y clickhouse-server clickhouse-client
|
||||||
|
sudo systemctl enable --now clickhouse-server
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vérification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clickhouse-client --query "SELECT version()"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 2 — Déploiement du schéma ClickHouse
|
||||||
|
|
||||||
|
Le schéma est géré par **10 fichiers SQL ordonnés** dans `shared/clickhouse/` et un script de déploiement automatisé.
|
||||||
|
|
||||||
|
### Architecture des bases de données
|
||||||
|
|
||||||
|
Le système utilise **deux bases de données** séparées :
|
||||||
|
|
||||||
|
| Base | Variable d'env | Défaut | Contenu |
|
||||||
|
|------|---------------|--------|---------|
|
||||||
|
| **Logs** | `CLICKHOUSE_DB_LOGS` | `ja4_logs` | Logs bruts et parsés |
|
||||||
|
| **Traitement** | `CLICKHOUSE_DB_PROCESSING` | `ja4_processing` | Agrégations, ML, dictionnaires, vues, audit |
|
||||||
|
|
||||||
|
### Fichiers de migration
|
||||||
|
|
||||||
|
```
|
||||||
|
shared/clickhouse/
|
||||||
|
├── 00_database.sql # Création des deux bases
|
||||||
|
├── 01_raw_tables.sql # ja4_logs.http_logs_raw (ingestion brute, TTL 1 jour)
|
||||||
|
├── 02_dictionaries.sql # ja4_processing : dict_iplocate_asn, ref_bot_networks, bot_ip, bot_ja4
|
||||||
|
├── 03_anubis_tables.sql # ja4_processing : tables et dictionnaires Anubis (UA, IP, ASN, Country)
|
||||||
|
├── 04_mv_http_logs.sql # ja4_logs.http_logs + mv_http_logs (parsing JSON → colonnes typées)
|
||||||
|
├── 05_aggregation_tables.sql # ja4_processing : agg_host_ip_ja4_1h, agg_header_fingerprint_1h + MVs
|
||||||
|
├── 06_ml_tables.sql # ja4_processing : ml_detected_anomalies, ml_all_scores, view_ip_recurrence
|
||||||
|
├── 07_ai_features_view.sql # ja4_processing : view_ai_features_1h (feature engineering)
|
||||||
|
├── 08_users.sql # Utilisateurs data_writer et analyst + permissions
|
||||||
|
├── 09_audit_table.sql # ja4_processing : audit_logs (trace SOC, TTL 90 jours)
|
||||||
|
└── deploy_schema.sh # Script de déploiement automatisé
|
||||||
|
```
|
||||||
|
|
||||||
|
### Déploiement avec les noms par défaut
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd shared/clickhouse/
|
||||||
|
chmod +x deploy_schema.sh
|
||||||
|
./deploy_schema.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Déploiement avec des noms personnalisés
|
||||||
|
|
||||||
|
```bash
|
||||||
|
CLICKHOUSE_DB_LOGS=prod_logs \
|
||||||
|
CLICKHOUSE_DB_PROCESSING=prod_analysis \
|
||||||
|
CLICKHOUSE_HOST=clickhouse.internal \
|
||||||
|
CLICKHOUSE_PORT=9000 \
|
||||||
|
CLICKHOUSE_USER=default \
|
||||||
|
CLICKHOUSE_PASSWORD=MonMotDePasse \
|
||||||
|
./deploy_schema.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Le script effectue une substitution `sed` sur chaque fichier SQL avant de l'envoyer à `clickhouse-client --multiquery`.
|
||||||
|
|
||||||
|
### Déploiement manuel (fichier par fichier)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clickhouse-client --multiquery < shared/clickhouse/00_database.sql
|
||||||
|
clickhouse-client --multiquery < shared/clickhouse/01_raw_tables.sql
|
||||||
|
# ... etc., dans l'ordre
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vérification du déploiement
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Tables dans ja4_logs
|
||||||
|
clickhouse-client --query "SHOW TABLES FROM ja4_logs"
|
||||||
|
# Attendu : http_logs, http_logs_raw
|
||||||
|
|
||||||
|
# Tables dans ja4_processing
|
||||||
|
clickhouse-client --query "SHOW TABLES FROM ja4_processing"
|
||||||
|
# Attendu : agg_header_fingerprint_1h, agg_host_ip_ja4_1h, anubis_asn_rules,
|
||||||
|
# anubis_country_rules, anubis_ip_rules, anubis_ua_rules, audit_logs,
|
||||||
|
# ml_all_scores, ml_detected_anomalies, ref_bot_networks, ...
|
||||||
|
|
||||||
|
# Dictionnaires chargés
|
||||||
|
clickhouse-client --query "SELECT name, status FROM system.dictionaries WHERE database IN ('ja4_logs', 'ja4_processing')"
|
||||||
|
|
||||||
|
# Utilisateurs créés
|
||||||
|
clickhouse-client --query "SHOW USERS"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 3 — Configuration des utilisateurs ClickHouse
|
||||||
|
|
||||||
|
Le fichier `08_users.sql` crée deux utilisateurs avec des mots de passe par défaut **à changer impérativement** :
|
||||||
|
|
||||||
|
### Utilisateurs et permissions
|
||||||
|
|
||||||
|
| Utilisateur | Mot de passe défaut | Rôle | Permissions |
|
||||||
|
|-------------|-------------------|------|-------------|
|
||||||
|
| `data_writer` | `ChangeMe` | Ingestion (correlator) | `INSERT`, `SELECT` sur `ja4_logs.http_logs_raw` |
|
||||||
|
| `analyst` | `ChangeMe` | Lecture (dashboard, bot-detector) | `SELECT` sur `ja4_logs.http_logs`, `ja4_processing.ml_*`, `ja4_processing.view_*`, `ja4_processing.audit_logs` |
|
||||||
|
|
||||||
|
### Changement des mots de passe (obligatoire en production)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Utiliser SHA256 pour la production
|
||||||
|
clickhouse-client --query "
|
||||||
|
ALTER USER data_writer IDENTIFIED WITH sha256_password BY 'VotreMotDePasseWriter';
|
||||||
|
ALTER USER analyst IDENTIFIED WITH sha256_password BY 'VotreMotDePasseAnalyst';
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Permissions supplémentaires pour le bot-detector
|
||||||
|
|
||||||
|
Le bot-detector a besoin d'écrire dans les tables ML. Ajoutez ces grants pour l'utilisateur utilisé par le bot-detector (ici `admin` ou un utilisateur dédié) :
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Créer un utilisateur dédié pour le bot-detector
|
||||||
|
CREATE USER IF NOT EXISTS bot_writer IDENTIFIED WITH sha256_password BY 'VotreMotDePasse';
|
||||||
|
|
||||||
|
-- Lectures : features et récurrence (ja4_processing)
|
||||||
|
GRANT SELECT ON ja4_processing.view_ai_features_1h TO bot_writer;
|
||||||
|
GRANT SELECT ON ja4_processing.view_ip_recurrence TO bot_writer;
|
||||||
|
GRANT SELECT ON ja4_processing.ml_detected_anomalies TO bot_writer;
|
||||||
|
|
||||||
|
-- Écritures : résultats ML (ja4_processing)
|
||||||
|
GRANT INSERT ON ja4_processing.ml_all_scores TO bot_writer;
|
||||||
|
GRANT INSERT ON ja4_processing.ml_detected_anomalies TO bot_writer;
|
||||||
|
|
||||||
|
-- Lectures/écritures : tables Anubis (ja4_processing) — pour fetch_rules.py
|
||||||
|
GRANT SELECT, INSERT, ALTER ON ja4_processing.anubis_ua_rules TO bot_writer;
|
||||||
|
GRANT SELECT, INSERT, ALTER ON ja4_processing.anubis_ip_rules TO bot_writer;
|
||||||
|
GRANT SELECT, INSERT, ALTER ON ja4_processing.anubis_asn_rules TO bot_writer;
|
||||||
|
GRANT SELECT, INSERT, ALTER ON ja4_processing.anubis_country_rules TO bot_writer;
|
||||||
|
GRANT SYSTEM RELOAD DICTIONARY ON *.* TO bot_writer;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Permissions pour l'audit (dashboard)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Le dashboard doit pouvoir écrire dans audit_logs
|
||||||
|
GRANT INSERT ON ja4_processing.audit_logs TO analyst;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 4 — Fichiers de données externes
|
||||||
|
|
||||||
|
Certains dictionnaires ClickHouse chargent des fichiers CSV depuis le disque :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Répertoire des fichiers utilisateur ClickHouse
|
||||||
|
sudo mkdir -p /var/lib/clickhouse/user_files/
|
||||||
|
|
||||||
|
# Fichiers requis (à fournir par l'opérateur) :
|
||||||
|
# - iplocate_asn.csv → dict_iplocate_asn (géolocalisation IP/ASN)
|
||||||
|
# - bot_ip.csv → bot_ip (IPs de bots connues)
|
||||||
|
# - bot_ja4.csv → bot_ja4 (signatures JA4 de bots)
|
||||||
|
# - asn_reputation.csv → dict_asn_reputation (réputation ASN)
|
||||||
|
|
||||||
|
# Exemple de format bot_ip.csv :
|
||||||
|
# prefix,bot_name
|
||||||
|
# 198.51.100.0/24,ExampleBot
|
||||||
|
# 203.0.113.42/32,ScannerBot
|
||||||
|
|
||||||
|
# Permissions
|
||||||
|
sudo chown -R clickhouse:clickhouse /var/lib/clickhouse/user_files/
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 5 — Installation des services Go (RPM)
|
||||||
|
|
||||||
|
### Build des RPMs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Tous les RPMs (sentinel + correlator + mod-reqin-log) × 3 distros
|
||||||
|
make rpm-all
|
||||||
|
|
||||||
|
# Ou individuellement
|
||||||
|
make rpm-sentinel # → services/sentinel/dist/
|
||||||
|
make rpm-correlator # → services/correlator/dist/
|
||||||
|
make rpm-mod-reqin-log # → services/mod-reqin-log/dist/
|
||||||
|
```
|
||||||
|
|
||||||
|
Les RPMs sont générés dans `services/<nom>/dist/` avec un sous-dossier par distro (el8, el9, el10).
|
||||||
|
|
||||||
|
### Installation des RPMs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sur le serveur cible (Rocky 9 par exemple)
|
||||||
|
sudo yum install -y ./ja4sentinel-*.el9.x86_64.rpm
|
||||||
|
sudo yum install -y ./logcorrelator-*.el9.x86_64.rpm
|
||||||
|
sudo yum install -y ./mod_reqin_log-*.el9.x86_64.rpm
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration du sentinel
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fichier de configuration principal
|
||||||
|
sudo cp /etc/ja4sentinel/config.yml.default /etc/ja4sentinel/config.yml
|
||||||
|
sudo vi /etc/ja4sentinel/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Variables d'environnement (dans `/etc/sysconfig/ja4sentinel` ou `.env`) :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
JA4SENTINEL_INTERFACE=eth0 # Interface réseau à capturer
|
||||||
|
JA4SENTINEL_PORTS=443,8443 # Ports TLS à surveiller
|
||||||
|
```
|
||||||
|
|
||||||
|
Le sentinel écrit vers le socket Unix du correlator : `/var/run/logcorrelator/network.socket`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl enable --now ja4sentinel
|
||||||
|
sudo systemctl status ja4sentinel
|
||||||
|
journalctl -u ja4sentinel -f
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration du correlator
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo cp /etc/logcorrelator/config.yml.default /etc/logcorrelator/config.yml
|
||||||
|
sudo vi /etc/logcorrelator/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Configuration minimale pour activer ClickHouse (`config.yml`) :
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
outputs:
|
||||||
|
clickhouse:
|
||||||
|
enabled: true
|
||||||
|
dsn: clickhouse://data_writer:VotreMotDePasse@localhost:9000/ja4_logs
|
||||||
|
table: http_logs_raw
|
||||||
|
batch_size: 500
|
||||||
|
flush_interval_ms: 200
|
||||||
|
|
||||||
|
file:
|
||||||
|
enabled: true
|
||||||
|
path: /var/log/logcorrelator/correlated.log
|
||||||
|
```
|
||||||
|
|
||||||
|
Variable d'environnement alternative (dans `/etc/sysconfig/logcorrelator`) :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
LOGCORRELATOR_CLICKHOUSE_DSN=clickhouse://data_writer:VotreMotDePasse@localhost:9000/ja4_logs
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl enable --now logcorrelator
|
||||||
|
sudo systemctl status logcorrelator
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration de mod-reqin-log
|
||||||
|
|
||||||
|
Le module Apache écrit les requêtes HTTP en JSON vers le socket Unix du correlator.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Le RPM installe automatiquement le module dans Apache
|
||||||
|
# Vérifier le chargement
|
||||||
|
httpd -M | grep reqin
|
||||||
|
|
||||||
|
# La configuration est dans /etc/httpd/conf.d/mod_reqin_log.conf
|
||||||
|
# Le socket par défaut : /var/run/logcorrelator/http.socket
|
||||||
|
sudo systemctl restart httpd
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 6 — Installation des services Python (Docker)
|
||||||
|
|
||||||
|
### Bot-detector
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd services/bot-detector
|
||||||
|
|
||||||
|
# Copier et configurer .env
|
||||||
|
cp .env.example .env
|
||||||
|
vi .env # Renseigner CLICKHOUSE_HOST, mots de passe, etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
Variables d'environnement clés :
|
||||||
|
|
||||||
|
| Variable | Défaut | Description |
|
||||||
|
|----------|--------|-------------|
|
||||||
|
| `CLICKHOUSE_HOST` | `clickhouse` | Hôte ClickHouse |
|
||||||
|
| `CLICKHOUSE_PORT` | `8123` | Port HTTP ClickHouse |
|
||||||
|
| `CLICKHOUSE_DB_PROCESSING` | `ja4_processing` | Base de traitement |
|
||||||
|
| `CLICKHOUSE_DB_LOGS` | `ja4_logs` | Base de logs |
|
||||||
|
| `CLICKHOUSE_USER` | `admin` | Utilisateur (utiliser `bot_writer` en prod) |
|
||||||
|
| `CLICKHOUSE_PASSWORD` | — | Mot de passe |
|
||||||
|
| `ANOMALY_THRESHOLD` | `-0.05` | Seuil de détection d'anomalies |
|
||||||
|
| `CYCLE_INTERVAL_SEC` | `300` | Intervalle entre cycles de détection (secondes) |
|
||||||
|
| `RETRAIN_INTERVAL_HOURS` | `24` | Intervalle de réentraînement du modèle |
|
||||||
|
| `HEALTH_PORT` | `8080` | Port du endpoint /health |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Dashboard
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd services/dashboard
|
||||||
|
|
||||||
|
# Copier et configurer .env
|
||||||
|
cp .env.example .env
|
||||||
|
vi .env # Renseigner CLICKHOUSE_HOST, mots de passe, etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
Variables d'environnement clés :
|
||||||
|
|
||||||
|
| Variable | Défaut | Description |
|
||||||
|
|----------|--------|-------------|
|
||||||
|
| `CLICKHOUSE_HOST` | `clickhouse` | Hôte ClickHouse |
|
||||||
|
| `CLICKHOUSE_PORT` | `8123` | Port HTTP ClickHouse |
|
||||||
|
| `CLICKHOUSE_DB_PROCESSING` | `ja4_processing` | Base de traitement |
|
||||||
|
| `CLICKHOUSE_DB_LOGS` | `ja4_logs` | Base de logs |
|
||||||
|
| `CLICKHOUSE_USER` | `analyst` | Utilisateur en lecture |
|
||||||
|
| `CLICKHOUSE_PASSWORD` | — | Mot de passe |
|
||||||
|
| `API_HOST` | `0.0.0.0` | Adresse d'écoute de l'API |
|
||||||
|
| `API_PORT` | `8000` | Port de l'API FastAPI |
|
||||||
|
| `CORS_ORIGINS` | `["http://localhost:3000"]` | Origines CORS autorisées |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Étape 7 — Vérification de bout en bout
|
||||||
|
|
||||||
|
### 1. Vérifier que les services tournent
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Services systemd (Go)
|
||||||
|
sudo systemctl status ja4sentinel logcorrelator httpd
|
||||||
|
|
||||||
|
# Services Docker (Python)
|
||||||
|
docker compose -f services/bot-detector/docker-compose.yml ps
|
||||||
|
docker compose -f services/dashboard/docker-compose.yaml ps
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Vérifier l'ingestion des logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Logs bruts ingérés par le correlator
|
||||||
|
clickhouse-client --query "SELECT count() FROM ja4_logs.http_logs_raw"
|
||||||
|
|
||||||
|
# Logs parsés par la vue matérialisée
|
||||||
|
clickhouse-client --query "SELECT count() FROM ja4_logs.http_logs"
|
||||||
|
|
||||||
|
# Derniers logs
|
||||||
|
clickhouse-client --query "
|
||||||
|
SELECT time, src_ip, method, host, path, ja4
|
||||||
|
FROM ja4_logs.http_logs
|
||||||
|
ORDER BY time DESC
|
||||||
|
LIMIT 5
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Vérifier les agrégations
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clickhouse-client --query "
|
||||||
|
SELECT window_start, count()
|
||||||
|
FROM ja4_processing.agg_host_ip_ja4_1h
|
||||||
|
GROUP BY window_start
|
||||||
|
ORDER BY window_start DESC
|
||||||
|
LIMIT 5
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Vérifier les détections ML
|
||||||
|
|
||||||
|
```bash
|
||||||
|
clickhouse-client --query "
|
||||||
|
SELECT src_ip, anomaly_score, label, detected_at
|
||||||
|
FROM ja4_processing.ml_detected_anomalies
|
||||||
|
ORDER BY detected_at DESC
|
||||||
|
LIMIT 5
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Tester le dashboard
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8000/api/metrics | python3 -m json.tool | head -20
|
||||||
|
curl -s http://localhost:8000/api/health
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Schéma réseau récapitulatif
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────┐ Unix socket ┌──────────────┐ ClickHouse ┌──────────────┐
|
||||||
|
│ mod-reqin-log│──── http.socket ─────→│ │ INSERT INTO │ │
|
||||||
|
│ (Apache) │ (source A) │ correlator │───→ ja4_logs. │ ClickHouse │
|
||||||
|
└──────────────┘ │ │ http_logs_raw │ │
|
||||||
|
│ │ │ ┌─────────┐ │
|
||||||
|
┌──────────────┐ Unix socket │ │ MV parse JSON │ │ja4_logs │ │
|
||||||
|
│ sentinel │──── network.socket ──→│ │ ↓ │ │ _raw │ │
|
||||||
|
│ (TLS capture)│ (source B) └──────────────┘ ja4_logs. │ │ _parsed │ │
|
||||||
|
└──────────────┘ http_logs │ └─────────┘ │
|
||||||
|
│ │ │
|
||||||
|
MV agrégation │ ┌─────────┐ │
|
||||||
|
↓ │ │ja4_ │ │
|
||||||
|
┌──────────────┐ SELECT features ┌──────────────┐ ja4_processing. │ │processing│
|
||||||
|
│ bot-detector │←─── view_ai_features ─│ │ agg_*, view_* │ │ _agg │ │
|
||||||
|
│ (ML/Python) │ │ ClickHouse │ │ │ _ml │ │
|
||||||
|
│ │───→ INSERT scores ────→│ │ ml_all_scores │ │ _views │ │
|
||||||
|
└──────────────┘ ml_detected_* └──────────────┘ ml_detected_* │ └─────────┘ │
|
||||||
|
└──────────────┘
|
||||||
|
┌──────────────┐ SELECT * ↑
|
||||||
|
│ dashboard │←─── ja4_processing.* ───────────────────────────┘
|
||||||
|
│ (FastAPI) │←─── ja4_logs.http_logs ─────────────────────────┘
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user