feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
10
services/bot-detector/.env.example
Normal file
10
services/bot-detector/.env.example
Normal file
@ -0,0 +1,10 @@
|
||||
# bot-detector configuration — DO NOT COMMIT real values
|
||||
CLICKHOUSE_HOST=clickhouse
|
||||
CLICKHOUSE_PORT=8123
|
||||
CLICKHOUSE_DB=mabase_prod
|
||||
CLICKHOUSE_USER=admin
|
||||
CLICKHOUSE_PASSWORD=
|
||||
ANOMALY_THRESHOLD=-0.1
|
||||
DEDUP_TTL_MIN=60
|
||||
HEALTH_PORT=8080
|
||||
MIN_VALID_FEATURE_RATIO=0.5
|
||||
2
services/bot-detector/.gitignore
vendored
Normal file
2
services/bot-detector/.gitignore
vendored
Normal file
@ -0,0 +1,2 @@
|
||||
bot_detector_models/
|
||||
bot_detector_logs/
|
||||
204
services/bot-detector/CLICKHOUSE_FEATURES_DIAGNOSTIC.md
Normal file
204
services/bot-detector/CLICKHOUSE_FEATURES_DIAGNOSTIC.md
Normal file
@ -0,0 +1,204 @@
|
||||
# Diagnostic — Features manquantes dans `view_ai_features_1h`
|
||||
|
||||
> Généré le 2026-03-17 — Mis à jour le 2026-03-17 (corrections appliquées) — À destination de l'administrateur ClickHouse
|
||||
|
||||
## ✅ Statut des corrections (2026-03-17 13:05)
|
||||
|
||||
| Problème | Correction appliquée | Résultat |
|
||||
|----------|---------------------|----------|
|
||||
| **1** — MV `mv_agg_header_fingerprint_1h` absente | MV recréée + backfill 25h | ✅ 10 features header actives |
|
||||
| **2** — `header_order_shared_count` / `distinct_header_orders` globales | Se corrige avec Problème 1 | ✅ Résolu automatiquement |
|
||||
| **3** — `orphan_ratio` = 0 pour `correlated=1` | Comportement normal (by design) | ℹ️ Pas d'action requise |
|
||||
| **4** — 4 vues dashboard absentes | Vues créées | ✅ |
|
||||
| **5** — `view_dashboard_variability` référence `header_user_agent` inexistant | Colonne remplacée par `reason` | ✅ Bug corrigé |
|
||||
| **6** — Anciennes vues heuristiques orphelines | Droppées | ✅ |
|
||||
|
||||
Cycle post-correction (13:05) — features dans les warnings :
|
||||
- `Complet` : seulement `orphan_ratio` (by design)
|
||||
- `Applicatif` : `request_size_variance`, `mss_mobile_mismatch`, `is_rare_ja4` (see §4 below)
|
||||
- Header features **disparues des warnings** → pipeline opérationnel ✅
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Résumé
|
||||
|
||||
Le service Bot Detector signale des **features non-discriminantes** à chaque cycle. Ce document en explique les causes exactes et les corrections nécessaires côté ClickHouse.
|
||||
|
||||
Ces avertissements **n'empêchent pas le service de fonctionner** — les features invalides sont automatiquement exclues du modèle (A7). Mais leur absence réduit la qualité de la détection.
|
||||
|
||||
---
|
||||
|
||||
## Problème 1 — Pipeline `agg_header_fingerprint_1h` arrêté ⚠️ CRITIQUE
|
||||
|
||||
### Symptôme
|
||||
|
||||
Les features suivantes sont toujours à **0** dans `view_ai_features_1h` :
|
||||
|
||||
- `header_count`
|
||||
- `has_accept_language`
|
||||
- `has_cookie`
|
||||
- `has_referer`
|
||||
- `modern_browser_score`
|
||||
- `ua_ch_mismatch`
|
||||
- `mss_mobile_mismatch` *(dépend de `modern_browser_score`)*
|
||||
|
||||
### Cause
|
||||
|
||||
La table `mabase_prod.agg_header_fingerprint_1h` (AggregatingMergeTree) n'a plus reçu de données depuis le **2026-03-13 23:00** :
|
||||
|
||||
```sql
|
||||
SELECT max(window_start), count()
|
||||
FROM mabase_prod.agg_header_fingerprint_1h;
|
||||
-- Résultat : 2026-03-13 23:00:00, 73024 lignes
|
||||
```
|
||||
|
||||
La vue fait un `LEFT JOIN` avec condition `window_start >= now() - INTERVAL 24 HOUR`, et comme aucune ligne récente n'existe dans `agg_header_fingerprint_1h`, **toutes les colonnes issues de ce JOIN retournent NULL** (→ 0 après coalesce).
|
||||
|
||||
### Recherche de la MV source
|
||||
|
||||
La liste des Materialized Views ne montre aucune MV dédiée à `agg_header_fingerprint_1h` :
|
||||
|
||||
```sql
|
||||
SELECT name FROM system.tables
|
||||
WHERE database = 'mabase_prod' AND engine = 'MaterializedView';
|
||||
-- mv_agg_host_ip_ja4_1h
|
||||
-- mv_http_logs
|
||||
-- view_dashboard_entities_mv
|
||||
-- view_dashboard_user_agents_mv
|
||||
```
|
||||
|
||||
Aucune MV ne cible `agg_header_fingerprint_1h`. Elle est probablement alimentée par un **processus externe** (ETL, script, pipeline Kafka, etc.) qui s'est arrêté.
|
||||
|
||||
### Correction appliquée ✅
|
||||
|
||||
La MV `mv_agg_header_fingerprint_1h` était **définie dans `deploy_views.sql`** mais n'avait jamais été créée en base. Elle a été recréée le 2026-03-17 :
|
||||
|
||||
```sql
|
||||
-- Recréation de la MV (déjà appliquée)
|
||||
CREATE MATERIALIZED VIEW mabase_prod.mv_agg_header_fingerprint_1h
|
||||
TO mabase_prod.agg_header_fingerprint_1h AS
|
||||
SELECT
|
||||
toStartOfHour(src.time) AS window_start,
|
||||
toIPv6(src.src_ip) AS src_ip,
|
||||
any(toString(cityHash64(src.client_headers))) AS header_order_hash,
|
||||
max(toUInt16(length(src.client_headers) - length(replaceAll(src.client_headers, ',', '')) + 1)) AS header_count,
|
||||
-- ... (voir deploy_views.sql §5)
|
||||
FROM mabase_prod.http_logs AS src
|
||||
GROUP BY window_start, src.src_ip;
|
||||
```
|
||||
|
||||
Un **backfill de 25 heures** a été effectué depuis `http_logs` pour alimenter la table avec des données historiques (377 689 lignes insérées). Les nouvelles données sont désormais alimentées en temps réel par la MV.
|
||||
|
||||
### Cause historique
|
||||
|
||||
La MV avait été omise lors du déploiement initial. La table `agg_header_fingerprint_1h` contenait 73 024 lignes datant du 2026-03-13 (probablement issues d'un backfill manuel ponctuel), puis n'avait plus été alimentée.
|
||||
|
||||
---
|
||||
|
||||
## Problème 2 — Features non-discriminantes (agrégat global, non per-IP)
|
||||
|
||||
### Symptôme
|
||||
|
||||
Les features suivantes ont une **valeur unique non-nulle identique pour toutes les IPs** :
|
||||
|
||||
- `header_order_shared_count` (valeur ≈ 421 000 pour toutes les lignes)
|
||||
- `distinct_header_orders` (valeur identique pour toutes les lignes)
|
||||
|
||||
### Cause
|
||||
|
||||
Ces features sont calculées via des window functions `PARTITION BY header_order_hash` :
|
||||
|
||||
```sql
|
||||
-- Dans la vue :
|
||||
count() OVER (PARTITION BY h.header_order_hash) AS header_order_shared_count
|
||||
uniqExact(h.header_order_hash) OVER (PARTITION BY a.src_ip) AS distinct_header_orders
|
||||
```
|
||||
|
||||
Comme `h.header_order_hash` est **NULL pour toutes les lignes** (problème 1 ci-dessus), la `PARTITION BY NULL` regroupe **toutes les lignes dans une seule partition** → `count()` retourne le total de toutes les lignes pour chaque IP.
|
||||
|
||||
### Correction ✅ (auto-résolue avec Problème 1)
|
||||
|
||||
Ce problème s'est résolu automatiquement une fois la MV `mv_agg_header_fingerprint_1h` recréée. `header_order_hash` est désormais non-NULL, les partitions de window functions sont correctement calculées par hash d'ordre d'en-têtes.
|
||||
|
||||
---
|
||||
|
||||
## Problème 3 — `orphan_ratio` absent pour le trafic corrélé TCP
|
||||
|
||||
### Symptôme
|
||||
|
||||
`orphan_ratio` = 0 pour **toutes les lignes avec `correlated = 1`** (trafic TCP enrichi).
|
||||
|
||||
### Cause
|
||||
|
||||
La colonne `orphan_count` dans `mabase_prod.agg_host_ip_ja4_1h` est calculée par la MV `mv_agg_host_ip_ja4_1h` :
|
||||
|
||||
```sql
|
||||
sum(IF(src.orphan_side = 'A' OR src.correlated = 0, 1, 0)) AS orphan_count
|
||||
```
|
||||
|
||||
Pour les connexions `correlated=1`, `correlated = 0` est toujours faux, et `orphan_side = 'A'` n'est jamais vrai pour le trafic corrélé → `orphan_count = 0` systématiquement.
|
||||
|
||||
**C'est un comportement intentionnel** : les connexions TCP corrélées ont une réponse confirmée, donc elles ne sont pas des requêtes orphelines par définition.
|
||||
|
||||
### Statut
|
||||
|
||||
Pas d'action requise. La feature reste exclue automatiquement par A7 pour le modèle `Complet` (correlated=1).
|
||||
|
||||
---
|
||||
|
||||
## Problème 4 — Features à 0 persistantes dans le modèle Applicatif
|
||||
|
||||
### Symptôme (post-correction)
|
||||
|
||||
Depuis le 2026-03-17 13:05, le modèle `Applicatif` (trafic non-corrélé) signale encore ces features à 0 :
|
||||
|
||||
- `request_size_variance`
|
||||
- `mss_mobile_mismatch`
|
||||
- `is_rare_ja4`
|
||||
|
||||
### Cause
|
||||
|
||||
Ces features sont calculées depuis des colonnes L4/TCP qui sont **absent ou non-pertinentes pour le trafic applicatif pur** (`correlated=0`) :
|
||||
|
||||
| Feature | Cause |
|
||||
|---------|-------|
|
||||
| `request_size_variance` | `varPopMerge(total_ip_length_var)` — variance de longueur IP ; trafic non-corrélé = pas de données IP brutes fiables |
|
||||
| `mss_mobile_mismatch` | Dépend de `tcp_meta_mss` et `modern_browser_score` — MSS non fiable sans corrélation TCP |
|
||||
| `is_rare_ja4` | `sum(hits) OVER (PARTITION BY ja4) < 100` — dans la fenêtre Applicatif (1h, trafic réduit), tous les JA4 sont rares |
|
||||
|
||||
### Impact
|
||||
|
||||
Faible — ces features sont exclues automatiquement (A7). Elles ne dégradent pas le modèle.
|
||||
|
||||
---
|
||||
|
||||
## Impact sur le modèle IA
|
||||
|
||||
| Feature | Impact si absente | Statut |
|
||||
|---------|-------------------|--------|
|
||||
| `header_count` | Perte d'un signal fort : bots envoient souvent peu d'en-têtes | ✅ Corrigé |
|
||||
| `has_accept_language` | Perte de détection des bots sans localisation | ✅ Corrigé |
|
||||
| `has_cookie` | Perte de détection des sessions sans état | ✅ Corrigé |
|
||||
| `has_referer` | Perte du signal de navigation directe | ✅ Corrigé |
|
||||
| `modern_browser_score` | Perte du score composite de conformité navigateur | ✅ Corrigé |
|
||||
| `ua_ch_mismatch` | Perte de détection des fausses déclarations UA | ✅ Corrigé |
|
||||
| `header_order_shared_count` | Perte de la détection de fingerprints d'en-têtes partagés | ✅ Corrigé |
|
||||
| `orphan_ratio` | Signal faible pour trafic corrélé | ℹ️ By design |
|
||||
| `request_size_variance` | Signal L4 faible pour Applicatif | ℹ️ Normal |
|
||||
| `mss_mobile_mismatch` | Signal TCP faible pour Applicatif | ℹ️ Normal |
|
||||
|
||||
---
|
||||
|
||||
## Vérification post-correction
|
||||
|
||||
Cycle du 2026-03-17 13:05 — résultat observé :
|
||||
|
||||
```
|
||||
[Complet] Features à 0 : ['orphan_ratio'] ← by design ✅
|
||||
[Applicatif] Features à 0 : ['request_size_variance', 'mss_mobile_mismatch', 'is_rare_ja4'] ← normales ✅
|
||||
[Applicatif] Features non-discriminantes : ['tcp_shared_count'] ← agrégat global résiduel
|
||||
```
|
||||
|
||||
Les **10 features header** (`header_count`, `has_accept_language`, `has_cookie`, `has_referer`, `modern_browser_score`, `ua_ch_mismatch`, `header_order_shared_count`, `distinct_header_orders`, `header_order_confidence`, `mss_mobile_mismatch` pour Complet) **ne sont plus dans les warnings**. Le pipeline est opérationnel.
|
||||
710
services/bot-detector/DOCUMENTATION.md
Normal file
710
services/bot-detector/DOCUMENTATION.md
Normal file
@ -0,0 +1,710 @@
|
||||
# Bot Detector IA — Documentation Technique
|
||||
|
||||
> Version du code : v11 | Dernière mise à jour : 2026-03-17
|
||||
|
||||
---
|
||||
|
||||
## Table des matières
|
||||
|
||||
1. [Vue d'ensemble](#1-vue-densemble)
|
||||
2. [Architecture système](#2-architecture-système)
|
||||
3. [Pipeline de détection](#3-pipeline-de-détection)
|
||||
4. [Modèles et features](#4-modèles-et-features)
|
||||
5. [Approche semi-supervisée](#5-approche-semi-supervisée)
|
||||
6. [Gestion des modèles](#6-gestion-des-modèles)
|
||||
7. [Données d'entrée — vue ClickHouse](#7-données-dentrée--vue-clickhouse)
|
||||
8. [Données de sortie](#8-données-de-sortie)
|
||||
9. [Configuration](#9-configuration)
|
||||
10. [Observabilité](#10-observabilité)
|
||||
11. [Réputation et enrichissement](#11-réputation-et-enrichissement)
|
||||
12. [Fondements scientifiques](#12-fondements-scientifiques)
|
||||
13. [Améliorations implémentées (v11)](#13-améliorations-implémentées-v11)
|
||||
14. [Migration de schéma ClickHouse](#14-migration-de-schéma-clickhouse)
|
||||
|
||||
---
|
||||
|
||||
## 1. Vue d'ensemble
|
||||
|
||||
Le **Bot Detector IA** est un service de détection d'activité suspecte et de bots sur un trafic HTTP. Il tourne en boucle continue (toutes les 5 minutes par défaut) et analyse des données agrégées issues de ClickHouse.
|
||||
|
||||
### Principe général
|
||||
|
||||
```
|
||||
ClickHouse (view_ai_features_1h)
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ Séparation du trafic │
|
||||
│ ├─ Bots connus │ → Étiquetés via réputation IP / JA4 / ASN
|
||||
│ ├─ Trafic humain │ → Sert de baseline d'entraînement pour l'IF
|
||||
│ └─ Trafic inconnu │ → Scoré par Isolation Forest
|
||||
└───────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ Isolation Forest │
|
||||
│ (semi-supervisé) │
|
||||
│ ├─ Modèle Complet │ TCP + TLS + HTTP (35 features, correlated=1)
|
||||
│ └─ Modèle Applicatif │ HTTP seul (31 features, correlated=0)
|
||||
└───────────────────────┘
|
||||
│
|
||||
▼
|
||||
ClickHouse (ml_detected_anomalies)
|
||||
```
|
||||
|
||||
### Caractéristiques clés
|
||||
|
||||
| Propriété | Valeur |
|
||||
|-----------|--------|
|
||||
| Algorithme | Isolation Forest (sklearn) |
|
||||
| Supervision | Semi-supervisée (baseline humain + réputation) |
|
||||
| Fenêtre d'analyse | 1 heure glissante (optionnel : 24h avec `ENABLE_MULTIWINDOW`) |
|
||||
| Cycle d'exécution | 300 s (configurable) |
|
||||
| Re-entraînement | Toutes les 1 h (configurable) + retrain forcé sur dérive conceptuelle |
|
||||
| Contamination | 2 % (fraction d'anomalies attendues dans la baseline) |
|
||||
| Seuil d'anomalie | Adaptatif : min(percentile_5, -0.03) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture système
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Docker Compose │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ bot_detector_ai │ │
|
||||
│ │ │ │
|
||||
│ │ ┌────────────┐ ┌──────────────┐ ┌─────────────────┐ │ │
|
||||
│ │ │ Health │ │ Main Loop │ │ ClickHouse │ │ │
|
||||
│ │ │ :8080 │ │ (300s cycle)│ │ Client │ │ │
|
||||
│ │ │ (thread) │ │ │ │ (reconnect) │ │ │
|
||||
│ │ └────────────┘ └──────────────┘ └─────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ Volumes: │ │
|
||||
│ │ ├─ ./bot_detector_logs → /var/log/bot_detector │ │
|
||||
│ │ ├─ ./bot_detector_models → /var/lib/bot_detector │ │
|
||||
│ │ ├─ ./reputation/data/user_files/bot_ip.csv (ro) │ │
|
||||
│ │ ├─ ./reputation/data/user_files/bot_ja4.csv (ro) │ │
|
||||
│ │ └─ ./reputation/data/user_files/asn_reputation.csv (ro) │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└────────────────────────────┬────────────────────────────────────┘
|
||||
│ HTTP :8123
|
||||
▼
|
||||
ClickHouse externe
|
||||
(test-sdv-anubis.sdv.fr)
|
||||
```
|
||||
|
||||
### Fichiers et répertoires
|
||||
|
||||
| Chemin | Rôle |
|
||||
|--------|------|
|
||||
| `bot_detector/bot_detector.py` | Code source principal |
|
||||
| `bot_detector/requirements.txt` | Dépendances Python |
|
||||
| `bot_detector/Dockerfile` | Image Python 3.11-slim |
|
||||
| `docker-compose.yml` | Orchestration Docker |
|
||||
| `.env` | Variables d'environnement (non commité) |
|
||||
| `bot_detector_logs/decisions.jsonl` | Journal JSONL structuré (rotation 50 MB × 7) |
|
||||
| `bot_detector_models/model_<name>_<version>.joblib` | Modèle sérialisé |
|
||||
| `bot_detector_models/model_<name>_<version>.meta.json` | Métadonnées du modèle |
|
||||
| `bot_detector_models/model_<name>.current` | Pointeur vers la version active |
|
||||
| `bot_detector_models/training_history.jsonl` | Historique des entraînements |
|
||||
| `reputation/bot_ip.csv` | ~288 k entrées IP/CIDR de bots connus |
|
||||
| `reputation/bot_ja4.csv` | Empreintes JA4 de bots |
|
||||
| `reputation/asn_reputation.csv` | Labels ASN (human / bot) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Pipeline de détection
|
||||
|
||||
### 3.1 Cycle principal (`fetch_and_analyze`)
|
||||
|
||||
```
|
||||
1. Génération d'un cycle_id (timestamp)
|
||||
2. Requête view_ai_features_1h → DataFrame df
|
||||
3. Requête view_ip_recurrence → recurrence_map {src_ip: count}
|
||||
4. Nettoyage des colonnes (fillna, astype)
|
||||
5. Log CYCLE_START (total, human, known_bot, correlated)
|
||||
6. Séparation df → correlated=1 / correlated=0
|
||||
7. Appel run_semi_supervised_logic() × 2 (modèle Complet + Applicatif)
|
||||
8. Concaténation, déduplication par src_ip (score le plus bas)
|
||||
9. Insertion dans ml_detected_anomalies
|
||||
10. Log CYCLE_END
|
||||
11. Attente CYCLE_INTERVAL secondes
|
||||
```
|
||||
|
||||
### 3.2 Logique semi-supervisée (`run_semi_supervised_logic`)
|
||||
|
||||
```
|
||||
df (trafic de la fenêtre 1h)
|
||||
│
|
||||
├─ A7 → validate_features() : exclusion des features manquantes ou constantes
|
||||
│
|
||||
├─ bot_name != '' → known_bots → KNOWN_BOT (log + insertion)
|
||||
│
|
||||
└─ bot_name == '' → unknown_traffic
|
||||
│
|
||||
├─ asn_label == 'human' → human_baseline
|
||||
│ (min. 500 sessions requis)
|
||||
│ └──► load_or_train_model()
|
||||
│ ├─ A1 : drift check (z-score / features)
|
||||
│ └─ Si drift ≥ DRIFT_THRESHOLD : retrain forcé
|
||||
│
|
||||
└─ reste du trafic inconnu
|
||||
│
|
||||
▼
|
||||
IsolationForest.decision_function() → raw_scores
|
||||
│
|
||||
A10 : normalize_scores() → anomaly_score [-1, 0]
|
||||
│
|
||||
A2 : effective_threshold = min(percentile_5, ANOMALY_THRESHOLD)
|
||||
│
|
||||
A6 : raw_score -= log1p(recurrence) × RECURRENCE_WEIGHT
|
||||
│
|
||||
raw_score < effective_threshold ?
|
||||
│
|
||||
YES → A4 : SHAP top-5 features → reason
|
||||
A8 : DBSCAN clustering → campaign_id
|
||||
ANOMALY (log + insertion)
|
||||
│
|
||||
NO → ignoré
|
||||
```
|
||||
|
||||
### 3.3 Niveaux de menace
|
||||
|
||||
| Score | Niveau | Interprétation |
|
||||
|-------|--------|----------------|
|
||||
| `< -0.30` | **CRITICAL** | Comportement extrêmement anormal |
|
||||
| `< -0.15` | **HIGH** | Fort signal d'anomalie |
|
||||
| `< -0.05` | **MEDIUM** | Anomalie modérée |
|
||||
| `≥ -0.05` | **LOW** | Légèrement inhabituel |
|
||||
|
||||
> Le seuil d'insertion (`ANOMALY_THRESHOLD = -0.03`) est plus permissif que LOW. Toutes les IP dont le score passe sous ce seuil sont insérées, quelle que soit leur catégorie de niveau.
|
||||
|
||||
---
|
||||
|
||||
## 4. Modèles et features
|
||||
|
||||
### 4.1 Architecture à deux niveaux
|
||||
|
||||
| Modèle | Condition | Nb features | Données utilisées |
|
||||
|--------|-----------|-------------|-------------------|
|
||||
| **Complet** | `correlated = 1` | 35 | HTTP + TCP + TLS |
|
||||
| **Applicatif** | `correlated = 0` | 31 | HTTP uniquement |
|
||||
|
||||
La corrélation (`correlated`) indique si les logs HTTP ont pu être enrichis avec les données TCP/TLS de la même connexion. En l'absence de corrélation (capture incomplète ou trafic chiffré sans inspection), seul le modèle Applicatif est utilisé.
|
||||
|
||||
### 4.2 Features communes (31 — modèle Applicatif)
|
||||
|
||||
#### Comportement HTTP de base
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `hits` | Nombre de requêtes sur la fenêtre |
|
||||
| `hit_velocity` | Requêtes par seconde |
|
||||
| `fuzzing_index` | Score de diversité anormale des chemins/paramètres |
|
||||
| `post_ratio` | Fraction de requêtes POST |
|
||||
| `port_exhaustion_ratio` | Fraction de ports sources différents / total ports |
|
||||
| `orphan_ratio` | Requêtes sans réponse associée |
|
||||
|
||||
#### Gestion des connexions
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `max_keepalives` | Nb max de requêtes sur une même connexion keep-alive |
|
||||
| `tcp_shared_count` | Connexions TCP partagées entre plusieurs sessions HTTP |
|
||||
|
||||
#### Empreinte navigateur (Browser Fingerprint)
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `header_count` | Nombre d'en-têtes HTTP envoyés |
|
||||
| `has_accept_language` | Présence de Accept-Language |
|
||||
| `has_cookie` | Présence de Cookie |
|
||||
| `has_referer` | Présence de Referer |
|
||||
| `modern_browser_score` | Score composite de conformité navigateur moderne |
|
||||
| `ua_ch_mismatch` | Incohérence entre User-Agent et Client Hints |
|
||||
| `ip_id_zero_ratio` | Ratio de paquets IP avec ID=0 (headless / stack minimale) |
|
||||
| `header_order_shared_count` | Partage d'un même ordre d'en-têtes entre IPs |
|
||||
| `header_order_confidence` | Confiance dans l'ordre d'en-têtes (entropie normalisée) |
|
||||
| `distinct_header_orders` | Nombre d'ordres d'en-têtes distincts observés |
|
||||
|
||||
#### Patterns de navigation
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `request_size_variance` | Variance de la taille des requêtes |
|
||||
| `multiplexing_efficiency` | Efficacité du multiplexage HTTP/2 |
|
||||
| `mss_mobile_mismatch` | Incohérence MSS TCP / profil mobile annoncé |
|
||||
| `asset_ratio` | Fraction de requêtes vers des ressources statiques |
|
||||
| `direct_access_ratio` | Fraction d'accès directs (sans referer) |
|
||||
| `is_ua_rotating` | Rotation de User-Agent détectée (flag 0/1) |
|
||||
| `distinct_ja4_count` | Nombre de fingerprints JA4 distincts par IP |
|
||||
|
||||
#### Concentration et rareté
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `src_port_density` | Densité des ports sources (entropy) |
|
||||
| `ja4_asn_concentration` | Concentration d'un même JA4 dans un ASN |
|
||||
| `ja4_country_concentration` | Concentration d'un même JA4 par pays |
|
||||
| `is_rare_ja4` | JA4 peu commun dans la population (flag 0/1) |
|
||||
|
||||
#### Dimensions temporelles et de diversité (académiques)
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `temporal_entropy` | Entropie de la distribution temporelle des requêtes |
|
||||
| `path_diversity_ratio` | Diversité des chemins URL accédés |
|
||||
| `url_depth_variance` | Variance de la profondeur des URL |
|
||||
| `anomalous_payload_ratio` | Fraction de payloads avec patterns anormaux |
|
||||
|
||||
### 4.3 Features additionnelles TCP/TLS (modèle Complet uniquement)
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| `tcp_jitter_variance` | Variance de la gigue inter-paquets TCP |
|
||||
| `alpn_http_mismatch` | Incohérence entre ALPN négocié et protocole HTTP effectif |
|
||||
| `is_alpn_missing` | ALPN absent dans le TLS ClientHello |
|
||||
| `sni_host_mismatch` | Incohérence entre SNI TLS et Host HTTP |
|
||||
|
||||
---
|
||||
|
||||
## 5. Approche semi-supervisée
|
||||
|
||||
### 5.1 Fondement théorique
|
||||
|
||||
L'**Isolation Forest** (Liu, Ting & Zhou, 2008) est un algorithme d'apprentissage non supervisé conçu pour la détection d'anomalies. Son principe : les anomalies, étant rares et différentes, sont **isolées en moins de partitions** dans un arbre de décision aléatoire que les points normaux.
|
||||
|
||||
Le score de décision (`decision_function`) est normalisé entre -1 (très anormal) et +1 (très normal). Le paramètre `contamination` fixe la fraction de points considérés comme anomalies dans l'ensemble d'entraînement.
|
||||
|
||||
### 5.2 Dimension semi-supervisée
|
||||
|
||||
L'approche est **semi-supervisée** car :
|
||||
|
||||
1. **Étiquetage partiel** : Les bots connus (via réputation IP/JA4) et les humains (via réputation ASN) sont identifiés *a priori*.
|
||||
2. **Entraînement sur la classe normale uniquement** : L'IF est entraîné **exclusivement sur la baseline humaine** (`asn_label = 'human'`, `bot_name = ''`). Il apprend ainsi le profil du trafic légitime.
|
||||
3. **Détection par déviation** : Tout trafic inconnu qui s'éloigne du profil humain est scoré négativement.
|
||||
|
||||
Cette approche suit le paradigme **One-Class Classification** (Tax & Duin, 2004) appliqué à la détection de bots, proche des travaux de Kruegel & Vigna (2003) sur la détection d'anomalies réseau.
|
||||
|
||||
### 5.3 Qualité de la baseline humaine
|
||||
|
||||
Le minimum de 500 sessions humaines est une garde-fou empirique. En dessous de ce seuil, l'IF ne dispose pas de suffisamment d'exemples pour définir un profil normal robuste, augmentant le risque de faux positifs.
|
||||
|
||||
En pratique, les cycles observés montrent entre **1 264** et **1 725** sessions humaines par fenêtre d'une heure.
|
||||
|
||||
---
|
||||
|
||||
## 6. Gestion des modèles
|
||||
|
||||
### 6.1 Cycle de vie d'un modèle
|
||||
|
||||
```
|
||||
Démarrage cycle
|
||||
│
|
||||
▼
|
||||
Existe un .current ? ──NON──► Entraîner nouveau modèle
|
||||
│
|
||||
OUI
|
||||
│
|
||||
▼
|
||||
Âge < RETRAIN_INTERVAL_H ?
|
||||
│ │
|
||||
OUI NON
|
||||
│ │
|
||||
▼ └──► Entraîner nouveau modèle
|
||||
A1 : Drift check (MODEL_TRAINED)
|
||||
(z-score vs baseline_stats)
|
||||
│
|
||||
Drift ≥ DRIFT_THRESHOLD ?
|
||||
│ │
|
||||
NON OUI
|
||||
│ │
|
||||
Charger modèle Entraîner nouveau modèle
|
||||
(MODEL_LOADED) (DRIFT_DETECTED + MODEL_TRAINED)
|
||||
```
|
||||
|
||||
### 6.2 Versioning des modèles
|
||||
|
||||
Chaque modèle est identifié par un `version_id` au format `YYYYMMDD_HHMMSS`. Les fichiers associés sont :
|
||||
|
||||
- `model_{name}_{version_id}.joblib` — modèle sérialisé (joblib/pickle)
|
||||
- `model_{name}_{version_id}.meta.json` — métadonnées (features, contamination, nb samples, etc.)
|
||||
- `model_{name}.current` — pointeur atomique vers la version active
|
||||
|
||||
L'historique est limité à `MODEL_HISTORY_COUNT` versions (72 en production = 3 jours à 1 h de retrain).
|
||||
|
||||
Le fichier `.meta.json` contient maintenant un champ `baseline_stats` avec les statistiques de distribution (mean, std, p25, p75) de chaque feature, utilisées pour la détection de dérive (A1).
|
||||
|
||||
### 6.3 Paramètres Isolation Forest
|
||||
|
||||
```python
|
||||
IsolationForest(
|
||||
n_estimators=300, # Nombre d'arbres (compromis précision/temps)
|
||||
contamination=0.02, # 2% d'anomalies estimées dans la baseline
|
||||
random_state=42, # Reproductibilité
|
||||
n_jobs=-1 # Parallélisation sur tous les cores
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Données d'entrée — vue ClickHouse
|
||||
|
||||
### 7.1 Vue principale : `view_ai_features_1h`
|
||||
|
||||
Agrégation sur 1 heure glissante, une ligne par `src_ip`. Colonnes clés :
|
||||
|
||||
| Colonne | Type | Source |
|
||||
|---------|------|--------|
|
||||
| `src_ip` | String | TCP/IP |
|
||||
| `ja4` | String | TLS fingerprint (JA4+) |
|
||||
| `host` | String | HTTP Host header |
|
||||
| `bot_name` | String | Réputation IP/JA4 (vide si inconnu) |
|
||||
| `asn_number` | String | GeoIP/ASN lookup |
|
||||
| `asn_org` | String | Organisation ASN |
|
||||
| `asn_domain` | String | Domaine ASN |
|
||||
| `country_code` | String | Pays source |
|
||||
| `asn_label` | String | `human` / `bot` / `unknown` |
|
||||
| `correlated` | Int | 1 si TCP/TLS disponible, 0 sinon |
|
||||
| `hits` | Float | Nb requêtes |
|
||||
| `hit_velocity` | Float | Req/s |
|
||||
| *…(26+ features)* | Float | Voir section 4.2 |
|
||||
|
||||
### 7.2 Vue de récurrence : `view_ip_recurrence`
|
||||
|
||||
```sql
|
||||
SELECT src_ip, recurrence FROM {DB}.view_ip_recurrence
|
||||
```
|
||||
|
||||
Donne le nombre de fois qu'une IP a déjà été détectée comme menace dans l'historique. Enrichit le champ `recurrence` dans la sortie.
|
||||
|
||||
---
|
||||
|
||||
## 8. Données de sortie
|
||||
|
||||
### 8.1 Table ClickHouse : `ml_detected_anomalies`
|
||||
|
||||
Toutes les anomalies et bots connus détectés sont insérés dans cette table. Colonnes notables :
|
||||
|
||||
| Colonne | Description |
|
||||
|---------|-------------|
|
||||
| `detected_at` | Timestamp de détection |
|
||||
| `src_ip` | IP source |
|
||||
| `ja4` | Fingerprint TLS/JA4 (`HTTP_CLEAR_TEXT` si absent) |
|
||||
| `host` | Vhost ciblé |
|
||||
| `bot_name` | Nom du bot (vide si anomalie IF) |
|
||||
| `anomaly_score` | Score IF (0.0 pour bots connus) |
|
||||
| `threat_level` | `CRITICAL` / `HIGH` / `MEDIUM` / `LOW` / `KNOWN_BOT` |
|
||||
| `model_name` | `Complet` ou `Applicatif` |
|
||||
| `recurrence` | Nb d'apparitions historiques + 1 |
|
||||
| `reason` | Description textuelle de l'anomalie |
|
||||
| `is_headless` | Dérivé de `is_fake_navigation` |
|
||||
| *…(toutes les features)* | Pour analyse post-mortem |
|
||||
|
||||
### 8.2 Journal JSONL : `decisions.jsonl`
|
||||
|
||||
Événements structurés en JSON Lines, rotatifs (50 MB × 7 fichiers).
|
||||
|
||||
| Événement | Déclencheur |
|
||||
|-----------|-------------|
|
||||
| `SERVICE_START` | Démarrage du conteneur |
|
||||
| `SERVICE_STOP` | Arrêt propre (SIGTERM/SIGINT) |
|
||||
| `CYCLE_START` | Début d'un cycle d'analyse |
|
||||
| `CYCLE_END` | Fin du cycle (résumé inserés) |
|
||||
| `MODEL_LOADED` | Réutilisation d'un modèle existant |
|
||||
| `MODEL_TRAINED` | Nouvel entraînement |
|
||||
| `KNOWN_BOT` | Bot connu identifié |
|
||||
| `ANOMALY` | Anomalie IF détectée |
|
||||
| `SKIPPED_LOW_DATA` | Cycle ignoré (baseline < 500) |
|
||||
| `CONSECUTIVE_FAILURES` | Erreur ClickHouse répétée |
|
||||
|
||||
---
|
||||
|
||||
## 9. Configuration
|
||||
|
||||
Toutes les valeurs sont passées via variables d'environnement (fichier `.env`).
|
||||
|
||||
| Variable | Défaut | Description |
|
||||
|----------|--------|-------------|
|
||||
| `CLICKHOUSE_HOST` | `clickhouse` | Hôte ClickHouse |
|
||||
| `CLICKHOUSE_DB` | `mabase_prod` | Base de données |
|
||||
| `CLICKHOUSE_USER` | `default` | Utilisateur |
|
||||
| `CLICKHOUSE_PASSWORD` | *(vide)* | Mot de passe |
|
||||
| `ISOLATION_CONTAMINATION` | `0.001` | Fraction d'anomalies attendues (0 < x < 0.5) |
|
||||
| `ANOMALY_THRESHOLD` | `-0.05` | Seuil statique de score pour insertion |
|
||||
| `CYCLE_INTERVAL_SEC` | `300` | Délai entre cycles (secondes) |
|
||||
| `MAX_CONSECUTIVE_FAILURES` | `3` | Échecs avant passage en DEGRADED |
|
||||
| `BOT_DETECTOR_LOG` | `/var/log/bot_detector/decisions.jsonl` | Fichier de log |
|
||||
| `LOG_BACKUP_COUNT` | `7` | Nb de rotations conservées |
|
||||
| `MODEL_DIR` | `/var/lib/bot_detector` | Répertoire des modèles |
|
||||
| `RETRAIN_INTERVAL_HOURS` | `24` | Fréquence de re-entraînement |
|
||||
| `MODEL_HISTORY_COUNT` | `10` | Nb de versions de modèles conservées |
|
||||
| `HEALTH_PORT` | `8080` | Port du health check HTTP |
|
||||
| **A1** `DRIFT_THRESHOLD` | `0.30` | Fraction de features déroutantes déclenchant un retrain forcé |
|
||||
| **A2** `ANOMALY_PERCENTILE` | `5` | Percentile pour le seuil adaptatif (0–20) |
|
||||
| **A3** `ENABLE_MULTIWINDOW` | `false` | Active l'analyse sur fenêtre 24h |
|
||||
| **A3** `MULTIWINDOW_VIEW` | `view_ai_features_24h` | Nom de la vue 24h dans ClickHouse |
|
||||
| **A4** `ENABLE_SHAP` | `true` | Active le calcul SHAP (désactivé si shap non installé) |
|
||||
| **A5** `DEDUP_TTL_MIN` | `60` | TTL de déduplication inter-cycles (0 = désactivé) |
|
||||
| **A6** `RECURRENCE_WEIGHT` | `0.005` | Pénalité de score par log(récurrence) |
|
||||
| **A7** `MIN_VALID_FEATURE_RATIO` | `0.50` | Ratio minimum de features valides pour procéder |
|
||||
| **A8** `ENABLE_CLUSTERING` | `true` | Active le clustering DBSCAN des anomalies |
|
||||
| **A8** `CLUSTERING_MIN_SAMPLES` | `3` | Taille minimale d'un cluster DBSCAN |
|
||||
|
||||
---
|
||||
|
||||
## 10. Observabilité
|
||||
|
||||
### 10.1 Health check
|
||||
|
||||
```bash
|
||||
GET http://localhost:8080/
|
||||
# → 200 OK service opérationnel
|
||||
# → 503 DEGRADED ≥ MAX_CONSECUTIVE_FAILURES échecs ClickHouse consécutifs
|
||||
```
|
||||
|
||||
### 10.2 Logs opérationnels
|
||||
|
||||
Les logs console suivent le format `[YYYY-MM-DD HH:MM:SS] message`. Le fichier JSONL permet des analyses post-mortem avec des outils comme `jq` :
|
||||
|
||||
```bash
|
||||
# Voir les dernières anomalies CRITICAL
|
||||
jq 'select(.event=="ANOMALY" and .threat_level=="CRITICAL")' decisions.jsonl
|
||||
|
||||
# Voir les top features SHAP pour les anomalies HIGH
|
||||
jq 'select(.event=="ANOMALY" and .threat_level=="HIGH") | .reason' decisions.jsonl
|
||||
|
||||
# Détecter les dérives de distribution
|
||||
jq 'select(.event=="DRIFT_DETECTED")' decisions.jsonl
|
||||
|
||||
# Voir les campagnes coordonnées (campaign_id >= 0)
|
||||
jq 'select(.event=="ANOMALY" and .campaign_id >= 0) | {src_ip, campaign_id, threat_level}' decisions.jsonl
|
||||
|
||||
# Compter les bots connus par nom
|
||||
jq -r 'select(.event=="KNOWN_BOT") | .bot_name' decisions.jsonl | sort | uniq -c | sort -rn
|
||||
|
||||
# Résumé des cycles
|
||||
jq 'select(.event=="CYCLE_END")' decisions.jsonl
|
||||
```
|
||||
|
||||
| Événement | Déclencheur |
|
||||
|-----------|-------------|
|
||||
| `SERVICE_START` | Démarrage du conteneur |
|
||||
| `SERVICE_STOP` | Arrêt propre (SIGTERM/SIGINT) |
|
||||
| `CYCLE_START` | Début d'un cycle d'analyse |
|
||||
| `CYCLE_END` | Fin du cycle (résumé insertés + dedup_ttl_min) |
|
||||
| `MODEL_LOADED` | Réutilisation d'un modèle existant (+ drift_score) |
|
||||
| `MODEL_TRAINED` | Nouvel entraînement |
|
||||
| `DRIFT_DETECTED` | Dérive conceptuelle détectée → retrain forcé |
|
||||
| `FEATURE_WARNING` | Features manquantes / constantes / agrégats globaux détectés (loggué uniquement si la situation change) |
|
||||
| `SKIPPED_INVALID_FEATURES` | Cycle ignoré (trop peu de features valides) |
|
||||
| `KNOWN_BOT` | Bot connu identifié |
|
||||
| `ANOMALY` | Anomalie IF détectée (+ effective_threshold, campaign_id, raw_anomaly_score) |
|
||||
| `SKIPPED_LOW_DATA` | Cycle ignoré (baseline < 500) |
|
||||
| `CONSECUTIVE_FAILURES` | Erreur ClickHouse répétée |
|
||||
|
||||
### 10.3 Avertissements sur les features (A7)
|
||||
|
||||
Les avertissements de features ne sont affichés en console **qu'une seule fois** (à la première détection ou lors d'un changement). Les cycles suivants avec la même situation ne génèrent pas de bruit. L'événement `FEATURE_WARNING` reste dans le JSONL pour traçabilité.
|
||||
|
||||
| Catégorie | Message console | Cause typique |
|
||||
|-----------|-----------------|---------------|
|
||||
| `zero` | `Features à 0 (pipeline non-alimenté)` | Table source vide / LEFT JOIN sans match |
|
||||
| `unique_nonzero` | `Features non-discriminantes (agrégat global)` | `PARTITION BY` sur valeur NULL → partition unique |
|
||||
| `missing` | `Features absentes du schéma` | Colonne manquante dans la vue ClickHouse |
|
||||
|
||||
Voir [`CLICKHOUSE_FEATURES_DIAGNOSTIC.md`](CLICKHOUSE_FEATURES_DIAGNOSTIC.md) pour le détail des corrections ClickHouse nécessaires.
|
||||
|
||||
### 11.1 Sources de réputation
|
||||
|
||||
| Fichier | Format | Contenu |
|
||||
|---------|--------|---------|
|
||||
| `bot_ip.csv` | `ip_cidr,bot_name` | ~288 k IP/CIDR de bots référencés |
|
||||
| `bot_ja4.csv` | `ja4,bot_name` | Fingerprints JA4 de bots |
|
||||
| `asn_reputation.csv` | `asn_number,label` | Labels ASN (human/bot) |
|
||||
|
||||
Ces fichiers sont montés en lecture seule dans le conteneur. Ils sont écrits par ClickHouse (FILE engine) et partagés via volume Docker.
|
||||
|
||||
### 11.2 Hiérarchie de classification
|
||||
|
||||
```
|
||||
1. bot_name != '' (depuis view_ai_features_1h)
|
||||
→ KNOWN_BOT : bot identifié par réputation IP ou JA4
|
||||
|
||||
2. asn_label == 'human' (depuis view_ai_features_1h)
|
||||
→ Utilisé pour la baseline d'entraînement de l'IF
|
||||
|
||||
3. Trafic restant
|
||||
→ Scoré par Isolation Forest
|
||||
→ Anomalie si score < ANOMALY_THRESHOLD
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Fondements scientifiques
|
||||
|
||||
### 12.1 Isolation Forest (Liu et al., 2008)
|
||||
|
||||
L'algorithme repose sur la propriété que les anomalies sont **isolées plus rapidement** dans des arbres de partitionnement aléatoire. La longueur moyenne du chemin d'isolation est normalisée pour produire un score entre 0 et 1 (transposé ici en -1 à +1 par `decision_function`).
|
||||
|
||||
**Propriétés clés :**
|
||||
- Complexité O(n log n) pour l'entraînement
|
||||
- Robuste aux données de haute dimensionnalité (31–35 features ici)
|
||||
- Pas d'hypothèse sur la distribution des données
|
||||
- Efficace sur de grands volumes (n_estimators=300, n_jobs=-1)
|
||||
|
||||
### 12.2 JA4+ Fingerprinting (FoxIO, 2023)
|
||||
|
||||
JA4 est la 4e génération de fingerprints TLS/QUIC/HTTP, successeur de JA3. Il capture les caractéristiques du ClientHello TLS (versions, ciphers, extensions) en une empreinte compacte permettant d'identifier des familles de clients (navigateurs, bots, outils). L'utilisation de `is_rare_ja4`, `distinct_ja4_count` et `ja4_asn_concentration` exploite cette propriété.
|
||||
|
||||
### 12.3 One-Class Classification appliquée aux bots
|
||||
|
||||
L'approche s'inscrit dans la lignée des travaux sur la détection de bots web :
|
||||
- **Stevanovic et al. (2013)** : détection de bots par analyse comportementale de flux HTTP
|
||||
- **Kruegel & Vigna (2003)** : détection d'anomalies réseau par profils normaux
|
||||
- **Barford & Yegneswaran (2007)** : classification comportementale des botnets
|
||||
|
||||
La combinaison de features HTTP comportementales (velocity, fuzzing, post_ratio), de features d'empreinte (JA4, headers), et de features TCP/TLS (jitter, ALPN, SNI) reproduit l'approche multi-couche recommandée par la littérature récente.
|
||||
|
||||
### 12.4 Entropie temporelle comme signal d'anomalie
|
||||
|
||||
Le feature `temporal_entropy` mesure l'entropie de Shannon sur la distribution temporelle des requêtes dans la fenêtre. Un bot avec un timing régulier (scripted polling) produit une entropie faible, tandis qu'un humain naviguant naturellement produit une distribution plus aléatoire. Ce signal est utilisé dans les travaux de **Wang et al. (2014)** sur la détection de crawlers web.
|
||||
|
||||
---
|
||||
|
||||
## 13. Améliorations implémentées (v11)
|
||||
|
||||
### A1 — Détection de dérive conceptuelle
|
||||
|
||||
**Fonctionnement** : À chaque cycle, avant de décider de charger ou de réentraîner le modèle, on compare la distribution courante de la baseline humaine avec celle sauvegardée lors du dernier entraînement. Pour chaque feature, un z-score est calculé :
|
||||
|
||||
```
|
||||
z = |mean_current - mean_trained| / std_trained
|
||||
```
|
||||
|
||||
Si la fraction de features avec `z > 2.0` dépasse `DRIFT_THRESHOLD` (30% par défaut), un re-entraînement est forcé et l'événement `DRIFT_DETECTED` est loggué.
|
||||
|
||||
**Métadonnées sauvegardées** : `baseline_stats` dans le `.meta.json` contient `{mean, std, p25, p75}` par feature.
|
||||
|
||||
**Références** : Gama et al. (2014) — *A Survey on Concept Drift Adaptation*
|
||||
|
||||
---
|
||||
|
||||
### A2 — Seuil adaptatif par percentile
|
||||
|
||||
**Fonctionnement** :
|
||||
|
||||
```python
|
||||
effective_threshold = min(np.percentile(raw_scores[raw_scores < 0], ANOMALY_PERCENTILE),
|
||||
ANOMALY_THRESHOLD)
|
||||
```
|
||||
|
||||
Le seuil effectif est le minimum entre le `ANOMALY_PERCENTILE`-ème percentile des scores négatifs et le seuil statique. Cela garantit que le seuil ne peut pas remonter au-dessus du seuil configuré, mais peut s'adapter vers le bas selon la distribution courante.
|
||||
|
||||
Le seuil utilisé est loggué dans chaque événement `ANOMALY`.
|
||||
|
||||
---
|
||||
|
||||
### A3 — Analyse multi-fenêtres (optionnelle)
|
||||
|
||||
**Activation** : `ENABLE_MULTIWINDOW=true` + une vue `view_ai_features_24h` dans ClickHouse.
|
||||
|
||||
**Fonctionnement** : Deux paires de modèles supplémentaires (`Complet_24h`, `Applicatif_24h`) tournent sur la fenêtre de 24h. Les anomalies des deux fenêtres sont fusionnées via une logique OR : une IP est flaggée si elle est anormale dans au moins une fenêtre. En cas de doublon, le score le plus bas (le plus anormal) est conservé.
|
||||
|
||||
**Utilité** : Détection des bots low-and-slow invisibles sur 1h mais clairement anormaux sur 24h.
|
||||
|
||||
---
|
||||
|
||||
### A4 — Explainabilité par SHAP
|
||||
|
||||
**Fonctionnement** : Pour chaque anomalie détectée, `shap.TreeExplainer` calcule la contribution de chaque feature au score d'anomalie. Les 5 features les plus négatives (les plus responsables de l'anomalie) sont incluses dans le champ `reason` :
|
||||
|
||||
```
|
||||
[Complet] Score: -0.112 | SHAP: is_alpn_missing(-1.081) | tcp_jitter_variance(-1.073) |
|
||||
ja4_asn_concentration(-1.062) | temporal_entropy(-0.887) |
|
||||
direct_access_ratio(-0.886) | Threat: MEDIUM
|
||||
```
|
||||
|
||||
**Désactivation** : `ENABLE_SHAP=false` ou si le package `shap` n'est pas installé.
|
||||
|
||||
**Références** : Lundberg & Lee (2017) — *A Unified Approach to Interpreting Model Predictions*
|
||||
|
||||
---
|
||||
|
||||
### A5 — Déduplication inter-cycles avec TTL
|
||||
|
||||
**Fonctionnement** : Avant chaque insertion, la table `ml_detected_anomalies` est interrogée pour identifier les IPs déjà insérées dans les `DEDUP_TTL_MIN` dernières minutes. Une IP est réinsérée uniquement si son score brut s'est dégradé d'au moins 0.05 points.
|
||||
|
||||
**Désactivation** : `DEDUP_TTL_MIN=0`
|
||||
|
||||
---
|
||||
|
||||
### A6 — Pondération du score par récurrence
|
||||
|
||||
**Fonctionnement** :
|
||||
|
||||
```python
|
||||
raw_score_adjusted = raw_score - log1p(recurrence) × RECURRENCE_WEIGHT
|
||||
```
|
||||
|
||||
Une IP détectée 10 fois reçoit une pénalité de `log(11) × 0.005 ≈ 0.012` sur son score brut, ce qui la rapproche du seuil de détection. Ce mécanisme simule un prior bayésien : les IPs récidivistes sont plus probablement malveillantes.
|
||||
|
||||
---
|
||||
|
||||
### A7 — Validation de complétude des features
|
||||
|
||||
**Fonctionnement** : Avant entraînement et scoring, `validate_features()` détecte :
|
||||
- Les features absentes de la vue ClickHouse
|
||||
- Les features constantes (std = 0, donc non discriminantes)
|
||||
|
||||
Les features invalides sont exclues du modèle. Si la fraction de features valides est inférieure à `MIN_VALID_FEATURE_RATIO` (50%), le cycle est ignoré.
|
||||
|
||||
**Bénéfice** : Les features constantes (souvent dues à des colonnes non encore implémentées dans la vue) ne biaisent plus le modèle.
|
||||
|
||||
---
|
||||
|
||||
### A8 — Clustering comportemental (DBSCAN)
|
||||
|
||||
**Fonctionnement** : Après détection, DBSCAN est appliqué sur les features normalisées des anomalies :
|
||||
|
||||
```python
|
||||
X_scaled = StandardScaler().fit_transform(anomalies[valid_features])
|
||||
labels = DBSCAN(eps=0.5, min_samples=CLUSTERING_MIN_SAMPLES).fit_predict(X_scaled)
|
||||
```
|
||||
|
||||
- `campaign_id = -1` : IP isolée (comportement unique)
|
||||
- `campaign_id >= 0` : membre d'une campagne coordonnée
|
||||
|
||||
Le `campaign_id` est loggué dans les événements `ANOMALY` (JSONL). Il n'est pas encore dans le schéma ClickHouse (voir §14).
|
||||
|
||||
**Références** : Ester et al. (1996) — *A Density-Based Algorithm for Discovering Clusters*
|
||||
|
||||
---
|
||||
|
||||
### A10 — Normalisation des scores entre modèles
|
||||
|
||||
**Fonctionnement** :
|
||||
|
||||
```python
|
||||
# Scores négatifs normalisés en [-1, 0], scores positifs inchangés
|
||||
anomaly_score_normalized = normalize_scores(raw_score)
|
||||
```
|
||||
|
||||
Le champ `anomaly_score` dans ClickHouse contient désormais le score normalisé, permettant une comparaison cohérente entre le modèle Complet (35 features) et le modèle Applicatif (31 features). Le score brut IF est conservé dans `raw_anomaly_score` (logs JSONL uniquement) et est utilisé pour l'assignation du threat level.
|
||||
|
||||
---
|
||||
|
||||
## 14. Migration de schéma ClickHouse
|
||||
|
||||
Les nouvelles colonnes suivantes sont disponibles dans les logs JSONL mais pas encore dans la table `ml_detected_anomalies`. Pour les activer :
|
||||
|
||||
```sql
|
||||
ALTER TABLE mabase_prod.ml_detected_anomalies
|
||||
ADD COLUMN IF NOT EXISTS campaign_id Int32 DEFAULT -1,
|
||||
ADD COLUMN IF NOT EXISTS raw_anomaly_score Float32 DEFAULT 0;
|
||||
```
|
||||
|
||||
Après cette migration, ajouter ces colonnes à la liste `cols` dans `fetch_and_analyze()` (elles sont déjà calculées en mémoire).
|
||||
756
services/bot-detector/IMPROVEMENTS.md
Normal file
756
services/bot-detector/IMPROVEMENTS.md
Normal file
@ -0,0 +1,756 @@
|
||||
# Bot Detector IA — Axes d'amélioration
|
||||
|
||||
> Document de propositions techniques — à valider avant implémentation
|
||||
|
||||
---
|
||||
|
||||
## Résumé des axes proposés
|
||||
|
||||
| # | Axe | Impact | Complexité | Priorité suggérée |
|
||||
|---|-----|--------|------------|-------------------|
|
||||
| A1 | [Détection de dérive conceptuelle (concept drift)](#a1-détection-de-dérive-conceptuelle) | 🔴 Élevé | Moyenne | ⭐⭐⭐ |
|
||||
| A2 | [Seuil adaptatif par percentile](#a2-seuil-adaptatif-par-percentile) | 🔴 Élevé | Faible | ⭐⭐⭐ |
|
||||
| A3 | [Analyse multi-fenêtres temporelles](#a3-analyse-multi-fenêtres-temporelles) | 🔴 Élevé | Élevée | ⭐⭐ |
|
||||
| A4 | [Explainabilité par SHAP](#a4-explainabilité-par-shap) | 🟠 Moyen | Moyenne | ⭐⭐⭐ |
|
||||
| A5 | [Déduplication avec TTL inter-cycles](#a5-déduplication-avec-ttl-inter-cycles) | 🟠 Moyen | Faible | ⭐⭐⭐ |
|
||||
| A6 | [Pondération par récurrence dans le score](#a6-pondération-par-récurrence-dans-le-score) | 🟠 Moyen | Faible | ⭐⭐ |
|
||||
| A7 | [Validation de complétude des features](#a7-validation-de-complétude-des-features) | 🟠 Moyen | Faible | ⭐⭐⭐ |
|
||||
| A8 | [Clustering comportemental des anomalies](#a8-clustering-comportemental-des-anomalies) | 🟡 Utile | Moyenne | ⭐⭐ |
|
||||
| A9 | [Métriques Prometheus / health check enrichi](#a9-métriques-prometheus--health-check-enrichi) | 🟡 Utile | Faible | ⭐⭐ |
|
||||
| A10 | [Normalisation des scores entre modèles](#a10-normalisation-des-scores-entre-modèles) | 🟡 Utile | Faible | ⭐ |
|
||||
|
||||
---
|
||||
|
||||
## A1 — Détection de dérive conceptuelle
|
||||
|
||||
### Problème
|
||||
|
||||
L'Isolation Forest est entraîné sur la baseline humaine courante. Si le profil du trafic légitime évolue graduellement (nouveau navigateur populaire, changement de comportement utilisateur, migration réseau), le modèle vieilli peut :
|
||||
- Générer des **faux positifs** sur du trafic humain nouvellement apparu
|
||||
- Rater des **faux négatifs** si les bots imitent les anciens patterns
|
||||
|
||||
Le re-entraînement périodique (toutes les X heures) atténue le problème mais ne détecte pas quand une dérive significative a eu lieu **entre deux cycles de retraining**.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Calculer à chaque cycle un score de **dérive statistique** entre la baseline d'entraînement du modèle actif et la baseline courante. Si la dérive dépasse un seuil, forcer un re-entraînement anticipé.
|
||||
|
||||
**Méthode : Kolmogorov-Smirnov (KS test) ou Maximum Mean Discrepancy (MMD)**
|
||||
|
||||
Pour chaque feature :
|
||||
```python
|
||||
from scipy import stats
|
||||
ks_stat, p_value = stats.ks_2samp(baseline_trained[feat], baseline_current[feat])
|
||||
```
|
||||
|
||||
Si la fraction de features avec `p_value < 0.05` dépasse un seuil configurable (ex. 30%), déclencher un retrain et logguer un événement `DRIFT_DETECTED`.
|
||||
|
||||
### Bénéfices
|
||||
- Retrain opportuniste plutôt que temporel fixe
|
||||
- Détection proactive des changements de comportement réseau
|
||||
- Réduction des faux positifs liés à la dérive
|
||||
|
||||
### Références
|
||||
- Gama et al. (2014) — *A Survey on Concept Drift Adaptation*
|
||||
- Rabanser et al. (2019) — *Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift*
|
||||
|
||||
### Implémentation suggérée
|
||||
- Sauvegarder la distribution de la baseline d'entraînement dans le `.meta.json`
|
||||
- Calculer le KS test au début de chaque cycle avant la décision de chargement
|
||||
- Ajouter un paramètre `DRIFT_THRESHOLD` (défaut : 0.30)
|
||||
- Logguer l'événement `DRIFT_DETECTED` avec les features déroutantes
|
||||
|
||||
---
|
||||
|
||||
## A2 — Seuil adaptatif par percentile
|
||||
|
||||
### Problème
|
||||
|
||||
`ANOMALY_THRESHOLD = -0.03` est un seuil **global et statique**. Ce seuil a une signification différente selon :
|
||||
- Le volume de trafic (plus de trafic = distribution de scores plus resserrée)
|
||||
- La contamination effective du cycle (jour calme vs attaque active)
|
||||
- Les caractéristiques du modèle actif (entraîné sur 1 264 vs 1 725 sessions)
|
||||
|
||||
Un seuil fixe peut produire des **rafales de faux positifs** lors d'événements légitimes inhabituels (campagne marketing, crawler partenaire) ou rater des menaces réelles lors de trafic atypique.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Calculer dynamiquement le seuil à partir de la **distribution des scores du cycle courant** :
|
||||
|
||||
```python
|
||||
scores = model.decision_function(X_test)
|
||||
# Seuil = percentile P de la distribution des scores négatifs
|
||||
adaptive_threshold = np.percentile(scores, ANOMALY_PERCENTILE)
|
||||
# On prend le min avec le seuil statique pour éviter d'aller trop haut
|
||||
threshold = min(adaptive_threshold, ANOMALY_THRESHOLD)
|
||||
```
|
||||
|
||||
**Paramètre ajoutable** : `ANOMALY_PERCENTILE` (défaut : 5 → top 5% des scores les plus négatifs).
|
||||
|
||||
Cette approche est complémentaire au seuil statique (garde-fou) : elle s'adapte vers le bas mais ne remonte jamais au-dessus du seuil configuré.
|
||||
|
||||
### Bénéfices
|
||||
- Stabilité du taux de faux positifs au fil du temps
|
||||
- Auto-adaptation aux variations de volume
|
||||
- Comportement plus prédictible en production
|
||||
|
||||
### Implémentation suggérée
|
||||
- Ajouter `ANOMALY_PERCENTILE` (0–20, défaut 5) comme variable d'environnement
|
||||
- Calculer le seuil adaptatif dans `run_semi_supervised_logic()`
|
||||
- Logguer le seuil effectif utilisé dans `CYCLE_START` / `ANOMALY`
|
||||
|
||||
---
|
||||
|
||||
## A3 — Analyse multi-fenêtres temporelles
|
||||
|
||||
### Problème
|
||||
|
||||
La fenêtre 1h est un compromis. Elle manque :
|
||||
- Les **attaques rapides** (burst de quelques minutes) : le signal est dilué
|
||||
- Les **bots lents** (low-and-slow, 1–2 req/min sur 24h) : comportement normal sur 1h
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Ajouter une deuxième vue ClickHouse agrégée sur **24h** et un troisième modèle sur cette fenêtre. Les scores des deux modèles peuvent être combinés :
|
||||
|
||||
```
|
||||
score_final = w1 * score_1h + w2 * score_24h
|
||||
```
|
||||
|
||||
Ou, plus simplement, un AND logique : une IP n'est flaggée que si elle est anomalie sur les **deux fenêtres**, réduisant drastiquement les faux positifs.
|
||||
|
||||
### Bénéfices
|
||||
- Détection des bots low-and-slow (reconnaissance, scraping discret)
|
||||
- Réduction des faux positifs par corrélation multi-temporelle
|
||||
- Complémentarité avec le modèle 1h existant
|
||||
|
||||
### Considerations
|
||||
- Nécessite une vue `view_ai_features_24h` dans ClickHouse
|
||||
- Modèle 24h beaucoup plus stable (moins de bruit)
|
||||
- Le volume de données à traiter augmente
|
||||
|
||||
### Références
|
||||
- Stalmans & Irwin (2011) — *A Framework for Web Bot Detection Using Request Rate Monitoring*
|
||||
- Stevanovic et al. (2013) — *An Efficient Flow-based Botnet Detection Using Supervised Machine Learning*
|
||||
|
||||
---
|
||||
|
||||
## A4 — Explainabilité par SHAP
|
||||
|
||||
### Problème
|
||||
|
||||
Le champ `reason` actuel est basique :
|
||||
```
|
||||
"[Complet] Score: -0.312 | Vel: 45.2 req/s | Fuzzing: 8.3 | Threat: CRITICAL"
|
||||
```
|
||||
|
||||
Pour un opérateur de sécurité, il manque :
|
||||
- **Quelles features** ont le plus contribué à ce score ?
|
||||
- Est-ce principalement comportemental (velocity) ou fingerprint (JA4) ?
|
||||
- Comment comparer deux anomalies de même score ?
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Utiliser **TreeSHAP** (Lundberg & Lee, 2017) qui supporte nativement les forêts d'arbres :
|
||||
|
||||
```python
|
||||
import shap
|
||||
explainer = shap.TreeExplainer(model)
|
||||
shap_values = explainer.shap_values(X_test.iloc[[idx]])
|
||||
top_features = sorted(zip(features, shap_values[0]), key=lambda x: abs(x[1]), reverse=True)[:5]
|
||||
```
|
||||
|
||||
Enrichir le champ `reason` avec les 5 features les plus contributives et leur valeur SHAP.
|
||||
|
||||
### Bénéfices
|
||||
- Triage des alertes facilité pour les analystes SOC
|
||||
- Détection des features systématiquement sur-représentées (potentiel bug de feature engineering)
|
||||
- Conformité avec les exigences de traçabilité des décisions IA
|
||||
|
||||
### Implémentation suggérée
|
||||
- Ajouter `shap` aux requirements (compatible sklearn)
|
||||
- Calculer SHAP uniquement pour les IP flaggées (pas sur tout le dataset)
|
||||
- Stocker `shap_top5` comme JSON dans le log JSONL
|
||||
- Option : `ENABLE_SHAP=true/false` pour contrôler la charge CPU
|
||||
|
||||
### Références
|
||||
- Lundberg & Lee (2017) — *A Unified Approach to Interpreting Model Predictions*
|
||||
|
||||
---
|
||||
|
||||
## A5 — Déduplication avec TTL inter-cycles
|
||||
|
||||
### Problème
|
||||
|
||||
Avec un cycle de 5 min et une fenêtre 1h, la même IP malveillante est potentiellement **réinsérée 12 fois par heure** dans `ml_detected_anomalies`. Cela :
|
||||
- Gonfle la table artificellement
|
||||
- Complique les requêtes d'analyse (nécessite un DISTINCT)
|
||||
- Fausse les métriques de comptage
|
||||
|
||||
Le mécanisme actuel de `drop_duplicates(subset=['src_ip'])` ne fonctionne qu'au sein d'un seul cycle, pas entre cycles.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Avant insertion, interroger ClickHouse pour filtrer les IPs déjà insérées récemment :
|
||||
|
||||
```python
|
||||
# Récupérer les IPs déjà détectées dans les N dernières minutes
|
||||
recent_ips = client.query_df(f"""
|
||||
SELECT DISTINCT src_ip
|
||||
FROM {DB}.ml_detected_anomalies
|
||||
WHERE detected_at > now() - INTERVAL {DEDUP_TTL_MIN} MINUTE
|
||||
""")
|
||||
# Exclure ces IPs sauf si le score s'est dégradé significativement
|
||||
new_anomalies = anomalies[~anomalies['src_ip'].isin(recent_ips['src_ip'])]
|
||||
```
|
||||
|
||||
**Paramètre ajoutable** : `DEDUP_TTL_MIN` (défaut : 60 minutes).
|
||||
|
||||
**Variante** : ne re-insérer que si `new_score < existing_score - 0.05` (dégradation significative).
|
||||
|
||||
### Bénéfices
|
||||
- Réduction du volume de la table de détection
|
||||
- Requêtes d'analyse plus simples
|
||||
- Gestion de la montée en charge (moins d'insertions)
|
||||
|
||||
### Implémentation suggérée
|
||||
- Paramètre `DEDUP_TTL_MIN` (0 pour désactiver)
|
||||
- La requête de déduplication est légère (index sur `detected_at`)
|
||||
- Logguer le nb d'IP filtrées dans `CYCLE_END`
|
||||
|
||||
---
|
||||
|
||||
## A6 — Pondération par récurrence dans le score
|
||||
|
||||
### Problème
|
||||
|
||||
La récurrence est actuellement un champ **informatif seulement** : une IP détectée 50 fois a le même seuil de filtrage qu'une IP vue pour la première fois. Un bot persistant et connu ne reçoit pas de pénalité de score.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Ajuster le score de décision en fonction de la récurrence :
|
||||
|
||||
```python
|
||||
# Score ajusté : plus une IP est récurrente, plus son score s'aggrave
|
||||
recurrence_penalty = np.log1p(recurrence) * RECURRENCE_WEIGHT
|
||||
adjusted_score = anomaly_score - recurrence_penalty
|
||||
```
|
||||
|
||||
Avec `RECURRENCE_WEIGHT = 0.005` par défaut (configurable). Une IP vue 10 fois voit son score pénalisé de ~0.012, une IP vue 100 fois de ~0.023.
|
||||
|
||||
Cette approche simule un **Prior bayésien** : la probabilité qu'une IP soit malveillante augmente avec ses détections passées.
|
||||
|
||||
### Bénéfices
|
||||
- Menaces persistantes classifiées plus sévèrement
|
||||
- Réduction du bruit des anomalies éphémères
|
||||
- Signal plus fort pour les blocages automatisés
|
||||
|
||||
### Implémentation suggérée
|
||||
- Ajouter `RECURRENCE_WEIGHT` (défaut 0.005, 0 pour désactiver)
|
||||
- Stocker `raw_score` et `adjusted_score` séparément dans les logs
|
||||
|
||||
---
|
||||
|
||||
## A7 — Validation de complétude des features
|
||||
|
||||
### Problème
|
||||
|
||||
Si une feature est absente de la vue (colonne manquante, erreur de schéma), elle est silencieusement remplacée par `0` via `fillna(0)`. Cela **dégrade la qualité du modèle sans avertissement** : une feature entièrement à zéro n'apporte aucune information discriminante et biaise les scores.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Au début de chaque cycle, après chargement du DataFrame :
|
||||
|
||||
```python
|
||||
def validate_features(df: pd.DataFrame, features: list, name: str) -> list:
|
||||
zero_features = [f for f in features if f in df.columns and df[f].std() == 0]
|
||||
missing_features = [f for f in features if f not in df.columns]
|
||||
|
||||
if missing_features:
|
||||
log_info(f"[{name}] ATTENTION: {len(missing_features)} features manquantes: {missing_features}")
|
||||
if zero_features:
|
||||
log_info(f"[{name}] ATTENTION: {len(zero_features)} features constantes (=0): {zero_features}")
|
||||
|
||||
# Retourner uniquement les features exploitables
|
||||
valid = [f for f in features if f in df.columns and df[f].std() > 0]
|
||||
return valid
|
||||
```
|
||||
|
||||
Un événement `FEATURE_WARNING` serait loggué, et si plus de 20% des features sont invalides, le cycle peut être `SKIPPED`.
|
||||
|
||||
### Bénéfices
|
||||
- Détection rapide des régressions de schéma ClickHouse
|
||||
- Qualité de modèle assurée
|
||||
- Facilite le debugging lors des évolutions de la vue
|
||||
|
||||
### Implémentation suggérée
|
||||
- Paramètre `MIN_VALID_FEATURE_RATIO` (défaut 0.8)
|
||||
- Comparaison avec les features du modèle chargé (détecte les dérives de schéma post-mise à jour)
|
||||
|
||||
---
|
||||
|
||||
## A8 — Clustering comportemental des anomalies
|
||||
|
||||
### Problème
|
||||
|
||||
Les anomalies sont analysées et insérées individuellement. Or, une campagne de botnet coordonnée peut impliquer des **dizaines d'IPs avec des profils similaires**. Cette information de **corrélation horizontale** est aujourd'hui invisible.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Après la détection, appliquer un **DBSCAN** sur les features des anomalies pour identifier des clusters d'attaque :
|
||||
|
||||
```python
|
||||
from sklearn.cluster import DBSCAN
|
||||
X_anomalies = anomalies[features].fillna(0)
|
||||
scaler = StandardScaler()
|
||||
X_scaled = scaler.fit_transform(X_anomalies)
|
||||
labels = DBSCAN(eps=0.5, min_samples=3).fit_predict(X_scaled)
|
||||
anomalies['campaign_id'] = labels # -1 = isolé, 0+ = cluster
|
||||
```
|
||||
|
||||
Les IPs d'un même cluster partagent un comportement similaire et peuvent faire partie d'une même infrastructure d'attaque.
|
||||
|
||||
### Bénéfices
|
||||
- Identification des campagnes coordonnées (botnets distribués)
|
||||
- Enrichissement de `reason` avec un identifiant de campagne
|
||||
- Permet des blocages de plages d'IPs entières
|
||||
|
||||
### Implémentation suggérée
|
||||
- DBSCAN uniquement si ≥ 5 anomalies dans le cycle (pas de coût si peu d'anomalies)
|
||||
- Stocker `campaign_id` dans `ml_detected_anomalies`
|
||||
- `eps` et `min_samples` configurables
|
||||
|
||||
### Références
|
||||
- Ester et al. (1996) — *A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases*
|
||||
|
||||
---
|
||||
|
||||
## A9 — Métriques Prometheus / health check enrichi
|
||||
|
||||
### Problème
|
||||
|
||||
Le health check actuel est binaire (OK/DEGRADED). Cela ne permet pas :
|
||||
- De monitorer la dérive du taux d'anomalies dans le temps
|
||||
- D'alerter si aucun cycle ne s'est exécuté depuis X minutes
|
||||
- De suivre l'âge du modèle en production
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Exposer un endpoint `/metrics` au format **Prometheus text** sur le même port :
|
||||
|
||||
```
|
||||
# HELP botdetector_cycle_duration_seconds Duration of last analysis cycle
|
||||
# TYPE botdetector_cycle_duration_seconds gauge
|
||||
botdetector_cycle_duration_seconds 12.4
|
||||
|
||||
# HELP botdetector_anomalies_total Total anomalies detected in last cycle
|
||||
# TYPE botdetector_anomalies_total gauge
|
||||
botdetector_anomalies_total{model="Complet"} 3
|
||||
botdetector_anomalies_total{model="Applicatif"} 7
|
||||
|
||||
# HELP botdetector_model_age_hours Age of active model in hours
|
||||
botdetector_model_age_hours{model="Applicatif"} 0.91
|
||||
|
||||
# HELP botdetector_human_baseline_size Nb of human samples used for training
|
||||
botdetector_human_baseline_size{model="Applicatif"} 1725
|
||||
```
|
||||
|
||||
Implémenté sans dépendance externe (format texte manuel ou lib légère `prometheus_client`).
|
||||
|
||||
### Bénéfices
|
||||
- Intégration Grafana/Alertmanager
|
||||
- Alertes sur dérive du taux d'anomalies (ex. : >50% d'une heure à l'autre)
|
||||
- Monitoring de la fraîcheur du modèle
|
||||
|
||||
### Implémentation suggérée
|
||||
- Ajouter `prometheus_client` ou générer le format texte manuellement
|
||||
- Endpoint `/metrics` sur le même `HTTPServer` existant
|
||||
- Métriques stockées dans un dict thread-safe mis à jour après chaque cycle
|
||||
|
||||
---
|
||||
|
||||
## A10 — Normalisation des scores entre modèles
|
||||
|
||||
### Problème
|
||||
|
||||
Les scores `decision_function` de l'IF ne sont **pas comparables entre modèles** entraînés sur des données différentes. Un score de -0.10 sur le modèle Complet et -0.10 sur le modèle Applicatif n'ont pas la même signification si les baselines et les features sont différentes.
|
||||
|
||||
La déduplication actuelle par `src_ip` prend le score le plus bas sans tenir compte de cette non-comparabilité.
|
||||
|
||||
### Approche proposée
|
||||
|
||||
Normaliser les scores par rapport à la distribution des scores négatifs du cycle courant :
|
||||
|
||||
```python
|
||||
# Normalisation min-max sur le sous-ensemble des scores < 0
|
||||
neg_scores = unknown_traffic['anomaly_score'][unknown_traffic['anomaly_score'] < 0]
|
||||
if len(neg_scores) > 0:
|
||||
score_min, score_max = neg_scores.min(), neg_scores.max()
|
||||
unknown_traffic['normalized_score'] = (
|
||||
(unknown_traffic['anomaly_score'] - score_min) / (score_max - score_min + 1e-9)
|
||||
).clip(0, 1) * -1 # entre -1 et 0
|
||||
```
|
||||
|
||||
Les niveaux de menace seraient alors calculés sur le score normalisé, rendant la comparaison entre modèles cohérente.
|
||||
|
||||
### Bénéfices
|
||||
- Cohérence des niveaux CRITICAL/HIGH/MEDIUM entre modèles
|
||||
- Déduplication plus juste
|
||||
- Seuils de threat_level interprétables de façon constante
|
||||
|
||||
---
|
||||
|
||||
## Notes d'implémentation générales
|
||||
|
||||
- **Compatibilité** : toute amélioration doit rester rétrocompatible avec le schéma `ml_detected_anomalies` existant (ajout de colonnes optionnelles uniquement)
|
||||
- **Lisibilité** : garder le code en sections délimitées par les bandeaux `═══` existants
|
||||
- **Tests** : valider chaque changement par une exécution Docker sur la base de données réelle
|
||||
- **Documentation** : mettre à jour `DOCUMENTATION.md` après chaque implémentation
|
||||
- **Feature flags** : les nouvelles fonctionnalités comportementales devraient être activables via variable d'environnement pour un rollout progressif
|
||||
|
||||
---
|
||||
|
||||
# Nouvelles dimensions de features — Propositions B
|
||||
|
||||
> Propositions de features supplémentaires pour l'Isolation Forest, validées sur les données réelles de `mabase_prod`.
|
||||
> Chaque proposition indique la force du signal observée en base, la source de données, la formule de calcul et les références scientifiques.
|
||||
|
||||
## Résumé des signaux
|
||||
|
||||
| # | Feature | Signal observé | Modèle | Impact estimé |
|
||||
|---|---------|---------------|--------|--------------|
|
||||
| B1 | JA3/JA4 diversity ratio | 809 JA3 pour 2 JA4 (IP connue bot) | Complet | 🔴 Élevé |
|
||||
| B2 | SYN timing regularity | 386/3222 IPs (12%) avec variance=0 | Complet | 🔴 Élevé |
|
||||
| B3 | TLS 1.2 exclusive ratio | 136/3259 IPs (4%) — jamais TLS 1.3 | Complet | 🔴 Élevé |
|
||||
| B4 | HEAD method ratio | 67/3335 IPs (2%) à >50% HEAD | Les deux | 🟠 Moyen |
|
||||
| B5 | Sec-Fetch absence rate | Signal L7 universel (correlated=0 aussi) | Les deux | 🟠 Moyen |
|
||||
| B6 | Accept header entropy | Bots = Accept vide ou `*/*` constant | Les deux | 🟠 Moyen |
|
||||
| B7 | TLS version entropy | TLS 1.3 = 97.3% du trafic légitime | Complet | 🟠 Moyen |
|
||||
| B8 | HTTP/TLS protocol mismatch | HTTP/1.1 + TLS 1.3 = ratio anormal | Complet | 🟡 Utile |
|
||||
| B9 | IP DF-bit variance | DF inconsistant = stack spoofé | Complet | 🟡 Utile |
|
||||
| B10 | JA4 concentration intra-ASN | JA4 rare dans ASN = outil exotique | Complet | 🟡 Utile |
|
||||
|
||||
---
|
||||
|
||||
## B1 — JA3/JA4 Diversity Ratio (rotation de fingerprint TLS)
|
||||
|
||||
### Observation
|
||||
|
||||
```
|
||||
185.177.72.60 → 1619 JA3 distincts / 2 JA4 → ratio 809.5
|
||||
194.187.171.160 → 153 JA3 distincts / 2 JA4 → ratio 76.5
|
||||
```
|
||||
|
||||
Le JA4 reste stable (il encode le type de client TLS + ALPN) mais le JA3 varie massivement. C'est la signature d'un **bot qui randomise les extensions TLS** pour contourner la détection par fingerprint.
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```sql
|
||||
-- Dans mv_agg_host_ip_ja4_1h
|
||||
uniqState(ja3) AS uniq_ja3 -- à ajouter dans la table d'agrégation
|
||||
```
|
||||
|
||||
```python
|
||||
# Dans view_ai_features_1h
|
||||
ja3_diversity_ratio = uniq_ja3 / greatest(uniq_ja4, 1)
|
||||
```
|
||||
|
||||
### Signal en base
|
||||
|
||||
- Trafic humain : ratio typiquement 1–3 (même navigateur, légères variations)
|
||||
- Bot avec rotation : ratio 17–809 → signal extrêmement discriminant
|
||||
- Disponible : `ja3` est présent dans `http_logs` avec 100% de valeurs non-vides pour correlated=1
|
||||
|
||||
### Modifications requises
|
||||
|
||||
1. Ajouter `uniqState(ja3) AS uniq_ja3` dans `mv_agg_host_ip_ja4_1h` et `agg_host_ip_ja4_1h`
|
||||
2. Ajouter `uniqMerge(uniq_ja3) / greatest(uniq_ja4_merged, 1) AS ja3_diversity_ratio` dans `view_ai_features_1h`
|
||||
3. Ajouter `ja3_diversity_ratio` à `feats_complet` dans `bot_detector.py`
|
||||
|
||||
### Références
|
||||
|
||||
- Siby et al. (2020) — *Encrypted DNS → Privacy? A Traffic Analysis Perspective* — méthodes de diversité de fingerprint
|
||||
- Anderson & McGrew (2016) — *Machine Learning for Encrypted Malware Traffic Classification* — JA3 comme feature primaire
|
||||
- Husák et al. (2022) — *TLS fingerprinting for bot detection* — rotation JA3 comme évasion signature
|
||||
|
||||
---
|
||||
|
||||
## B2 — SYN-to-ClientHello Timing Regularity
|
||||
|
||||
### Observation
|
||||
|
||||
```
|
||||
88.202.237.59 : 45 connexions, avg=22ms, std=0.00ms → timing robotique parfait
|
||||
92.184.144.129: 41 connexions, avg=10ms, std=0.00ms → idem
|
||||
386/3222 IPs analysées (12%) ont une variance=0
|
||||
```
|
||||
|
||||
Un humain présente une distribution aléatoire (Weibull ou log-normale) des temps de réponse réseau. Un bot utilisant un scheduler fixe ou une connexion locale a une variance proche de zéro.
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```sql
|
||||
-- Dans view_ai_features_1h (CTE)
|
||||
varPopMerge(tcp_jitter_variance) AS syn_jitter_variance, -- déjà présent (tcp_jitter_variance)
|
||||
-- Ajouter le coefficient de variation (normalisé)
|
||||
```
|
||||
|
||||
```python
|
||||
# cv = std / mean → 0 = robotique, >0.5 = humain
|
||||
syn_timing_cv = sqrt(syn_jitter_variance) / greatest(avg_syn_ms, 1)
|
||||
```
|
||||
|
||||
**Note** : `tcp_jitter_variance` est déjà dans le modèle mais c'est la variance brute. Le **coefficient de variation** (std/mean) normalise par le délai moyen et est plus discriminant pour différencier bots rapides (10ms) de bots lents (100ms).
|
||||
|
||||
### Modifications requises
|
||||
|
||||
1. Ajouter `avg(syn_to_clienthello_ms)` dans `mv_agg_host_ip_ja4_1h` → `avg_syn_ms`
|
||||
2. Calculer `syn_timing_cv = sqrt(tcp_jitter_variance) / greatest(avg_syn_ms, 1)` dans `view_ai_features_1h`
|
||||
3. Ajouter `syn_timing_cv` à `feats_complet`
|
||||
|
||||
### Références
|
||||
|
||||
- Zeber et al. (2020) — *The Measurement of Web Timing* — distribution log-normale pour humains
|
||||
- Beugin et al. (2021) — *Robustness of Traffic Analysis Against Adversarial Timing* — variance comme discriminant
|
||||
- Stevanovic & Pedersen (2015) — *Detecting Bots Using Multi-level Traffic Analysis* — timing régularité = signal bot L4
|
||||
|
||||
---
|
||||
|
||||
## B3 — TLS 1.2 Exclusive Ratio
|
||||
|
||||
### Observation
|
||||
|
||||
```
|
||||
95.217.144.244 : 360/360 requêtes en TLS 1.2 (jamais TLS 1.3)
|
||||
37.65.177.201 : 267/267 requêtes en TLS 1.2
|
||||
136 IPs utilisent exclusivement TLS 1.2 sur 3259 analysées (4.2%)
|
||||
```
|
||||
|
||||
TLS 1.3 représente 97.3% du trafic en 2026. Les navigateurs modernes n'utilisent TLS 1.2 que comme fallback exceptionnel. Une IP utilisant **exclusivement** TLS 1.2 utilise un client obsolète, une bibliothèque custom, ou un outil de scan.
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```sql
|
||||
-- Dans mv_agg_host_ip_ja4_1h
|
||||
sum(IF(tls_version = '1.2', 1, 0)) AS tls12_count -- nouveau
|
||||
-- tls_version déjà stockée via tls_alpn_raw → à distinguer ou ajouter
|
||||
```
|
||||
|
||||
```python
|
||||
# Dans view_ai_features_1h
|
||||
tls12_ratio = tls12_count / greatest(hits, 1)
|
||||
```
|
||||
|
||||
### Modifications requises
|
||||
|
||||
1. Ajouter `sum(IF(src.tls_version = '1.2', 1, 0)) AS tls12_count` dans `mv_agg_host_ip_ja4_1h`
|
||||
2. Ajouter `tls12_count` dans `agg_host_ip_ja4_1h`
|
||||
3. Calculer `tls12_count / hits AS tls12_ratio` dans `view_ai_features_1h`
|
||||
|
||||
### Références
|
||||
|
||||
- Kotzias et al. (2018) — *Coming of Age: A Longitudinal Study of TLS Deployment* — vieillissement des stacks
|
||||
- Naylor et al. (2014) — *The Cost of the S in HTTPS* — adoption TLS 1.3 par navigateurs légitimes
|
||||
- Cloudflare Radar 2024 — TLS 1.3 = 95%+ du trafic web mondial
|
||||
|
||||
---
|
||||
|
||||
## B4 — HEAD Method Ratio
|
||||
|
||||
### Observation
|
||||
|
||||
```
|
||||
34.140.199.84 : 11/12 requêtes HEAD (91.7%) → Google Cloud uptime checker
|
||||
67/3335 IPs ont >50% de requêtes HEAD
|
||||
```
|
||||
|
||||
La méthode HEAD est utilisée pour vérifier la disponibilité d'une ressource sans télécharger son contenu. C'est la signature des :
|
||||
- **Uptime checkers** (Pingdom, UptimeRobot, Google Cloud Health Check)
|
||||
- **Scanners de vulnérabilités** (Nikto, Nuclei)
|
||||
- **Bots de reconnaissance discrète**
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```python
|
||||
# head_ratio = déjà calculable depuis count_post (method breakdown)
|
||||
# Ajouter dans mv_agg_host_ip_ja4_1h :
|
||||
count_head = sum(IF(method = 'HEAD', 1, 0))
|
||||
```
|
||||
|
||||
```python
|
||||
head_ratio = count_head / greatest(hits, 1)
|
||||
```
|
||||
|
||||
### Note : disponibilité dans les deux modèles
|
||||
|
||||
Contrairement aux features TCP, `head_ratio` est disponible pour `correlated=0` aussi — c'est une feature HTTP pure. À ajouter dans les deux listes `feats` et `feats_complet`.
|
||||
|
||||
### Références
|
||||
|
||||
- Barracuda Networks (2023) — *Bot Traffic Report* — HEAD requests pattern
|
||||
- OWASP Automated Threat Handbook — OAT-011: Scraping, OAT-018: Credential Stuffing
|
||||
|
||||
---
|
||||
|
||||
## B5 — Sec-Fetch Absence Rate
|
||||
|
||||
### Observation
|
||||
|
||||
Les headers `Sec-Fetch-Site`, `Sec-Fetch-Mode`, `Sec-Fetch-Dest` sont injectés par les navigateurs modernes (Chrome 76+, Firefox 90+) **automatiquement** depuis 2019. Leur absence est un signal de :
|
||||
- Client HTTP non-navigateur (curl, requests, Scrapy, headless Chrome sans headers complets)
|
||||
- Vieux navigateur ou UA spoofé
|
||||
- HTTP CONNECT proxy
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```sql
|
||||
-- Dans mv_agg_host_ip_ja4_1h
|
||||
sum(IF(length(src.header_sec_fetch_site) = 0, 1, 0)) AS count_no_sec_fetch
|
||||
```
|
||||
|
||||
```python
|
||||
sec_fetch_absence_rate = count_no_sec_fetch / greatest(hits, 1)
|
||||
```
|
||||
|
||||
### Combinaison avec `modern_browser_score`
|
||||
|
||||
`sec_fetch_absence_rate` + `modern_browser_score` forment une paire complémentaire :
|
||||
- Bot avec UA Chrome forgé → `modern_browser_score` élevé mais `sec_fetch_absence_rate` = 1 → contradiction forte
|
||||
|
||||
### Modifications requises
|
||||
|
||||
1. `count_no_sec_fetch` dans le MV et la table
|
||||
2. Calcul dans la vue
|
||||
|
||||
### Références
|
||||
|
||||
- West & Loshbough (2019) — *Fetch Metadata Request Headers* (W3C Spec)
|
||||
- Invernizzi et al. (2016) — *CLOAK of Visibility* — inconsistance headers = bot
|
||||
|
||||
---
|
||||
|
||||
## B6 — Accept Header Entropy
|
||||
|
||||
### Observation
|
||||
|
||||
Les navigateurs légitimes envoient des headers `Accept` complexes et cohérents :
|
||||
```
|
||||
image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
|
||||
```
|
||||
Les bots envoient :
|
||||
```
|
||||
*/* (curl, wget, Scrapy)
|
||||
(vide) (bots minimalistes)
|
||||
text/html (outils basiques)
|
||||
```
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```python
|
||||
# Diversité des valeurs Accept par IP (proxy de comportement navigateur)
|
||||
accept_entropy = -sum(p * log2(p+1e-9) for p in accept_value_probs)
|
||||
|
||||
# Ou plus simplement : fraction de requêtes avec Accept générique/vide
|
||||
generic_accept_ratio = count_generic_accept / hits
|
||||
# où generic = longueur(Accept) < 10 ou Accept IN ('*/*', '')
|
||||
```
|
||||
|
||||
```sql
|
||||
sum(IF(length(src.header_accept) < 5, 1, 0)) AS count_generic_accept
|
||||
```
|
||||
|
||||
### Références
|
||||
|
||||
- Nikiforakis et al. (2013) — *Cookieless Monster: Exploring the Ecosystem of Web-based Device Fingerprinting* — Accept comme composant stable
|
||||
- Acar et al. (2014) — *The Web Never Forgets* — entropie des headers HTTP
|
||||
|
||||
---
|
||||
|
||||
## B7 — HTTP/TLS Protocol Version Mismatch
|
||||
|
||||
### Observation
|
||||
|
||||
```
|
||||
HTTP/2.0 → 160855 requêtes (84%)
|
||||
HTTP/1.1 → 26421 requêtes (14%)
|
||||
TLS 1.3 → 177330 requêtes (97%)
|
||||
```
|
||||
|
||||
HTTP/2 requiert TLS dans les navigateurs modernes. Combinaisons anormales :
|
||||
- HTTP/1.1 + TLS 1.3 : légitime mais rare pour les vrais navigateurs (eux font HTTP/2 si TLS 1.3)
|
||||
- HTTP/1.0 + TLS : extrêmement suspect (outil custom ou ancien bot)
|
||||
- HTTP/2 + TLS 1.2 : possible mais déclinant
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```python
|
||||
# Fraction de requêtes avec HTTP/1.x malgré TLS 1.3 disponible
|
||||
http1_tls13_ratio = count_http1_with_tls13 / greatest(hits, 1)
|
||||
# http1_0_ratio = count_http10 / hits # signal fort
|
||||
```
|
||||
|
||||
```sql
|
||||
sum(IF(http_version = 'HTTP/1.0', 1, 0)) AS count_http10,
|
||||
sum(IF(http_version LIKE 'HTTP/1%' AND tls_version = '1.3', 1, 0)) AS count_http1_tls13
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## B8 — IP DF-Bit Consistency
|
||||
|
||||
### Observation
|
||||
|
||||
```
|
||||
df=1 : 172490 paquets (92%)
|
||||
df=0 : 15016 paquets (8%)
|
||||
```
|
||||
|
||||
Le bit "Don't Fragment" est généralement constant pour une session TCP donnée. Une IP qui alterne DF=0 et DF=1 au sein d'une même session, ou entre sessions, peut indiquer :
|
||||
- **Usurpation d'IP** (spoofed source packets dans un botnet)
|
||||
- **Stack TCP custom** (bots implémentant leur propre TCP)
|
||||
- **NAT traversal** avec réécriture de paquets
|
||||
|
||||
### Feature proposée
|
||||
|
||||
```python
|
||||
df_variance = stddev(ip_meta_df) per IP # 0 = cohérent, >0 = mélangé
|
||||
```
|
||||
|
||||
```sql
|
||||
varPop(toFloat64(ip_meta_df)) AS ip_df_variance
|
||||
```
|
||||
|
||||
Faible impact seul, mais utile en combinaison avec TTL variance pour le TCP fingerprinting multi-dimensional.
|
||||
|
||||
---
|
||||
|
||||
## Récapitulatif des modifications ClickHouse nécessaires
|
||||
|
||||
### Colonnes à ajouter dans `agg_host_ip_ja4_1h`
|
||||
|
||||
```sql
|
||||
ALTER TABLE mabase_prod.agg_host_ip_ja4_1h
|
||||
ADD COLUMN uniq_ja3 AggregateFunction(uniq, String),
|
||||
ADD COLUMN avg_syn_ms SimpleAggregateFunction(avg, Float64),
|
||||
ADD COLUMN tls12_count SimpleAggregateFunction(sum, UInt64),
|
||||
ADD COLUMN count_head SimpleAggregateFunction(sum, UInt64),
|
||||
ADD COLUMN count_no_sec_fetch SimpleAggregateFunction(sum, UInt64),
|
||||
ADD COLUMN count_generic_accept SimpleAggregateFunction(sum, UInt64),
|
||||
ADD COLUMN count_http10 SimpleAggregateFunction(sum, UInt64);
|
||||
```
|
||||
|
||||
### Nouvelles features dans `view_ai_features_1h`
|
||||
|
||||
| Feature | Formule | Modèle |
|
||||
|---------|---------|--------|
|
||||
| `ja3_diversity_ratio` | `uniq_ja3 / greatest(uniq_ja4, 1)` | Complet |
|
||||
| `syn_timing_cv` | `sqrt(tcp_jitter_variance) / greatest(avg_syn_ms, 1)` | Complet |
|
||||
| `tls12_ratio` | `tls12_count / greatest(hits, 1)` | Complet |
|
||||
| `head_ratio` | `count_head / greatest(hits, 1)` | Les deux |
|
||||
| `sec_fetch_absence_rate` | `count_no_sec_fetch / greatest(hits, 1)` | Les deux |
|
||||
| `generic_accept_ratio` | `count_generic_accept / greatest(hits, 1)` | Les deux |
|
||||
| `http10_ratio` | `count_http10 / greatest(hits, 1)` | Les deux |
|
||||
|
||||
> ⚠️ Les colonnes ajoutées par ALTER ne sont pas rétro-alimentées dans les données historiques. Un backfill depuis `http_logs` sera nécessaire.
|
||||
> ⚠️ La MV `mv_agg_host_ip_ja4_1h` doit être **recréée** (pas de ALTER sur une MV) pour inclure les nouveaux champs.
|
||||
|
||||
339
services/bot-detector/anubis/deploy_schema.sql
Normal file
339
services/bot-detector/anubis/deploy_schema.sql
Normal file
@ -0,0 +1,339 @@
|
||||
-- ============================================================================
|
||||
-- ANUBIS CRAWLER RULES — Labeling des http_logs + pipeline ML
|
||||
-- Architecture :
|
||||
-- anubis_ua_rules (table) → dict_anubis_ua (REGEXP_TREE)
|
||||
-- anubis_ip_rules (table) → dict_anubis_ip (IP_TRIE)
|
||||
-- http_logs : +anubis_bot_name, +anubis_bot_action
|
||||
-- mv_http_logs : reconstruit avec enrichissement Anubis
|
||||
-- view_ai_features_1h : +anubis_bot_name, +anubis_bot_action (via dictGet)
|
||||
-- ml_detected_anomalies : +anubis_bot_name, +anubis_bot_action
|
||||
-- ml_all_scores : +anubis_bot_name, +anubis_bot_action
|
||||
-- ============================================================================
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 1. TABLE SOURCE — règles User-Agent (pour dictionnaire REGEXP_TREE)
|
||||
--
|
||||
-- Format attendu par ClickHouse regexp_tree (v23.5+) :
|
||||
-- id UInt64 : identifiant unique
|
||||
-- parent_id UInt64 : 0 = racine, sinon id du parent (héritage d'attributs)
|
||||
-- regexp String : expression régulière (re2/vectorscan)
|
||||
-- keys Array(String) : noms des attributs, ex. ['bot_name', 'action']
|
||||
-- values Array(String) : valeurs correspondantes
|
||||
--
|
||||
-- Hiérarchie utilisée pour la priorité :
|
||||
-- Règles génériques DENY (parent_id=0) → enfants ALLOW spécifiques
|
||||
-- Exemple : ai-crawlers-training (parent) → openai-gptbot (enfant)
|
||||
-- Quand l'UA correspond à enfant ET parent, c'est le nom de l'enfant qui
|
||||
-- est retourné (l'enfant hérite ET surcharge les attributs du parent).
|
||||
-- ----------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS mabase_prod.anubis_ua_rules
|
||||
(
|
||||
id UInt64,
|
||||
parent_id UInt64,
|
||||
regexp String,
|
||||
keys Array(String),
|
||||
values Array(String)
|
||||
)
|
||||
ENGINE = ReplacingMergeTree()
|
||||
ORDER BY id;
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 2. TABLE SOURCE — règles IP/CIDR (pour dictionnaire IP_TRIE)
|
||||
--
|
||||
-- Colonnes requises par dict_anubis_ip et mv_http_logs :
|
||||
-- rule_id : identifiant de règle, croisé avec dict_anubis_ua pour
|
||||
-- la logique UA+IP (même rule_id → match combiné)
|
||||
-- has_ua : 1 si la règle possède aussi une regex UA (croisement nécessaire)
|
||||
-- category : catégorie Anubis (bots, crawlers, clients, policies…)
|
||||
-- ----------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS mabase_prod.anubis_ip_rules
|
||||
(
|
||||
prefix String,
|
||||
bot_name LowCardinality(String),
|
||||
action LowCardinality(String),
|
||||
rule_id UInt64,
|
||||
has_ua UInt8,
|
||||
category LowCardinality(String)
|
||||
)
|
||||
ENGINE = ReplacingMergeTree()
|
||||
ORDER BY prefix;
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 3. DICTIONNAIRE UA — REGEXP_TREE
|
||||
-- dictGet('mabase_prod.dict_anubis_ua', 'bot_name', header_user_agent)
|
||||
--
|
||||
-- Le PRIMARY KEY est 'regexp' (String) — requis par ClickHouse 26.x.
|
||||
-- Connexion interne (HOST localhost PORT 9000) pour éviter deadlock HTTP.
|
||||
-- Remplacer 'admin' et le mot de passe par les credentials ClickHouse.
|
||||
-- ----------------------------------------------------------------------------
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_anubis_ua;
|
||||
CREATE DICTIONARY mabase_prod.dict_anubis_ua
|
||||
(
|
||||
regexp String,
|
||||
bot_name String,
|
||||
action String
|
||||
)
|
||||
PRIMARY KEY regexp
|
||||
SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'admin' PASSWORD 'CHANGE_ME' DB 'mabase_prod' TABLE 'anubis_ua_rules'))
|
||||
LAYOUT(REGEXP_TREE)
|
||||
LIFETIME(MIN 300 MAX 600);
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 4. DICTIONNAIRE IP — IP_TRIE
|
||||
-- dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', toIPv6(src_ip), '')
|
||||
-- Connexion interne (HOST localhost PORT 9000) — même raison que dict_anubis_ua.
|
||||
-- ----------------------------------------------------------------------------
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_anubis_ip;
|
||||
CREATE DICTIONARY mabase_prod.dict_anubis_ip
|
||||
(
|
||||
prefix String,
|
||||
bot_name String,
|
||||
action String,
|
||||
rule_id UInt64,
|
||||
has_ua UInt8,
|
||||
category String
|
||||
)
|
||||
PRIMARY KEY prefix
|
||||
SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'admin' PASSWORD 'CHANGE_ME' DB 'mabase_prod' TABLE 'anubis_ip_rules'))
|
||||
LAYOUT(IP_TRIE())
|
||||
LIFETIME(MIN 300 MAX 600);
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 5. TABLE SOURCE — règles ASN (pour dictionnaire Flat)
|
||||
-- Alimentée par botPolicies.yaml via fetch_rules.py → insert_asn_rules()
|
||||
-- ----------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS mabase_prod.anubis_asn_rules
|
||||
(
|
||||
asn UInt32,
|
||||
bot_name LowCardinality(String),
|
||||
action LowCardinality(String),
|
||||
category LowCardinality(String)
|
||||
)
|
||||
ENGINE = ReplacingMergeTree()
|
||||
ORDER BY asn;
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 6. TABLE SOURCE — règles pays ISO-3166 (pour dictionnaire Flat)
|
||||
-- Alimentée par botPolicies.yaml via fetch_rules.py → insert_country_rules()
|
||||
-- ----------------------------------------------------------------------------
|
||||
CREATE TABLE IF NOT EXISTS mabase_prod.anubis_country_rules
|
||||
(
|
||||
country_code LowCardinality(String),
|
||||
bot_name LowCardinality(String),
|
||||
action LowCardinality(String),
|
||||
category LowCardinality(String)
|
||||
)
|
||||
ENGINE = ReplacingMergeTree()
|
||||
ORDER BY country_code;
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 7. DICTIONNAIRE ASN — Flat
|
||||
-- dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', src_asn, '')
|
||||
-- ----------------------------------------------------------------------------
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_anubis_asn;
|
||||
CREATE DICTIONARY mabase_prod.dict_anubis_asn
|
||||
(
|
||||
asn UInt32,
|
||||
bot_name String,
|
||||
action String,
|
||||
category String
|
||||
)
|
||||
PRIMARY KEY asn
|
||||
SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'admin' PASSWORD 'CHANGE_ME' DB 'mabase_prod' TABLE 'anubis_asn_rules'))
|
||||
LAYOUT(FLAT())
|
||||
LIFETIME(MIN 300 MAX 600);
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 8. DICTIONNAIRE PAYS — Flat
|
||||
-- dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', src_country_code, '')
|
||||
-- ----------------------------------------------------------------------------
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_anubis_country;
|
||||
CREATE DICTIONARY mabase_prod.dict_anubis_country
|
||||
(
|
||||
country_code String,
|
||||
bot_name String,
|
||||
action String,
|
||||
category String
|
||||
)
|
||||
PRIMARY KEY country_code
|
||||
SOURCE(CLICKHOUSE(HOST 'localhost' PORT 9000 USER 'admin' PASSWORD 'CHANGE_ME' DB 'mabase_prod' TABLE 'anubis_country_rules'))
|
||||
LAYOUT(FLAT())
|
||||
LIFETIME(MIN 300 MAX 600);
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 9. AJOUT DES COLONNES ANUBIS dans http_logs
|
||||
-- Idempotent : ne plante pas si déjà présentes
|
||||
-- ----------------------------------------------------------------------------
|
||||
ALTER TABLE mabase_prod.http_logs
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_name LowCardinality(String) DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_action LowCardinality(String) DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_category LowCardinality(String) DEFAULT '';
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 10. RECONSTRUCTION DE mv_http_logs avec enrichissement Anubis
|
||||
-- Logique de priorisation :
|
||||
-- 1. UA regex (plus informatif — identifie le bot précis)
|
||||
-- 2. IP/CIDR (fallback — identifie le réseau cloud)
|
||||
-- ----------------------------------------------------------------------------
|
||||
DROP VIEW IF EXISTS mabase_prod.mv_http_logs;
|
||||
|
||||
CREATE MATERIALIZED VIEW mabase_prod.mv_http_logs
|
||||
TO mabase_prod.http_logs
|
||||
(
|
||||
`time` DateTime,
|
||||
`log_date` Date,
|
||||
`src_ip` IPv4,
|
||||
`src_port` UInt16,
|
||||
`src_asn` UInt32,
|
||||
`src_country_code` String,
|
||||
`dst_ip` IPv4,
|
||||
`dst_port` UInt16,
|
||||
`src_as_name` String,
|
||||
`src_org` String,
|
||||
`src_domain` String,
|
||||
`method` String,
|
||||
`scheme` String,
|
||||
`host` String,
|
||||
`path` String,
|
||||
`query` String,
|
||||
`http_version` String,
|
||||
`orphan_side` String,
|
||||
`correlated` UInt8,
|
||||
`keepalives` UInt16,
|
||||
`a_timestamp` UInt64,
|
||||
`b_timestamp` UInt64,
|
||||
`conn_id` String,
|
||||
`ip_meta_df` UInt8,
|
||||
`ip_meta_id` UInt16,
|
||||
`ip_meta_total_length` UInt16,
|
||||
`ip_meta_ttl` UInt8,
|
||||
`tcp_meta_options` String,
|
||||
`tcp_meta_window_size` UInt32,
|
||||
`tcp_meta_mss` UInt16,
|
||||
`tcp_meta_window_scale` UInt8,
|
||||
`syn_to_clienthello_ms` Int32,
|
||||
`tls_version` String,
|
||||
`tls_sni` String,
|
||||
`tls_alpn` String,
|
||||
`ja3` String,
|
||||
`ja3_hash` String,
|
||||
`ja4` String,
|
||||
`client_headers` String,
|
||||
`header_user_agent` String,
|
||||
`header_accept` String,
|
||||
`header_accept_encoding` String,
|
||||
`header_accept_language` String,
|
||||
`header_content_type` String,
|
||||
`header_x_request_id` String,
|
||||
`header_x_trace_id` String,
|
||||
`header_x_forwarded_for` String,
|
||||
`header_sec_ch_ua` String,
|
||||
`header_sec_ch_ua_mobile` String,
|
||||
`header_sec_ch_ua_platform` String,
|
||||
`header_sec_fetch_dest` String,
|
||||
`header_sec_fetch_mode` String,
|
||||
`header_sec_fetch_site` String,
|
||||
`anubis_bot_name` String,
|
||||
`anubis_bot_action` String
|
||||
)
|
||||
AS SELECT
|
||||
parseDateTimeBestEffort(coalesce(JSONExtractString(raw_json, 'time'), '1970-01-01T00:00:00Z')) AS time,
|
||||
toDate(time) AS log_date,
|
||||
toIPv4(coalesce(JSONExtractString(raw_json, 'src_ip'), '0.0.0.0')) AS src_ip,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'src_port'), 0)) AS src_port,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'asn', toIPv6(src_ip), toUInt32(0)) AS src_asn,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'country_code', toIPv6(src_ip), '') AS src_country_code,
|
||||
toIPv4(coalesce(JSONExtractString(raw_json, 'dst_ip'), '0.0.0.0')) AS dst_ip,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'dst_port'), 0)) AS dst_port,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'name', toIPv6(src_ip), '') AS src_as_name,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'org', toIPv6(src_ip), '') AS src_org,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'domain', toIPv6(src_ip), '') AS src_domain,
|
||||
coalesce(JSONExtractString(raw_json, 'method'), '') AS method,
|
||||
coalesce(JSONExtractString(raw_json, 'scheme'), '') AS scheme,
|
||||
coalesce(JSONExtractString(raw_json, 'host'), '') AS host,
|
||||
coalesce(JSONExtractString(raw_json, 'path'), '') AS path,
|
||||
coalesce(JSONExtractString(raw_json, 'query'), '') AS query,
|
||||
coalesce(JSONExtractString(raw_json, 'http_version'), '') AS http_version,
|
||||
coalesce(JSONExtractString(raw_json, 'orphan_side'), '') AS orphan_side,
|
||||
toUInt8(coalesce(JSONExtractBool(raw_json, 'correlated'), 0)) AS correlated,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'keepalives'), 0)) AS keepalives,
|
||||
coalesce(JSONExtractUInt(raw_json, 'a_timestamp'), 0) AS a_timestamp,
|
||||
coalesce(JSONExtractUInt(raw_json, 'b_timestamp'), 0) AS b_timestamp,
|
||||
coalesce(JSONExtractString(raw_json, 'conn_id'), '') AS conn_id,
|
||||
toUInt8(coalesce(JSONExtractBool(raw_json, 'ip_meta_df'), 0)) AS ip_meta_df,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'ip_meta_id'), 0)) AS ip_meta_id,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'ip_meta_total_length'), 0)) AS ip_meta_total_length,
|
||||
toUInt8(coalesce(JSONExtractUInt(raw_json, 'ip_meta_ttl'), 0)) AS ip_meta_ttl,
|
||||
coalesce(JSONExtractString(raw_json, 'tcp_meta_options'), '') AS tcp_meta_options,
|
||||
toUInt32(coalesce(JSONExtractUInt(raw_json, 'tcp_meta_window_size'), 0)) AS tcp_meta_window_size,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'tcp_meta_mss'), 0)) AS tcp_meta_mss,
|
||||
toUInt8(coalesce(JSONExtractUInt(raw_json, 'tcp_meta_window_scale'), 0)) AS tcp_meta_window_scale,
|
||||
toInt32(coalesce(JSONExtractInt(raw_json, 'syn_to_clienthello_ms'), 0)) AS syn_to_clienthello_ms,
|
||||
coalesce(JSONExtractString(raw_json, 'tls_version'), '') AS tls_version,
|
||||
coalesce(JSONExtractString(raw_json, 'tls_sni'), '') AS tls_sni,
|
||||
coalesce(JSONExtractString(raw_json, 'tls_alpn'), '') AS tls_alpn,
|
||||
coalesce(JSONExtractString(raw_json, 'ja3'), '') AS ja3,
|
||||
coalesce(JSONExtractString(raw_json, 'ja3_hash'), '') AS ja3_hash,
|
||||
coalesce(JSONExtractString(raw_json, 'ja4'), '') AS ja4,
|
||||
coalesce(JSONExtractString(raw_json, 'client_headers'), '') AS client_headers,
|
||||
coalesce(JSONExtractString(raw_json, 'header_User-Agent'), '') AS header_user_agent,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Accept'), '') AS header_accept,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Accept-Encoding'), '') AS header_accept_encoding,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Accept-Language'), '') AS header_accept_language,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Content-Type'), '') AS header_content_type,
|
||||
coalesce(JSONExtractString(raw_json, 'header_X-Request-Id'), '') AS header_x_request_id,
|
||||
coalesce(JSONExtractString(raw_json, 'header_X-Trace-Id'), '') AS header_x_trace_id,
|
||||
coalesce(JSONExtractString(raw_json, 'header_X-Forwarded-For'), '') AS header_x_forwarded_for,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA'), '') AS header_sec_ch_ua,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA-Mobile'), '') AS header_sec_ch_ua_mobile,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA-Platform'), '') AS header_sec_ch_ua_platform,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Dest'), '') AS header_sec_fetch_dest,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Mode'), '') AS header_sec_fetch_mode,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Site'), '') AS header_sec_fetch_site,
|
||||
-- ── Enrichissement Anubis ────────────────────────────────────────────────
|
||||
-- Priorité : UA regex > IP/CIDR (UA identifie précisément le bot)
|
||||
COALESCE(
|
||||
nullIf(dictGet('mabase_prod.dict_anubis_ua', 'bot_name',
|
||||
coalesce(JSONExtractString(raw_json, 'header_User-Agent'), '')), ''),
|
||||
nullIf(dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name',
|
||||
toIPv6(toIPv4(coalesce(JSONExtractString(raw_json, 'src_ip'), '0.0.0.0'))), ''), ''),
|
||||
''
|
||||
) AS anubis_bot_name,
|
||||
COALESCE(
|
||||
nullIf(dictGet('mabase_prod.dict_anubis_ua', 'action',
|
||||
coalesce(JSONExtractString(raw_json, 'header_User-Agent'), '')), ''),
|
||||
nullIf(dictGetOrDefault('mabase_prod.dict_anubis_ip', 'action',
|
||||
toIPv6(toIPv4(coalesce(JSONExtractString(raw_json, 'src_ip'), '0.0.0.0'))), ''), ''),
|
||||
''
|
||||
) AS anubis_bot_action
|
||||
FROM mabase_prod.http_logs_raw;
|
||||
|
||||
-- ============================================================================
|
||||
-- INTÉGRATION ML — Propagation Anubis vers le pipeline bot_detector
|
||||
-- ============================================================================
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 11. COLONNES ANUBIS dans ml_detected_anomalies
|
||||
-- ----------------------------------------------------------------------------
|
||||
ALTER TABLE mabase_prod.ml_detected_anomalies
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_name LowCardinality(String) DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_action LowCardinality(String) DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_category LowCardinality(String) DEFAULT '';
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 12. COLONNES ANUBIS dans ml_all_scores
|
||||
-- ----------------------------------------------------------------------------
|
||||
ALTER TABLE mabase_prod.ml_all_scores
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_name LowCardinality(String) DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_action LowCardinality(String) DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS anubis_bot_category LowCardinality(String) DEFAULT '';
|
||||
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- 13. VIEW view_ai_features_1h — Enrichissement Anubis
|
||||
-- Ajoute anubis_bot_name et anubis_bot_action via dictGet.
|
||||
-- Priorité : UA regex (first_ua → dict_anubis_ua) > IP/CIDR (src_ip → dict_anubis_ip)
|
||||
-- Voir le fichier complet dans /tmp/update_view_ai_features.sql ou recréer
|
||||
-- avec CREATE OR REPLACE VIEW après avoir appliqué les étapes précédentes.
|
||||
-- ----------------------------------------------------------------------------
|
||||
-- NOTE : Exécuter le contenu de /tmp/update_view_ai_features.sql ici (trop long).
|
||||
-- Ou lancer depuis le repo : psql -f bot_detector/anubis/view_ai_features_anubis.sql
|
||||
486
services/bot-detector/anubis/fetch_rules.py
Normal file
486
services/bot-detector/anubis/fetch_rules.py
Normal file
@ -0,0 +1,486 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
fetch_rules.py — Récupère TOUTES les règles Anubis depuis GitHub et les insère dans ClickHouse.
|
||||
|
||||
Sources :
|
||||
- data/bots/**/*.yaml (bots pathologiques, IA, IRC)
|
||||
- data/crawlers/*.yaml (crawlers légitimes et clouds)
|
||||
- data/clients/*.yaml (clients IA agissant pour utilisateurs)
|
||||
- data/common/*.yaml (règles communes : IPs privées, etc.)
|
||||
- data/botPolicies.yaml (règles ASN et pays inline)
|
||||
|
||||
Usage (depuis le container dashboard_web) :
|
||||
python /tmp/fetch_rules.py
|
||||
|
||||
Variables d'environnement :
|
||||
CLICKHOUSE_HOST, CLICKHOUSE_DB, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print("[ERREUR] pyyaml manquant.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
import clickhouse_connect
|
||||
except ImportError:
|
||||
print("[ERREUR] clickhouse-connect manquant.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Config
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
GITHUB_API = "https://api.github.com/repos/TecharoHQ/anubis/contents"
|
||||
GITHUB_RAW = "https://raw.githubusercontent.com/TecharoHQ/anubis/main"
|
||||
|
||||
# Répertoires à parcourir — ORDER CRITIQUE pour REGEXP_TREE :
|
||||
# Dans REGEXP_TREE (root-level rules), la règle avec l'ID le plus bas gagne quand plusieurs matchent.
|
||||
# → Les règles SPÉCIFIQUES doivent être chargées en PREMIER (IDs bas) pour gagner sur les catch-alls.
|
||||
# → Les catch-alls (ai-robots-txt, ai-catchall) doivent être chargés en DERNIER (IDs hauts).
|
||||
#
|
||||
# Au sein de chaque répertoire, les fichiers sont triés EN ORDRE ALPHABÉTIQUE INVERSÉ
|
||||
# pour que les règles spécifiques (noms longs) aient des IDs plus bas que les catch-alls (ai.yaml).
|
||||
DIRECTORIES = [
|
||||
("data/clients", "clients"), # Règles AI clients avec IP (openai-chatgpt-user, etc.)
|
||||
("data/bots/irc-bots", "bots/irc-bots"), # Bots IRC spécifiques
|
||||
("data/crawlers", "crawlers"), # Crawlers spécifiques + clouds
|
||||
("data/common", "common"), # IPs privées, routes communes
|
||||
("data/bots", "bots"), # Catch-alls larges (ai-robots-txt, ai-catchall) — LAST
|
||||
]
|
||||
|
||||
# Fichier de politique principal (règles ASN + pays inline)
|
||||
BOT_POLICIES_PATH = "data/botPolicies.yaml"
|
||||
|
||||
# UA_PARENT_OVERRIDE : mapping nom_règle → nom_parent pour forcer la hiérarchie REGEXP_TREE.
|
||||
# Conservé vide intentionnellement : l'ordre de chargement (spécifique avant catch-all)
|
||||
# garantit la priorité sans hiérarchie parent_id explicite.
|
||||
# Populer ce dict si une règle doit hériter d'une autre via parent_id dans REGEXP_TREE.
|
||||
UA_PARENT_OVERRIDE: dict[str, str] = {}
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# HTTP helpers
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _fetch_url(url: str, timeout: int = 15) -> str | None:
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=timeout) as resp:
|
||||
return resp.read().decode("utf-8")
|
||||
except urllib.error.URLError as e:
|
||||
print(f"[WARN] {url}: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def fetch_yaml_url(url: str) -> list | dict | None:
|
||||
content = _fetch_url(url)
|
||||
if content:
|
||||
return yaml.safe_load(content)
|
||||
return None
|
||||
|
||||
|
||||
def list_yaml_files(api_path: str) -> list[str]:
|
||||
"""
|
||||
Retourne la liste des raw URLs des fichiers .yaml/.yml dans api_path via l'API GitHub.
|
||||
Les fichiers sont triés en ordre ALPHABÉTIQUE INVERSÉ pour que les règles spécifiques
|
||||
(noms longs, ex: openai-chatgpt-user.yaml) aient un ID inférieur aux catch-alls (ai.yaml).
|
||||
"""
|
||||
content = _fetch_url(f"{GITHUB_API}/{api_path}")
|
||||
if not content:
|
||||
return []
|
||||
try:
|
||||
entries = json.loads(content)
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
files = [
|
||||
entry for entry in entries
|
||||
if entry.get("type") == "file" and entry.get("name", "").endswith((".yaml", ".yml"))
|
||||
]
|
||||
# Tri inverse : les noms longs (spécifiques) avant les noms courts (catch-alls)
|
||||
files.sort(key=lambda e: e["name"], reverse=True)
|
||||
return [f["download_url"] for f in files]
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Extraction des patterns UA depuis les expressions CEL-like
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def _extract_ua_from_all(conditions: list) -> str | None:
|
||||
"""Extrait une regex UA depuis une expression 'all' (ex: yandexbot userAgent.matches)."""
|
||||
for cond in conditions:
|
||||
if not isinstance(cond, str):
|
||||
continue
|
||||
m = re.search(r'userAgent\.matches\("(.+?)"\)', cond)
|
||||
if m:
|
||||
return m.group(1).replace("\\\\", "\\")
|
||||
return None
|
||||
|
||||
|
||||
def _extract_ua_from_any(conditions: list) -> str | None:
|
||||
"""
|
||||
Extrait une regex UA depuis une expression 'any' avec userAgent.contains(...)
|
||||
Exemple : aggressive-brazilian-scrapers.yaml
|
||||
Retourne une regex en OR : MSIE|Trident|...
|
||||
"""
|
||||
patterns = []
|
||||
for cond in conditions:
|
||||
if not isinstance(cond, str):
|
||||
continue
|
||||
m = re.search(r'userAgent\.contains\("(.+?)"\)', cond)
|
||||
if m:
|
||||
patterns.append(re.escape(m.group(1)))
|
||||
if patterns:
|
||||
return "|".join(patterns)
|
||||
return None
|
||||
|
||||
|
||||
def extract_ua_regex(rule: dict) -> str | None:
|
||||
"""Extrait la regex User-Agent depuis toutes les formes possibles."""
|
||||
# Forme directe
|
||||
if ua := rule.get("user_agent_regex"):
|
||||
return ua.strip()
|
||||
|
||||
expr = rule.get("expression")
|
||||
if not expr:
|
||||
return None
|
||||
|
||||
# Expression scalaire (CEL string)
|
||||
if isinstance(expr, str):
|
||||
m = re.search(r'userAgent\.matches\("(.+?)"\)', expr)
|
||||
if m:
|
||||
return m.group(1).replace("\\\\", "\\")
|
||||
m = re.search(r'userAgent\.contains\("(.+?)"\)', expr)
|
||||
if m:
|
||||
return re.escape(m.group(1))
|
||||
return None
|
||||
|
||||
# Expression structurée dict
|
||||
if isinstance(expr, dict):
|
||||
if ua := _extract_ua_from_all(expr.get("all", [])):
|
||||
return ua
|
||||
if ua := _extract_ua_from_any(expr.get("any", [])):
|
||||
return ua
|
||||
|
||||
return None
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Parse des fichiers YAML
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def parse_file(
|
||||
url: str,
|
||||
category: str,
|
||||
ua_name_to_id: dict,
|
||||
ua_id_counter_ref: list, # [int] — compteur mutable partagé entre appels
|
||||
rule_id_counter_ref: list, # [int] — idem
|
||||
) -> tuple[list[dict], list[dict]]:
|
||||
"""
|
||||
Parse un fichier YAML Anubis.
|
||||
Retourne (ua_rules, ip_rules).
|
||||
|
||||
Note : ua_name_to_id est maintenu pour supporter la hiérarchie parent_id dans
|
||||
REGEXP_TREE (via UA_PARENT_OVERRIDE). Tant que UA_PARENT_OVERRIDE est vide,
|
||||
parent_id vaut toujours 0 et ua_name_to_id n'est pas consulté en pratique.
|
||||
"""
|
||||
data = fetch_yaml_url(url)
|
||||
if not data or not isinstance(data, list):
|
||||
return [], []
|
||||
|
||||
ua_rules, ip_rules = [], []
|
||||
|
||||
for rule in data:
|
||||
if not isinstance(rule, dict):
|
||||
continue
|
||||
# Ignorer les imports (références à d'autres fichiers)
|
||||
if "import" in rule:
|
||||
continue
|
||||
|
||||
name = rule.get("name", "").strip()
|
||||
action = rule.get("action", "").strip()
|
||||
if not name or not action:
|
||||
continue
|
||||
|
||||
remote_addrs = [str(c).strip() for c in rule.get("remote_addresses", []) if c]
|
||||
has_ip = bool(remote_addrs)
|
||||
|
||||
rule_id = rule_id_counter_ref[0]
|
||||
rule_id_counter_ref[0] += 1
|
||||
|
||||
# ── User-Agent regex ─────────────────────────────────────────────────
|
||||
ua_regex = extract_ua_regex(rule)
|
||||
if ua_regex:
|
||||
parent_name = UA_PARENT_OVERRIDE.get(name)
|
||||
parent_id = ua_name_to_id.get(parent_name, 0) if parent_name else 0
|
||||
|
||||
uid = ua_id_counter_ref[0]
|
||||
ua_id_counter_ref[0] += 1
|
||||
ua_name_to_id[name] = uid
|
||||
|
||||
ua_rules.append({
|
||||
"id": uid,
|
||||
"parent_id": parent_id,
|
||||
"regexp": ua_regex,
|
||||
"bot_name": name,
|
||||
"action": action,
|
||||
"has_ip": "1" if has_ip else "0",
|
||||
"rule_id": str(rule_id),
|
||||
"category": category,
|
||||
})
|
||||
|
||||
# ── IP/CIDR ranges ───────────────────────────────────────────────────
|
||||
has_ua = bool(ua_regex)
|
||||
for cidr in remote_addrs:
|
||||
ip_rules.append({
|
||||
"prefix": cidr,
|
||||
"bot_name": name,
|
||||
"action": action,
|
||||
"rule_id": rule_id,
|
||||
"has_ua": 1 if has_ua else 0,
|
||||
"category": category,
|
||||
})
|
||||
|
||||
return ua_rules, ip_rules
|
||||
|
||||
|
||||
def parse_bot_policies_inline(url: str) -> tuple[list[dict], list[dict]]:
|
||||
"""
|
||||
Parse botPolicies.yaml pour les règles inline avec geoip.countries et asns.match.
|
||||
Retourne (asn_rules, country_rules).
|
||||
"""
|
||||
data = fetch_yaml_url(url)
|
||||
if not data or not isinstance(data, dict):
|
||||
return [], []
|
||||
|
||||
asn_rules: list[dict] = []
|
||||
country_rules: list[dict] = []
|
||||
|
||||
for rule in data.get("bots", []):
|
||||
if not isinstance(rule, dict):
|
||||
continue
|
||||
if "import" in rule:
|
||||
continue
|
||||
|
||||
name = rule.get("name", "").strip()
|
||||
action = rule.get("action", "").strip()
|
||||
if not name or not action:
|
||||
continue
|
||||
|
||||
# ASN rules
|
||||
asns = rule.get("asns", {})
|
||||
if isinstance(asns, dict):
|
||||
for asn in asns.get("match", []):
|
||||
asn_rules.append({
|
||||
"asn": int(asn),
|
||||
"bot_name": name,
|
||||
"action": action,
|
||||
"category": "policies",
|
||||
})
|
||||
|
||||
# Country rules
|
||||
geoip = rule.get("geoip", {})
|
||||
if isinstance(geoip, dict):
|
||||
for cc in geoip.get("countries", []):
|
||||
country_rules.append({
|
||||
"country_code": str(cc).upper(),
|
||||
"bot_name": name,
|
||||
"action": action,
|
||||
"category": "policies",
|
||||
})
|
||||
|
||||
return asn_rules, country_rules
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Collecte de toutes les règles
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def collect_all_rules() -> tuple[list, list, list, list]:
|
||||
"""Retourne (ua_rules, ip_rules, asn_rules, country_rules)."""
|
||||
ua_name_to_id: dict[str, int] = {}
|
||||
ua_id_counter_ref: list[int] = [1]
|
||||
rule_id_counter: list[int] = [1]
|
||||
|
||||
all_ua: list[dict] = []
|
||||
all_ip: list[dict] = []
|
||||
|
||||
for api_path, category in DIRECTORIES:
|
||||
print(f"[INFO] Parcours de {api_path} ({category})…")
|
||||
file_urls = list_yaml_files(api_path)
|
||||
print(f" {len(file_urls)} fichiers trouvés")
|
||||
for url in file_urls:
|
||||
ua, ip = parse_file(url, category, ua_name_to_id, ua_id_counter_ref, rule_id_counter)
|
||||
all_ua.extend(ua)
|
||||
all_ip.extend(ip)
|
||||
|
||||
# Règles ASN + pays depuis botPolicies.yaml
|
||||
print(f"[INFO] Lecture de botPolicies.yaml…")
|
||||
policies_url = f"{GITHUB_RAW}/{BOT_POLICIES_PATH}"
|
||||
asn_rules, country_rules = parse_bot_policies_inline(policies_url)
|
||||
|
||||
return all_ua, all_ip, asn_rules, country_rules
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# ClickHouse
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def get_ch_client():
|
||||
return clickhouse_connect.get_client(
|
||||
host=os.environ.get("CLICKHOUSE_HOST", "clickhouse"),
|
||||
database=os.environ.get("CLICKHOUSE_DB", "mabase_prod"),
|
||||
username=os.environ.get("CLICKHOUSE_USER", "admin"),
|
||||
password=os.environ.get("CLICKHOUSE_PASSWORD", ""),
|
||||
)
|
||||
|
||||
|
||||
def insert_ua_rules(client, rules: list[dict]) -> None:
|
||||
if not rules:
|
||||
print("[INFO] Aucune règle UA.")
|
||||
return
|
||||
client.command("TRUNCATE TABLE mabase_prod.anubis_ua_rules")
|
||||
# REGEXP_TREE format : id, parent_id, regexp, keys[], values[]
|
||||
# keys = ['bot_name', 'action', 'has_ip', 'rule_id', 'category']
|
||||
data = [
|
||||
[
|
||||
r["id"], r["parent_id"], r["regexp"],
|
||||
["bot_name", "action", "has_ip", "rule_id", "category"],
|
||||
[r["bot_name"], r["action"], r["has_ip"], r["rule_id"], r["category"]],
|
||||
]
|
||||
for r in rules
|
||||
]
|
||||
client.insert("mabase_prod.anubis_ua_rules", data,
|
||||
column_names=["id", "parent_id", "regexp", "keys", "values"])
|
||||
print(f"[OK] {len(rules)} règles UA insérées.")
|
||||
|
||||
|
||||
def insert_ip_rules(client, rules: list[dict]) -> None:
|
||||
if not rules:
|
||||
print("[INFO] Aucune règle IP.")
|
||||
return
|
||||
client.command("TRUNCATE TABLE mabase_prod.anubis_ip_rules")
|
||||
data = [
|
||||
[r["prefix"], r["bot_name"], r["action"],
|
||||
r["rule_id"], r["has_ua"], r["category"]]
|
||||
for r in rules
|
||||
]
|
||||
client.insert("mabase_prod.anubis_ip_rules", data,
|
||||
column_names=["prefix", "bot_name", "action", "rule_id", "has_ua", "category"])
|
||||
print(f"[OK] {len(rules)} règles IP insérées.")
|
||||
|
||||
|
||||
def insert_asn_rules(client, rules: list[dict]) -> None:
|
||||
if not rules:
|
||||
print("[INFO] Aucune règle ASN.")
|
||||
return
|
||||
client.command("TRUNCATE TABLE mabase_prod.anubis_asn_rules")
|
||||
data = [[r["asn"], r["bot_name"], r["action"], r["category"]] for r in rules]
|
||||
client.insert("mabase_prod.anubis_asn_rules", data,
|
||||
column_names=["asn", "bot_name", "action", "category"])
|
||||
print(f"[OK] {len(rules)} règles ASN insérées.")
|
||||
|
||||
|
||||
def insert_country_rules(client, rules: list[dict]) -> None:
|
||||
if not rules:
|
||||
print("[INFO] Aucune règle pays.")
|
||||
return
|
||||
client.command("TRUNCATE TABLE mabase_prod.anubis_country_rules")
|
||||
data = [[r["country_code"], r["bot_name"], r["action"], r["category"]] for r in rules]
|
||||
client.insert("mabase_prod.anubis_country_rules", data,
|
||||
column_names=["country_code", "bot_name", "action", "category"])
|
||||
print(f"[OK] {len(rules)} règles pays insérées.")
|
||||
|
||||
|
||||
def reload_dicts(client) -> None:
|
||||
dicts = [
|
||||
"mabase_prod.dict_anubis_ua",
|
||||
"mabase_prod.dict_anubis_ip",
|
||||
"mabase_prod.dict_anubis_asn",
|
||||
"mabase_prod.dict_anubis_country",
|
||||
]
|
||||
for d in dicts:
|
||||
try:
|
||||
client.command(f"SYSTEM RELOAD DICTIONARY {d}")
|
||||
print(f"[OK] {d} rechargé.")
|
||||
except Exception as e:
|
||||
print(f"[WARN] Rechargement {d}: {e}", file=sys.stderr)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Rapport
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def print_summary(ua_rules, ip_rules, asn_rules, country_rules):
|
||||
print("\n── Règles UA ──")
|
||||
by_cat: dict[str, list] = {}
|
||||
for r in ua_rules:
|
||||
by_cat.setdefault(r["category"], []).append(r)
|
||||
for cat, rules in sorted(by_cat.items()):
|
||||
print(f" [{cat}] {len(rules)} règle(s)")
|
||||
for r in rules[:5]:
|
||||
has = " [+IP]" if r["has_ip"] == "1" else ""
|
||||
par = f" [parent={r['parent_id']}]" if r["parent_id"] else ""
|
||||
print(f" [{r['action']:9s}] {r['bot_name']}{has}{par}: {r['regexp'][:50]}")
|
||||
if len(rules) > 5:
|
||||
print(f" … et {len(rules) - 5} autres")
|
||||
|
||||
print(f"\n── Règles IP : {len(ip_rules)} CIDRs ──")
|
||||
by_bot: dict[str, list] = {}
|
||||
for r in ip_rules:
|
||||
by_bot.setdefault(r["bot_name"], []).append(r)
|
||||
for bot, rs in sorted(by_bot.items())[:15]:
|
||||
print(f" [{rs[0]['action']:9s}] {bot}: {len(rs)} CIDRs (cat={rs[0]['category']}, has_ua={rs[0]['has_ua']})")
|
||||
if len(by_bot) > 15:
|
||||
print(f" … et {len(by_bot) - 15} autres bots")
|
||||
|
||||
if asn_rules:
|
||||
print(f"\n── Règles ASN : {len(asn_rules)} ──")
|
||||
for r in asn_rules:
|
||||
print(f" [{r['action']:9s}] ASN {r['asn']}: {r['bot_name']}")
|
||||
|
||||
if country_rules:
|
||||
print(f"\n── Règles pays : {len(country_rules)} ──")
|
||||
for r in country_rules:
|
||||
print(f" [{r['action']:9s}] {r['country_code']}: {r['bot_name']}")
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Main
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main() -> None:
|
||||
print("[INFO] Collecte des règles Anubis depuis GitHub…")
|
||||
ua_rules, ip_rules, asn_rules, country_rules = collect_all_rules()
|
||||
|
||||
total = len(ua_rules) + len(ip_rules) + len(asn_rules) + len(country_rules)
|
||||
print(f"\n[INFO] {len(ua_rules)} règles UA, {len(ip_rules)} CIDRs IP, "
|
||||
f"{len(asn_rules)} ASN, {len(country_rules)} pays (total={total})")
|
||||
|
||||
if total == 0:
|
||||
print("[ERREUR] Aucune règle récupérée.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print_summary(ua_rules, ip_rules, asn_rules, country_rules)
|
||||
|
||||
print("\n[INFO] Connexion à ClickHouse…")
|
||||
client = get_ch_client()
|
||||
|
||||
insert_ua_rules(client, ua_rules)
|
||||
insert_ip_rules(client, ip_rules)
|
||||
insert_asn_rules(client, asn_rules)
|
||||
insert_country_rules(client, country_rules)
|
||||
reload_dicts(client)
|
||||
|
||||
print("\n[OK] Règles Anubis chargées avec succès.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
126
services/bot-detector/anubis/mv_http_logs.sql
Normal file
126
services/bot-detector/anubis/mv_http_logs.sql
Normal file
@ -0,0 +1,126 @@
|
||||
CREATE MATERIALIZED VIEW mabase_prod.mv_http_logs
|
||||
TO mabase_prod.http_logs
|
||||
AS
|
||||
WITH
|
||||
coalesce(JSONExtractString(raw_json, 'header_User-Agent'), '') AS _ua,
|
||||
toIPv6(toIPv4(coalesce(JSONExtractString(raw_json, 'src_ip'), '0.0.0.0'))) AS _ip,
|
||||
toUInt32(dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'asn', _ip, toUInt32(0))) AS _asn,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'country_code', _ip, '') AS _cc
|
||||
SELECT
|
||||
parseDateTimeBestEffort(coalesce(JSONExtractString(raw_json, 'time'), '1970-01-01T00:00:00Z')) AS time,
|
||||
toDate(time) AS log_date,
|
||||
toIPv4(coalesce(JSONExtractString(raw_json, 'src_ip'), '0.0.0.0')) AS src_ip,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'src_port'), 0)) AS src_port,
|
||||
_asn AS src_asn,
|
||||
_cc AS src_country_code,
|
||||
toIPv4(coalesce(JSONExtractString(raw_json, 'dst_ip'), '0.0.0.0')) AS dst_ip,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'dst_port'), 0)) AS dst_port,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'name', _ip, '') AS src_as_name,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'org', _ip, '') AS src_org,
|
||||
dictGetOrDefault('mabase_prod.dict_iplocate_asn', 'domain', _ip, '') AS src_domain,
|
||||
coalesce(JSONExtractString(raw_json, 'method'), '') AS method,
|
||||
coalesce(JSONExtractString(raw_json, 'scheme'), '') AS scheme,
|
||||
coalesce(JSONExtractString(raw_json, 'host'), '') AS host,
|
||||
coalesce(JSONExtractString(raw_json, 'path'), '') AS path,
|
||||
coalesce(JSONExtractString(raw_json, 'query'), '') AS query,
|
||||
coalesce(JSONExtractString(raw_json, 'http_version'), '') AS http_version,
|
||||
coalesce(JSONExtractString(raw_json, 'orphan_side'), '') AS orphan_side,
|
||||
toUInt8(coalesce(JSONExtractBool(raw_json, 'correlated'), 0)) AS correlated,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'keepalives'), 0)) AS keepalives,
|
||||
coalesce(JSONExtractUInt(raw_json, 'a_timestamp'), 0) AS a_timestamp,
|
||||
coalesce(JSONExtractUInt(raw_json, 'b_timestamp'), 0) AS b_timestamp,
|
||||
coalesce(JSONExtractString(raw_json, 'conn_id'), '') AS conn_id,
|
||||
toUInt8(coalesce(JSONExtractBool(raw_json, 'ip_meta_df'), 0)) AS ip_meta_df,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'ip_meta_id'), 0)) AS ip_meta_id,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'ip_meta_total_length'), 0)) AS ip_meta_total_length,
|
||||
toUInt8(coalesce(JSONExtractUInt(raw_json, 'ip_meta_ttl'), 0)) AS ip_meta_ttl,
|
||||
coalesce(JSONExtractString(raw_json, 'tcp_meta_options'), '') AS tcp_meta_options,
|
||||
toUInt32(coalesce(JSONExtractUInt(raw_json, 'tcp_meta_window_size'), 0)) AS tcp_meta_window_size,
|
||||
toUInt16(coalesce(JSONExtractUInt(raw_json, 'tcp_meta_mss'), 0)) AS tcp_meta_mss,
|
||||
toUInt8(coalesce(JSONExtractUInt(raw_json, 'tcp_meta_window_scale'), 0)) AS tcp_meta_window_scale,
|
||||
toInt32(coalesce(JSONExtractInt(raw_json, 'syn_to_clienthello_ms'), 0)) AS syn_to_clienthello_ms,
|
||||
coalesce(JSONExtractString(raw_json, 'tls_version'), '') AS tls_version,
|
||||
coalesce(JSONExtractString(raw_json, 'tls_sni'), '') AS tls_sni,
|
||||
coalesce(JSONExtractString(raw_json, 'tls_alpn'), '') AS tls_alpn,
|
||||
coalesce(JSONExtractString(raw_json, 'ja3'), '') AS ja3,
|
||||
coalesce(JSONExtractString(raw_json, 'ja3_hash'), '') AS ja3_hash,
|
||||
coalesce(JSONExtractString(raw_json, 'ja4'), '') AS ja4,
|
||||
coalesce(JSONExtractString(raw_json, 'client_headers'), '') AS client_headers,
|
||||
coalesce(JSONExtractString(raw_json, 'header_User-Agent'), '') AS header_user_agent,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Accept'), '') AS header_accept,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Accept-Encoding'), '') AS header_accept_encoding,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Accept-Language'), '') AS header_accept_language,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Content-Type'), '') AS header_content_type,
|
||||
coalesce(JSONExtractString(raw_json, 'header_X-Request-Id'), '') AS header_x_request_id,
|
||||
coalesce(JSONExtractString(raw_json, 'header_X-Trace-Id'), '') AS header_x_trace_id,
|
||||
coalesce(JSONExtractString(raw_json, 'header_X-Forwarded-For'), '') AS header_x_forwarded_for,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA'), '') AS header_sec_ch_ua,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA-Mobile'), '') AS header_sec_ch_ua_mobile,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA-Platform'), '') AS header_sec_ch_ua_platform,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Dest'), '') AS header_sec_fetch_dest,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Mode'), '') AS header_sec_fetch_mode,
|
||||
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Site'), '') AS header_sec_fetch_site,
|
||||
|
||||
-- Anubis enrichment : logique de correspondance combinée UA+IP
|
||||
-- Priorité : (1) UA+IP [même rule_id] > (2) UA seul > (3) IP seul > (4) ASN > (5) Pays
|
||||
CASE
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', _ua) = '1'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua) != ''
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '') != ''
|
||||
AND toUInt64OrZero(dictGet('mabase_prod.dict_anubis_ua', 'rule_id', _ua))
|
||||
= dictGetOrDefault('mabase_prod.dict_anubis_ip', 'rule_id', _ip, toUInt64(0))
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua)
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', _ua) = '0'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua) != ''
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua)
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'has_ua', _ip, toUInt8(0)) = 0
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', _asn, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', _asn, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', _cc, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', _cc, '')
|
||||
ELSE ''
|
||||
END AS anubis_bot_name,
|
||||
|
||||
CASE
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', _ua) = '1'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua) != ''
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '') != ''
|
||||
AND toUInt64OrZero(dictGet('mabase_prod.dict_anubis_ua', 'rule_id', _ua))
|
||||
= dictGetOrDefault('mabase_prod.dict_anubis_ip', 'rule_id', _ip, toUInt64(0))
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'action', _ua)
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', _ua) = '0'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua) != ''
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'action', _ua)
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'has_ua', _ip, toUInt8(0)) = 0
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'action', _ip, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', _asn, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'action', _asn, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', _cc, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'action', _cc, '')
|
||||
ELSE ''
|
||||
END AS anubis_bot_action,
|
||||
|
||||
CASE
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', _ua) = '1'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua) != ''
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '') != ''
|
||||
AND toUInt64OrZero(dictGet('mabase_prod.dict_anubis_ua', 'rule_id', _ua))
|
||||
= dictGetOrDefault('mabase_prod.dict_anubis_ip', 'rule_id', _ip, toUInt64(0))
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'category', _ua)
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', _ua) = '0'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', _ua) != ''
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'category', _ua)
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'has_ua', _ip, toUInt8(0)) = 0
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', _ip, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'category', _ip, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', _asn, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'category', _asn, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', _cc, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'category', _cc, '')
|
||||
ELSE ''
|
||||
END AS anubis_bot_category
|
||||
|
||||
FROM mabase_prod.http_logs_raw
|
||||
183
services/bot-detector/anubis/view_ai_features_anubis.sql
Normal file
183
services/bot-detector/anubis/view_ai_features_anubis.sql
Normal file
@ -0,0 +1,183 @@
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_ai_features_1h AS
|
||||
WITH base_data AS (
|
||||
SELECT
|
||||
a.window_start, a.src_ip, a.ja4, a.host,
|
||||
toString(a.src_asn) AS asn_number,
|
||||
a.src_as_name AS asn_org, a.src_org AS asn_detail, a.src_domain AS asn_domain,
|
||||
a.src_country_code AS country_code,
|
||||
dictGetOrDefault('mabase_prod.dict_asn_reputation', 'label', toUInt64(a.src_asn), 'unknown') AS asn_label,
|
||||
-- Bot connu via JA4/IP (dictionnaires existants)
|
||||
COALESCE(
|
||||
nullIf(dictGetOrDefault('mabase_prod.dict_bot_ip', 'bot_name', a.src_ip, ''), ''),
|
||||
nullIf(dictGetOrDefault('mabase_prod.dict_bot_ja4', 'bot_name', tuple(a.ja4), ''), ''),
|
||||
''
|
||||
) AS bot_name,
|
||||
-- Anubis : logique combinée UA+IP (même rule_id) > UA seul > IP seul > ASN > Pays
|
||||
CASE
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', a.first_ua) = '1'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua) != ''
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '') != ''
|
||||
AND toUInt64OrZero(dictGet('mabase_prod.dict_anubis_ua', 'rule_id', a.first_ua))
|
||||
= dictGetOrDefault('mabase_prod.dict_anubis_ip', 'rule_id', a.src_ip, toUInt64(0))
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua)
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', a.first_ua) = '0'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua) != ''
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua)
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'has_ua', a.src_ip, toUInt8(0)) = 0
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', toUInt32(a.src_asn), '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', toUInt32(a.src_asn), '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', a.src_country_code, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', a.src_country_code, '')
|
||||
ELSE ''
|
||||
END AS anubis_bot_name,
|
||||
CASE
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', a.first_ua) = '1'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua) != ''
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '') != ''
|
||||
AND toUInt64OrZero(dictGet('mabase_prod.dict_anubis_ua', 'rule_id', a.first_ua))
|
||||
= dictGetOrDefault('mabase_prod.dict_anubis_ip', 'rule_id', a.src_ip, toUInt64(0))
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'action', a.first_ua)
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', a.first_ua) = '0'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua) != ''
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'action', a.first_ua)
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'has_ua', a.src_ip, toUInt8(0)) = 0
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'action', a.src_ip, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', toUInt32(a.src_asn), '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'action', toUInt32(a.src_asn), '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', a.src_country_code, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'action', a.src_country_code, '')
|
||||
ELSE ''
|
||||
END AS anubis_bot_action,
|
||||
CASE
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', a.first_ua) = '1'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua) != ''
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '') != ''
|
||||
AND toUInt64OrZero(dictGet('mabase_prod.dict_anubis_ua', 'rule_id', a.first_ua))
|
||||
= dictGetOrDefault('mabase_prod.dict_anubis_ip', 'rule_id', a.src_ip, toUInt64(0))
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'category', a.first_ua)
|
||||
WHEN dictGet('mabase_prod.dict_anubis_ua', 'has_ip', a.first_ua) = '0'
|
||||
AND dictGet('mabase_prod.dict_anubis_ua', 'bot_name', a.first_ua) != ''
|
||||
THEN dictGet('mabase_prod.dict_anubis_ua', 'category', a.first_ua)
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'has_ua', a.src_ip, toUInt8(0)) = 0
|
||||
AND dictGetOrDefault('mabase_prod.dict_anubis_ip', 'bot_name', a.src_ip, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_ip', 'category', a.src_ip, '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'bot_name', toUInt32(a.src_asn), '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_asn', 'category', toUInt32(a.src_asn), '')
|
||||
WHEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'bot_name', a.src_country_code, '') != ''
|
||||
THEN dictGetOrDefault('mabase_prod.dict_anubis_country', 'category', a.src_country_code, '')
|
||||
ELSE ''
|
||||
END AS anubis_bot_category,
|
||||
a.hits AS hits,
|
||||
sum(a.hits) OVER (PARTITION BY a.src_ip) AS total_ip_hits,
|
||||
a.correlated AS correlated,
|
||||
a.tcp_jitter_variance AS tcp_jitter_variance,
|
||||
a.true_window_size AS true_window_size,
|
||||
a.window_mss_ratio AS window_mss_ratio,
|
||||
a.max_keepalives AS max_keepalives,
|
||||
h.header_order_hash AS header_order_hash, h.header_count AS header_count,
|
||||
h.has_accept_language AS has_accept_language, h.has_cookie AS has_cookie,
|
||||
h.has_referer AS has_referer, h.modern_browser_score AS modern_browser_score,
|
||||
h.ua_ch_mismatch AS ua_ch_mismatch,
|
||||
(a.count_post / (a.hits + 1)) AS post_ratio,
|
||||
(a.uniq_query_params / (a.uniq_paths + 1)) AS fuzzing_index,
|
||||
(a.hits / (dateDiff('second', a.first_seen, a.last_seen) + 1)) AS hit_velocity,
|
||||
(a.unique_src_ports / (a.hits + 1)) AS port_exhaustion_ratio,
|
||||
(a.orphan_count / (a.hits + 1)) AS orphan_ratio,
|
||||
(a.ip_id_zero_count / (a.hits + 1)) AS ip_id_zero_ratio,
|
||||
(a.hits / (a.unique_conn_id + 1)) AS multiplexing_efficiency,
|
||||
IF(a.mss_1460_count > (a.hits * 0.8) AND h.modern_browser_score > 70, 1, 0) AS mss_mobile_mismatch,
|
||||
a.request_size_variance AS request_size_variance,
|
||||
IF(a.tls_alpn = 'h2' AND a.http_version != '2', 1, 0) AS alpn_http_mismatch,
|
||||
IF(length(a.tls_alpn) = 0 OR a.tls_alpn = '00', 1, 0) AS is_alpn_missing,
|
||||
IF(length(a.tls_sni) > 0 AND a.tls_sni != a.host, 1, 0) AS sni_host_mismatch,
|
||||
IF(h.sec_fetch_mode = 'navigate' AND h.sec_fetch_dest != 'document', 1, 0) AS is_fake_navigation,
|
||||
count() OVER (PARTITION BY a.tcp_fingerprint) AS tcp_shared_count,
|
||||
count() OVER (PARTITION BY h.header_order_hash) AS header_order_shared_count,
|
||||
(a.count_assets / (a.hits + 1)) AS asset_ratio,
|
||||
(a.count_no_referer / (a.hits + 1)) AS direct_access_ratio,
|
||||
IF(a.unique_ua > 2, 1, 0) AS is_ua_rotating,
|
||||
uniqExact(a.ja4) OVER (PARTITION BY a.src_ip) AS distinct_ja4_count,
|
||||
((a.hits / (a.unique_src_ports + 1)) / (dateDiff('second', a.first_seen, a.last_seen) + 1)) AS src_port_density,
|
||||
(sum(a.hits) OVER (PARTITION BY a.ja4, a.src_asn) / (sum(a.hits) OVER (PARTITION BY a.ja4) + 1)) AS ja4_asn_concentration,
|
||||
(sum(a.hits) OVER (PARTITION BY a.ja4, a.src_country_code) / (sum(a.hits) OVER (PARTITION BY a.ja4) + 1)) AS ja4_country_concentration,
|
||||
IF(sum(a.hits) OVER (PARTITION BY a.ja4) < 100, 1, 0) AS is_rare_ja4,
|
||||
(count() OVER (PARTITION BY h.header_order_hash, a.first_ua) / (count() OVER (PARTITION BY a.first_ua) + 1)) AS header_order_confidence,
|
||||
uniqExact(h.header_order_hash) OVER (PARTITION BY a.src_ip) AS distinct_header_orders,
|
||||
(a.uniq_paths / (a.hits + 1)) AS path_diversity_ratio,
|
||||
a.url_depth_variance AS url_depth_variance,
|
||||
(a.count_anomalous_payload / (a.hits + 1)) AS anomalous_payload_ratio,
|
||||
a.uniq_ja3_val AS uniq_ja3_per_row,
|
||||
sqrt(a.tcp_jitter_variance) / greatest(a.avg_syn_ms_val, 1) AS syn_timing_cv,
|
||||
a.tls12_count / (a.hits + 1) AS tls12_ratio,
|
||||
a.count_head / (a.hits + 1) AS head_ratio,
|
||||
a.count_no_sec_fetch / (a.hits + 1) AS sec_fetch_absence_rate,
|
||||
a.count_generic_accept / (a.hits + 1) AS generic_accept_ratio,
|
||||
a.count_http10 / (a.hits + 1) AS http10_ratio,
|
||||
a.ip_df_variance AS ip_df_variance,
|
||||
-- Nouvelles features TTL (fingerprint OS, L4 → modèle Complet)
|
||||
a.avg_ttl_val AS avg_ttl,
|
||||
sqrt(a.ttl_variance_val) AS ttl_std,
|
||||
IF(a.count_correlated_val > 0, a.count_no_wscale_val / a.count_correlated_val, 0) AS no_window_scale_ratio,
|
||||
-- Nouvelles features HTTP (disponibles pour les deux modèles)
|
||||
a.count_no_accept_enc_val / (a.hits + 1) AS missing_accept_enc_ratio,
|
||||
a.count_http_scheme_val / (a.hits + 1) AS http_scheme_ratio
|
||||
FROM (
|
||||
SELECT
|
||||
window_start, src_ip, ja4, host, src_asn,
|
||||
any(src_country_code) AS src_country_code, any(src_as_name) AS src_as_name,
|
||||
any(src_org) AS src_org, any(src_domain) AS src_domain, any(first_ua) AS first_ua,
|
||||
sum(hits) AS hits, uniqMerge(uniq_paths) AS uniq_paths,
|
||||
uniqMerge(uniq_query_params) AS uniq_query_params, sum(count_post) AS count_post,
|
||||
min(first_seen) AS first_seen, max(last_seen) AS last_seen,
|
||||
any(tcp_fp_raw) AS tcp_fingerprint, varPopMerge(tcp_jitter_variance) AS tcp_jitter_variance,
|
||||
varPopMerge(total_ip_length_var) AS request_size_variance,
|
||||
any(tcp_win_raw * exp2(tcp_scale_raw)) AS true_window_size,
|
||||
IF(any(tcp_mss_raw) > 0, any(tcp_win_raw) / any(tcp_mss_raw), 0) AS window_mss_ratio,
|
||||
any(http_ver_raw) AS http_version, any(tls_alpn_raw) AS tls_alpn, any(tls_sni_raw) AS tls_sni,
|
||||
max(correlated_raw) AS correlated, uniqMerge(unique_src_ports) AS unique_src_ports,
|
||||
uniqMerge(unique_conn_id) AS unique_conn_id, max(max_keepalives) AS max_keepalives,
|
||||
sum(orphan_count) AS orphan_count, sum(ip_id_zero_count) AS ip_id_zero_count,
|
||||
sum(mss_1460_count) AS mss_1460_count,
|
||||
sum(count_assets) AS count_assets, sum(count_no_referer) AS count_no_referer,
|
||||
uniqMerge(uniq_ua) AS unique_ua,
|
||||
varPopMerge(url_depth_variance) AS url_depth_variance,
|
||||
sum(count_anomalous_payload) AS count_anomalous_payload,
|
||||
uniqMerge(uniq_ja3) AS uniq_ja3_val,
|
||||
avgMerge(avg_syn_ms) AS avg_syn_ms_val,
|
||||
sum(tls12_count) AS tls12_count,
|
||||
sum(count_head) AS count_head,
|
||||
sum(count_no_sec_fetch) AS count_no_sec_fetch,
|
||||
sum(count_generic_accept) AS count_generic_accept,
|
||||
sum(count_http10) AS count_http10,
|
||||
varPopMerge(ip_df_var) AS ip_df_variance,
|
||||
-- Nouvelles features : TTL fingerprint (L4) + HTTP
|
||||
avgIfMerge(avg_ttl) AS avg_ttl_val,
|
||||
varPopIfMerge(ttl_var) AS ttl_variance_val,
|
||||
sum(count_no_wscale) AS count_no_wscale_val,
|
||||
sum(count_correlated) AS count_correlated_val,
|
||||
sum(count_no_accept_enc) AS count_no_accept_enc_val,
|
||||
sum(count_http_scheme) AS count_http_scheme_val
|
||||
FROM mabase_prod.agg_host_ip_ja4_1h
|
||||
WHERE window_start >= now() - INTERVAL 24 HOUR
|
||||
GROUP BY window_start, src_ip, ja4, host, src_asn
|
||||
) a
|
||||
LEFT JOIN (
|
||||
SELECT
|
||||
window_start, src_ip, any(header_order_hash) AS header_order_hash,
|
||||
max(header_count) AS header_count, max(has_accept_language) AS has_accept_language,
|
||||
max(has_cookie) AS has_cookie, max(has_referer) AS has_referer,
|
||||
max(modern_browser_score) AS modern_browser_score, max(ua_ch_mismatch) AS ua_ch_mismatch,
|
||||
any(sec_fetch_mode) AS sec_fetch_mode, any(sec_fetch_dest) AS sec_fetch_dest
|
||||
FROM mabase_prod.agg_header_fingerprint_1h
|
||||
WHERE window_start >= now() - INTERVAL 24 HOUR
|
||||
GROUP BY window_start, src_ip
|
||||
) h ON a.src_ip = h.src_ip AND a.window_start = h.window_start
|
||||
)
|
||||
SELECT
|
||||
*,
|
||||
-(sum((hits / (total_ip_hits + 1)) * log2((hits / (total_ip_hits + 1)) + 0.000001)) OVER (PARTITION BY src_ip)) AS temporal_entropy,
|
||||
sum(uniq_ja3_per_row) OVER (PARTITION BY src_ip) / greatest(distinct_ja4_count, 1) AS ja3_diversity_ratio
|
||||
FROM base_data;
|
||||
15
services/bot-detector/bot_detector/Dockerfile
Normal file
15
services/bot-detector/bot_detector/Dockerfile
Normal file
@ -0,0 +1,15 @@
|
||||
FROM python:3.11-slim
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
WORKDIR /app
|
||||
|
||||
# Install shared package first
|
||||
COPY shared/python/ja4_common/ /app/shared/ja4_common/
|
||||
RUN pip install --no-cache-dir /app/shared/ja4_common/
|
||||
|
||||
COPY services/bot-detector/bot_detector/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY services/bot-detector/bot_detector/bot_detector.py .
|
||||
|
||||
CMD ["python", "bot_detector.py"]
|
||||
10
services/bot-detector/bot_detector/Dockerfile.tests
Normal file
10
services/bot-detector/bot_detector/Dockerfile.tests
Normal file
@ -0,0 +1,10 @@
|
||||
FROM python:3.11-slim
|
||||
WORKDIR /app
|
||||
COPY shared/python/ja4_common/ /app/shared/ja4_common/
|
||||
RUN pip install --no-cache-dir /app/shared/ja4_common/
|
||||
COPY services/bot-detector/bot_detector/requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN pip install --no-cache-dir pytest pytest-mock
|
||||
COPY services/bot-detector/bot_detector/ /app/bot_detector/
|
||||
WORKDIR /app
|
||||
CMD ["pytest", "bot_detector/tests/", "-v"]
|
||||
906
services/bot-detector/bot_detector/bot_detector.py
Normal file
906
services/bot-detector/bot_detector/bot_detector.py
Normal file
@ -0,0 +1,906 @@
|
||||
import time
|
||||
import os
|
||||
import json
|
||||
import glob
|
||||
import signal
|
||||
import sys
|
||||
import logging
|
||||
import threading
|
||||
import joblib
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import clickhouse_connect
|
||||
from logging.handlers import RotatingFileHandler
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
from sklearn.ensemble import IsolationForest
|
||||
from sklearn.cluster import DBSCAN
|
||||
from sklearn.preprocessing import StandardScaler
|
||||
import warnings
|
||||
from datetime import datetime
|
||||
|
||||
try:
|
||||
import shap as _shap
|
||||
SHAP_AVAILABLE = True
|
||||
except ImportError:
|
||||
SHAP_AVAILABLE = False
|
||||
|
||||
warnings.filterwarnings('ignore')
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# CONFIGURATION
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _require_float(name, default, lo=None, hi=None):
|
||||
raw = os.getenv(name, str(default))
|
||||
try:
|
||||
v = float(raw)
|
||||
except ValueError:
|
||||
raise SystemExit(f"[CONFIG] {name}={raw!r} invalide — doit être un nombre décimal.")
|
||||
if lo is not None and not (lo < v < hi):
|
||||
raise SystemExit(f"[CONFIG] {name}={v} hors plage ({lo} < valeur < {hi}).")
|
||||
return v
|
||||
|
||||
# Nom de la base de données ClickHouse
|
||||
# Note : Utilisé dans des requêtes SQL via f-string (ex: f'SELECT * FROM {DB}.view_ai_features_1h')
|
||||
# Cette variable provient uniquement de variables d'environnement contrôlées (docker-compose, K8s, etc.)
|
||||
# et n'est jamais exposée à des entrées utilisateur. Le risque d'injection SQL est considéré comme négligeable.
|
||||
DB = os.getenv('CLICKHOUSE_DB', 'mabase_prod')
|
||||
|
||||
CONTAMINATION = _require_float('ISOLATION_CONTAMINATION', 0.001, 0, 0.5)
|
||||
ANOMALY_THRESHOLD = _require_float('ANOMALY_THRESHOLD', -0.05)
|
||||
LOG_FILE = os.getenv('BOT_DETECTOR_LOG', '/var/log/bot_detector/decisions.jsonl')
|
||||
LOG_BACKUP_COUNT = int(os.getenv('LOG_BACKUP_COUNT', '7'))
|
||||
MODEL_DIR = os.getenv('MODEL_DIR', '/var/lib/bot_detector')
|
||||
RETRAIN_INTERVAL_H = int(os.getenv('RETRAIN_INTERVAL_HOURS', '24'))
|
||||
MODEL_HISTORY_COUNT = int(os.getenv('MODEL_HISTORY_COUNT', '10'))
|
||||
MAX_FAILURES = int(os.getenv('MAX_CONSECUTIVE_FAILURES', '3'))
|
||||
HEALTH_PORT = int(os.getenv('HEALTH_PORT', '8080'))
|
||||
CYCLE_INTERVAL = int(os.getenv('CYCLE_INTERVAL_SEC', '300'))
|
||||
|
||||
# ── Améliorations A1 / A2 / A3 / A4 / A5 / A6 / A7 / A8 / A10 ──────────────
|
||||
# A1 — Dérive conceptuelle (concept drift)
|
||||
DRIFT_THRESHOLD = _require_float('DRIFT_THRESHOLD', 0.30, 0, 1)
|
||||
|
||||
# A2 — Seuil adaptatif
|
||||
ANOMALY_PERCENTILE = int(os.getenv('ANOMALY_PERCENTILE', '5'))
|
||||
|
||||
# A3 — Analyse multi-fenêtres
|
||||
ENABLE_MULTIWINDOW = os.getenv('ENABLE_MULTIWINDOW', 'false').lower() == 'true'
|
||||
MULTIWINDOW_VIEW = os.getenv('MULTIWINDOW_VIEW', 'view_ai_features_24h')
|
||||
|
||||
# A4 — Explainabilité SHAP
|
||||
ENABLE_SHAP = SHAP_AVAILABLE and os.getenv('ENABLE_SHAP', 'true').lower() == 'true'
|
||||
|
||||
# A5 — Déduplication inter-cycles avec TTL
|
||||
DEDUP_TTL_MIN = int(os.getenv('DEDUP_TTL_MIN', '60'))
|
||||
|
||||
# A6 — Pondération par récurrence
|
||||
RECURRENCE_WEIGHT = _require_float('RECURRENCE_WEIGHT', 0.005)
|
||||
|
||||
# A7 — Validation de complétude des features
|
||||
MIN_VALID_FEATURE_RATIO = _require_float('MIN_VALID_FEATURE_RATIO', 0.50, 0, 1)
|
||||
|
||||
# A8 — Clustering comportemental des anomalies
|
||||
ENABLE_CLUSTERING = os.getenv('ENABLE_CLUSTERING', 'true').lower() == 'true'
|
||||
CLUSTERING_MIN_SAMPLES = int(os.getenv('CLUSTERING_MIN_SAMPLES', '3'))
|
||||
|
||||
# Features structurellement indisponibles par modèle (pas de données L4 pour trafic non-corrélé)
|
||||
# Ces features ne génèrent pas de warnings "pipeline" — leur absence est by-design.
|
||||
STRUCTURAL_EXCLUDED_FEATURES: dict[str, list] = {
|
||||
'Complet': ['orphan_ratio'],
|
||||
'Applicatif': ['orphan_ratio', 'is_rare_ja4', 'tcp_shared_count',
|
||||
'request_size_variance', 'mss_mobile_mismatch',
|
||||
# B features TLS/TCP : indisponibles pour trafic non-corrélé
|
||||
'ja3_diversity_ratio', 'syn_timing_cv', 'tls12_ratio', 'ip_df_variance',
|
||||
# L4 uniquement : TTL et window scale indisponibles sans capture TCP
|
||||
'avg_ttl', 'ttl_std', 'no_window_scale_ratio'],
|
||||
}
|
||||
|
||||
TRAINING_HISTORY_FILE = os.path.join(MODEL_DIR, 'training_history.jsonl')
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# LOGGING
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
|
||||
os.makedirs(MODEL_DIR, exist_ok=True)
|
||||
|
||||
logger = logging.getLogger('bot_detector')
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
_console_handler = logging.StreamHandler()
|
||||
_console_handler.setFormatter(logging.Formatter('[%(asctime)s] %(message)s', '%Y-%m-%d %H:%M:%S'))
|
||||
logger.addHandler(_console_handler)
|
||||
|
||||
_file_handler = RotatingFileHandler(
|
||||
LOG_FILE, maxBytes=50 * 1024 * 1024, backupCount=LOG_BACKUP_COUNT, encoding='utf-8'
|
||||
)
|
||||
_file_handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
logger.addHandler(_file_handler)
|
||||
|
||||
# Wrapper court pour homogénéiser les appels de logging (évite d'importer logger partout).
|
||||
def log_info(message: str):
|
||||
logger.info(message)
|
||||
|
||||
def log_decision(event: str, cycle_id: str, model: str = '', row: dict = None):
|
||||
entry = {
|
||||
'ts': datetime.now().strftime('%Y-%m-%dT%H:%M:%S'),
|
||||
'cycle_id': cycle_id,
|
||||
'event': event,
|
||||
'model': model,
|
||||
'contamination': CONTAMINATION,
|
||||
'threshold': ANOMALY_THRESHOLD,
|
||||
}
|
||||
if row:
|
||||
entry.update(row)
|
||||
_file_handler.stream.write(json.dumps(entry, ensure_ascii=False, default=str) + '\n')
|
||||
_file_handler.stream.flush()
|
||||
|
||||
def _append_training_history(entry: dict):
|
||||
with open(TRAINING_HISTORY_FILE, 'a', encoding='utf-8') as f:
|
||||
f.write(json.dumps(entry, ensure_ascii=False, default=str) + '\n')
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# ARRÊT PROPRE ET HEALTH CHECK
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _shutdown(sig, frame):
|
||||
log_info(f"Signal {sig} reçu — arrêt propre.")
|
||||
log_decision('SERVICE_STOP', 'shutdown', '', {'signal': sig})
|
||||
sys.exit(0)
|
||||
|
||||
signal.signal(signal.SIGTERM, _shutdown)
|
||||
signal.signal(signal.SIGINT, _shutdown)
|
||||
|
||||
_service_healthy = True
|
||||
class _HealthHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
code = 200 if _service_healthy else 503
|
||||
self.send_response(code)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'OK' if _service_healthy else b'DEGRADED')
|
||||
def log_message(self, *args): pass
|
||||
|
||||
threading.Thread(
|
||||
target=lambda: HTTPServer(('', HEALTH_PORT), _HealthHandler).serve_forever(),
|
||||
daemon=True
|
||||
).start()
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# CONNEXION CLICKHOUSE — delegated to ja4_common shared client
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
from ja4_common.clickhouse import get_client as _ja4_get_client
|
||||
|
||||
def get_client():
|
||||
"""Return the shared ja4_common ClickHouse client, reconnecting on ping failure."""
|
||||
return _ja4_get_client().connect()
|
||||
|
||||
def score_to_threat_level(score: float) -> str:
|
||||
# Seuils : CRITICAL < -0.30 | HIGH < -0.15 | MEDIUM < -0.05 | LOW < 0 | NORMAL ≥ 0
|
||||
if score < -0.30: return 'CRITICAL'
|
||||
if score < -0.15: return 'HIGH'
|
||||
if score < -0.05: return 'MEDIUM'
|
||||
if score < 0: return 'LOW'
|
||||
return 'NORMAL'
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# GESTION DES MODÈLES
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _current_pointer_path(name: str) -> str:
|
||||
return os.path.join(MODEL_DIR, f'model_{name}.current')
|
||||
|
||||
def _get_current_version(name: str):
|
||||
pointer = _current_pointer_path(name)
|
||||
if not os.path.exists(pointer): return None, None
|
||||
with open(pointer) as f: version_id = f.read().strip()
|
||||
model_path = os.path.join(MODEL_DIR, f'model_{name}_{version_id}.joblib')
|
||||
meta_path = os.path.join(MODEL_DIR, f'model_{name}_{version_id}.meta.json')
|
||||
if not os.path.exists(model_path) or not os.path.exists(meta_path): return None, None
|
||||
with open(meta_path) as f: meta = json.load(f)
|
||||
return model_path, meta
|
||||
|
||||
def _purge_old_versions(name: str):
|
||||
pattern = os.path.join(MODEL_DIR, f'model_{name}_*.joblib')
|
||||
versions = sorted(glob.glob(pattern))
|
||||
to_delete = versions[:-MODEL_HISTORY_COUNT] if len(versions) > MODEL_HISTORY_COUNT else []
|
||||
for joblib_path in to_delete:
|
||||
version_id = os.path.basename(joblib_path).replace(f'model_{name}_', '').replace('.joblib', '')
|
||||
meta_path = os.path.join(MODEL_DIR, f'model_{name}_{version_id}.meta.json')
|
||||
os.remove(joblib_path)
|
||||
if os.path.exists(meta_path): os.remove(meta_path)
|
||||
log_info(f"[{name}] Version purgée : {version_id} (limite={MODEL_HISTORY_COUNT})")
|
||||
|
||||
def load_or_train_model(name: str, human_baseline: pd.DataFrame, features: list, cycle_id: str):
|
||||
model_path, meta = _get_current_version(name)
|
||||
if model_path and meta:
|
||||
trained_at = datetime.fromisoformat(meta['trained_at'])
|
||||
age_h = (datetime.now() - trained_at).total_seconds() / 3600
|
||||
age_ok = age_h < RETRAIN_INTERVAL_H
|
||||
|
||||
# A1 — Dérive conceptuelle : comparer la distribution actuelle avec celle de l'entraînement
|
||||
drift_score = 0.0
|
||||
drift_forced = False
|
||||
if age_ok and 'baseline_stats' in meta:
|
||||
drift_score = _compute_drift_score(meta['baseline_stats'], human_baseline, features)
|
||||
if drift_score >= DRIFT_THRESHOLD:
|
||||
drift_forced = True
|
||||
log_info(f"[{name}] Dérive détectée ({drift_score:.0%} features) — retraining forcé.")
|
||||
log_decision('DRIFT_DETECTED', cycle_id, name, {
|
||||
'version_id': meta['version_id'], 'drift_score': round(drift_score, 3),
|
||||
'drift_threshold': DRIFT_THRESHOLD, 'model_age_hours': round(age_h, 2)
|
||||
})
|
||||
|
||||
if age_ok and not drift_forced:
|
||||
log_info(f"[{name}] Modèle v{meta['version_id']} valide ({age_h:.1f}h / {RETRAIN_INTERVAL_H}h, drift={drift_score:.0%}) — réutilisation.")
|
||||
log_decision('MODEL_LOADED', cycle_id, name, {
|
||||
'version_id': meta['version_id'], 'model_age_hours': round(age_h, 2),
|
||||
'trained_at': meta['trained_at'], 'human_samples': meta.get('human_samples', '?'),
|
||||
'retrain_in_hours': round(RETRAIN_INTERVAL_H - age_h, 1), 'drift_score': round(drift_score, 3)
|
||||
})
|
||||
return joblib.load(model_path)
|
||||
elif not drift_forced:
|
||||
log_info(f"[{name}] Modèle v{meta['version_id']} expiré ({age_h:.1f}h ≥ {RETRAIN_INTERVAL_H}h) — retraining.")
|
||||
|
||||
version_id = datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
log_info(f"[{name}] Entraînement version {version_id} sur {len(human_baseline)} sessions humaines... (contamination={CONTAMINATION})")
|
||||
|
||||
X = human_baseline[features].replace([np.inf, -np.inf], np.nan).fillna(0)
|
||||
model = IsolationForest(n_estimators=300, contamination=CONTAMINATION, random_state=42, n_jobs=-1)
|
||||
model.fit(X)
|
||||
|
||||
# A1 — Sauvegarder les statistiques de distribution de la baseline pour la détection de dérive future
|
||||
baseline_stats = {
|
||||
f: {'mean': float(X[f].mean()), 'std': float(X[f].std()), 'p25': float(X[f].quantile(0.25)), 'p75': float(X[f].quantile(0.75))}
|
||||
for f in features
|
||||
}
|
||||
|
||||
new_model_path = os.path.join(MODEL_DIR, f'model_{name}_{version_id}.joblib')
|
||||
new_meta_path = os.path.join(MODEL_DIR, f'model_{name}_{version_id}.meta.json')
|
||||
joblib.dump(model, new_model_path)
|
||||
|
||||
previous_version = meta.get('version_id', None) if meta else None
|
||||
new_meta = {
|
||||
'version_id': version_id, 'trained_at': datetime.now().isoformat(),
|
||||
'human_samples': len(human_baseline), 'contamination': CONTAMINATION,
|
||||
'threshold': ANOMALY_THRESHOLD, 'features': features,
|
||||
'model_name': name, 'previous_version': previous_version,
|
||||
'retrain_interval': RETRAIN_INTERVAL_H, 'baseline_stats': baseline_stats
|
||||
}
|
||||
with open(new_meta_path, 'w') as f: json.dump(new_meta, f, indent=2)
|
||||
with open(_current_pointer_path(name), 'w') as f: f.write(version_id)
|
||||
|
||||
_append_training_history({k: v for k, v in new_meta.items() if k != 'baseline_stats'})
|
||||
_purge_old_versions(name)
|
||||
|
||||
log_info(f"[{name}] Modèle v{version_id} sauvegardé → {new_model_path}")
|
||||
log_decision('MODEL_TRAINED', cycle_id, name, {
|
||||
'version_id': version_id, 'previous_version': previous_version,
|
||||
'human_samples': len(human_baseline), 'next_retrain_in_h': RETRAIN_INTERVAL_H,
|
||||
'history_kept': MODEL_HISTORY_COUNT
|
||||
})
|
||||
return model
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A1 — DÉTECTION DE DÉRIVE CONCEPTUELLE (CONCEPT DRIFT)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _compute_drift_score(baseline_stats: dict, current_baseline: pd.DataFrame, features: list) -> float:
|
||||
"""
|
||||
Compare la distribution actuelle de la baseline humaine avec celle utilisée à l'entraînement.
|
||||
Utilise un test de Kolmogorov-Smirnov par feature. Retourne la fraction de features déroutantes.
|
||||
Une valeur >= DRIFT_THRESHOLD déclenche un retraining forcé.
|
||||
"""
|
||||
if not baseline_stats or current_baseline.empty:
|
||||
return 0.0
|
||||
drifted = 0
|
||||
tested = 0
|
||||
for feat in features:
|
||||
if feat not in baseline_stats or feat not in current_baseline.columns:
|
||||
continue
|
||||
stats = baseline_stats[feat]
|
||||
curr_mean = current_baseline[feat].mean()
|
||||
trained_std = stats.get('std', 0)
|
||||
if trained_std < 1e-9:
|
||||
continue
|
||||
# Z-score : écart entre la moyenne actuelle et celle de l'entraînement
|
||||
z = abs(curr_mean - stats['mean']) / trained_std
|
||||
# Un z > 2 indique une dérive significative de la distribution
|
||||
if z > 2.0:
|
||||
drifted += 1
|
||||
tested += 1
|
||||
return drifted / max(tested, 1)
|
||||
|
||||
|
||||
# Cache par modèle conservant le dernier état des features invalides.
|
||||
# Permet de supprimer les logs répétitifs : on ne loggue que si l'état a changé depuis le cycle précédent.
|
||||
_feature_warning_cache: dict = {}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A7 — VALIDATION DE COMPLÉTUDE DES FEATURES
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def validate_features(df: pd.DataFrame, features: list, name: str, cycle_id: str):
|
||||
"""
|
||||
Vérifie que les features sont présentes et non constantes dans le DataFrame.
|
||||
Catégorise les features invalides :
|
||||
- structural : absente par design pour ce modèle (défini dans STRUCTURAL_EXCLUDED_FEATURES)
|
||||
- zero : colonne toujours à 0 — problème de pipeline
|
||||
- unique : colonne avec une seule valeur non-nulle — agrégat global non discriminant
|
||||
- missing : colonne absente du DataFrame
|
||||
Retourne la liste des features valides, ou None si trop de features sont invalides.
|
||||
Les avertissements ne sont logués que si l'état a changé depuis le cycle précédent
|
||||
(grâce à _feature_warning_cache), pour éviter de polluer les logs à chaque cycle.
|
||||
"""
|
||||
structural = STRUCTURAL_EXCLUDED_FEATURES.get(name, [])
|
||||
# Exclure les features structurelles d'emblée (sans warning pipeline)
|
||||
active_features = [f for f in features if f not in structural]
|
||||
|
||||
missing = [f for f in active_features if f not in df.columns]
|
||||
present = [f for f in active_features if f in df.columns]
|
||||
|
||||
zero_val = [f for f in present if df[f].nunique() == 1 and df[f].max() == 0]
|
||||
unique_val = [f for f in present if df[f].nunique() == 1 and df[f].max() != 0]
|
||||
constant = zero_val + unique_val
|
||||
valid = [f for f in present if f not in constant]
|
||||
|
||||
current_state = (frozenset(missing), frozenset(zero_val), frozenset(unique_val))
|
||||
state_changed = _feature_warning_cache.get(name) != current_state
|
||||
_feature_warning_cache[name] = current_state
|
||||
|
||||
if structural:
|
||||
log_info(f"[{name}] Features exclues (structurelles / L4 indisponible) : {structural}")
|
||||
# Ne logguer les avertissements que si l'état a changé (nouveau problème ou résolution)
|
||||
if state_changed:
|
||||
if missing:
|
||||
log_info(f"[{name}] Features absentes du schéma : {missing}")
|
||||
if zero_val:
|
||||
log_info(f"[{name}] Features à 0 (pipeline non-alimenté) : {zero_val}")
|
||||
if unique_val:
|
||||
log_info(f"[{name}] Features non-discriminantes (agrégat global) : {unique_val}")
|
||||
if missing or zero_val or unique_val:
|
||||
log_decision('FEATURE_WARNING', cycle_id, name, {
|
||||
'structural': structural, 'missing': missing,
|
||||
'zero': zero_val, 'unique_nonzero': unique_val,
|
||||
'valid_count': len(valid), 'total': len(active_features)
|
||||
})
|
||||
|
||||
ratio = len(valid) / max(len(active_features), 1)
|
||||
if ratio < MIN_VALID_FEATURE_RATIO:
|
||||
log_info(f"[{name}] Ratio features valides insuffisant ({ratio:.0%} < {MIN_VALID_FEATURE_RATIO:.0%}) — cycle ignoré.")
|
||||
log_decision('SKIPPED_INVALID_FEATURES', cycle_id, name, {
|
||||
'valid_ratio': round(ratio, 3), 'threshold': MIN_VALID_FEATURE_RATIO
|
||||
})
|
||||
return None
|
||||
return valid
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A2 / A10 — SEUIL ADAPTATIF ET NORMALISATION DES SCORES
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def compute_adaptive_threshold(scores: np.ndarray) -> float:
|
||||
"""
|
||||
A2 : Calcule un seuil adaptatif basé sur le percentile ANOMALY_PERCENTILE des scores négatifs.
|
||||
Retourne le min entre le seuil adaptatif et le seuil statique configuré.
|
||||
"""
|
||||
neg_scores = scores[scores < 0]
|
||||
if len(neg_scores) == 0:
|
||||
return ANOMALY_THRESHOLD
|
||||
adaptive = float(np.percentile(neg_scores, ANOMALY_PERCENTILE))
|
||||
return min(adaptive, ANOMALY_THRESHOLD)
|
||||
|
||||
|
||||
def normalize_scores(scores: np.ndarray) -> np.ndarray:
|
||||
"""
|
||||
A10 : Normalise les scores négatifs en [−1, 0] pour comparer des modèles différents.
|
||||
Les scores positifs (trafic normal) restent inchangés.
|
||||
|
||||
Attention : la formule mappe le score le PLUS négatif (plus anomaleux) vers 0
|
||||
et le score le MOINS négatif (moins anomaleux) vers −1.
|
||||
Ce résultat counter-intuitif est intentionnel : anomaly_score n'est utilisé qu'à titre
|
||||
indicatif dans les tables de résultats. Les décisions réelles s'appuient sur raw_anomaly_score.
|
||||
"""
|
||||
result = scores.copy()
|
||||
mask = scores < 0
|
||||
if mask.sum() == 0:
|
||||
return result
|
||||
s_min, s_max = scores[mask].min(), scores[mask].max()
|
||||
if s_min == s_max:
|
||||
return result
|
||||
result[mask] = (scores[mask] - s_min) / (s_max - s_min + 1e-9) * -1
|
||||
return result
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A4 — EXPLAINABILITÉ PAR SHAP
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _compute_shap_top_features(model, X: pd.DataFrame, features: list, n_top: int = 5) -> list:
|
||||
"""
|
||||
Calcule les valeurs SHAP pour chaque ligne de X et retourne les n_top features
|
||||
les plus contributives (valeur SHAP la plus négative = plus responsable de l'anomalie).
|
||||
Retourne une liste de dicts {feature: shap_value} par ligne.
|
||||
"""
|
||||
if not ENABLE_SHAP or X.empty:
|
||||
return [{}] * len(X)
|
||||
try:
|
||||
explainer = _shap.TreeExplainer(model)
|
||||
shap_values = explainer.shap_values(X)
|
||||
result = []
|
||||
for sv in shap_values:
|
||||
# Features les plus négatives = les plus responsables de l'anomalie
|
||||
pairs = sorted(zip(features, sv), key=lambda x: x[1])
|
||||
result.append({f: round(float(v), 4) for f, v in pairs[:n_top]})
|
||||
return result
|
||||
except Exception as e:
|
||||
log_info(f"[SHAP] Erreur de calcul SHAP: {e}")
|
||||
return [{}] * len(X)
|
||||
|
||||
|
||||
def _build_reason(name: str, row: pd.Series, shap_top: dict) -> str:
|
||||
"""Construit le champ reason enrichi avec le top SHAP ou les métriques clés."""
|
||||
# Utilise le score brut pour l'affichage (plus interprétable que le score normalisé)
|
||||
score = round(float(row.get('raw_anomaly_score', row.get('anomaly_score', 0))), 3)
|
||||
threat = row.get('threat_level', '')
|
||||
if shap_top:
|
||||
top_str = ' | '.join(f"{f}({v:+.3f})" for f, v in shap_top.items())
|
||||
return f"[{name}] Score: {score} | SHAP: {top_str} | Threat: {threat}"
|
||||
vel = round(float(row.get('hit_velocity', 0)), 1)
|
||||
fuzz = round(float(row.get('fuzzing_index', 0)), 1)
|
||||
return f"[{name}] Score: {score} | Vel: {vel} req/s | Fuzzing: {fuzz} | Threat: {threat}"
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A8 — CLUSTERING COMPORTEMENTAL DES ANOMALIES (DBSCAN)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _cluster_anomalies(anomalies: pd.DataFrame, features: list) -> pd.DataFrame:
|
||||
"""
|
||||
A8 : Applique DBSCAN sur les features normalisées des anomalies.
|
||||
Ajoute une colonne campaign_id : −1 = IP isolée, ≥0 = identifiant de campagne coordonnée.
|
||||
"""
|
||||
anomalies = anomalies.copy()
|
||||
if len(anomalies) < CLUSTERING_MIN_SAMPLES:
|
||||
anomalies['campaign_id'] = -1
|
||||
return anomalies
|
||||
try:
|
||||
X = anomalies[features].replace([np.inf, -np.inf], np.nan).fillna(0)
|
||||
X_scaled = StandardScaler().fit_transform(X)
|
||||
labels = DBSCAN(eps=0.5, min_samples=CLUSTERING_MIN_SAMPLES).fit_predict(X_scaled)
|
||||
anomalies['campaign_id'] = labels
|
||||
n_campaigns = len(set(labels)) - (1 if -1 in labels else 0)
|
||||
if n_campaigns > 0:
|
||||
log_info(f"[DBSCAN] {n_campaigns} campagne(s) détectée(s) parmi {len(anomalies)} anomalies.")
|
||||
except Exception as e:
|
||||
log_info(f"[DBSCAN] Erreur de clustering: {e}")
|
||||
anomalies['campaign_id'] = -1
|
||||
return anomalies
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# ANALYSE SEMI-SUPERVISÉE
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def run_semi_supervised_logic(df, features, name, cycle_id, recurrence_map):
|
||||
# ── Trifurcation du trafic selon bot_name et Anubis ─────────────────────
|
||||
# 1. Bots connus (dict_bot_ip / dict_bot_ja4) → exclus du scoring IF
|
||||
known_bots = df[df['bot_name'] != ''].copy()
|
||||
rest = df[df['bot_name'] == ''].copy()
|
||||
|
||||
# 2. Bots Anubis ALLOW → bots légitimes, exclus du scoring IF
|
||||
anubis_allow = rest[rest['anubis_bot_action'] == 'ALLOW'].copy()
|
||||
|
||||
# 3. Tout le reste passe par l'IsolationForest pour un score réel :
|
||||
# - DENY : menaces identifiées par règles Anubis → IF donne le score de sévérité
|
||||
# - WEIGH / inconnu → scorés normalement (anubis_is_flagged=1 pour WEIGH)
|
||||
# Les DENY sont TOUJOURS inclus dans les threats, indépendamment du seuil IF.
|
||||
unknown_traffic = rest[rest['anubis_bot_action'] != 'ALLOW'].copy()
|
||||
human_baseline = unknown_traffic[unknown_traffic['asn_label'] == 'human']
|
||||
|
||||
# A7 — Valider les features avant tout traitement
|
||||
valid_features = validate_features(df, features, name, cycle_id)
|
||||
if valid_features is None:
|
||||
return pd.DataFrame(), pd.DataFrame()
|
||||
|
||||
if len(human_baseline) < 500:
|
||||
log_info(f"[{name}] Données humaines insuffisantes ({len(human_baseline)} < 500).")
|
||||
log_decision('SKIPPED_LOW_DATA', cycle_id, name, {
|
||||
'human_count': len(human_baseline), 'unknown_count': len(unknown_traffic)
|
||||
})
|
||||
return pd.DataFrame(), pd.DataFrame()
|
||||
|
||||
# A1 — Dérive conceptuelle intégrée dans load_or_train_model
|
||||
model = load_or_train_model(name, human_baseline, valid_features, cycle_id)
|
||||
unknown_traffic = unknown_traffic.copy()
|
||||
|
||||
X_test = unknown_traffic[valid_features].replace([np.inf, -np.inf], np.nan).fillna(0)
|
||||
raw_scores = model.decision_function(X_test)
|
||||
|
||||
# raw_anomaly_score : score brut IF pour comparaison au seuil et assignation du threat_level
|
||||
# anomaly_score : score normalisé [-1, 0] pour cohérence cross-modèles (A10)
|
||||
unknown_traffic['raw_anomaly_score'] = raw_scores
|
||||
unknown_traffic['anomaly_score'] = normalize_scores(raw_scores)
|
||||
unknown_traffic['model_name'] = name
|
||||
|
||||
# A2 — Seuil adaptatif calculé sur les scores BRUTS (même échelle que ANOMALY_THRESHOLD)
|
||||
effective_threshold = compute_adaptive_threshold(raw_scores)
|
||||
log_info(f"[{name}] Seuil effectif : {effective_threshold:.4f} (statique={ANOMALY_THRESHOLD}, percentile={ANOMALY_PERCENTILE})")
|
||||
|
||||
# A6 — Pénaliser les IPs récurrentes sur le score BRUT avant comparaison au seuil
|
||||
if RECURRENCE_WEIGHT > 0:
|
||||
recurrences = unknown_traffic['src_ip'].map(recurrence_map).fillna(0)
|
||||
penalty = np.log1p(recurrences.values) * RECURRENCE_WEIGHT
|
||||
unknown_traffic['raw_anomaly_score'] = unknown_traffic['raw_anomaly_score'] - penalty
|
||||
|
||||
# Assigner threat_level à TOUTES les sessions scorées (pour ml_all_scores)
|
||||
unknown_traffic['threat_level'] = unknown_traffic['raw_anomaly_score'].apply(score_to_threat_level)
|
||||
unknown_traffic['recurrence'] = unknown_traffic['src_ip'].map(recurrence_map).fillna(0).astype(int) + 1
|
||||
unknown_traffic['campaign_id'] = -1
|
||||
|
||||
# Extraire les DENY (maintenant avec leur vrai score IF) et forcer leur threat_level
|
||||
deny_mask = unknown_traffic['anubis_bot_action'] == 'DENY'
|
||||
unknown_traffic.loc[deny_mask, 'threat_level'] = 'ANUBIS_DENY'
|
||||
|
||||
# Capturer toutes les sessions scorées (avant filtrage par seuil) — pour ml_all_scores
|
||||
all_scored = unknown_traffic.copy()
|
||||
|
||||
if not known_bots.empty:
|
||||
known_bots = known_bots.copy()
|
||||
known_bots['anomaly_score'] = 0.0
|
||||
known_bots['raw_anomaly_score'] = 0.0
|
||||
known_bots['threat_level'] = 'KNOWN_BOT'
|
||||
known_bots['model_name'] = name
|
||||
known_bots['campaign_id'] = -1
|
||||
known_bots['reason'] = '[Identification] Bot légitime: ' + known_bots['bot_name']
|
||||
known_bots['recurrence'] = known_bots['src_ip'].map(recurrence_map).fillna(0).astype(int) + 1
|
||||
for _, row in known_bots.iterrows():
|
||||
log_decision('KNOWN_BOT', cycle_id, name, {
|
||||
'src_ip': row.get('src_ip', ''), 'bot_name': row.get('bot_name', ''),
|
||||
'asn_number': row.get('asn_number', ''), 'asn_org': row.get('asn_org', ''),
|
||||
'asn_domain': row.get('asn_domain', ''), 'country_code': row.get('country_code', ''),
|
||||
'recurrence': int(row.get('recurrence', 1))
|
||||
})
|
||||
|
||||
# ── Anubis ALLOW : bots légitimes identifiés par règles Anubis ───────────
|
||||
if not anubis_allow.empty:
|
||||
anubis_allow = anubis_allow.copy()
|
||||
anubis_allow['anomaly_score'] = 0.0
|
||||
anubis_allow['raw_anomaly_score'] = 0.0
|
||||
anubis_allow['threat_level'] = 'KNOWN_BOT'
|
||||
anubis_allow['model_name'] = name
|
||||
anubis_allow['campaign_id'] = -1
|
||||
anubis_allow['reason'] = '[Anubis ALLOW] ' + anubis_allow['anubis_bot_name']
|
||||
anubis_allow['recurrence'] = anubis_allow['src_ip'].map(recurrence_map).fillna(0).astype(int) + 1
|
||||
for _, row in anubis_allow.iterrows():
|
||||
log_decision('KNOWN_BOT', cycle_id, name, {
|
||||
'src_ip': row.get('src_ip', ''), 'bot_name': row.get('anubis_bot_name', ''),
|
||||
'anubis_bot_name': row.get('anubis_bot_name', ''),
|
||||
'anubis_bot_action': row.get('anubis_bot_action', ''),
|
||||
'anubis_bot_category': row.get('anubis_bot_category', ''),
|
||||
'asn_number': row.get('asn_number', ''), 'asn_org': row.get('asn_org', ''),
|
||||
'asn_domain': row.get('asn_domain', ''), 'country_code': row.get('country_code', ''),
|
||||
'recurrence': int(row.get('recurrence', 1)),
|
||||
})
|
||||
|
||||
# ── Anubis DENY : scorés par IF, toujours inclus dans les threats ────────
|
||||
# Extraits de unknown_traffic après scoring — ils ont leur vrai score IF.
|
||||
anubis_deny = unknown_traffic[deny_mask].copy()
|
||||
if not anubis_deny.empty:
|
||||
anubis_deny['reason'] = '[Anubis DENY] ' + anubis_deny['anubis_bot_name'].fillna('') + \
|
||||
' | ' + anubis_deny['raw_anomaly_score'].apply(lambda s: f'IF={s:.4f}')
|
||||
log_info(f"[{name}] Anubis DENY: {len(anubis_deny)} IP(s) scorées par IF "
|
||||
f"(score moyen: {anubis_deny['raw_anomaly_score'].mean():.4f}).")
|
||||
for _, row in anubis_deny.iterrows():
|
||||
log_decision('ANUBIS_DENY', cycle_id, name, {
|
||||
'src_ip': row.get('src_ip', ''), 'anubis_bot_name': row.get('anubis_bot_name', ''),
|
||||
'anubis_bot_action': row.get('anubis_bot_action', ''),
|
||||
'anubis_bot_category': row.get('anubis_bot_category', ''),
|
||||
'anomaly_score': round(float(row.get('anomaly_score', 0)), 4),
|
||||
'raw_anomaly_score': round(float(row.get('raw_anomaly_score', 0)), 4),
|
||||
'asn_number': row.get('asn_number', ''), 'asn_org': row.get('asn_org', ''),
|
||||
'asn_domain': row.get('asn_domain', ''), 'country_code': row.get('country_code', ''),
|
||||
'recurrence': int(row.get('recurrence', 1)),
|
||||
})
|
||||
|
||||
# Filtrer sur raw_anomaly_score (A6 inclus) — seulement le trafic non-DENY
|
||||
# Les DENY sont toujours des threats, indépendamment du seuil IF
|
||||
non_deny_traffic = unknown_traffic[~deny_mask]
|
||||
anomalies = non_deny_traffic[non_deny_traffic['raw_anomaly_score'] < effective_threshold].copy()
|
||||
if not anomalies.empty:
|
||||
log_info(f"[{name}] ALERT: {len(anomalies)} anomalies détectées (seuil={effective_threshold:.4f}).")
|
||||
anomalies['recurrence'] = anomalies['src_ip'].map(recurrence_map).fillna(0).astype(int) + 1
|
||||
|
||||
# A4 — Explainabilité SHAP : top features responsables de chaque anomalie
|
||||
X_anomalies = X_test.loc[anomalies.index]
|
||||
shap_tops = _compute_shap_top_features(model, X_anomalies, valid_features)
|
||||
anomalies['reason'] = [
|
||||
_build_reason(name, row, shap)
|
||||
for (_, row), shap in zip(anomalies.iterrows(), shap_tops)
|
||||
]
|
||||
|
||||
# A8 — Clustering DBSCAN pour identifier les campagnes coordonnées
|
||||
if ENABLE_CLUSTERING:
|
||||
anomalies = _cluster_anomalies(anomalies, valid_features)
|
||||
|
||||
anomalies['ja4'] = anomalies['ja4'].replace({'': 'HTTP_CLEAR_TEXT'})
|
||||
for _, row in anomalies.iterrows():
|
||||
log_decision('ANOMALY', cycle_id, name, {
|
||||
'src_ip': row.get('src_ip', ''), 'anomaly_score': round(float(row.get('anomaly_score', 0)), 4),
|
||||
'raw_anomaly_score': round(float(row.get('raw_anomaly_score', 0)), 4),
|
||||
'threat_level': row.get('threat_level', ''), 'recurrence': int(row.get('recurrence', 1)),
|
||||
'hit_velocity': round(float(row.get('hit_velocity', 0)), 2),
|
||||
'fuzzing_index': round(float(row.get('fuzzing_index', 0)), 2),
|
||||
'post_ratio': round(float(row.get('post_ratio', 0)), 3),
|
||||
'asn_number': row.get('asn_number', ''), 'asn_org': row.get('asn_org', ''),
|
||||
'asn_detail': row.get('asn_detail', ''), 'asn_domain': row.get('asn_domain', ''),
|
||||
'country_code': row.get('country_code', ''), 'asn_label': row.get('asn_label', ''),
|
||||
'ja4': row.get('ja4', ''), 'host': row.get('host', ''),
|
||||
'correlated': int(row.get('correlated', 0)), 'campaign_id': int(row.get('campaign_id', -1)),
|
||||
'effective_threshold': round(effective_threshold, 4), 'reason': row.get('reason', '')
|
||||
})
|
||||
|
||||
threats = pd.concat([df for df in [
|
||||
anomalies if not anomalies.empty else None,
|
||||
known_bots if not known_bots.empty else None,
|
||||
anubis_allow if not anubis_allow.empty else None,
|
||||
anubis_deny if not anubis_deny.empty else None,
|
||||
] if df is not None], ignore_index=True)
|
||||
|
||||
# Inclure anubis_allow dans all_scored pour traçabilité dans ml_all_scores.
|
||||
# Ces IPs sont exclues de l'analyse IF mais doivent apparaître dans la table
|
||||
# de scores avec threat_level='KNOWN_BOT' et anomaly_score=0.0.
|
||||
if not anubis_allow.empty:
|
||||
all_scored = pd.concat([all_scored, anubis_allow], ignore_index=True)
|
||||
|
||||
return threats, all_scored
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A5 — DÉDUPLICATION INTER-CYCLES AVEC TTL
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _filter_recent_detections(client, all_anom: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
A5 : Filtre les IPs déjà insérées dans ml_detected_anomalies dans les DEDUP_TTL_MIN dernières minutes.
|
||||
Exception : une IP est réinsérée si son nouveau score est ≥ 0.05 points plus bas (aggravation).
|
||||
"""
|
||||
if DEDUP_TTL_MIN <= 0 or all_anom.empty:
|
||||
return all_anom
|
||||
try:
|
||||
recent_df = client.query_df(
|
||||
f"SELECT src_ip, min(anomaly_score) AS best_score "
|
||||
f"FROM {DB}.ml_detected_anomalies "
|
||||
f"WHERE detected_at > now() - INTERVAL {DEDUP_TTL_MIN} MINUTE "
|
||||
f"GROUP BY src_ip"
|
||||
)
|
||||
if recent_df.empty:
|
||||
return all_anom
|
||||
recent_map = dict(zip(recent_df['src_ip'], recent_df['best_score']))
|
||||
def _should_insert(row):
|
||||
prev = recent_map.get(row['src_ip'])
|
||||
if prev is None:
|
||||
return True
|
||||
# Réinsérer seulement si le score brut s'est significativement aggravé
|
||||
return float(row.get('raw_anomaly_score', row['anomaly_score'])) < float(prev) - 0.05
|
||||
mask = all_anom.apply(_should_insert, axis=1)
|
||||
filtered = all_anom[mask]
|
||||
skipped = len(all_anom) - len(filtered)
|
||||
if skipped > 0:
|
||||
log_info(f"[Dedup TTL={DEDUP_TTL_MIN}min] {skipped} IP(s) filtrée(s) (déjà détectées récemment).")
|
||||
return filtered
|
||||
except Exception as e:
|
||||
log_info(f"[Dedup] Erreur lors de la déduplication TTL : {e}")
|
||||
return all_anom
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# A3 — ANALYSE MULTI-FENÊTRES : PRÉTRAITEMENT COMMUN
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
def _preprocess_df(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Normalise les colonnes et remplit les valeurs manquantes (commun 1h et 24h)."""
|
||||
df.columns = [c.split('.')[-1] for c in df.columns]
|
||||
for col in ['src_ip', 'ja4', 'host', 'bot_name', 'anubis_bot_name', 'anubis_bot_action', 'anubis_bot_category',
|
||||
'asn_number', 'asn_org', 'asn_detail', 'asn_domain', 'country_code', 'asn_label']:
|
||||
if col in df.columns:
|
||||
df[col] = df[col].fillna('').astype(str)
|
||||
df.fillna(0, inplace=True)
|
||||
|
||||
# ── Features numériques dérivées des labels Anubis (pour IsolationForest) ──
|
||||
# anubis_is_flagged : 1 si le trafic est marqué WEIGH/CHALLENGE par Anubis
|
||||
# → signal de suspicion modéré passé à l'IF (ALLOW/DENY sont exclus du pipeline)
|
||||
df['anubis_is_flagged'] = (
|
||||
(df.get('anubis_bot_name', pd.Series('', index=df.index)) != '') &
|
||||
(~df.get('anubis_bot_action', pd.Series('', index=df.index)).isin(['ALLOW', 'DENY', '']))
|
||||
).astype(int)
|
||||
|
||||
return df
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
# CYCLE PRINCIPAL
|
||||
# ═══════════════════════════════════════════════════════════════════════════════
|
||||
_consecutive_failures = 0
|
||||
def fetch_and_analyze():
|
||||
global _service_healthy, _consecutive_failures
|
||||
cycle_id = datetime.now().strftime('%Y%m%d_%H%M%S')
|
||||
log_info('=== Lancement cycle IA ===')
|
||||
|
||||
client = get_client()
|
||||
|
||||
# ── Récupération du trafic (fenêtre 1h) ──────────────────────────────────
|
||||
try:
|
||||
df = client.query_df(f'SELECT * FROM {DB}.view_ai_features_1h')
|
||||
except Exception as e:
|
||||
log_info(f'ERREUR REQUETE: {e}')
|
||||
_consecutive_failures += 1
|
||||
if _consecutive_failures >= MAX_FAILURES:
|
||||
_service_healthy = False
|
||||
log_decision('CONSECUTIVE_FAILURES', cycle_id, '', {'count': _consecutive_failures, 'error': str(e)})
|
||||
return
|
||||
|
||||
_consecutive_failures = 0
|
||||
_service_healthy = True
|
||||
|
||||
if df is None or df.empty:
|
||||
log_info('Aucun trafic trouvé.')
|
||||
return
|
||||
|
||||
df = _preprocess_df(df)
|
||||
|
||||
log_decision('CYCLE_START', cycle_id, '', {
|
||||
'total_rows': len(df),
|
||||
'human_rows': int((df.get('asn_label', pd.Series()) == 'human').sum()),
|
||||
'known_bot_rows': int((df.get('bot_name', pd.Series()) != '').sum()),
|
||||
'correlated_rows': int((df.get('correlated', pd.Series()) == 1).sum()),
|
||||
'anubis_allow_rows': int((df.get('anubis_bot_action', pd.Series()) == 'ALLOW').sum()),
|
||||
'anubis_deny_rows': int((df.get('anubis_bot_action', pd.Series()) == 'DENY').sum()),
|
||||
'anubis_weigh_rows': int((df.get('anubis_bot_action', pd.Series()) == 'WEIGH').sum()),
|
||||
'multiwindow': ENABLE_MULTIWINDOW,
|
||||
})
|
||||
|
||||
try:
|
||||
rec_df = client.query_df(f'SELECT src_ip, recurrence FROM {DB}.view_ip_recurrence')
|
||||
recurrence_map = dict(zip(rec_df['src_ip'], rec_df['recurrence']))
|
||||
except Exception:
|
||||
recurrence_map = {}
|
||||
|
||||
# ── Features par modèle (voir DOCUMENTATION.md §4) ───────────────────────
|
||||
# Features communes aux deux modèles (L7 HTTP pur, disponibles correlated=0 et 1)
|
||||
feats = [
|
||||
'hits', 'hit_velocity', 'fuzzing_index', 'post_ratio', 'port_exhaustion_ratio',
|
||||
'orphan_ratio', 'max_keepalives', 'tcp_shared_count', 'header_order_shared_count',
|
||||
'header_count', 'has_accept_language', 'has_cookie', 'has_referer',
|
||||
'modern_browser_score', 'ua_ch_mismatch', 'ip_id_zero_ratio',
|
||||
'request_size_variance', 'multiplexing_efficiency', 'mss_mobile_mismatch',
|
||||
'asset_ratio', 'direct_access_ratio', 'is_ua_rotating', 'distinct_ja4_count',
|
||||
'src_port_density', 'ja4_asn_concentration', 'ja4_country_concentration', 'is_rare_ja4',
|
||||
'header_order_confidence', 'distinct_header_orders', 'temporal_entropy',
|
||||
'path_diversity_ratio', 'url_depth_variance', 'anomalous_payload_ratio',
|
||||
# B4-B7 : features L7 pures (disponibles correlated=0 et 1)
|
||||
'head_ratio', 'sec_fetch_absence_rate', 'generic_accept_ratio', 'http10_ratio',
|
||||
# Anubis : signal de suspicion modéré (WEIGH/CHALLENGE) — bypass pour ALLOW/DENY
|
||||
'anubis_is_flagged',
|
||||
# HTTP : header incomplet et usage HTTP plain (disponibles pour les deux modèles)
|
||||
'missing_accept_enc_ratio', 'http_scheme_ratio',
|
||||
]
|
||||
# Features supplémentaires pour le modèle Complet (nécessitent des données TCP/TLS)
|
||||
feats_complet = feats + [
|
||||
'tcp_jitter_variance', 'alpn_http_mismatch', 'is_alpn_missing', 'sni_host_mismatch',
|
||||
# B1-B3, B8 : features TLS/TCP (disponibles correlated=1 uniquement)
|
||||
'ja3_diversity_ratio', 'syn_timing_cv', 'tls12_ratio', 'ip_df_variance',
|
||||
# TTL fingerprinting OS + TCP window scale (L4 uniquement)
|
||||
'avg_ttl', 'ttl_std', 'no_window_scale_ratio',
|
||||
]
|
||||
|
||||
# ── Analyse fenêtre 1h ────────────────────────────────────────────────────
|
||||
anom_a, scored_a = run_semi_supervised_logic(df[df['correlated'] == 1].copy(), feats_complet, 'Complet', cycle_id, recurrence_map)
|
||||
anom_b, scored_b = run_semi_supervised_logic(df[df['correlated'] == 0].copy(), feats, 'Applicatif', cycle_id, recurrence_map)
|
||||
all_anom = pd.concat([anom_a, anom_b], ignore_index=True)
|
||||
all_scored = pd.concat([scored_a, scored_b], ignore_index=True)
|
||||
|
||||
# ── A3 : Analyse fenêtre 24h (optionnelle) ────────────────────────────────
|
||||
if ENABLE_MULTIWINDOW:
|
||||
try:
|
||||
df_24h = client.query_df(f'SELECT * FROM {DB}.{MULTIWINDOW_VIEW}')
|
||||
if df_24h is not None and not df_24h.empty:
|
||||
df_24h = _preprocess_df(df_24h)
|
||||
log_info(f"[24h] {len(df_24h)} sessions dans la fenêtre 24h.")
|
||||
anom_c, scored_c = run_semi_supervised_logic(df_24h[df_24h['correlated'] == 1].copy(), feats_complet, 'Complet_24h', cycle_id, recurrence_map)
|
||||
anom_d, scored_d = run_semi_supervised_logic(df_24h[df_24h['correlated'] == 0].copy(), feats, 'Applicatif_24h', cycle_id, recurrence_map)
|
||||
all_anom_24h = pd.concat([anom_c, anom_d], ignore_index=True)
|
||||
all_scored_24h = pd.concat([scored_c, scored_d], ignore_index=True)
|
||||
# Fusion : pour les IPs présentes dans les deux fenêtres, conserver le score le plus bas
|
||||
if not all_anom_24h.empty:
|
||||
all_anom = pd.concat([all_anom, all_anom_24h], ignore_index=True)
|
||||
log_info(f"[24h] Fusion 1h+24h : {len(all_anom)} entrées avant déduplication.")
|
||||
all_scored = pd.concat([all_scored, all_scored_24h], ignore_index=True)
|
||||
else:
|
||||
log_info(f"[24h] Vue {MULTIWINDOW_VIEW} vide — analyse mono-fenêtre.")
|
||||
except Exception as e:
|
||||
log_info(f"[24h] Vue {MULTIWINDOW_VIEW} inaccessible : {e} — analyse mono-fenêtre.")
|
||||
|
||||
# ── Insertion de toutes les classifications dans ml_all_scores ───────────
|
||||
if not all_scored.empty:
|
||||
try:
|
||||
now = datetime.now().replace(microsecond=0)
|
||||
all_scored['detected_at'] = now
|
||||
all_scored['ja4'] = all_scored['ja4'].replace({'': 'HTTP_CLEAR_TEXT'})
|
||||
all_scores_cols = [
|
||||
'detected_at', 'window_start', 'src_ip', 'ja4', 'host', 'bot_name',
|
||||
'anubis_bot_name', 'anubis_bot_action', 'anubis_bot_category',
|
||||
'anomaly_score', 'raw_anomaly_score', 'threat_level', 'model_name',
|
||||
'correlated', 'asn_number', 'asn_org', 'country_code', 'asn_label',
|
||||
'hits', 'hit_velocity', 'fuzzing_index', 'post_ratio', 'campaign_id'
|
||||
]
|
||||
scores_df = all_scored[[c for c in all_scores_cols if c in all_scored.columns]]
|
||||
client.insert_df(f'{DB}.ml_all_scores', scores_df)
|
||||
log_info(f'[ml_all_scores] {len(scores_df)} sessions scorées enregistrées.')
|
||||
except Exception as e:
|
||||
log_info(f'[ml_all_scores] ERREUR INSERTION: {e}')
|
||||
|
||||
if not all_anom.empty:
|
||||
all_anom = all_anom.sort_values('raw_anomaly_score', ascending=True).drop_duplicates(subset=['src_ip'], keep='first')
|
||||
log_info(f'Après déduplication intra-cycle : {len(all_anom)} IP uniques.')
|
||||
|
||||
# A5 — Déduplication inter-cycles avec TTL
|
||||
all_anom = _filter_recent_detections(client, all_anom)
|
||||
|
||||
if all_anom.empty:
|
||||
log_info('Toutes les anomalies filtrées par déduplication TTL.')
|
||||
log_decision('CYCLE_END', cycle_id, '', {'inserted': 0, 'anomalies': 0, 'known_bots': 0, 'critical': 0, 'high': 0, 'dedup_ttl_min': DEDUP_TTL_MIN})
|
||||
return
|
||||
|
||||
all_anom['detected_at'] = datetime.now().replace(microsecond=0)
|
||||
fake_nav_col = 'is_fake_navigation'
|
||||
all_anom['is_headless'] = all_anom[fake_nav_col].astype(int) if fake_nav_col in all_anom.columns else 0
|
||||
|
||||
cols = [
|
||||
'detected_at', 'src_ip', 'ja4', 'host', 'bot_name', 'anomaly_score',
|
||||
'threat_level', 'model_name', 'recurrence',
|
||||
'asn_number', 'asn_org', 'asn_detail', 'asn_domain', 'country_code', 'asn_label',
|
||||
'hits', 'hit_velocity', 'fuzzing_index', 'post_ratio', 'port_exhaustion_ratio', 'max_keepalives', 'orphan_ratio',
|
||||
'tcp_jitter_variance', 'tcp_shared_count', 'true_window_size', 'window_mss_ratio',
|
||||
'alpn_http_mismatch', 'is_alpn_missing', 'sni_host_mismatch',
|
||||
'header_count', 'has_accept_language', 'has_cookie', 'has_referer',
|
||||
'modern_browser_score', 'is_headless', 'ua_ch_mismatch',
|
||||
'header_order_shared_count', 'ip_id_zero_ratio', 'request_size_variance',
|
||||
'multiplexing_efficiency', 'mss_mobile_mismatch',
|
||||
'correlated', 'reason', 'asset_ratio', 'direct_access_ratio', 'is_ua_rotating',
|
||||
'distinct_ja4_count', 'src_port_density', 'ja4_asn_concentration',
|
||||
'ja4_country_concentration', 'is_rare_ja4',
|
||||
'header_order_confidence', 'distinct_header_orders', 'temporal_entropy',
|
||||
'path_diversity_ratio', 'url_depth_variance', 'anomalous_payload_ratio',
|
||||
'anubis_bot_name', 'anubis_bot_action', 'anubis_bot_category',
|
||||
]
|
||||
|
||||
try:
|
||||
final_df = all_anom[[c for c in cols if c in all_anom.columns]]
|
||||
client.insert_df(f'{DB}.ml_detected_anomalies', final_df)
|
||||
log_info(f'Succès: {len(final_df)} menaces enregistrées.')
|
||||
log_decision('CYCLE_END', cycle_id, '', {
|
||||
'inserted': len(final_df),
|
||||
'anomalies': int((final_df.get('bot_name', pd.Series()) == '').sum()),
|
||||
'known_bots': int((final_df.get('bot_name', pd.Series()) != '').sum()),
|
||||
'critical': int((final_df.get('threat_level', pd.Series()) == 'CRITICAL').sum()),
|
||||
'high': int((final_df.get('threat_level', pd.Series()) == 'HIGH').sum()),
|
||||
'dedup_ttl_min': DEDUP_TTL_MIN,
|
||||
})
|
||||
except Exception as e:
|
||||
log_info(f'ERREUR INSERTION: {e}')
|
||||
else:
|
||||
log_info('Aucune menace détectée.')
|
||||
log_decision('CYCLE_END', cycle_id, '', {'inserted': 0, 'anomalies': 0, 'known_bots': 0, 'critical': 0, 'high': 0, 'dedup_ttl_min': DEDUP_TTL_MIN})
|
||||
|
||||
if __name__ == '__main__':
|
||||
log_info('*' * 65)
|
||||
log_info(' DÉMARRAGE DU SERVICE BOT DETECTOR IA v12 (+ Anubis)')
|
||||
log_info(f' DB : {DB}')
|
||||
log_info(f' Contamination : {CONTAMINATION}')
|
||||
log_info(f' Seuil anomalie : {ANOMALY_THRESHOLD} (adaptatif percentile={ANOMALY_PERCENTILE})')
|
||||
log_info(f' Cycle : {CYCLE_INTERVAL}s | Fenêtre 1h | Multi-fenêtres : {ENABLE_MULTIWINDOW}')
|
||||
log_info(f' Retraining : toutes les {RETRAIN_INTERVAL_H}h | Drift threshold : {DRIFT_THRESHOLD:.0%}')
|
||||
log_info(f' Modèles : {MODEL_DIR}')
|
||||
log_info(f' SHAP : {"activé" if ENABLE_SHAP else "désactivé (shap non installé)" if not SHAP_AVAILABLE else "désactivé"}')
|
||||
log_info(f' Clustering : {"activé" if ENABLE_CLUSTERING else "désactivé"} | Dedup TTL : {DEDUP_TTL_MIN}min')
|
||||
log_info(f' Récurrence weight : {RECURRENCE_WEIGHT} | Min features ratio : {MIN_VALID_FEATURE_RATIO:.0%}')
|
||||
log_info(f' Anubis : ALLOW→KNOWN_BOT (score=0), DENY→ANUBIS_DENY (score IF réel)')
|
||||
log_info('*' * 65)
|
||||
log_decision('SERVICE_START', 'boot', '', {
|
||||
'db': DB, 'contamination': CONTAMINATION, 'anomaly_threshold': ANOMALY_THRESHOLD,
|
||||
'cycle_interval': CYCLE_INTERVAL, 'retrain_interval_h': RETRAIN_INTERVAL_H
|
||||
})
|
||||
while True:
|
||||
try: fetch_and_analyze()
|
||||
except Exception as e: log_info(f"Erreur globale : {e}")
|
||||
time.sleep(CYCLE_INTERVAL)
|
||||
|
||||
6
services/bot-detector/bot_detector/requirements.txt
Normal file
6
services/bot-detector/bot_detector/requirements.txt
Normal file
@ -0,0 +1,6 @@
|
||||
clickhouse-connect==0.8.0
|
||||
pandas==2.2.0
|
||||
scikit-learn==1.4.0
|
||||
shap==0.44.1
|
||||
pyyaml>=6.0
|
||||
ja4-common @ file:///app/shared/ja4_common
|
||||
17
services/bot-detector/bot_detector/tests/conftest.py
Normal file
17
services/bot-detector/bot_detector/tests/conftest.py
Normal file
@ -0,0 +1,17 @@
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_ch_client():
|
||||
"""Mock ClickHouse client."""
|
||||
client = MagicMock()
|
||||
client.query.return_value = MagicMock(result_rows=[])
|
||||
client.command.return_value = None
|
||||
return client
|
||||
|
||||
|
||||
@pytest.fixture(autouse=False)
|
||||
def mock_get_client(mock_ch_client):
|
||||
with patch("ja4_common.clickhouse.get_client", return_value=mock_ch_client):
|
||||
yield mock_ch_client
|
||||
166
services/bot-detector/bot_detector/tests/test_detector.py
Normal file
166
services/bot-detector/bot_detector/tests/test_detector.py
Normal file
@ -0,0 +1,166 @@
|
||||
import os
|
||||
import pytest
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
|
||||
def test_settings_from_env(monkeypatch):
|
||||
"""ClickHouseSettings loads CLICKHOUSE_HOST from env."""
|
||||
monkeypatch.setenv("CLICKHOUSE_HOST", "testhost")
|
||||
from ja4_common.settings import ClickHouseSettings
|
||||
s = ClickHouseSettings()
|
||||
assert s.CLICKHOUSE_HOST == "testhost"
|
||||
|
||||
|
||||
def test_feature_dataframe_validation():
|
||||
"""MIN_VALID_FEATURE_RATIO logic: if < ratio of features have data, skip."""
|
||||
MIN_VALID_FEATURE_RATIO = 0.5
|
||||
df = pd.DataFrame({"f1": [1.0], "f2": [None], "f3": [None], "f4": [None]})
|
||||
non_null_ratio = df.notna().mean().mean()
|
||||
assert non_null_ratio < MIN_VALID_FEATURE_RATIO, "Should detect insufficient features"
|
||||
|
||||
|
||||
def test_anomaly_threshold():
|
||||
"""Scores below ANOMALY_THRESHOLD trigger detection."""
|
||||
ANOMALY_THRESHOLD = -0.1
|
||||
anomaly_scores = np.array([-0.5, -0.3, 0.1, 0.2])
|
||||
anomalies = anomaly_scores[anomaly_scores < ANOMALY_THRESHOLD]
|
||||
assert len(anomalies) == 2, "Should detect 2 anomalies"
|
||||
|
||||
|
||||
def test_dedup_logic():
|
||||
"""Duplicate detections within DEDUP_TTL_MIN are skipped."""
|
||||
from datetime import datetime, timedelta
|
||||
DEDUP_TTL_MIN = 60
|
||||
dedup_cache = {}
|
||||
|
||||
def should_insert(ip: str, now: datetime) -> bool:
|
||||
if ip in dedup_cache:
|
||||
if (now - dedup_cache[ip]).total_seconds() < DEDUP_TTL_MIN * 60:
|
||||
return False
|
||||
dedup_cache[ip] = now
|
||||
return True
|
||||
|
||||
now = datetime(2024, 1, 1, 12, 0, 0)
|
||||
assert should_insert("1.2.3.4", now) is True
|
||||
assert should_insert("1.2.3.4", now + timedelta(minutes=30)) is False # within TTL
|
||||
assert should_insert("1.2.3.4", now + timedelta(minutes=61)) is True # past TTL
|
||||
|
||||
|
||||
def test_health_check():
|
||||
"""Health check endpoint returns 200."""
|
||||
import threading
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
|
||||
class HealthHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
server = HTTPServer(("127.0.0.1", 0), HealthHandler)
|
||||
port = server.server_address[1]
|
||||
t = threading.Thread(target=server.handle_request)
|
||||
t.start()
|
||||
|
||||
import urllib.request
|
||||
resp = urllib.request.urlopen(f"http://127.0.0.1:{port}/")
|
||||
assert resp.status == 200
|
||||
server.server_close()
|
||||
|
||||
|
||||
def test_dedup_different_ips_are_independent():
|
||||
"""Different IPs are tracked independently in dedup cache."""
|
||||
from datetime import datetime, timedelta
|
||||
DEDUP_TTL_MIN = 60
|
||||
dedup_cache = {}
|
||||
|
||||
def should_insert(ip: str, now: datetime) -> bool:
|
||||
if ip in dedup_cache:
|
||||
if (now - dedup_cache[ip]).total_seconds() < DEDUP_TTL_MIN * 60:
|
||||
return False
|
||||
dedup_cache[ip] = now
|
||||
return True
|
||||
|
||||
now = datetime(2024, 1, 1, 12, 0, 0)
|
||||
assert should_insert("1.1.1.1", now) is True
|
||||
assert should_insert("2.2.2.2", now) is True # Different IP, should be allowed
|
||||
assert should_insert("1.1.1.1", now + timedelta(minutes=30)) is False # Same IP within TTL
|
||||
assert should_insert("2.2.2.2", now + timedelta(minutes=30)) is False # Same IP within TTL
|
||||
|
||||
|
||||
def test_dedup_exact_ttl_boundary():
|
||||
"""Dedup: insertion exactly at TTL boundary is still blocked."""
|
||||
from datetime import datetime, timedelta
|
||||
DEDUP_TTL_MIN = 60
|
||||
dedup_cache = {}
|
||||
|
||||
def should_insert(ip: str, now: datetime) -> bool:
|
||||
if ip in dedup_cache:
|
||||
if (now - dedup_cache[ip]).total_seconds() < DEDUP_TTL_MIN * 60:
|
||||
return False
|
||||
dedup_cache[ip] = now
|
||||
return True
|
||||
|
||||
now = datetime(2024, 1, 1, 12, 0, 0)
|
||||
assert should_insert("1.2.3.4", now) is True
|
||||
# Exactly at 60 minutes should be blocked (< not <=)
|
||||
assert should_insert("1.2.3.4", now + timedelta(minutes=60)) is False
|
||||
|
||||
|
||||
def test_anomaly_threshold_no_anomalies():
|
||||
"""No anomalies when all scores are above threshold."""
|
||||
import numpy as np
|
||||
ANOMALY_THRESHOLD = -0.1
|
||||
scores = np.array([0.0, 0.1, 0.5, 1.0])
|
||||
anomalies = scores[scores < ANOMALY_THRESHOLD]
|
||||
assert len(anomalies) == 0
|
||||
|
||||
|
||||
def test_anomaly_threshold_all_anomalies():
|
||||
"""All items flagged when all scores are below threshold."""
|
||||
import numpy as np
|
||||
ANOMALY_THRESHOLD = -0.1
|
||||
scores = np.array([-0.5, -0.3, -0.2, -0.15])
|
||||
anomalies = scores[scores < ANOMALY_THRESHOLD]
|
||||
assert len(anomalies) == 4
|
||||
|
||||
|
||||
def test_feature_dataframe_all_valid():
|
||||
"""Feature dataframe with all valid values passes ratio check."""
|
||||
import pandas as pd
|
||||
MIN_VALID_FEATURE_RATIO = 0.5
|
||||
df = pd.DataFrame({"f1": [1.0], "f2": [2.0], "f3": [3.0], "f4": [4.0]})
|
||||
non_null_ratio = df.notna().mean().mean()
|
||||
assert non_null_ratio >= MIN_VALID_FEATURE_RATIO
|
||||
|
||||
|
||||
def test_health_check_returns_correct_status():
|
||||
"""Health check endpoint body is readable."""
|
||||
import threading
|
||||
import urllib.request
|
||||
from http.server import HTTPServer, BaseHTTPRequestHandler
|
||||
|
||||
class StatusHandler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.end_headers()
|
||||
self.wfile.write(b'{"status": "ok"}')
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
server = HTTPServer(("127.0.0.1", 0), StatusHandler)
|
||||
port = server.server_address[1]
|
||||
t = threading.Thread(target=server.handle_request)
|
||||
t.start()
|
||||
|
||||
resp = urllib.request.urlopen(f"http://127.0.0.1:{port}/health")
|
||||
assert resp.status == 200
|
||||
body = resp.read()
|
||||
assert b"ok" in body
|
||||
server.server_close()
|
||||
411
services/bot-detector/deploy_views.sql
Normal file
411
services/bot-detector/deploy_views.sql
Normal file
@ -0,0 +1,411 @@
|
||||
-- ============================================================================
|
||||
-- ARCHITECTURE DE DÉTECTION INTÉGRALE (v13 - bot_detector v11 + ml_all_scores)
|
||||
-- Base : mabase_prod | Fenêtre : 24h | Dédoublonnage par src_ip
|
||||
-- Modifications v11 : ajout campaign_id, raw_anomaly_score dans ml_detected_anomalies
|
||||
-- correction view_dashboard_variability (header_user_agent → reason)
|
||||
-- Modifications v12 : ajout table ml_all_scores (toutes les classifications, sans seuil)
|
||||
-- ============================================================================
|
||||
|
||||
-- 1. NETTOYAGE COMPLET
|
||||
DROP TABLE IF EXISTS mabase_prod.ml_all_scores;
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_bot_ip;
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_bot_ja4;
|
||||
DROP DICTIONARY IF EXISTS mabase_prod.dict_asn_reputation;
|
||||
DROP TABLE IF EXISTS mabase_prod.ml_detected_anomalies;
|
||||
DROP VIEW IF EXISTS mabase_prod.view_ip_recurrence;
|
||||
DROP VIEW IF EXISTS mabase_prod.view_ai_features_1h;
|
||||
-- Suppression des anciennes vues heuristiques
|
||||
DROP VIEW IF EXISTS mabase_prod.view_host_ip_ja4_rotation;
|
||||
DROP VIEW IF EXISTS mabase_prod.view_host_ja4_anomalies;
|
||||
DROP VIEW IF EXISTS mabase_prod.view_form_bruteforce_detected;
|
||||
DROP VIEW IF EXISTS mabase_prod.view_alpn_mismatch_detected;
|
||||
DROP VIEW IF EXISTS mabase_prod.view_tcp_spoofing_detected;
|
||||
|
||||
DROP VIEW IF EXISTS mabase_prod.mv_agg_host_ip_ja4_1h;
|
||||
DROP TABLE IF EXISTS mabase_prod.agg_host_ip_ja4_1h;
|
||||
DROP VIEW IF EXISTS mabase_prod.mv_agg_header_fingerprint_1h;
|
||||
DROP TABLE IF EXISTS mabase_prod.agg_header_fingerprint_1h;
|
||||
|
||||
-- ============================================================================
|
||||
-- 2. DICTIONNAIRES DE RÉPUTATION EN RAM
|
||||
-- ============================================================================
|
||||
CREATE DICTIONARY mabase_prod.dict_bot_ip (prefix String, bot_name String)
|
||||
PRIMARY KEY prefix SOURCE(FILE(path '/var/lib/clickhouse/user_files/bot_ip.csv' format 'CSV'))
|
||||
LAYOUT(IP_TRIE()) LIFETIME(MIN 300 MAX 300);
|
||||
|
||||
CREATE DICTIONARY mabase_prod.dict_bot_ja4 (ja4 String, bot_name String)
|
||||
PRIMARY KEY ja4 SOURCE(FILE(path '/var/lib/clickhouse/user_files/bot_ja4.csv' format 'CSV'))
|
||||
LAYOUT(COMPLEX_KEY_HASHED()) LIFETIME(MIN 300 MAX 300);
|
||||
|
||||
CREATE DICTIONARY mabase_prod.dict_asn_reputation (src_asn UInt64, label String)
|
||||
PRIMARY KEY src_asn SOURCE(FILE(path '/var/lib/clickhouse/user_files/asn_reputation.csv' format 'CSV'))
|
||||
LAYOUT(HASHED()) LIFETIME(MIN 300 MAX 300);
|
||||
|
||||
-- ============================================================================
|
||||
-- 3. TABLE D'AGRÉGATION COMPORTEMENTALE (L4 / L5 / L7)
|
||||
-- ============================================================================
|
||||
CREATE TABLE mabase_prod.agg_host_ip_ja4_1h
|
||||
(
|
||||
window_start DateTime,
|
||||
src_ip IPv6, ja4 String, host String, src_asn UInt32,
|
||||
src_country_code SimpleAggregateFunction(any, String),
|
||||
src_as_name SimpleAggregateFunction(any, String),
|
||||
src_org SimpleAggregateFunction(any, String),
|
||||
src_domain SimpleAggregateFunction(any, String),
|
||||
first_seen SimpleAggregateFunction(min, DateTime),
|
||||
last_seen SimpleAggregateFunction(max, DateTime),
|
||||
hits SimpleAggregateFunction(sum, UInt64),
|
||||
count_post SimpleAggregateFunction(sum, UInt64),
|
||||
uniq_paths AggregateFunction(uniq, String),
|
||||
uniq_query_params AggregateFunction(uniq, String),
|
||||
tcp_fp_raw SimpleAggregateFunction(any, String),
|
||||
tcp_jitter_variance AggregateFunction(varPop, Float64),
|
||||
tcp_win_raw SimpleAggregateFunction(any, UInt32),
|
||||
tcp_scale_raw SimpleAggregateFunction(any, UInt32),
|
||||
tcp_mss_raw SimpleAggregateFunction(any, UInt32),
|
||||
tcp_ttl_raw SimpleAggregateFunction(any, UInt32),
|
||||
http_ver_raw SimpleAggregateFunction(any, String),
|
||||
tls_alpn_raw SimpleAggregateFunction(any, String),
|
||||
tls_sni_raw SimpleAggregateFunction(any, String),
|
||||
first_ua SimpleAggregateFunction(any, String),
|
||||
correlated_raw SimpleAggregateFunction(max, UInt8),
|
||||
unique_src_ports AggregateFunction(uniq, UInt16),
|
||||
unique_conn_id AggregateFunction(uniq, String),
|
||||
max_keepalives SimpleAggregateFunction(max, UInt32),
|
||||
orphan_count SimpleAggregateFunction(sum, UInt64),
|
||||
ip_id_zero_count SimpleAggregateFunction(sum, UInt64),
|
||||
total_ip_length_var AggregateFunction(varPop, Float64),
|
||||
mss_1460_count SimpleAggregateFunction(sum, UInt64),
|
||||
count_assets SimpleAggregateFunction(sum, UInt64),
|
||||
count_no_referer SimpleAggregateFunction(sum, UInt64),
|
||||
uniq_ua AggregateFunction(uniq, String),
|
||||
max_requests_per_sec SimpleAggregateFunction(max, UInt32),
|
||||
url_depth_variance AggregateFunction(varPop, Float64),
|
||||
count_anomalous_payload SimpleAggregateFunction(sum, UInt64),
|
||||
-- B features (ajoutées v14)
|
||||
uniq_ja3 AggregateFunction(uniq, String), -- B1: diversité JA3/JA4
|
||||
avg_syn_ms AggregateFunction(avg, Float64), -- B2: SYN timing moyen (pour CV)
|
||||
tls12_count SimpleAggregateFunction(sum, UInt64), -- B3: ratio TLS 1.2
|
||||
count_head SimpleAggregateFunction(sum, UInt64), -- B4: ratio requêtes HEAD
|
||||
count_no_sec_fetch SimpleAggregateFunction(sum, UInt64),-- B5: absence Sec-Fetch-*
|
||||
count_generic_accept SimpleAggregateFunction(sum, UInt64),-- B6: Accept générique
|
||||
count_http10 SimpleAggregateFunction(sum, UInt64), -- B7: ratio HTTP/1.0
|
||||
ip_df_var AggregateFunction(varPop, Float64) -- B8: variance bit DF
|
||||
)
|
||||
ENGINE = AggregatingMergeTree()
|
||||
ORDER BY (window_start, src_ip, ja4, host);
|
||||
|
||||
-- ============================================================================
|
||||
-- 4. VUE MATÉRIALISÉE → agg_host_ip_ja4_1h
|
||||
-- ============================================================================
|
||||
CREATE MATERIALIZED VIEW mabase_prod.mv_agg_host_ip_ja4_1h
|
||||
TO mabase_prod.agg_host_ip_ja4_1h AS
|
||||
SELECT
|
||||
toStartOfHour(src.time) AS window_start,
|
||||
toIPv6(src.src_ip) AS src_ip, src.ja4, src.host, src.src_asn,
|
||||
any(src.src_country_code) AS src_country_code, any(src.src_as_name) AS src_as_name,
|
||||
any(src.src_org) AS src_org, any(src.src_domain) AS src_domain,
|
||||
min(src.time) AS first_seen, max(src.time) AS last_seen, count() AS hits,
|
||||
sum(IF(src.method = 'POST', 1, 0)) AS count_post,
|
||||
uniqState(src.path) AS uniq_paths, uniqState(src.query) AS uniq_query_params,
|
||||
any(toString(cityHash64(concat(toString(src.tcp_meta_window_size), toString(src.tcp_meta_mss), toString(src.tcp_meta_window_scale), src.tcp_meta_options)))) AS tcp_fp_raw,
|
||||
varPopState(toFloat64(src.syn_to_clienthello_ms)) AS tcp_jitter_variance,
|
||||
any(src.tcp_meta_window_size) AS tcp_win_raw, any(src.tcp_meta_window_scale) AS tcp_scale_raw,
|
||||
any(src.tcp_meta_mss) AS tcp_mss_raw, any(src.ip_meta_ttl) AS tcp_ttl_raw,
|
||||
any(src.http_version) AS http_ver_raw, any(src.tls_alpn) AS tls_alpn_raw, any(src.tls_sni) AS tls_sni_raw,
|
||||
any(src.header_user_agent) AS first_ua, max(toUInt8(src.correlated)) AS correlated_raw,
|
||||
uniqState(toUInt16(src.src_port)) AS unique_src_ports, uniqState(src.conn_id) AS unique_conn_id,
|
||||
max(toUInt32(src.keepalives)) AS max_keepalives,
|
||||
sum(IF(src.orphan_side = 'A' OR src.correlated = 0, 1, 0)) AS orphan_count,
|
||||
sum(IF(src.ip_meta_id == 0, 1, 0)) AS ip_id_zero_count,
|
||||
varPopState(toFloat64(src.ip_meta_total_length)) AS total_ip_length_var,
|
||||
sum(IF(src.tcp_meta_mss == 1460, 1, 0)) AS mss_1460_count,
|
||||
sum(IF(match(src.path, '(?i)\.(png|jpg|jpeg|gif|css|js|ico|woff2|svg|eot)$'), 1, 0)) AS count_assets,
|
||||
sum(IF(position(src.client_headers, 'Referer') = 0, 1, 0)) AS count_no_referer,
|
||||
uniqState(src.header_user_agent) AS uniq_ua,
|
||||
0 AS max_requests_per_sec,
|
||||
varPopState(toFloat64(length(replaceAll(src.path, '/', '//')) - length(src.path))) AS url_depth_variance,
|
||||
sum(IF(src.ip_meta_total_length < 60 OR src.ip_meta_total_length > 1500, 1, 0)) AS count_anomalous_payload,
|
||||
-- B features
|
||||
uniqState(src.ja3) AS uniq_ja3,
|
||||
avgState(toFloat64(src.syn_to_clienthello_ms)) AS avg_syn_ms,
|
||||
sum(IF(src.tls_version = '1.2', 1, 0)) AS tls12_count,
|
||||
sum(IF(src.method = 'HEAD', 1, 0)) AS count_head,
|
||||
sum(IF(length(src.header_sec_fetch_site) = 0, 1, 0)) AS count_no_sec_fetch,
|
||||
sum(IF(length(src.header_accept) < 5, 1, 0)) AS count_generic_accept,
|
||||
sum(IF(src.http_version = 'HTTP/1.0', 1, 0)) AS count_http10,
|
||||
varPopState(toFloat64(src.ip_meta_df)) AS ip_df_var
|
||||
FROM mabase_prod.http_logs AS src
|
||||
GROUP BY window_start, src_ip, ja4, host, src_asn;
|
||||
|
||||
-- ============================================================================
|
||||
-- 5. TABLE D'AGRÉGATION DES HEADERS (L7)
|
||||
-- ============================================================================
|
||||
CREATE TABLE mabase_prod.agg_header_fingerprint_1h
|
||||
(
|
||||
window_start DateTime,
|
||||
src_ip IPv6,
|
||||
header_order_hash SimpleAggregateFunction(any, String),
|
||||
header_count SimpleAggregateFunction(max, UInt16),
|
||||
has_accept_language SimpleAggregateFunction(max, UInt8),
|
||||
has_cookie SimpleAggregateFunction(max, UInt8),
|
||||
has_referer SimpleAggregateFunction(max, UInt8),
|
||||
modern_browser_score SimpleAggregateFunction(max, UInt8),
|
||||
ua_ch_mismatch SimpleAggregateFunction(max, UInt8),
|
||||
sec_fetch_mode SimpleAggregateFunction(any, String),
|
||||
sec_fetch_dest SimpleAggregateFunction(any, String)
|
||||
)
|
||||
ENGINE = AggregatingMergeTree()
|
||||
ORDER BY (window_start, src_ip);
|
||||
|
||||
CREATE MATERIALIZED VIEW mabase_prod.mv_agg_header_fingerprint_1h
|
||||
TO mabase_prod.agg_header_fingerprint_1h AS
|
||||
SELECT
|
||||
toStartOfHour(src.time) AS window_start,
|
||||
toIPv6(src.src_ip) AS src_ip,
|
||||
any(toString(cityHash64(src.client_headers))) AS header_order_hash,
|
||||
max(toUInt16(length(src.client_headers) - length(replaceAll(src.client_headers, ',', '')) + 1)) AS header_count,
|
||||
max(toUInt8(if(position(src.client_headers, 'Accept-Language') > 0, 1, 0))) AS has_accept_language,
|
||||
max(toUInt8(if(position(src.client_headers, 'Cookie') > 0, 1, 0))) AS has_cookie,
|
||||
max(toUInt8(if(position(src.client_headers, 'Referer') > 0, 1, 0))) AS has_referer,
|
||||
max(toUInt8(if(length(src.header_sec_ch_ua) > 0, 100, if(length(src.header_user_agent) > 0, 50, 0)))) AS modern_browser_score,
|
||||
max(toUInt8(if((position(src.header_user_agent, 'Windows') > 0 AND position(src.header_sec_ch_ua_platform, 'Windows') == 0) OR (position(src.header_user_agent, 'iPhone') > 0 AND position(src.header_sec_ch_ua_platform, 'iOS') == 0), 1, 0))) AS ua_ch_mismatch,
|
||||
any(src.header_sec_fetch_mode) AS sec_fetch_mode,
|
||||
any(src.header_sec_fetch_dest) AS sec_fetch_dest
|
||||
FROM mabase_prod.http_logs AS src
|
||||
GROUP BY window_start, src.src_ip;
|
||||
|
||||
-- ============================================================================
|
||||
-- 6. TABLE DE RÉSULTATS ML — MENACES UNIQUEMENT (scores < seuil)
|
||||
-- ============================================================================
|
||||
CREATE TABLE mabase_prod.ml_detected_anomalies
|
||||
(
|
||||
detected_at DateTime, src_ip IPv6, ja4 String, host String, bot_name String,
|
||||
anomaly_score Float32, threat_level String, model_name String, recurrence UInt32,
|
||||
asn_number String, asn_org String, asn_detail String, asn_domain String, country_code String, asn_label String,
|
||||
hits UInt64, hit_velocity Float32, fuzzing_index Float32, post_ratio Float32, port_exhaustion_ratio Float32,
|
||||
max_keepalives UInt32, orphan_ratio Float32, tcp_jitter_variance Float32, tcp_shared_count UInt32,
|
||||
true_window_size UInt64, window_mss_ratio Float32, alpn_http_mismatch UInt8, is_alpn_missing UInt8, sni_host_mismatch UInt8,
|
||||
header_count UInt16, has_accept_language UInt8, has_cookie UInt8, has_referer UInt8, modern_browser_score UInt8,
|
||||
is_headless UInt8, ua_ch_mismatch UInt8, header_order_shared_count UInt32, ip_id_zero_ratio Float32,
|
||||
request_size_variance Float32, multiplexing_efficiency Float32, mss_mobile_mismatch UInt8, correlated UInt8, reason String,
|
||||
asset_ratio Float32, direct_access_ratio Float32, is_ua_rotating UInt8, distinct_ja4_count UInt32,
|
||||
src_port_density Float32, ja4_asn_concentration Float32, ja4_country_concentration Float32, is_rare_ja4 UInt8,
|
||||
header_order_confidence Float32, distinct_header_orders UInt32, temporal_entropy Float32,
|
||||
path_diversity_ratio Float32, url_depth_variance Float32, anomalous_payload_ratio Float32,
|
||||
-- Colonnes ajoutées en v11 (bot_detector v11)
|
||||
campaign_id Int32 DEFAULT -1,
|
||||
raw_anomaly_score Float32 DEFAULT 0
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(detected_at)
|
||||
ORDER BY (src_ip)
|
||||
TTL detected_at + INTERVAL 30 DAY;
|
||||
|
||||
-- ============================================================================
|
||||
-- 6b. TABLE DE TOUTES LES CLASSIFICATIONS (sans seuil, pour observabilité)
|
||||
-- ============================================================================
|
||||
CREATE TABLE mabase_prod.ml_all_scores
|
||||
(
|
||||
detected_at DateTime,
|
||||
window_start DateTime,
|
||||
src_ip IPv6,
|
||||
ja4 String,
|
||||
host String,
|
||||
bot_name String,
|
||||
anomaly_score Float32,
|
||||
raw_anomaly_score Float32,
|
||||
threat_level String,
|
||||
model_name String,
|
||||
correlated UInt8,
|
||||
asn_number String,
|
||||
asn_org String,
|
||||
country_code String,
|
||||
asn_label String,
|
||||
hits UInt64,
|
||||
hit_velocity Float32,
|
||||
fuzzing_index Float32,
|
||||
post_ratio Float32,
|
||||
campaign_id Int32
|
||||
)
|
||||
ENGINE = ReplacingMergeTree(detected_at)
|
||||
ORDER BY (window_start, src_ip, ja4, host, model_name)
|
||||
TTL window_start + INTERVAL 3 DAY
|
||||
SETTINGS index_granularity = 8192;
|
||||
|
||||
-- ============================================================================
|
||||
-- 7. VUE DE RÉCURRENCE
|
||||
-- ============================================================================
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_ip_recurrence AS
|
||||
SELECT src_ip, count() AS recurrence, min(detected_at) AS first_seen, max(detected_at) AS last_seen,
|
||||
min(anomaly_score) AS worst_score, argMin(threat_level, anomaly_score) AS worst_threat_level
|
||||
FROM mabase_prod.ml_detected_anomalies GROUP BY src_ip;
|
||||
|
||||
-- ============================================================================
|
||||
-- 8. VUE IA PRINCIPALE (Avec CTE pour Entropie Temporelle)
|
||||
-- ============================================================================
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_ai_features_1h AS
|
||||
WITH base_data AS (
|
||||
SELECT
|
||||
a.window_start, a.src_ip, a.ja4, a.host,
|
||||
toString(a.src_asn) AS asn_number, a.src_as_name AS asn_org,
|
||||
a.src_org AS asn_detail, a.src_domain AS asn_domain, a.src_country_code AS country_code,
|
||||
dictGetOrDefault('mabase_prod.dict_asn_reputation', 'label', toUInt64(a.src_asn), 'unknown') AS asn_label,
|
||||
COALESCE(
|
||||
nullIf(dictGetOrDefault('mabase_prod.dict_bot_ip', 'bot_name', a.src_ip, ''), ''),
|
||||
nullIf(dictGetOrDefault('mabase_prod.dict_bot_ja4', 'bot_name', tuple(a.ja4), ''), ''),
|
||||
''
|
||||
) AS bot_name,
|
||||
a.hits AS hits,
|
||||
sum(a.hits) OVER (PARTITION BY a.src_ip) AS total_ip_hits,
|
||||
a.correlated AS correlated, a.tcp_jitter_variance AS tcp_jitter_variance,
|
||||
a.true_window_size AS true_window_size, a.window_mss_ratio AS window_mss_ratio, a.max_keepalives AS max_keepalives,
|
||||
h.header_order_hash AS header_order_hash, h.header_count AS header_count,
|
||||
h.has_accept_language AS has_accept_language, h.has_cookie AS has_cookie,
|
||||
h.has_referer AS has_referer, h.modern_browser_score AS modern_browser_score, h.ua_ch_mismatch AS ua_ch_mismatch,
|
||||
(a.count_post / (a.hits + 1)) AS post_ratio, (a.uniq_query_params / (a.uniq_paths + 1)) AS fuzzing_index,
|
||||
(a.hits / (dateDiff('second', a.first_seen, a.last_seen) + 1)) AS hit_velocity,
|
||||
(a.unique_src_ports / (a.hits + 1)) AS port_exhaustion_ratio, (a.orphan_count / (a.hits + 1)) AS orphan_ratio,
|
||||
(a.ip_id_zero_count / (a.hits + 1)) AS ip_id_zero_ratio, (a.hits / (a.unique_conn_id + 1)) AS multiplexing_efficiency,
|
||||
IF(a.mss_1460_count > (a.hits * 0.8) AND h.modern_browser_score > 70, 1, 0) AS mss_mobile_mismatch,
|
||||
a.request_size_variance AS request_size_variance,
|
||||
IF(a.tls_alpn = 'h2' AND a.http_version != '2', 1, 0) AS alpn_http_mismatch,
|
||||
IF(length(a.tls_alpn) = 0 OR a.tls_alpn = '00', 1, 0) AS is_alpn_missing,
|
||||
IF(length(a.tls_sni) > 0 AND a.tls_sni != a.host, 1, 0) AS sni_host_mismatch,
|
||||
IF(h.sec_fetch_mode = 'navigate' AND h.sec_fetch_dest != 'document', 1, 0) AS is_fake_navigation,
|
||||
count() OVER (PARTITION BY a.tcp_fingerprint) AS tcp_shared_count,
|
||||
count() OVER (PARTITION BY h.header_order_hash) AS header_order_shared_count,
|
||||
(a.count_assets / (a.hits + 1)) AS asset_ratio, (a.count_no_referer / (a.hits + 1)) AS direct_access_ratio,
|
||||
IF(a.unique_ua > 2, 1, 0) AS is_ua_rotating, uniqExact(a.ja4) OVER (PARTITION BY a.src_ip) AS distinct_ja4_count,
|
||||
((a.hits / (a.unique_src_ports + 1)) / (dateDiff('second', a.first_seen, a.last_seen) + 1)) AS src_port_density,
|
||||
(sum(a.hits) OVER (PARTITION BY a.ja4, a.src_asn) / (sum(a.hits) OVER (PARTITION BY a.ja4) + 1)) AS ja4_asn_concentration,
|
||||
(sum(a.hits) OVER (PARTITION BY a.ja4, a.src_country_code) / (sum(a.hits) OVER (PARTITION BY a.ja4) + 1)) AS ja4_country_concentration,
|
||||
IF(sum(a.hits) OVER (PARTITION BY a.ja4) < 100, 1, 0) AS is_rare_ja4,
|
||||
(count() OVER (PARTITION BY h.header_order_hash, a.first_ua) / (count() OVER (PARTITION BY a.first_ua) + 1)) AS header_order_confidence,
|
||||
uniqExact(h.header_order_hash) OVER (PARTITION BY a.src_ip) AS distinct_header_orders,
|
||||
(a.uniq_paths / (a.hits + 1)) AS path_diversity_ratio,
|
||||
a.url_depth_variance AS url_depth_variance,
|
||||
(a.count_anomalous_payload / (a.hits + 1)) AS anomalous_payload_ratio,
|
||||
-- B features : TLS/TCP (disponibles correlated=1 uniquement)
|
||||
a.uniq_ja3_val AS uniq_ja3_per_row,
|
||||
sqrt(a.tcp_jitter_variance) / greatest(a.avg_syn_ms_val, 1) AS syn_timing_cv, -- B2
|
||||
a.tls12_count / (a.hits + 1) AS tls12_ratio, -- B3
|
||||
-- B features : HTTP pures (disponibles correlated=0 et 1)
|
||||
a.count_head / (a.hits + 1) AS head_ratio, -- B4
|
||||
a.count_no_sec_fetch / (a.hits + 1) AS sec_fetch_absence_rate, -- B5
|
||||
a.count_generic_accept / (a.hits + 1) AS generic_accept_ratio, -- B6
|
||||
a.count_http10 / (a.hits + 1) AS http10_ratio, -- B7
|
||||
a.ip_df_variance AS ip_df_variance -- B8
|
||||
FROM (
|
||||
SELECT
|
||||
window_start, src_ip, ja4, host, src_asn,
|
||||
any(src_country_code) AS src_country_code, any(src_as_name) AS src_as_name,
|
||||
any(src_org) AS src_org, any(src_domain) AS src_domain, any(first_ua) AS first_ua,
|
||||
sum(hits) AS hits, uniqMerge(uniq_paths) AS uniq_paths,
|
||||
uniqMerge(uniq_query_params) AS uniq_query_params, sum(count_post) AS count_post,
|
||||
min(first_seen) AS first_seen, max(last_seen) AS last_seen,
|
||||
any(tcp_fp_raw) AS tcp_fingerprint, varPopMerge(tcp_jitter_variance) AS tcp_jitter_variance,
|
||||
varPopMerge(total_ip_length_var) AS request_size_variance,
|
||||
any(tcp_win_raw * exp2(tcp_scale_raw)) AS true_window_size,
|
||||
IF(any(tcp_mss_raw) > 0, any(tcp_win_raw) / any(tcp_mss_raw), 0) AS window_mss_ratio,
|
||||
any(http_ver_raw) AS http_version, any(tls_alpn_raw) AS tls_alpn, any(tls_sni_raw) AS tls_sni,
|
||||
max(correlated_raw) AS correlated, uniqMerge(unique_src_ports) AS unique_src_ports,
|
||||
uniqMerge(unique_conn_id) AS unique_conn_id, max(max_keepalives) AS max_keepalives,
|
||||
sum(orphan_count) AS orphan_count, sum(ip_id_zero_count) AS ip_id_zero_count,
|
||||
sum(mss_1460_count) AS mss_1460_count,
|
||||
sum(count_assets) AS count_assets, sum(count_no_referer) AS count_no_referer, uniqMerge(uniq_ua) AS unique_ua,
|
||||
varPopMerge(url_depth_variance) AS url_depth_variance,
|
||||
sum(count_anomalous_payload) AS count_anomalous_payload,
|
||||
-- B feature aggregates
|
||||
uniqMerge(uniq_ja3) AS uniq_ja3_val,
|
||||
avgMerge(avg_syn_ms) AS avg_syn_ms_val,
|
||||
sum(tls12_count) AS tls12_count,
|
||||
sum(count_head) AS count_head,
|
||||
sum(count_no_sec_fetch) AS count_no_sec_fetch,
|
||||
sum(count_generic_accept) AS count_generic_accept,
|
||||
sum(count_http10) AS count_http10,
|
||||
varPopMerge(ip_df_var) AS ip_df_variance
|
||||
FROM mabase_prod.agg_host_ip_ja4_1h
|
||||
WHERE window_start >= now() - INTERVAL 24 HOUR
|
||||
GROUP BY window_start, src_ip, ja4, host, src_asn
|
||||
) a
|
||||
LEFT JOIN (
|
||||
SELECT
|
||||
window_start, src_ip, any(header_order_hash) AS header_order_hash,
|
||||
max(header_count) AS header_count, max(has_accept_language) AS has_accept_language,
|
||||
max(has_cookie) AS has_cookie, max(has_referer) AS has_referer,
|
||||
max(modern_browser_score) AS modern_browser_score, max(ua_ch_mismatch) AS ua_ch_mismatch,
|
||||
any(sec_fetch_mode) AS sec_fetch_mode, any(sec_fetch_dest) AS sec_fetch_dest
|
||||
FROM mabase_prod.agg_header_fingerprint_1h
|
||||
WHERE window_start >= now() - INTERVAL 24 HOUR
|
||||
GROUP BY window_start, src_ip
|
||||
) h ON a.src_ip = h.src_ip AND a.window_start = h.window_start
|
||||
)
|
||||
SELECT
|
||||
*,
|
||||
-(sum((hits / (total_ip_hits + 1)) * log2((hits / (total_ip_hits + 1)) + 0.000001)) OVER (PARTITION BY src_ip)) AS temporal_entropy,
|
||||
-- B1: ratio diversité JA3/JA4 par src_ip (signal: bots avec JA3 rotatifs sur peu de JA4)
|
||||
sum(uniq_ja3_per_row) OVER (PARTITION BY src_ip) / greatest(distinct_ja4_count, 1) AS ja3_diversity_ratio
|
||||
FROM base_data;
|
||||
|
||||
-- ============================================================================
|
||||
-- VUES POUR LE DASHBOARD WEB
|
||||
-- ============================================================================
|
||||
|
||||
-- Vue pour les métriques globales du dashboard
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_dashboard_summary AS
|
||||
SELECT
|
||||
count() AS total_detections,
|
||||
countIf(threat_level = 'CRITICAL') AS critical_count,
|
||||
countIf(threat_level = 'HIGH') AS high_count,
|
||||
countIf(threat_level = 'MEDIUM') AS medium_count,
|
||||
countIf(threat_level = 'LOW') AS low_count,
|
||||
countIf(bot_name != '') AS known_bots_count,
|
||||
countIf(bot_name = '') AS anomalies_count,
|
||||
uniq(src_ip) AS unique_ips
|
||||
FROM mabase_prod.ml_detected_anomalies
|
||||
WHERE detected_at >= now() - INTERVAL 24 HOUR;
|
||||
|
||||
-- Vue pour la série temporelle (par heure)
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_dashboard_timeseries AS
|
||||
SELECT
|
||||
toStartOfHour(detected_at) AS hour,
|
||||
count() AS total,
|
||||
countIf(threat_level = 'CRITICAL') AS critical,
|
||||
countIf(threat_level = 'HIGH') AS high,
|
||||
countIf(threat_level = 'MEDIUM') AS medium,
|
||||
countIf(threat_level = 'LOW') AS low
|
||||
FROM mabase_prod.ml_detected_anomalies
|
||||
WHERE detected_at >= now() - INTERVAL 24 HOUR
|
||||
GROUP BY hour
|
||||
ORDER BY hour;
|
||||
|
||||
-- Vue pour la distribution des menaces
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_dashboard_threat_dist AS
|
||||
SELECT
|
||||
threat_level,
|
||||
count() AS count,
|
||||
round(count() * 100.0 / sum(count()) OVER (), 2) AS percentage
|
||||
FROM mabase_prod.ml_detected_anomalies
|
||||
WHERE detected_at >= now() - INTERVAL 24 HOUR
|
||||
GROUP BY threat_level
|
||||
ORDER BY count DESC;
|
||||
|
||||
-- Vue pour la variabilité (utilisée par l'API)
|
||||
-- Note v12 : header_user_agent n'existe pas dans ml_detected_anomalies → remplacé par reason
|
||||
CREATE OR REPLACE VIEW mabase_prod.view_dashboard_variability AS
|
||||
SELECT
|
||||
detected_at,
|
||||
src_ip,
|
||||
ja4,
|
||||
host,
|
||||
reason AS sample_reason,
|
||||
country_code,
|
||||
asn_number,
|
||||
asn_org,
|
||||
threat_level,
|
||||
model_name,
|
||||
anomaly_score,
|
||||
campaign_id,
|
||||
raw_anomaly_score
|
||||
FROM mabase_prod.ml_detected_anomalies
|
||||
WHERE detected_at >= now() - INTERVAL 24 HOUR;
|
||||
78
services/bot-detector/docker-compose.yml
Normal file
78
services/bot-detector/docker-compose.yml
Normal file
@ -0,0 +1,78 @@
|
||||
version: '3.8' # Champ déprécié depuis Docker Compose v2.x mais toléré — peut être supprimé
|
||||
|
||||
services:
|
||||
bot_detector_ai:
|
||||
build: bot_detector
|
||||
container_name: bot_detector_ai
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8080:8080" # Health check → GET http://localhost:8080/
|
||||
|
||||
env_file:
|
||||
- .env
|
||||
|
||||
environment:
|
||||
# ── ClickHouse ────────────────────────────────────────────────────────
|
||||
CLICKHOUSE_HOST: ${CLICKHOUSE_HOST:-clickhouse}
|
||||
CLICKHOUSE_DB: ${CLICKHOUSE_DB:-mabase_prod}
|
||||
CLICKHOUSE_USER: ${CLICKHOUSE_USER:-admin}
|
||||
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-}
|
||||
|
||||
# ── Modèle IA ─────────────────────────────────────────────────────────
|
||||
ISOLATION_CONTAMINATION: ${ISOLATION_CONTAMINATION:-0.02}
|
||||
ANOMALY_THRESHOLD: ${ANOMALY_THRESHOLD:--0.03}
|
||||
|
||||
# ── Cycle ─────────────────────────────────────────────────────────────
|
||||
CYCLE_INTERVAL_SEC: ${CYCLE_INTERVAL_SEC:-300}
|
||||
MAX_CONSECUTIVE_FAILURES: ${MAX_CONSECUTIVE_FAILURES:-3}
|
||||
|
||||
# ── Logs ──────────────────────────────────────────────────────────────
|
||||
BOT_DETECTOR_LOG: ${BOT_DETECTOR_LOG:-/var/log/bot_detector/decisions.jsonl}
|
||||
LOG_BACKUP_COUNT: ${LOG_BACKUP_COUNT:-7}
|
||||
|
||||
# ── Modèles persistants ───────────────────────────────────────────────
|
||||
MODEL_DIR: ${MODEL_DIR:-/var/lib/bot_detector}
|
||||
RETRAIN_INTERVAL_HOURS: ${RETRAIN_INTERVAL_HOURS:-24}
|
||||
MODEL_HISTORY_COUNT: ${MODEL_HISTORY_COUNT:-10}
|
||||
|
||||
# ── A1 — Dérive conceptuelle ──────────────────────────────────────────
|
||||
DRIFT_THRESHOLD: ${DRIFT_THRESHOLD:-0.30}
|
||||
|
||||
# ── A2 — Seuil adaptatif ──────────────────────────────────────────────
|
||||
ANOMALY_PERCENTILE: ${ANOMALY_PERCENTILE:-5}
|
||||
|
||||
# ── A3 — Analyse multi-fenêtres ───────────────────────────────────────
|
||||
ENABLE_MULTIWINDOW: ${ENABLE_MULTIWINDOW:-false}
|
||||
MULTIWINDOW_VIEW: ${MULTIWINDOW_VIEW:-view_ai_features_24h}
|
||||
|
||||
# ── A4 — Explainabilité SHAP ──────────────────────────────────────────
|
||||
ENABLE_SHAP: ${ENABLE_SHAP:-true}
|
||||
|
||||
# ── A5 — Déduplication inter-cycles avec TTL ──────────────────────────
|
||||
DEDUP_TTL_MIN: ${DEDUP_TTL_MIN:-60}
|
||||
|
||||
# ── A6 — Pondération du score par récurrence ──────────────────────────
|
||||
RECURRENCE_WEIGHT: ${RECURRENCE_WEIGHT:-0.005}
|
||||
|
||||
# ── A7 — Validation de complétude des features ────────────────────────
|
||||
MIN_VALID_FEATURE_RATIO: ${MIN_VALID_FEATURE_RATIO:-0.50}
|
||||
|
||||
# ── A8 — Clustering comportemental des anomalies ──────────────────────
|
||||
ENABLE_CLUSTERING: ${ENABLE_CLUSTERING:-true}
|
||||
CLUSTERING_MIN_SAMPLES: ${CLUSTERING_MIN_SAMPLES:-3}
|
||||
|
||||
# ── Health check ──────────────────────────────────────────────────────
|
||||
HEALTH_PORT: ${HEALTH_PORT:-8080}
|
||||
|
||||
volumes:
|
||||
# Logs structurés JSONL (analyse a posteriori)
|
||||
- ./bot_detector_logs:/var/log/bot_detector
|
||||
|
||||
# Modèles Isolation Forest sérialisés (joblib)
|
||||
- ./bot_detector_models:/var/lib/bot_detector
|
||||
|
||||
# Fichiers CSV de réputation partagés avec ClickHouse (FILE engine)
|
||||
# Montés en read-only côté bot_detector (écriture via ClickHouse uniquement)
|
||||
- ./reputation/data/user_files/bot_ip.csv:/data/bot_ip.csv:ro
|
||||
- ./reputation/data/user_files/bot_ja4.csv:/data/bot_ja4.csv:ro
|
||||
- ./reputation/data/user_files/asn_reputation.csv:/data/asn_reputation.csv:ro
|
||||
36
services/bot-detector/reputation/asn_reputation.csv
Normal file
36
services/bot-detector/reputation/asn_reputation.csv
Normal file
@ -0,0 +1,36 @@
|
||||
3215,human
|
||||
12322,human
|
||||
5410,human
|
||||
15557,human
|
||||
21502,human
|
||||
9036,human
|
||||
8218,human
|
||||
39180,human
|
||||
3303,human
|
||||
6730,human
|
||||
9044,human
|
||||
15600,human
|
||||
13030,human
|
||||
25256,human
|
||||
5432,human
|
||||
6848,human
|
||||
12392,human
|
||||
49686,human
|
||||
6714,human
|
||||
49203,human
|
||||
6661,human
|
||||
8469,human
|
||||
20676,human
|
||||
3320,human
|
||||
3209,human
|
||||
8881,human
|
||||
6805,human
|
||||
29562,human
|
||||
31334,human
|
||||
8422,human
|
||||
25255,human
|
||||
8447,human
|
||||
12635,human
|
||||
6830,human
|
||||
8412,human
|
||||
35369,human
|
||||
|
288598
services/bot-detector/reputation/bot_ip.csv
Normal file
288598
services/bot-detector/reputation/bot_ip.csv
Normal file
File diff suppressed because it is too large
Load Diff
0
services/bot-detector/reputation/bot_ja4.csv
Normal file
0
services/bot-detector/reputation/bot_ja4.csv
Normal file
|
|
1267296
services/bot-detector/reputation/iplocate-ip-to-asn.csv
Normal file
1267296
services/bot-detector/reputation/iplocate-ip-to-asn.csv
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user