85d3b95b7b
Complete implementation of HTTP/2 passive fingerprinting per thesis §2.5.3: mod-reqin-log (C module): - Replace connection-level filter with ap_hook_process_connection (APR_HOOK_FIRST) to capture H2 preface before mod_http2 takes over the connection - AP_MODE_SPECULATIVE read of 512 bytes from c->input_filters - Parse SETTINGS, WINDOW_UPDATE, PRIORITY flags, pseudo-header order - Output individual SETTINGS params as separate JSON fields (IDs 1-6, 8) - Read H2 notes from c1 (master connection) for mod_http2 secondary conns - Fix header_order_signature JSON length bug (26→strlen) ClickHouse schema: - Add 8 new columns to http_logs: h2_has_priority, h2_header_table_size, h2_enable_push, h2_max_concurrent_streams, h2_initial_window_size, h2_max_frame_size, h2_max_header_list_size, h2_enable_connect_protocol - Use Int32/Int64 with DEFAULT -1 to distinguish absent vs zero - Update mv_http_logs to extract individual fields via JSONHas/JSONExtractInt - Migration 04_http2_fields.sql updated for existing deployments Correlator: - Accept both timestamp_ns and timestamp field names (backward compat) Integration: - Enable HTTP/2 in Apache: Protocols h2 http/1.1 in httpd-integration.conf Validated end-to-end via Playwright: H2 curl traffic → mod-reqin-log → correlator → ClickHouse with all 12 H2 columns populated correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tests d'intégration full-stack — ja4-platform
Architecture
┌─────────────────────────────────────────────────────┐
│ platform (Rocky Linux 9) │
│ │
│ ┌──────────┐ http.socket ┌────────────┐ │
│ │ Apache │───────────────→│ │ │
│ │+ mod-reqin│ │ correlator │──→ ClickHouse
│ └──────────┘ │ │ │
│ ┌──────────┐ network.socket │ │ │
│ │ sentinel │───────────────→│ │ │
│ │(TLS pcap) │ └────────────┘ │
│ └──────────┘ │
│ cap_add: NET_RAW, NET_ADMIN │
└─────────────────────────────────────────────────────┘
↑ HTTPS │
test traffic ja4_logs.http_logs_raw
↓
┌──────────────────┐
│ ClickHouse │
│ ja4_logs │
│ ja4_processing │
└──────────────────┘
↑ ↑
┌──────┘ └──────┐
┌──────────────┐ ┌──────────────┐
│ bot-detector │ │ dashboard │
│ (ML/Python) │ │ (FastAPI) │
└──────────────┘ └──────────────┘
Utilisation
# Lancer les tests (build + start + test + teardown)
./run-tests.sh
# Garder le stack actif après les tests (debug)
./run-tests.sh --no-down
# Build uniquement (pas de tests)
./run-tests.sh --build-only
# Ou depuis la racine du monorepo :
make test-integration
Conteneurs
| Conteneur | Image | Rôle |
|---|---|---|
clickhouse |
clickhouse/clickhouse-server:24.8 | Base de données, schema auto-init |
platform |
Rocky Linux 9 (build custom) | Apache HTTPS + mod-reqin-log + sentinel + correlator |
bot-detector |
Python 3.11 | Détection d'anomalies ML |
dashboard |
Python 3.11 / FastAPI | API SOC |
Capabilities réseau
Le conteneur platform a besoin de :
NET_RAW— pour la capture de paquets réseau (sentinel/pcap)NET_ADMIN— pour la configuration de l'interface réseau
Ces capabilities sont déclarées dans docker-compose.yml :
platform:
cap_add:
- NET_RAW
- NET_ADMIN
Phases de test
- Schema ClickHouse — vérifie les 2 bases, tables clés, utilisateurs
- Génération de trafic — 50+ requêtes HTTPS vers Apache
- Pipeline de données — vérifie les logs bruts et parsés dans ClickHouse
- Dashboard API — vérifie /health et /api/metrics
- Bot-detector — vérifie que le processus tourne
- Sentinel — vérifie la capture réseau
Debug
# Logs du platform (Apache + correlator + sentinel)
docker compose logs platform
# Logs corrélés
docker compose exec platform cat /var/log/logcorrelator/correlated.log
# Requête ClickHouse directe
docker compose exec clickhouse clickhouse-client \
-q "SELECT time, src_ip, method, host, path FROM ja4_logs.http_logs ORDER BY time DESC LIMIT 10"
# Shell dans le platform
docker compose exec platform bash