feat: full-stack Docker Compose integration tests
- 4-container stack: ClickHouse, platform (Rocky 9), bot-detector, dashboard - Platform builds sentinel on Rocky (CGO+libpcap native), correlator static - mod-reqin-log compiled with apxs on Rocky (matching RPM build target) - ClickHouse init script patches credentials for test env (sed-based) - 8-phase test runner: schema, traffic gen, pipeline, dashboard API, bot-detector, sentinel - All 13 checks pass, 3 non-blocking warnings (empty dicts, log paths) SQL schema fixes discovered during integration: - 02_dictionaries: IPv6CIDR → String (not a valid ClickHouse type) - 03_anubis_tables: dict_anubis_ua missing has_ip/rule_id/category attrs - 03_anubis_tables: dict_anubis_country FLAT() → COMPLEX_KEY_HASHED() (String key) - 09_audit_table: CODEC before DEFAULT → DEFAULT before CODEC Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
98
tests/integration/README.md
Normal file
98
tests/integration/README.md
Normal file
@ -0,0 +1,98 @@
|
||||
# Tests d'intégration full-stack — ja4-platform
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ platform (Rocky Linux 9) │
|
||||
│ │
|
||||
│ ┌──────────┐ http.socket ┌────────────┐ │
|
||||
│ │ Apache │───────────────→│ │ │
|
||||
│ │+ mod-reqin│ │ correlator │──→ ClickHouse
|
||||
│ └──────────┘ │ │ │
|
||||
│ ┌──────────┐ network.socket │ │ │
|
||||
│ │ sentinel │───────────────→│ │ │
|
||||
│ │(TLS pcap) │ └────────────┘ │
|
||||
│ └──────────┘ │
|
||||
│ cap_add: NET_RAW, NET_ADMIN │
|
||||
└─────────────────────────────────────────────────────┘
|
||||
↑ HTTPS │
|
||||
test traffic ja4_logs.http_logs_raw
|
||||
↓
|
||||
┌──────────────────┐
|
||||
│ ClickHouse │
|
||||
│ ja4_logs │
|
||||
│ ja4_processing │
|
||||
└──────────────────┘
|
||||
↑ ↑
|
||||
┌──────┘ └──────┐
|
||||
┌──────────────┐ ┌──────────────┐
|
||||
│ bot-detector │ │ dashboard │
|
||||
│ (ML/Python) │ │ (FastAPI) │
|
||||
└──────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## Utilisation
|
||||
|
||||
```bash
|
||||
# Lancer les tests (build + start + test + teardown)
|
||||
./run-tests.sh
|
||||
|
||||
# Garder le stack actif après les tests (debug)
|
||||
./run-tests.sh --no-down
|
||||
|
||||
# Build uniquement (pas de tests)
|
||||
./run-tests.sh --build-only
|
||||
|
||||
# Ou depuis la racine du monorepo :
|
||||
make test-integration
|
||||
```
|
||||
|
||||
## Conteneurs
|
||||
|
||||
| Conteneur | Image | Rôle |
|
||||
|-----------|-------|------|
|
||||
| `clickhouse` | clickhouse/clickhouse-server:24.8 | Base de données, schema auto-init |
|
||||
| `platform` | Rocky Linux 9 (build custom) | Apache HTTPS + mod-reqin-log + sentinel + correlator |
|
||||
| `bot-detector` | Python 3.11 | Détection d'anomalies ML |
|
||||
| `dashboard` | Python 3.11 / FastAPI | API SOC |
|
||||
|
||||
## Capabilities réseau
|
||||
|
||||
Le conteneur `platform` a besoin de :
|
||||
- `NET_RAW` — pour la capture de paquets réseau (sentinel/pcap)
|
||||
- `NET_ADMIN` — pour la configuration de l'interface réseau
|
||||
|
||||
Ces capabilities sont déclarées dans `docker-compose.yml` :
|
||||
```yaml
|
||||
platform:
|
||||
cap_add:
|
||||
- NET_RAW
|
||||
- NET_ADMIN
|
||||
```
|
||||
|
||||
## Phases de test
|
||||
|
||||
1. **Schema ClickHouse** — vérifie les 2 bases, tables clés, utilisateurs
|
||||
2. **Génération de trafic** — 50+ requêtes HTTPS vers Apache
|
||||
3. **Pipeline de données** — vérifie les logs bruts et parsés dans ClickHouse
|
||||
4. **Dashboard API** — vérifie /health et /api/metrics
|
||||
5. **Bot-detector** — vérifie que le processus tourne
|
||||
6. **Sentinel** — vérifie la capture réseau
|
||||
|
||||
## Debug
|
||||
|
||||
```bash
|
||||
# Logs du platform (Apache + correlator + sentinel)
|
||||
docker compose logs platform
|
||||
|
||||
# Logs corrélés
|
||||
docker compose exec platform cat /var/log/logcorrelator/correlated.log
|
||||
|
||||
# Requête ClickHouse directe
|
||||
docker compose exec clickhouse clickhouse-client \
|
||||
-q "SELECT time, src_ip, method, host, path FROM ja4_logs.http_logs ORDER BY time DESC LIMIT 10"
|
||||
|
||||
# Shell dans le platform
|
||||
docker compose exec platform bash
|
||||
```
|
||||
Reference in New Issue
Block a user