feat(ebpf): Apache HTTP capture + nginx multi-kernel validation
**Apache HTTP capture via apr_socket_recv** : - Uprobe sur libapr-1.so.0 (Apache Portable Runtime) - Compatible tous kernels 4.18+ (CentOS 8, Rocky 9/10) - Configuration unifiée : servers: ["nginx", "apache"] **nginx HTTP capture validation multi-kernel** : - Kretprobe __x64_sys_recvfrom validé sur CentOS 8 (4.18) - Rocky 9 (5.14) et Rocky 10 (6.12) confirmés - Contourne limitation tracepoint sys_exit_recvfrom **Documentation** : - docs/TEST_BUILD_STACK.md : stack complète test/build (VMs, Docker, RPMs) - services/ja4ebpf/docs/APACHE_HTTP_VALIDATION.md : validation Apache - services/ja4ebpf/docs/NGINX_MULTI_KERNEL_VALIDATION.md : validation nginx - docs/architecture.md + docs/services/ja4ebpf.md mis à jour **Tests unitaires Apache** : - internal/loader/apache_test.go : tests libapr, paths, structures BPF - internal/correlation/apache_test.go : tests corrélation HTTP Apache **Packaging** : - RPM spec mis à jour (version 0.3.0-1, changelog complet) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Jacquin Antoine
399
docs/TEST_BUILD_STACK.md
Normal file
399
docs/TEST_BUILD_STACK.md
Normal file
@ -0,0 +1,399 @@
|
||||
# Stack de Test et Build — ja4-platform
|
||||
|
||||
## Vue d'ensemble
|
||||
|
||||
La plateforme ja4 utilise une infrastructure de tests multi-niveaux :
|
||||
- **Docker** : Build et tests unitaires
|
||||
- **VM Vagrant** : Tests d'intégration eBPF sur kernel réel
|
||||
- **RPMs** : Packaging multi-distro (el8/el9/el10)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Host Ubuntu (libvirt/KVM) │
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────┐│
|
||||
│ │ centos8 VM │ │ rocky9 VM │ │ rocky10 VM ││
|
||||
│ │ (kernel 4.18) │ │ (kernel 5.14) │ │ (kernel 6.12) ││
|
||||
│ │ IP: DHCP │ │ IP: DHCP │ │ IP: DHCP ││
|
||||
│ └──────────────────┘ └──────────────────┘ └─────────────────┘│
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ │
|
||||
│ │ traffic VM │ │ analysis VM │ │
|
||||
│ │ (curl-impersonate)│ │ (ClickHouse + ML)│ │
|
||||
│ │ IP: DHCP │ │ IP: 192.168.42.10│ │
|
||||
│ └──────────────────┘ └──────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Docker (build/tests) │ │
|
||||
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
|
||||
│ │ │ ja4ebpf │ │bot-detector│ │ dashboard │ │ │
|
||||
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Réseau privé
|
||||
|
||||
**Nom du réseau** : `ja4-e2e` (libvirt)
|
||||
**Plage d'adresses** : 192.168.42.0/24 (DHCP)
|
||||
**Routeur** : Libvirt NAT vers host
|
||||
|
||||
| VM | IP | Rôle |
|
||||
|----|----|----|
|
||||
| analysis | 192.168.42.10 | ClickHouse + bot-detector + dashboard |
|
||||
| centos8 | DHCP (192.168.42.x) | Tests eBPF kernel 4.18 |
|
||||
| rocky9 | DHCP (192.168.42.x) | Tests eBPF kernel 5.14 (défaut) |
|
||||
| rocky10 | DHCP (192.168.42.x) | Tests eBPF kernel 6.12 |
|
||||
| traffic | DHCP (192.168.42.x) | Générateur de trafic |
|
||||
|
||||
## VMs de Test
|
||||
|
||||
### 1. centos8 (CentOS 8 Stream)
|
||||
|
||||
**Box Vagrant** : `centos/8`
|
||||
**Kernel** : 4.18.0-240.el8_3.x86_64
|
||||
**Mémoire** : 2 Go RAM, 2 CPU
|
||||
**Rôle** : Tests eBPF sur kernel 4.18 (RHEL 8 minimal)
|
||||
|
||||
**Services installés** :
|
||||
- nginx (port 80/443)
|
||||
- Apache httpd (port 8080/8443)
|
||||
- ja4ebpf (service systemd)
|
||||
- ClickHouse (Docker)
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
make vm-up # Démarrer centos8
|
||||
vagrant ssh centos8 # Connexion SSH
|
||||
make test-vm-centos8 # Tests complets
|
||||
```
|
||||
|
||||
### 2. rocky9 (Rocky Linux 9) — VM par défaut
|
||||
|
||||
**Box Vagrant** : `generic/rocky9`
|
||||
**Kernel** : 5.14.0-427.el9.x86_64
|
||||
**Mémoire** : 2 Go RAM, 2 CPU
|
||||
**Rôle** : Tests eBPF principaux (défaut)
|
||||
|
||||
**Services installés** :
|
||||
- nginx (port 80/443)
|
||||
- Apache httpd (port 8080/8443)
|
||||
- Varnish (port 6081)
|
||||
- Hitch (port 8443)
|
||||
- ja4ebpf (service systemd)
|
||||
- ClickHouse (Docker)
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
make vm-up # Démarrer rocky9
|
||||
make vm-ssh # Connexion SSH
|
||||
make test-vm-nginx # Test nginx
|
||||
make test-vm-apache # Test Apache
|
||||
make test-vm-all # Tous les tests
|
||||
```
|
||||
|
||||
### 3. rocky10 (Rocky Linux 10 / AlmaLinux 10)
|
||||
|
||||
**Box Vagrant** : `almalinux/10`
|
||||
**Kernel** : 6.12.0-124.el10_1.x86_64
|
||||
**Mémoire** : 2 Go RAM, 2 CPU
|
||||
**Rôle** : Tests eBPF sur kernel récent (6.12+)
|
||||
|
||||
**Services installés** :
|
||||
- nginx (port 80/443)
|
||||
- Apache httpd (port 8080/8443)
|
||||
- ja4ebpf (service systemd)
|
||||
- ClickHouse (Docker)
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
vagrant up rocky10 # Démarrer rocky10
|
||||
vagrant ssh rocky10 # Connexion SSH
|
||||
make test-vm-rocky10 # Tests complets
|
||||
```
|
||||
|
||||
### 4. analysis (VM centralisée)
|
||||
|
||||
**Box Vagrant** : `generic/rocky9`
|
||||
**IP fixe** : 192.168.42.10
|
||||
**Mémoire** : 8 Go RAM (torch + isotree build), 2 CPU
|
||||
**Rôle** : Serveur central pour tests E2E distribués
|
||||
|
||||
**Services Docker** :
|
||||
- **ClickHouse** (ports 9000, 8123) : Base de données temps réel
|
||||
- **bot-detector** (port 8080) : ML Python pour détection de bots
|
||||
- **dashboard** (port 8000) : Interface web Flask
|
||||
|
||||
**Réception des logs** :
|
||||
- Les VMs centos8/rocky9/rocky10 envoient leurs logs ja4ebpf vers 192.168.42.10:9000
|
||||
- ClickHouse agrège les données de tous les endpoints
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
make e2e-up # Démarrer analysis + endpoints
|
||||
make test-e2e # Test E2E complet
|
||||
make test-e2e-quick # Test E2E rapide
|
||||
```
|
||||
|
||||
### 5. traffic (Générateur de trafic)
|
||||
|
||||
**Box Vagrant** : `generic/rocky9`
|
||||
**Mémoire** : 1 Go RAM, 2 CPU
|
||||
**Rôle** : Génération de trafic réaliste vers endpoints
|
||||
|
||||
**Outils installés** :
|
||||
- curl-impersonate : TLS fingerprints réalistes (Chrome, Firefox, Safari...)
|
||||
- httpx : HTTP/2, HTTP/3, HEADERS optimisés
|
||||
- Scripts de génération de trafic
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
vagrant up traffic # Démarrer traffic
|
||||
vagrant ssh traffic # Connexion SSH
|
||||
./generate-traffic.sh # Générer du trafic vers endpoints
|
||||
```
|
||||
|
||||
## Build Docker
|
||||
|
||||
### Image de production ja4ebpf
|
||||
|
||||
**Dockerfile** : `services/ja4ebpf/Dockerfile`
|
||||
**Base image** : `alpine:3.19` (dynamique)
|
||||
**Builder** : `rockylinux:9` (compilation eBPF)
|
||||
|
||||
**Étapes de build** :
|
||||
```bash
|
||||
# 1. Génération bytecode eBPF (requiert clang + bpf2go)
|
||||
make generate
|
||||
|
||||
# 2. Build image Docker
|
||||
make build
|
||||
|
||||
# Sortie : antitbone/ja4ebpf:dev
|
||||
```
|
||||
|
||||
**Variables de build** :
|
||||
- `BUILD_VERSION` : Version du binaire (défaut: dev)
|
||||
- `GO_VERSION` : Version Go (défaut: 1.24.3)
|
||||
|
||||
### Image de tests
|
||||
|
||||
**Dockerfile** : `services/ja4ebpf/Dockerfile.tests`
|
||||
**Base image** : `rockylinux:9` (clang + Go)
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
make test # Lance les tests unitaires Go
|
||||
```
|
||||
|
||||
### Image multi-distro RPM
|
||||
|
||||
**Dockerfile** : `services/ja4ebpf/Dockerfile.package`
|
||||
**Cible** : RPMs el8, el9, el10
|
||||
|
||||
**Stages** :
|
||||
1. **go-builder** : Rocky Linux 9 + clang + Go
|
||||
2. **rpm-el8** : AlmaLinux 8 + rpmbuild
|
||||
3. **rpm-el9** : Rocky Linux 9 + rpmbuild
|
||||
4. **rpm-el10** : AlmaLinux 10 + rpmbuild
|
||||
5. **output** : Collecte tous les RPMs
|
||||
|
||||
**Commandes** :
|
||||
```bash
|
||||
make rpm-ja4ebpf # Construit les 3 RPMs
|
||||
make dist # Copie RPMs dans dist/
|
||||
|
||||
# Sortie :
|
||||
# services/ja4ebpf/dist/el8/ja4ebpf-0.3.0-1.el8.x86_64.rpm
|
||||
# services/ja4ebpf/dist/el9/ja4ebpf-0.3.0-1.el9.x86_64.rpm
|
||||
# services/ja4ebpf/dist/el10/ja4ebpf-0.3.0-1.el10.x86_64.rpm
|
||||
```
|
||||
|
||||
## Services Docker (VM analysis)
|
||||
|
||||
### Stack complète
|
||||
|
||||
**Fichier** : `tests/vm/analysis/docker-compose.yml`
|
||||
|
||||
**Services** :
|
||||
|
||||
#### 1. ClickHouse
|
||||
|
||||
- **Image** : `clickhouse/clickhouse-server:24.8`
|
||||
- **Ports** : 9000 (native), 8123 (HTTP)
|
||||
- **Volumes** :
|
||||
- Schéma SQL (12 fichiers)
|
||||
- Données CSV (dictionnaires, browser_h2)
|
||||
- Scripts d'initialisation
|
||||
- **Healthcheck** : `clickhouse-client --query "SELECT 1"`
|
||||
|
||||
#### 2. bot-detector
|
||||
|
||||
- **Build** : `services/bot-detector/bot_detector/Dockerfile`
|
||||
- **Port** : 8080
|
||||
- **Dépendances** : ClickHouse
|
||||
- **Variables** :
|
||||
- Cycle accéléré : 30s (production: 300s)
|
||||
- ML : isolation_forest, XGBoost
|
||||
- Logs : JSONL dans volume
|
||||
|
||||
#### 3. dashboard
|
||||
|
||||
- **Build** : `services/dashboard/Dockerfile`
|
||||
- **Port** : 8000
|
||||
- **Dépendances** : ClickHouse
|
||||
- **Framework** : Flask + Plotly
|
||||
|
||||
## Commandes principales
|
||||
|
||||
### VMs (Vagrant)
|
||||
|
||||
```bash
|
||||
# Création
|
||||
make vm-up # rocky9 seulement (défaut)
|
||||
make vm-up-all # centos8 + rocky9 + rocky10
|
||||
|
||||
# Gestion
|
||||
make vm-ssh # Connexion rocky9
|
||||
vagrant ssh centos8 # Connexion centos8
|
||||
make vm-down # Détruire rocky9
|
||||
make vm-down-all # Détruire toutes les VMs
|
||||
|
||||
# Provisionnement
|
||||
make vm-reprovision # Re-provisionner toutes les VMs
|
||||
vagrant provision centos8 # Re-provisionner centos8
|
||||
```
|
||||
|
||||
### Tests VM
|
||||
|
||||
```bash
|
||||
# Tests simples (host → VM)
|
||||
make test-vm-nginx # nginx sur rocky9
|
||||
make test-vm-apache # Apache sur rocky9
|
||||
make test-vm-hitch-varnish # hitch+varnish sur rocky9
|
||||
make test-vm-all # Tous les stacks sur rocky9
|
||||
|
||||
# Tests multi-distros
|
||||
make test-vm-centos8 # Tous les stacks sur centos8
|
||||
make test-vm-rocky10 # Tous les stacks sur rocky10
|
||||
make test-vm-matrix # 3 stacks × 3 distros (9 tests)
|
||||
|
||||
# Tests unitaires Go dans VMs
|
||||
make test-vm-all-distros # Tests Go sur centos8 + rocky9 + rocky10
|
||||
```
|
||||
|
||||
### Build et Packaging
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
make generate # Génération bytecode eBPF
|
||||
make build # Build image Docker
|
||||
make test # Tests unitaires Docker
|
||||
|
||||
# RPMs
|
||||
make rpm-ja4ebpf # RPMs el8 + el9 + el10
|
||||
make dist # Copie RPMs → dist/
|
||||
```
|
||||
|
||||
### Tests E2E
|
||||
|
||||
```bash
|
||||
make e2e-up # Démarrer analysis + endpoints
|
||||
make test-e2e # Test complet (trafic → capture → ML → dashboard)
|
||||
make test-e2e-quick # Test rapide (1 cycle ML)
|
||||
make e2e-down # Arrêter analysis + endpoints
|
||||
```
|
||||
|
||||
## Flux de test typique
|
||||
|
||||
### Développement itératif
|
||||
|
||||
```bash
|
||||
# 1. Modifier code Go/C sur host
|
||||
vim services/ja4ebpf/internal/loader/loader.go
|
||||
|
||||
# 2. Synchroniser + recompiler dans VM
|
||||
make vm-rebuild-ja4ebpf
|
||||
|
||||
# 3. Lancer les tests
|
||||
make test-vm-nginx
|
||||
|
||||
# 4. Vérifier les logs
|
||||
vagrant ssh -- -t 'sudo journalctl -u ja4ebpf -n 50'
|
||||
```
|
||||
|
||||
### Test multi-kernel
|
||||
|
||||
```bash
|
||||
# 1. Démarrer toutes les VMs
|
||||
make vm-up-all
|
||||
|
||||
# 2. Lancer la matrice de tests
|
||||
make test-vm-matrix
|
||||
|
||||
# Résultat attendu :
|
||||
# - centos8: nginx ✅ apache ✅ hitch-varnish ✅
|
||||
# - rocky9: nginx ✅ apache ✅ hitch-varnish ✅
|
||||
# - rocky10: nginx ✅ apache ✅ hitch-varnish ✅
|
||||
```
|
||||
|
||||
### Build RPM pour production
|
||||
|
||||
```bash
|
||||
# 1. Builder les RPMs multi-distro
|
||||
make rpm-ja4ebpf
|
||||
|
||||
# 2. Vérifier les RPMs créés
|
||||
find services/ja4ebpf/dist -name '*.rpm'
|
||||
|
||||
# 3. Tester un RPM sur VM
|
||||
vagrant ssh rocky9 -- 'sudo rpm -Uvh /vagrant/dist/el9/ja4ebpf-*.rpm'
|
||||
```
|
||||
|
||||
## Dépannage
|
||||
|
||||
### VM ne démarre pas
|
||||
|
||||
```bash
|
||||
# Vérifier libvirt
|
||||
sudo systemctl status libvirtd
|
||||
virsh list --all
|
||||
|
||||
# Vérifier réseau
|
||||
virsh net-list --all
|
||||
virsh net-info ja4-e2e
|
||||
```
|
||||
|
||||
### Tests échouent
|
||||
|
||||
```bash
|
||||
# Vérifier que ja4ebpf tourne
|
||||
vagrant ssh -- 'systemctl status ja4ebpf'
|
||||
|
||||
# Vérifier les logs
|
||||
vagrant ssh -- 'sudo journalctl -u ja4ebpf -n 100 -f'
|
||||
|
||||
# Vérifier ClickHouse
|
||||
vagrant ssh -- 'docker ps | grep clickhouse'
|
||||
vagrant ssh -- 'docker exec clickhouse clickhouse-client --query "SELECT 1"'
|
||||
```
|
||||
|
||||
### Build échoue
|
||||
|
||||
```bash
|
||||
# Vérifier Docker
|
||||
docker ps
|
||||
docker images
|
||||
|
||||
# Nettoyer et recommencer
|
||||
docker system prune -a
|
||||
make generate
|
||||
```
|
||||
|
||||
## Références
|
||||
|
||||
- **Vagrantfile** : `tests/vm/Vagrantfile`
|
||||
- **Docker Compose analysis** : `tests/vm/analysis/docker-compose.yml`
|
||||
- **RPM spec** : `services/ja4ebpf/packaging/rpm/ja4ebpf.spec`
|
||||
- **Makefile principal** : `Makefile`
|
||||
- **Scripts de test** : `tests/vm/*.sh`
|
||||
Reference in New Issue
Block a user