f85a10b012
Correctifs pipeline L7 (uprobe SSL_read) :
- uprobe_ssl.c : ssl_set_fd ne retourne plus tôt quand fd_conn_map est
vide (accept4 non disponible en Docker). Sauvegarde ssl_ptr→{fd,0,0}
pour permettre le fallback /proc côté Go.
- main.go : consumeSSLEvents reécrit avec routeur magic-bytes complet :
* HTTP/2 preface → extraction SETTINGS + conversion correlation.HTTP2Settings
* HTTP/1.x requête → method, path, query, headers, header_order_sig
* HTTP/1.x réponse → status_code
* Fallback /proc/<tgid>/fd/<fd> quand src_ip=0 (accept4 absent)
- writer/clickhouse.go : export header_order_signature ajouté
Nouveaux packages :
- internal/parser/http1.go : parseur HTTP/1.x (IsHTTP1Request,
ParseHTTP1Request, IsHTTP1Response, ParseHTTP1Response)
- internal/parser/http1_test.go : 11 tests unitaires (28 total passent)
- internal/procutil/proc_lookup.go : résolution fd→IP via /proc avec cache
TTL 5s (FDCache). Supporte /proc/PID/net/tcp et tcp6, IPv4-mappé IPv6.
Infrastructure tests VM (tests/vm/) :
- Vagrantfile : VM Rocky Linux 9 KVM, 4 CPU / 4 GB RAM
- provision.sh : installation toolchain eBPF + Go + Docker + nginx
- run-tests-vm.sh : suite de test complète dans la VM (L3/L4+TLS+L7)
- README.md : guide d'installation et d'utilisation
- Makefile : cibles vm-up, vm-down, vm-ssh, test-vm-nginx, test-vm-all,
vm-rebuild-ja4ebpf
Corrections stack Docker :
- Dockerfiles nginx/apache/nginx-varnish/hitch-varnish : suppression des
références à shared/go/ja4common/ (répertoire supprimé)
- clickhouse-init.sh : restauré depuis git, seed anubis_ua_rules obsolète
supprimé (table REGEXP_TREE supprimée du schéma)
- traffic-gen : ajout HTTP/1.0 (http.client) et HTTP/2 (httpx)
- verify_db.py : script de vérification 35 checks (L3/L4/TLS/L7/corrélation)
- run-stack-tests.sh : phase 6 verify_db ajoutée
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
214 lines
8.3 KiB
Makefile
214 lines
8.3 KiB
Makefile
# =============================================================================
|
||
# ja4-platform — Monorepo Makefile
|
||
# Service de capture : ja4ebpf (eBPF CO-RE)
|
||
# =============================================================================
|
||
|
||
VERSION ?= $(shell git describe --tags --always 2>/dev/null || echo dev)
|
||
|
||
.PHONY: help \
|
||
build-all test-all rpm-all dist \
|
||
build-ja4ebpf test-ja4ebpf rpm-ja4ebpf \
|
||
build-bot-detector test-bot-detector \
|
||
build-dashboard test-dashboard \
|
||
test-all-stacks test-nginx test-nginx-varnish test-hitch-varnish test-apache \
|
||
test-matrix \
|
||
test-vm-nginx test-vm-all vm-up vm-down vm-ssh \
|
||
reload-prod-logs init-stack import-prod-data init-and-import \
|
||
purge-db
|
||
|
||
.DEFAULT_GOAL := help
|
||
|
||
help: ## Affiche cette aide
|
||
@echo ""
|
||
@echo " ja4-platform — Makefile"
|
||
@echo " ─────────────────────────────────────────────────────"
|
||
@echo ""
|
||
@echo " Build"
|
||
@echo " make build-all Construit toutes les images Docker"
|
||
@echo " make build-ja4ebpf Image ja4ebpf (agent eBPF CO-RE)"
|
||
@echo " make build-bot-detector Image bot-detector (détection ML)"
|
||
@echo " make build-dashboard Image dashboard (SOC UI)"
|
||
@echo ""
|
||
@echo " Tests unitaires"
|
||
@echo " make test-all Lance tous les tests unitaires"
|
||
@echo " make test-ja4ebpf Tests Go ja4ebpf"
|
||
@echo " make test-bot-detector Tests Python bot-detector"
|
||
@echo " make test-dashboard Tests Python dashboard"
|
||
@echo ""
|
||
@echo " Tests VM (eBPF sur kernel réel — nécessite 'make vm-up' d'abord)"
|
||
@echo " make vm-up Créer la VM Rocky Linux 9 (vagrant up)"
|
||
@echo " make vm-down Détruire la VM (vagrant destroy)"
|
||
@echo " make vm-ssh Connexion SSH à la VM"
|
||
@echo " make test-vm-nginx Test nginx dans la VM (L7 complet)"
|
||
@echo " make test-vm-all Tous les tests dans la VM"
|
||
@echo ""
|
||
@echo " Tests d'intégration (par stack, Docker — L3/L4/TLS uniquement)"
|
||
@echo " make test-all-stacks Toutes les stacks sur Rocky Linux 9"
|
||
@echo " make test-apache Stack Apache + ja4ebpf"
|
||
@echo " make test-nginx Stack nginx + ja4ebpf"
|
||
@echo " make test-nginx-varnish Stack nginx + Varnish + ja4ebpf"
|
||
@echo " make test-hitch-varnish Stack hitch + Varnish + ja4ebpf"
|
||
@echo ""
|
||
@echo " Matrice multi-distro"
|
||
@echo " make test-matrix Toutes stacks × el8/el9/el10"
|
||
@echo " make test-matrix MATRIX_STACKS=nginx,apache MATRIX_DISTROS=el9,el10"
|
||
@echo ""
|
||
@echo " RPM"
|
||
@echo " make rpm-all Construit tous les RPMs ja4ebpf (el8/el9/el10)"
|
||
@echo " make rpm-ja4ebpf RPMs ja4ebpf (el8, el9, el10)"
|
||
@echo " make dist Alias de rpm-all"
|
||
@echo ""
|
||
@echo " Base de données"
|
||
@echo " make init-stack Initialise ClickHouse (schéma complet)"
|
||
@echo " make import-prod-data Importe les données prod (date shift)"
|
||
@echo " make init-and-import init-stack + import-prod-data"
|
||
@echo " make reload-prod-logs Exporte prod → importe en dev"
|
||
@echo " make purge-db Supprime et recrée les bases ja4_*"
|
||
@echo ""
|
||
|
||
# ── Cibles agrégées ──────────────────────────────────────────────────────────
|
||
|
||
build-all: build-ja4ebpf build-bot-detector build-dashboard
|
||
@echo "All services built."
|
||
|
||
test-all: test-ja4ebpf test-bot-detector test-dashboard
|
||
@echo "All unit tests completed."
|
||
|
||
rpm-all: rpm-ja4ebpf
|
||
@echo "All RPMs built."
|
||
|
||
dist: rpm-all
|
||
@echo "RPMs disponibles dans services/ja4ebpf/dist/"
|
||
|
||
# ── ja4ebpf (agent eBPF CO-RE) ───────────────────────────────────────────────
|
||
|
||
build-ja4ebpf:
|
||
docker build \
|
||
-f services/ja4ebpf/Dockerfile \
|
||
--build-arg BUILD_VERSION=$(VERSION) \
|
||
-t ja4-platform/ja4ebpf:latest \
|
||
.
|
||
|
||
test-ja4ebpf:
|
||
docker build \
|
||
-f services/ja4ebpf/Dockerfile.tests \
|
||
-t ja4-platform/ja4ebpf-tests:latest \
|
||
.
|
||
docker run --rm ja4-platform/ja4ebpf-tests:latest
|
||
|
||
rpm-ja4ebpf:
|
||
# Build multi-distro : el8 (AlmaLinux 8) + el9 (Rocky 9) + el10 (AlmaLinux 10)
|
||
# Sortie : services/ja4ebpf/dist/el{8,9,10}/ja4ebpf-*.rpm
|
||
docker build \
|
||
-f services/ja4ebpf/Dockerfile.package \
|
||
--target output \
|
||
--output type=local,dest=services/ja4ebpf/dist \
|
||
--build-arg BUILD_VERSION=$(VERSION) \
|
||
.
|
||
@echo ""
|
||
@echo "RPMs produits :"
|
||
@find services/ja4ebpf/dist -name '*.rpm' | sort | sed 's/^/ /'
|
||
|
||
# ── bot-detector ─────────────────────────────────────────────────────────────
|
||
|
||
build-bot-detector:
|
||
docker build \
|
||
-f services/bot-detector/bot_detector/Dockerfile \
|
||
-t ja4-platform/bot-detector:latest \
|
||
.
|
||
|
||
test-bot-detector:
|
||
docker build \
|
||
-f services/bot-detector/bot_detector/Dockerfile.tests \
|
||
-t ja4-platform/bot-detector-tests:latest \
|
||
.
|
||
docker run --rm ja4-platform/bot-detector-tests:latest
|
||
|
||
# ── dashboard ────────────────────────────────────────────────────────────────
|
||
|
||
build-dashboard:
|
||
docker build \
|
||
-f services/dashboard/Dockerfile \
|
||
-t ja4-platform/dashboard:latest \
|
||
.
|
||
|
||
test-dashboard:
|
||
docker build \
|
||
-f services/dashboard/Dockerfile.tests \
|
||
-t ja4-platform/dashboard-tests:latest \
|
||
.
|
||
docker run --rm ja4-platform/dashboard-tests:latest
|
||
|
||
# ── Tests d'intégration par stack ────────────────────────────────────────────
|
||
|
||
test-all-stacks: ## Toutes les stacks sur la distro par défaut (Rocky Linux 9)
|
||
cd tests/integration && bash run-all-stacks.sh
|
||
|
||
test-apache:
|
||
cd tests/integration && bash apache/run-tests.sh
|
||
|
||
test-nginx:
|
||
cd tests/integration && bash nginx/run-tests.sh
|
||
|
||
test-nginx-varnish:
|
||
cd tests/integration && bash nginx-varnish/run-tests.sh
|
||
|
||
test-hitch-varnish:
|
||
cd tests/integration && bash hitch-varnish/run-tests.sh
|
||
|
||
# ── Tests VM (kernel réel, eBPF complet) ─────────────────────────────────────
|
||
|
||
# Répertoire Vagrantfile
|
||
VM_DIR := tests/vm
|
||
|
||
vm-up: ## Créer la VM Rocky Linux 9 pour les tests eBPF
|
||
cd $(VM_DIR) && vagrant up
|
||
|
||
vm-down: ## Détruire la VM
|
||
cd $(VM_DIR) && vagrant destroy -f
|
||
|
||
vm-ssh: ## Connexion SSH à la VM
|
||
cd $(VM_DIR) && vagrant ssh
|
||
|
||
vm-rebuild-ja4ebpf: ## Recompiler ja4ebpf dans la VM (après modifications)
|
||
cd $(VM_DIR) && vagrant rsync && vagrant ssh -- \
|
||
'export PATH=/usr/local/go/bin:$$PATH && \
|
||
cd /ja4-platform/services/ja4ebpf && \
|
||
GOWORK=off go generate ./internal/loader/ && \
|
||
GOWORK=off CGO_ENABLED=0 go build -o /usr/local/bin/ja4ebpf ./cmd/ja4ebpf/ && \
|
||
echo "ja4ebpf rebuilt OK"'
|
||
|
||
test-vm-nginx: ## Test nginx dans la VM (L3/L4/TLS/L7 HTTP complet)
|
||
@echo "=== Test VM nginx (kernel réel) ==="
|
||
cd $(VM_DIR) && vagrant rsync && vagrant ssh -- \
|
||
'sudo bash /ja4-platform/tests/vm/run-tests-vm.sh nginx'
|
||
|
||
test-vm-all: ## Tous les tests dans la VM
|
||
@echo "=== Tests VM (toutes stacks) ==="
|
||
cd $(VM_DIR) && vagrant rsync && vagrant ssh -- \
|
||
'sudo bash /ja4-platform/tests/vm/run-tests-vm.sh all'
|
||
|
||
# ── Matrice multi-distro ─────────────────────────────────────────────────────
|
||
|
||
test-matrix: ## Toutes stacks × el8 + el9 + el10
|
||
cd tests/integration && bash run-distro-matrix.sh \
|
||
$${MATRIX_STACKS:+--stacks=$${MATRIX_STACKS}} \
|
||
$${MATRIX_DISTROS:+--distros=$${MATRIX_DISTROS}}
|
||
|
||
# ── Base de données ───────────────────────────────────────────────────────────
|
||
|
||
reload-prod-logs:
|
||
./scripts/reload-prod-logs.sh
|
||
|
||
init-stack:
|
||
./scripts/init-stack.sh
|
||
|
||
import-prod-data:
|
||
./scripts/import-prod-data.sh
|
||
|
||
init-and-import:
|
||
./scripts/init-stack.sh --import-prod
|
||
|
||
purge-db:
|
||
./scripts/init-stack.sh --reset
|