d75825278e
feat: multi-distro VM tests, ja4ebpf eBPF improvements, bot-detector scoring
...
ja4ebpf:
- Refactor BPF TC capture with improved SYN offset handling and TCP option parsing
- Enhance TLS uprobe SSL hooking for better key extraction
- Add ClickHouse writer improvements for HTTP log materialized views
- Update RPM spec for Rocky Linux 8/9/10, fix systemd service
- Simplify loader with cleaner bpf2go integration
bot-detector:
- Add H2 SETTINGS per-parameter comparison in browser_matcher
- Enhance browser signatures and scoring pipeline
- Improve preprocessing and cycle detection
infra:
- Multi-distro Vagrantfile (centos8, rocky9, rocky10) with per-distro provisioning
- New Makefile targets: vm-up-all, test-vm-matrix, test-vm-centos8/rocky10
- Add debug helpers and run-test-from-host.sh for host-driven VM testing
- Update run-tests-vm.sh for cross-distro compatibility
- Remove accidental binary blob (\004)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com >
2026-04-13 01:09:33 +02:00
957918c565
fix(ja4ebpf): Rocky Linux RPM builder, remove correlated field, fix thesis
...
- Dockerfile.package: migre go-builder de golang:bookworm (Debian) vers
rockylinux:9, installe Go depuis le tarball officiel, remplace apt par
dnf (clang llvm libbpf-devel bpftool)
- Suppression du champ 'correlated' de l'agent ja4ebpf : avec eBPF/XDP,
la corrélation L3/L4↔L7 est toujours implicite par présence des champs.
Supprimé de : session.go, manager.go, main.go (x5), clickhouse.go
- Thèse (6 corrections listées + cohérence correlated) :
1. §3.5 + §3.9.1 : SSL_read retourne des octets bruts sans respecter les
frontières H2 → buffer circulaire de réassemblage en Go userspace
2. §3.1 : supprimé libpcap + CAP_NET_RAW, remplacé par définition uprobe
3. §4 + §7 : compte exact 96 features en 8 familles (Famille 1–8),
supprimé taxonomie F1–F11 obsolète, tous les totaux mis à jour
4. §2.4 + §8 : remplacé 7 fausses URLs arXiv par [Référence à vérifier]
5. §4 Famille 2 : ja4_drift_ratio → renvoi à Famille 8 (définition complète)
6. §6.4 : ajouté limite 'Overhead de l'uprobe SSL_read'
+ §3.6 : supprimé correlated=0/1 du texte architectural
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-04-12 04:48:40 +02:00
9734e21fe3
chore: suppression des services obsolètes (sentinel, correlator, mod-reqin-log)
...
Remplacés par l'agent ja4ebpf (eBPF CO-RE). Nettoyage complet :
Supprimé :
- old/ (archive de l'ancienne architecture)
- services/correlator/ (logcorrelator Go)
- services/sentinel/ (capture pcap Go)
- services/mod-reqin-log/ (module Apache C)
- shared/go/ja4common/ (lib Go partagée — plus importée par ja4ebpf)
- tests/integration/platform/ (test correlator+sentinel+httpd)
- tests/integration/docker-compose.yml (compose ancienne archi)
- tests/integration/run-tests.sh (runner correlator/sentinel)
- tests/integration/verify_mvs.py (script orphelin)
Nettoyé :
- go.work : retire ./shared/go/ja4common
- services/ja4ebpf/go.mod : retire replace ja4common (jamais importé)
- services/ja4ebpf/Dockerfile* : retire les COPY ja4common inutiles
- Makefile : retire test-ja4common-python, test-integration*, targets obsolètes
- tests/integration/README.md : réécrit pour l'architecture ja4ebpf
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-04-12 01:48:14 +02:00
d469e39da7
feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
...
Services:
- ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap)
- logcorrelator: JA4 log correlation engine (Go, ClickHouse)
- mod_reqin_log: Apache module (C, JSON request logging)
- bot_detector: ML bot detection pipeline (Python)
- dashboard: FastAPI/Streamlit analytics UI (Python)
Shared libraries:
- shared/go/ja4common: logger, config, shutdown, ipfilter (Go module)
- shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package)
- shared/clickhouse/: canonical SQL migrations (10 files)
Build & packaging:
- Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10)
- go.work workspace linking sentinel, correlator, ja4common
- Makefile with test-all, build-all, rpm-* targets
Fixes applied:
- go.work: 1.21 → 1.24.6 (required by sentinel)
- correlator Dockerfiles: golang:1.21 → golang:1.24
- replace directives in go.mod for ja4common local path
- pyproject.toml: setuptools.backends → setuptools.build_meta
- Removed static libpcap linking (unavailable on Rocky 9)
- Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32)
- Rewrote corrupted test files (logger_test.go × 2)
Test coverage:
- correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%)
- sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse)
Documentation:
- README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-04-07 16:42:59 +02:00
