code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
122 Commits 1 Branch 0 Tags
a02423fd18a390eaf266b26a5be1ec60dc0dff69
Commit Graph

5 Commits

Author SHA1 Message Date
Jacquin Antoine
a02423fd18 feat: maximize data completeness across L3/L4/TLS/HTTP layers and add E2E test infra 
Add SSL_write uprobe for HTTP response capture, HPACK decoder for HTTP/2
header extraction, and AcceptCache for reliable SSL/TC session correlation.
Populate all ClickHouse fields including tcp_meta_options, ip_meta_total_length,
syn_to_clienthello_ms, client_headers, TLS cipher suites/extensions, and
h2_enable_connect_protocol. Increase BPF capture buffers (HTTP 512B, TLS 1024B).
Add distributed E2E testing infrastructure with multi-VM Vagrant setup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-15 03:34:33 +02:00
Jacquin Antoine
61addc8cfa feat: JA3 fingerprinting, SSL correlation fix, ML pipeline overhaul, E2E test infra 
ja4ebpf:
- Add JA3 raw + MD5 hash fingerprinting (ComputeJA3 in TLS parser)
- Fix accept4 port double-swap bug (__builtin_bswap16 on already-host-order value)
- Fix scheme override bug in ClickHouse writer (HTTP block clearing HTTPS)
- Add HTTP/2 passive fingerprinting (Akamai H2 FP, SETTINGS, pseudo-header order)
- Enrich ClickHouse schema with IP/TCP metadata, H2 settings, Sec-* headers
- Ensure maximum data completeness: all available L3/L4, TLS, HTTP fields emitted

bot-detector:
- Replace logistic regression with MLP fusion classifier
- Replace KS drift detection with ADWIN online learning
- Replace NetworkX/Louvain with PyTorch Geometric GraphSAGE for fleet detection
- Replace autoencoder with RealNVP normalizing flow + SessionTransformer embeddings

infra:
- Add distributed E2E test infrastructure (4 VMs: endpoints + analysis)
- Add Vagrant provisioning for analysis VM, e2e Makefile targets, run scripts

docs:
- Restructure thesis into chapter files with corrected references
- Add E2E testing documentation
- Update architecture, schema, deployment, service docs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-15 02:57:07 +02:00
Jacquin Antoine
f88b739992 feat(e2e): add distributed E2E test framework with parametric traffic generation 
Add run-e2e-test.sh with CLI parameters (--hits, --http-ratio, --dns, --tls,
--src-ips, --keep-analysis, --up) for configurable traffic generation. Traffic
runs from VM endpoints with multiple source IPs (alias IPs on eth0) to produce
distinct sessions for the ML pipeline. Fix curl TLS flags (--tlsv1.2 instead
of --tls-v1-2), skip redundant local verification in distributed mode, and
fix dashboard is_available() cache that never retried after ClickHouse recovery.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-15 00:09:32 +02:00
toto
957918c565 fix(ja4ebpf): Rocky Linux RPM builder, remove correlated field, fix thesis 
- Dockerfile.package: migre go-builder de golang:bookworm (Debian) vers
  rockylinux:9, installe Go depuis le tarball officiel, remplace apt par
  dnf (clang llvm libbpf-devel bpftool)

- Suppression du champ 'correlated' de l'agent ja4ebpf : avec eBPF/XDP,
  la corrélation L3/L4↔L7 est toujours implicite par présence des champs.
  Supprimé de : session.go, manager.go, main.go (x5), clickhouse.go

- Thèse (6 corrections listées + cohérence correlated) :
  1. §3.5 + §3.9.1 : SSL_read retourne des octets bruts sans respecter les
     frontières H2 → buffer circulaire de réassemblage en Go userspace
  2. §3.1 : supprimé libpcap + CAP_NET_RAW, remplacé par définition uprobe
  3. §4 + §7 : compte exact 96 features en 8 familles (Famille 1–8),
     supprimé taxonomie F1–F11 obsolète, tous les totaux mis à jour
  4. §2.4 + §8 : remplacé 7 fausses URLs arXiv par [Référence à vérifier]
  5. §4 Famille 2 : ja4_drift_ratio → renvoi à Famille 8 (définition complète)
  6. §6.4 : ajouté limite 'Overhead de l'uprobe SSL_read'
  + §3.6 : supprimé correlated=0/1 du texte architectural

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-12 04:48:40 +02:00
toto
a1e4c1dad5 feat: add ja4ebpf service — eBPF-based TLS/TCP fingerprinting daemon 
- TC ingress hook captures TCP SYN (L3/L4) and TLS ClientHello
- Uprobes on SSL_read/SSL_set_fd capture decrypted TLS data
- Kprobes on accept4 correlate socket FDs to client IP:port
- JA4 fingerprint computed from parsed TLS ClientHello
- HTTP/2 SETTINGS and WINDOW_UPDATE extracted from decrypted streams
- Session manager with sharded map (256 shards) and GC goroutine
- Slowloris detection: sessions with no requests after 10s threshold
- ClickHouse batch writer to ja4_logs.http_logs_raw (raw_json)
- All tests pass: 17 parser + 10 correlation tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-11 22:43:26 +02:00