Commit Graph

6 Commits

Author SHA1 Message Date
b1218a2367 fix(ja4ebpf): fix TLS capture, SYN offsets, TCP option parsing
- Increase MAX_TLS_PAYLOAD from 512 to 2048 bytes to capture full
  TLS ClientHellos (modern browsers/curl send 1000-1543 byte ClientHellos)
- Fix ParseClientHello to tolerate XDP-truncated payloads: clamp
  recordLength and chLen to available data instead of returning error
- Fix cipher suites, compression, extensions truncation to use clamping
- Fix consumeSynEvents struct field offsets: dst_ip (4 bytes at offset 4)
  was not accounted for, causing all L3/L4 metadata to be read from
  wrong positions (TTL was actually dst_ip[0], windowSize was dst_port, etc.)
- Add parseTCPOptions() to extract MSS and Window Scale from raw TCP options
  (C code sets defaults of mss=0, window_scale=0xFF, expects Go to parse)
- Fix consumeAcceptEvents: skip zero-IP events to avoid phantom sessions
- Fix consumeSSLEvents: filter zero-IP/port events when proc fallback fails
- Add missing consumeHTTPPlainEvents goroutine (was defined but never called)
- Fix race condition: SYN consumer sets Correlated=true if TLS already present
- Update tls_hello_event struct offsets in Go consumer (payload_len now at
  offset 2054, was 518, due to payload array growing from 512 to 2048 bytes)
- Remove debug logging from consumers and GC

E2E verified: HTTP plain (port 80) and HTTPS (port 443) both produce
fully correlated sessions in ClickHouse with correct:
  - ip_meta_ttl=64, ip_meta_df=true, ip_meta_id
  - tcp_meta_window_size=64240, tcp_meta_window_scale=10, tcp_meta_mss=1460
  - ja4=t13i3010_1d37bd780c83_95d2a80e6515
  - tls_alpn=http/1.1
  - method=GET, path=/, header_order_signature=Host;User-Agent;Accept
  - correlated=1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-12 04:16:44 +02:00
f85a10b012 feat: pipeline L7 HTTP complet + infrastructure tests VM
Correctifs pipeline L7 (uprobe SSL_read) :
- uprobe_ssl.c : ssl_set_fd ne retourne plus tôt quand fd_conn_map est
  vide (accept4 non disponible en Docker). Sauvegarde ssl_ptr→{fd,0,0}
  pour permettre le fallback /proc côté Go.
- main.go : consumeSSLEvents reécrit avec routeur magic-bytes complet :
  * HTTP/2 preface → extraction SETTINGS + conversion correlation.HTTP2Settings
  * HTTP/1.x requête → method, path, query, headers, header_order_sig
  * HTTP/1.x réponse → status_code
  * Fallback /proc/<tgid>/fd/<fd> quand src_ip=0 (accept4 absent)
- writer/clickhouse.go : export header_order_signature ajouté

Nouveaux packages :
- internal/parser/http1.go : parseur HTTP/1.x (IsHTTP1Request,
  ParseHTTP1Request, IsHTTP1Response, ParseHTTP1Response)
- internal/parser/http1_test.go : 11 tests unitaires (28 total passent)
- internal/procutil/proc_lookup.go : résolution fd→IP via /proc avec cache
  TTL 5s (FDCache). Supporte /proc/PID/net/tcp et tcp6, IPv4-mappé IPv6.

Infrastructure tests VM (tests/vm/) :
- Vagrantfile : VM Rocky Linux 9 KVM, 4 CPU / 4 GB RAM
- provision.sh : installation toolchain eBPF + Go + Docker + nginx
- run-tests-vm.sh : suite de test complète dans la VM (L3/L4+TLS+L7)
- README.md : guide d'installation et d'utilisation
- Makefile : cibles vm-up, vm-down, vm-ssh, test-vm-nginx, test-vm-all,
  vm-rebuild-ja4ebpf

Corrections stack Docker :
- Dockerfiles nginx/apache/nginx-varnish/hitch-varnish : suppression des
  références à shared/go/ja4common/ (répertoire supprimé)
- clickhouse-init.sh : restauré depuis git, seed anubis_ua_rules obsolète
  supprimé (table REGEXP_TREE supprimée du schéma)
- traffic-gen : ajout HTTP/1.0 (http.client) et HTTP/2 (httpx)
- verify_db.py : script de vérification 35 checks (L3/L4/TLS/L7/corrélation)
- run-stack-tests.sh : phase 6 verify_db ajoutée

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-12 02:37:00 +02:00
9734e21fe3 chore: suppression des services obsolètes (sentinel, correlator, mod-reqin-log)
Remplacés par l'agent ja4ebpf (eBPF CO-RE). Nettoyage complet :

Supprimé :
- old/ (archive de l'ancienne architecture)
- services/correlator/ (logcorrelator Go)
- services/sentinel/ (capture pcap Go)
- services/mod-reqin-log/ (module Apache C)
- shared/go/ja4common/ (lib Go partagée — plus importée par ja4ebpf)
- tests/integration/platform/ (test correlator+sentinel+httpd)
- tests/integration/docker-compose.yml (compose ancienne archi)
- tests/integration/run-tests.sh (runner correlator/sentinel)
- tests/integration/verify_mvs.py (script orphelin)

Nettoyé :
- go.work : retire ./shared/go/ja4common
- services/ja4ebpf/go.mod : retire replace ja4common (jamais importé)
- services/ja4ebpf/Dockerfile* : retire les COPY ja4common inutiles
- Makefile : retire test-ja4common-python, test-integration*, targets obsolètes
- tests/integration/README.md : réécrit pour l'architecture ja4ebpf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-12 01:48:14 +02:00
dc6ffd6474 fix: tests intégration matrix — procps-ng, varnish h2, hitch ALPN, pgrep→ps
- Ajout de procps-ng dans les 4 Dockerfiles runtime (ps/pgrep disponibles)
- Remplacement de pgrep par ps -C dans tous les run-tests.sh
- Correction entrypoint nginx-varnish : pgrep nginx → cat nginx.pid (exit 127)
- Activation HTTP/2 dans Varnish : ajout de -p feature=+http2 dans les
  entrypoints nginx-varnish et hitch-varnish
- Restauration ALPN h2,http/1.1 dans hitch.conf (varnish supporte maintenant h2)
- Correction healthcheck hitch-varnish : curl sans --http1.1 (h2 fonctionnel)
- Correction requêtes phase_verify : http_logs_raw → http_logs, colonnes correctes
- Correction writer clickhouse.go : noms JSON alignés avec la MV (ip_meta_*, tls_sni…)
- Fix toStartOfSecond(DateTime) → toStartOfSecond(toDateTime64(col, 3))
- Retrait du SKIP el8/nginx-varnish (varnish s'installe bien sur AlmaLinux 8)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-12 01:29:01 +02:00
3b047b680a fix(ja4ebpf): split bpf2go generate into Ja4Tc + Ja4Ssl, fix RPM systemd-rpm-macros
- Use two separate //go:generate directives (Ja4Tc for tc_capture.c, Ja4Ssl
  for uprobe_ssl.c) to avoid duplicate LICENSE symbol and multi-file clang issue
- Update loader.go to hold tcObjs/sslObjs separately with correct field names:
  UprobeSslSetFd, UprobeSslReadEntry, UretprobeSslReadExit,
  KprobeAccept4Entry, KretprobeAccept4Exit
- Add systemd-rpm-macros to all three RPM build stages (el8/el9/el10)
  so that %{_unitdir} macro resolves correctly
- RPMs now build successfully for el8, el9, el10

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 23:21:11 +02:00
a1e4c1dad5 feat: add ja4ebpf service — eBPF-based TLS/TCP fingerprinting daemon
- TC ingress hook captures TCP SYN (L3/L4) and TLS ClientHello
- Uprobes on SSL_read/SSL_set_fd capture decrypted TLS data
- Kprobes on accept4 correlate socket FDs to client IP:port
- JA4 fingerprint computed from parsed TLS ClientHello
- HTTP/2 SETTINGS and WINDOW_UPDATE extracted from decrypted streams
- Session manager with sharded map (256 shards) and GC goroutine
- Slowloris detection: sessions with no requests after 10s threshold
- ClickHouse batch writer to ja4_logs.http_logs_raw (raw_json)
- All tests pass: 17 parser + 10 correlation tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 22:43:26 +02:00