code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
145 Commits 1 Branch 0 Tags
master
Commit Graph

4 Commits

Author SHA1 Message Date
Jacquin Antoine
742f4420c0 fix(test): add ClickHouse readiness check before starting ja4ebpf 
Fixed race condition where ja4ebpf would fail to connect to
ClickHouse at startup because ClickHouse HTTP port wasn't ready yet,
even though Docker healthcheck passed.

Changes:
- Add 30s wait loop with ClickHouse /ping endpoint check
- Log success message when ClickHouse is ready
- Applied to all 4 stacks: nginx, apache, nginx-varnish, hitch-varnish

Test results after fix:
- nginx: 240 rows, 175 JA4 fingerprints ✅
- apache: 257 rows, 191 JA4 fingerprints ✅
- nginx-varnish: 298 rows, 242 JA4 fingerprints ✅
- hitch-varnish: 247 rows, 177 JA4 fingerprints ✅

All L3/L4 metadata (TTL, MSS, Window), TLS fingerprinting (JA4, SNI),
and HTTP layer data are correctly captured and persisted.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-19 15:11:22 +02:00
toto
f85a10b012 feat: pipeline L7 HTTP complet + infrastructure tests VM 
Correctifs pipeline L7 (uprobe SSL_read) :
- uprobe_ssl.c : ssl_set_fd ne retourne plus tôt quand fd_conn_map est
  vide (accept4 non disponible en Docker). Sauvegarde ssl_ptr→{fd,0,0}
  pour permettre le fallback /proc côté Go.
- main.go : consumeSSLEvents reécrit avec routeur magic-bytes complet :
  * HTTP/2 preface → extraction SETTINGS + conversion correlation.HTTP2Settings
  * HTTP/1.x requête → method, path, query, headers, header_order_sig
  * HTTP/1.x réponse → status_code
  * Fallback /proc/<tgid>/fd/<fd> quand src_ip=0 (accept4 absent)
- writer/clickhouse.go : export header_order_signature ajouté

Nouveaux packages :
- internal/parser/http1.go : parseur HTTP/1.x (IsHTTP1Request,
  ParseHTTP1Request, IsHTTP1Response, ParseHTTP1Response)
- internal/parser/http1_test.go : 11 tests unitaires (28 total passent)
- internal/procutil/proc_lookup.go : résolution fd→IP via /proc avec cache
  TTL 5s (FDCache). Supporte /proc/PID/net/tcp et tcp6, IPv4-mappé IPv6.

Infrastructure tests VM (tests/vm/) :
- Vagrantfile : VM Rocky Linux 9 KVM, 4 CPU / 4 GB RAM
- provision.sh : installation toolchain eBPF + Go + Docker + nginx
- run-tests-vm.sh : suite de test complète dans la VM (L3/L4+TLS+L7)
- README.md : guide d'installation et d'utilisation
- Makefile : cibles vm-up, vm-down, vm-ssh, test-vm-nginx, test-vm-all,
  vm-rebuild-ja4ebpf

Corrections stack Docker :
- Dockerfiles nginx/apache/nginx-varnish/hitch-varnish : suppression des
  références à shared/go/ja4common/ (répertoire supprimé)
- clickhouse-init.sh : restauré depuis git, seed anubis_ua_rules obsolète
  supprimé (table REGEXP_TREE supprimée du schéma)
- traffic-gen : ajout HTTP/1.0 (http.client) et HTTP/2 (httpx)
- verify_db.py : script de vérification 35 checks (L3/L4/TLS/L7/corrélation)
- run-stack-tests.sh : phase 6 verify_db ajoutée

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-12 02:37:00 +02:00
toto
dc6ffd6474 fix: tests intégration matrix — procps-ng, varnish h2, hitch ALPN, pgrep→ps 
- Ajout de procps-ng dans les 4 Dockerfiles runtime (ps/pgrep disponibles)
- Remplacement de pgrep par ps -C dans tous les run-tests.sh
- Correction entrypoint nginx-varnish : pgrep nginx → cat nginx.pid (exit 127)
- Activation HTTP/2 dans Varnish : ajout de -p feature=+http2 dans les
  entrypoints nginx-varnish et hitch-varnish
- Restauration ALPN h2,http/1.1 dans hitch.conf (varnish supporte maintenant h2)
- Correction healthcheck hitch-varnish : curl sans --http1.1 (h2 fonctionnel)
- Correction requêtes phase_verify : http_logs_raw → http_logs, colonnes correctes
- Correction writer clickhouse.go : noms JSON alignés avec la MV (ip_meta_*, tls_sni…)
- Fix toStartOfSecond(DateTime) → toStartOfSecond(toDateTime64(col, 3))
- Retrait du SKIP el8/nginx-varnish (varnish s'installe bien sur AlmaLinux 8)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-12 01:29:01 +02:00
toto
3b047b680a fix(ja4ebpf): split bpf2go generate into Ja4Tc + Ja4Ssl, fix RPM systemd-rpm-macros 
- Use two separate //go:generate directives (Ja4Tc for tc_capture.c, Ja4Ssl
  for uprobe_ssl.c) to avoid duplicate LICENSE symbol and multi-file clang issue
- Update loader.go to hold tcObjs/sslObjs separately with correct field names:
  UprobeSslSetFd, UprobeSslReadEntry, UretprobeSslReadExit,
  KprobeAccept4Entry, KretprobeAccept4Exit
- Add systemd-rpm-macros to all three RPM build stages (el8/el9/el10)
  so that %{_unitdir} macro resolves correctly
- RPMs now build successfully for el8, el9, el10

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-11 23:21:11 +02:00