fix: TLS capture end-to-end in integration tests

- Add traffic-gen container (curlimages/curl) to send HTTPS traffic
  across Docker network so sentinel (pcap on eth0) captures ClientHello
- Seed anubis_ua_rules with catch-all rule (REGEXP_TREE needs ≥1 entry)
  so MV mv_http_logs processes raw logs without errors
- Add JA4/JA3 fingerprint verification in Phase 5 tests
- Dashboard healthcheck via python urllib (no curl in image)

Results: 59 raw logs, 59 parsed, 53 with JA4+JA3 fingerprints (TLS 1.3)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-07 21:04:14 +02:00
parent a985661369
commit da8357f43d
3 changed files with 56 additions and 11 deletions

View File

@ -27,4 +27,16 @@ for f in "$TMP_DIR"/*.sql; do
clickhouse-client --multiquery < "$f"
done
echo "[init] All SQL files executed successfully"
# ---------------------------------------------------------------------------
# Seed data required for dictionaries to function
# REGEXP_TREE dictionaries require at least one rule; without it, any INSERT
# into http_logs_raw fails because the MV mv_http_logs calls dictGet() on
# the empty dict. Insert a catch-all "unknown" rule so the pipeline works.
# ---------------------------------------------------------------------------
echo "[init] Seeding anubis_ua_rules (REGEXP_TREE needs ≥1 rule)..."
clickhouse-client --multiquery <<'SEED'
INSERT INTO ja4_processing.anubis_ua_rules (id, parent_id, regexp, keys, values) VALUES
(1, 0, '.*', ['bot_name','action','has_ip','rule_id','category'], ['','','0','0','']);
SEED
echo "[init] All SQL files executed and seed data inserted"