code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: pipeline L7 HTTP complet + infrastructure tests VM

Browse Source 
Correctifs pipeline L7 (uprobe SSL_read) :
- uprobe_ssl.c : ssl_set_fd ne retourne plus tôt quand fd_conn_map est
  vide (accept4 non disponible en Docker). Sauvegarde ssl_ptr→{fd,0,0}
  pour permettre le fallback /proc côté Go.
- main.go : consumeSSLEvents reécrit avec routeur magic-bytes complet :
  * HTTP/2 preface → extraction SETTINGS + conversion correlation.HTTP2Settings
  * HTTP/1.x requête → method, path, query, headers, header_order_sig
  * HTTP/1.x réponse → status_code
  * Fallback /proc/<tgid>/fd/<fd> quand src_ip=0 (accept4 absent)
- writer/clickhouse.go : export header_order_signature ajouté

Nouveaux packages :
- internal/parser/http1.go : parseur HTTP/1.x (IsHTTP1Request,
  ParseHTTP1Request, IsHTTP1Response, ParseHTTP1Response)
- internal/parser/http1_test.go : 11 tests unitaires (28 total passent)
- internal/procutil/proc_lookup.go : résolution fd→IP via /proc avec cache
  TTL 5s (FDCache). Supporte /proc/PID/net/tcp et tcp6, IPv4-mappé IPv6.

Infrastructure tests VM (tests/vm/) :
- Vagrantfile : VM Rocky Linux 9 KVM, 4 CPU / 4 GB RAM
- provision.sh : installation toolchain eBPF + Go + Docker + nginx
- run-tests-vm.sh : suite de test complète dans la VM (L3/L4+TLS+L7)
- README.md : guide d'installation et d'utilisation
- Makefile : cibles vm-up, vm-down, vm-ssh, test-vm-nginx, test-vm-all,
  vm-rebuild-ja4ebpf

Corrections stack Docker :
- Dockerfiles nginx/apache/nginx-varnish/hitch-varnish : suppression des
  références à shared/go/ja4common/ (répertoire supprimé)
- clickhouse-init.sh : restauré depuis git, seed anubis_ua_rules obsolète
  supprimé (table REGEXP_TREE supprimée du schéma)
- traffic-gen : ajout HTTP/1.0 (http.client) et HTTP/2 (httpx)
- verify_db.py : script de vérification 35 checks (L3/L4/TLS/L7/corrélation)
- run-stack-tests.sh : phase 6 verify_db ajoutée

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-12 02:37:00 +02:00
parent 9734e21fe3
commit f85a10b012
21 changed files with 1868 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
tests/integration/traffic-gen/generate_traffic.py
View File

@ -8,18 +8,19 @@ Simulates varied web traffic including:
- Multiple HTTP methods (GET, POST, PUT, DELETE, HEAD, OPTIONS, PATCH)
- Varied paths, query strings, form data, JSON payloads
- Both HTTP (port 80) and HTTPS (port 443)
- HTTP/1.0, HTTP/1.1, HTTP/2.0 (via httpx[http2])
- Different Accept/Language/Encoding headers
- Cookie / Referer / X-Forwarded-For always set — ensures src_ip diversity
in ClickHouse via mod_remoteip (r->useragent_ip updated from XFF)
- Multiple SSL contexts to vary TLS ClientHello parameters
Usage:
python generate_traffic.py [--host platform] [--http-port 80] [--https-port 443]
[--requests 500] [--workers 10] [--scenario all]
[--requests 500] [--workers 10]
"""
import argparse
import concurrent.futures
import http.client
import json
import random
import ssl
@ -435,6 +436,45 @@ def build_scenarios(host: str, http_port: int, https_port: int, count: int) -> l
label="options-cors",
))
# --- HTTP/1.0 explicite sur HTTP (port 80) ---
# http.client permet de forcer le protocole HTTP/1.0 via _http_vsn
h10_count = max(10, int(count * 0.05))
for _ in range(h10_count):
ua = random.choice(BROWSERS + BOTS)
path = random.choice(["/", "/health", "/index.html", "/robots.txt"])
scenarios.append(RequestScenario(
method="GET",
url=f"{base_http}{path}",
headers=_random_headers(ua, xff_ip=random.choice(HUMAN_IPS + BOT_IPS)),
label="http10-plain",
))
# --- HTTP/1.0 explicite sur HTTPS ---
for _ in range(max(5, int(count * 0.03))):
ua = random.choice(BROWSERS + BOTS)
_, ssl_ctx = random.choice(SSL_CONTEXTS)
scenarios.append(RequestScenario(
method="GET",
url=f"{base_https}/health",
headers=_random_headers(ua, xff_ip=random.choice(HUMAN_IPS)),
ssl_ctx=ssl_ctx,
label="http10-tls",
))
# --- HTTP/2 explicite (httpx[http2]) ---
h2_count = max(20, int(count * 0.10))
for _ in range(h2_count):
ua = random.choice(BROWSERS)
path = random.choice(PATHS)
qs = random.choice(QUERY_PARAMS)
scenarios.append(RequestScenario(
method=random.choice(["GET", "GET", "GET", "POST"]),
url=f"{base_https}{path}{qs}",
headers=_random_headers(ua, xff_ip=random.choice(HUMAN_IPS)),
body=json.dumps({"h2": True}).encode() if random.random() < 0.2 else None,
label="http2-explicit",
))
# Fill remaining with browser HTTPS GETs
while len(scenarios) < count:
ua = random.choice(BROWSERS)
@ -457,8 +497,78 @@ def build_scenarios(host: str, http_port: int, https_port: int, count: int) -> l
stats = {"ok": 0, "err": 0, "by_label": {}}
def _send_http10(scenario: RequestScenario) -> dict:
"""Envoie une requête en HTTP/1.0 pur via http.client."""
t0 = time.monotonic()
try:
from urllib.parse import urlparse
parsed = urlparse(scenario.url)
host = parsed.hostname
port = parsed.port or (443 if parsed.scheme == "https" else 80)
path = parsed.path or "/"
if parsed.query:
path += "?" + parsed.query
if parsed.scheme == "https":
ctx = scenario.ssl_ctx or ssl.create_default_context()
if hasattr(ctx, "check_hostname"):
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
conn = http.client.HTTPSConnection(host, port, timeout=5, context=ctx)
else:
conn = http.client.HTTPConnection(host, port, timeout=5)
# Forcer HTTP/1.0
conn._http_vsn = 10
conn._http_vsn_str = "HTTP/1.0"
hdrs = {k: v for k, v in scenario.headers.items()
if k.lower() not in ("connection",)}
conn.request(scenario.method, path, body=scenario.body, headers=hdrs)
resp = conn.getresponse()
resp.read(4096)
return {"ok": True, "status": resp.status, "label": scenario.label,
"ms": int((time.monotonic() - t0) * 1000)}
except Exception as e:
return {"ok": False, "error": str(e)[:80], "label": scenario.label,
"ms": int((time.monotonic() - t0) * 1000)}
finally:
try:
conn.close()
except Exception:
pass
def _send_http2(scenario: RequestScenario) -> dict:
"""Envoie une requête HTTP/2 via httpx (négociation ALPN h2)."""
t0 = time.monotonic()
try:
import httpx
with httpx.Client(http2=True, verify=False, timeout=5.0) as client:
hdrs = {k: v for k, v in scenario.headers.items()
if k.lower() not in ("connection", "content-length")}
resp = client.request(
method=scenario.method,
url=scenario.url,
headers=hdrs,
content=scenario.body,
)
return {"ok": True, "status": resp.status_code, "label": scenario.label,
"ms": int((time.monotonic() - t0) * 1000),
"http_version": resp.http_version}
except Exception as e:
return {"ok": False, "error": str(e)[:80], "label": scenario.label,
"ms": int((time.monotonic() - t0) * 1000)}
def send_request(scenario: RequestScenario) -> dict:
"""Send a single request, return result dict."""
"""Dispatcher : HTTP/1.0, HTTP/2, ou HTTP/1.1 selon le label."""
if scenario.label.startswith("http10"):
return _send_http10(scenario)
if scenario.label == "http2-explicit":
return _send_http2(scenario)
# HTTP/1.1 via urllib (chemin existant)
t0 = time.monotonic()
try:
req = urllib.request.Request(
@ -469,11 +579,10 @@ def send_request(scenario: RequestScenario) -> dict:
)
ctx = scenario.ssl_ctx
with urllib.request.urlopen(req, context=ctx, timeout=5) as resp:
_ = resp.read(4096) # consume partial body
_ = resp.read(4096)
return {"ok": True, "status": resp.status, "label": scenario.label,
"ms": int((time.monotonic() - t0) * 1000)}
except urllib.error.HTTPError as e:
# HTTP errors (4xx/5xx) are still valid responses — Apache served them
return {"ok": True, "status": e.code, "label": scenario.label,
"ms": int((time.monotonic() - t0) * 1000)}
except Exception as e: