Fixed potential use-after-free in Update() when gcRound deletes
a session between GetOrCreate() and acquiring the session lock.
Changes:
- Add 'deleted' flag to SessionState
- Mark sessions as deleted before removing from map in gcRound
- Check deleted flag in Update and recreate session if needed
This ensures updates to deleted sessions create a new session
instead of modifying a freed/dangling reference.
Race detector verified: go test ./internal/correlation/... -race
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ReadyCh consumer goroutine could block indefinitely during shutdown
because it only checked for channel closure but not context cancellation.
Changes:
- Add select statement to check both ctx.Done() and ReadyCh closure
- Add shutdown logging for better debugging
- Add 100ms grace period for goroutines to exit
Fixes potential hangs when receiving SIGTERM/SIGINT.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace single-service-per-endpoint with all-ips mode running nginx, apache,
and hitch+varnish simultaneously on 3 dedicated IPs per VM (eth1 alias IPs).
Add a dedicated traffic VM with curl-impersonate for realistic TLS fingerprints,
parallelized traffic generation, and paired SNI_HOSTS/TARGET_IPS lists for
per-VM per-service hostname identification (e.g. rocky9-nginx-platform.test).
Key changes:
- run-tests-vm.sh: add setup_all_ips(), IP-specific Listen/bind directives
with reset-before-apply pattern, graceful service availability checks
- run-e2e-test.sh: traffic VM architecture, all-ips mode, eth1 network,
paired IP/SNI lists, updated cleanup for alias IPs
- generate-traffic.sh: parallel background jobs, curl-impersonate detection,
auto source interface detection via ip route get, Host header in HTTP traffic
- Vagrantfile: add traffic VM with provision-traffic.sh
- provision-traffic.sh: install curl-impersonate and httpx for traffic gen
- test-rpm.sh: multi-interface TC check, updated ja4ebpf config
- clickhouse-init.sh: load CSV stubs for Anubis/bot-networks dictionaries
- Remove obsolete correlator/sentinel/mod-reqin-log docs
- Add h2_settings_ack column to http_logs schema
- Upgrade Go toolchain to 1.25.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The HPACK static table was completely wrong from index 15 onwards — entries
were shifted and missing, causing all header name lookups to return wrong
names (e.g. index 19 returned "cookie" instead of "accept"). Rewrite the
entire table as hpackStaticEntry{Name,Value} structs matching RFC 7541 Appendix
A (indices 1-61) plus browser extensions (62-100). Fix DecodeH2HeadersBlock to
properly decode fully-indexed representations (6.1) which were silently dropped
before — now both name and value are extracted from the static table entry.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add dst_ip and dst_port fields to tls_hello_event BPF struct and populate
them in tc_capture.c. Update Go TLS event handler with new byte offsets
(payload[2048]+src_ip(4)+dst_ip(4)+src_port(2)+dst_port(2)+payload_len(2)+
timestamp_ns(8) = 2070 bytes). Read dst_ip/dst_port from HTTP plain events
and use them to populate L3L4 when SYN was not captured, ensuring dst_ip
and dst_port are always available in ClickHouse for both TLS and HTTP sessions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix two critical offset bugs introduced when ip_total_length was added to
tcp_syn_event: tcp_options_raw offset 21→23 and tcp_options_len offset 61→63,
plus minimum size check 70→72. Fix ssl_data_event direction field offset from
4118 (inside timestamp_ns) to 4126. Simplify attachSSLWrite to use generated
objects directly instead of dynamic spec loading. Regenerate BPF objects with
SSL_write uprobe programs included.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add uprobe_ssl_write_entry/uretprobe_ssl_write_exit to capture server HTTP
responses via SSL_write with direction=1. Implement full HPACK decoder
(RFC 7541 static table, multi-byte integers, literal representations) for
HTTP/2 header extraction. Add AcceptCache mapping {tgid,fd}→SessionKey
from accept4 events as authoritative source for SSL correlation when BPF
ssl_conn_map has src_ip=0. Add ip_total_length to tcp_syn_event BPF struct.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The manual byte assembly (sa_buf[2]<<8 | sa_buf[3]) already produces
a host-byte-order port value; __builtin_bswap16 was swapping it again,
causing SSL events to use wrong source ports and preventing TLS/HTTP
session correlation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add run-e2e-test.sh with CLI parameters (--hits, --http-ratio, --dns, --tls,
--src-ips, --keep-analysis, --up) for configurable traffic generation. Traffic
runs from VM endpoints with multiple source IPs (alias IPs on eth0) to produce
distinct sessions for the ML pipeline. Fix curl TLS flags (--tlsv1.2 instead
of --tls-v1-2), skip redundant local verification in distributed mode, and
fix dashboard is_available() cache that never retried after ClickHouse recovery.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the LogisticRegression meta-learner with a PyTorch MetaFusionMLP
(Linear(3,16)->BN->ReLU->Dropout->Linear(16,1)->Sigmoid) for non-linear
fusion of EIF, NF, and XGBoost scores. Replace KS-test + quantile digest
drift detection with ADWIN (adaptive sliding window, Hoeffding bound).
Replace weekly XGBoost batch retraining with River HoeffdingAdaptiveTree
for incremental online learning (learn_one per cycle). Update all thesis
documentation sections (2.4.2c, 2.4.3, 3.8, discussion, conclusion).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite fleet.py to use a GNN-based approach: nodes are src_ip with ML feature
vectors, edges connect IPs sharing (JA4, ASN) pairs, GraphSAGE (2 SAGEConv
layers, in→64→32) produces 32D embeddings clustered by HDBSCAN. PyG NeighborLoader
activates for >50k nodes. Update thesis docs (§5.2, §6.4, §2, §8) to reflect
GraphSAGE architecture and PyG scalability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split monolithic thesis into separate chapter markdown files under
docs/thesis/. Remove fabricated bibliography entries, correct inflated
claims, add GNN/Transformers section, and rename MetaLearner to Fusion LR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add .gitignore rules for generated CSV data, eBPF compiled objects,
and vmlinux.h header. Remove 19 tracked files (~175 MB) that can be
regenerated from scripts (generate_*.py), bpftool, or bpf2go.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement offline profile building (profile_builder.py) and real-time
dynamic scoring (browser_matcher_dynamic.py) using HDBSCAN-based browser
fingerprint clustering. Add ClickHouse materialized view (13_h2_profiling.sql)
for h2_profile_stats aggregation. Update thesis and project documentation
to cover the new dynamic profiling architecture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Dockerfile.package: migre go-builder de golang:bookworm (Debian) vers
rockylinux:9, installe Go depuis le tarball officiel, remplace apt par
dnf (clang llvm libbpf-devel bpftool)
- Suppression du champ 'correlated' de l'agent ja4ebpf : avec eBPF/XDP,
la corrélation L3/L4↔L7 est toujours implicite par présence des champs.
Supprimé de : session.go, manager.go, main.go (x5), clickhouse.go
- Thèse (6 corrections listées + cohérence correlated) :
1. §3.5 + §3.9.1 : SSL_read retourne des octets bruts sans respecter les
frontières H2 → buffer circulaire de réassemblage en Go userspace
2. §3.1 : supprimé libpcap + CAP_NET_RAW, remplacé par définition uprobe
3. §4 + §7 : compte exact 96 features en 8 familles (Famille 1–8),
supprimé taxonomie F1–F11 obsolète, tous les totaux mis à jour
4. §2.4 + §8 : remplacé 7 fausses URLs arXiv par [Référence à vérifier]
5. §4 Famille 2 : ja4_drift_ratio → renvoi à Famille 8 (définition complète)
6. §6.4 : ajouté limite 'Overhead de l'uprobe SSL_read'
+ §3.6 : supprimé correlated=0/1 du texte architectural
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Increase MAX_TLS_PAYLOAD from 512 to 2048 bytes to capture full
TLS ClientHellos (modern browsers/curl send 1000-1543 byte ClientHellos)
- Fix ParseClientHello to tolerate XDP-truncated payloads: clamp
recordLength and chLen to available data instead of returning error
- Fix cipher suites, compression, extensions truncation to use clamping
- Fix consumeSynEvents struct field offsets: dst_ip (4 bytes at offset 4)
was not accounted for, causing all L3/L4 metadata to be read from
wrong positions (TTL was actually dst_ip[0], windowSize was dst_port, etc.)
- Add parseTCPOptions() to extract MSS and Window Scale from raw TCP options
(C code sets defaults of mss=0, window_scale=0xFF, expects Go to parse)
- Fix consumeAcceptEvents: skip zero-IP events to avoid phantom sessions
- Fix consumeSSLEvents: filter zero-IP/port events when proc fallback fails
- Add missing consumeHTTPPlainEvents goroutine (was defined but never called)
- Fix race condition: SYN consumer sets Correlated=true if TLS already present
- Update tls_hello_event struct offsets in Go consumer (payload_len now at
offset 2054, was 518, due to payload array growing from 512 to 2048 bytes)
- Remove debug logging from consumers and GC
E2E verified: HTTP plain (port 80) and HTTPS (port 443) both produce
fully correlated sessions in ClickHouse with correct:
- ip_meta_ttl=64, ip_meta_df=true, ip_meta_id
- tcp_meta_window_size=64240, tcp_meta_window_scale=10, tcp_meta_mss=1460
- ja4=t13i3010_1d37bd780c83_95d2a80e6515
- tls_alpn=http/1.1
- method=GET, path=/, header_order_signature=Host;User-Agent;Accept
- correlated=1
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use two separate //go:generate directives (Ja4Tc for tc_capture.c, Ja4Ssl
for uprobe_ssl.c) to avoid duplicate LICENSE symbol and multi-file clang issue
- Update loader.go to hold tcObjs/sslObjs separately with correct field names:
UprobeSslSetFd, UprobeSslReadEntry, UretprobeSslReadExit,
KprobeAccept4Entry, KretprobeAccept4Exit
- Add systemd-rpm-macros to all three RPM build stages (el8/el9/el10)
so that %{_unitdir} macro resolves correctly
- RPMs now build successfully for el8, el9, el10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When has_xff=1, the H2 connection is terminated by the reverse proxy/CDN,
so client H2 fingerprints are lost. Previously only D1 (h2_settings) was
neutralized; D2 (window_update), D3 (pseudo_order), and D4 (priority)
still penalized proxied traffic — a real Chrome behind Cloudflare scored
0.0 on 3 dimensions (45% of total weight).
Now all 4 H2 dimensions return 0.5 (neutral) when has_xff>0, and
non-browser H2 detection is also disabled behind proxies.
Tests: 10/10 passed including 3 new XFF-specific cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- API /api/campaigns/scatter: aggregate by campaign_id instead of per-IP
Returns avg_score, avg_velocity, unique_ips, ja4_list, asn_list, country_list
- Template: one bubble per campaign, sized by IP count
- Tooltip: campaign-level info (IPs, score, velocity, ASNs, pays, JA4s)
- Click navigates to campaign detail (not IP detail)
- Updated doc panel text
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Ajoute dict_browser_h2 dans /reflists (lecture seule via dict_browser_h2)
- Nouveaux endpoints API :
GET /api/browser-signatures/entries — liste browser_h2_signatures
(fallback dict CSV si migration 06 non appliquée)
POST /api/browser-signatures/entries — ajout fingerprint + reload dict
DELETE /api/browser-signatures/entries — suppression + reload dict
- Page /browsers : 2 nouvelles sections
'Base de signatures H2' — tableau des 10 fingerprints, form d'ajout,
mode lecture seule automatique si migration 06 non appliquée
'Règles de scoring browser_matcher.py' — tableau statique des 7 dimensions
(poids, valeurs par famille, seuils de bypass)
- Integration : browser_h2.csv copié dans user_files au démarrage ClickHouse
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
La route /api/browsers existait déjà (distribution JA4 par famille).
La nouvelle route du browser_matcher était en conflit — FastAPI utilisait
la première définition. Renommage en /api/browser-signatures.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Nouvelle page dédiée à l'analyse passive des signatures navigateur (§4) :
API — GET /api/browsers :
Requête view_ai_features_1h pour :
- Compteurs globaux (total, sessions_with_h2, matched, mismatch %)
- Distribution h2_dict_family (Chrome/Firefox/Safari/Edge)
- Répartition des signaux WINDOW_UPDATE (chrome/firefox/safari/absent/autre)
- Mismatch TLS↔H2 par famille JA4 (total + count + %)
- Top 20 sessions suspectes (tls_h2_family_mismatch=1, triées par hits)
Page /browsers :
- 6 KPI header (sessions, avec H2, famille connue, taux match, mismatch, % mismatch)
- Doc banner expliquant browser_matcher §4 et le mode DUAL_MODE
- Donut : familles H2 (dict_browser_h2 lookup)
- Bar horizontal : WINDOW_UPDATE signals par famille
- Bar groupé + ligne : mismatch TLS↔H2 par famille JA4 (count + %)
- Table : top 20 imposteurs potentiels avec IP cliquable, pseudo-order, cohérence
- Mini-KPIs : ordres pseudo-headers Chrome/Safari, Firefox, inconnu, PRIORITY frames
- Lien nav 'Navigateurs' dans le groupe Surveillance de base.html
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avant : toutes les vues de campagnes étaient fixes à 7 jours.
Après : sélecteur 1j / 7j (défaut) / 14j / 30j / 90j en haut à droite.
- Ajout du paramètre ?days= (1–90, défaut 7) à :
GET /api/campaigns
GET /api/campaigns/graph
GET /api/campaigns/scatter
GET /api/campaigns/{cid}
- Le sélecteur recharge simultanément les 3 vues (cartes, scatter, graphe)
et le panneau de détail avec la même fenêtre temporelle
- Le compteur de campagnes indique la plage active : (4 campagnes — 30j)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CH 24.8 refuse MODIFY ORDER BY sur des colonnes existantes (erreur BAD_ARGUMENTS 36).
La migration 01 ne pouvait donc pas corriger l'ORDER BY en post-init.
Correctif :
- 06_ml_tables.sql : ORDER BY (src_ip) → ORDER BY (src_ip, ja4, host, model_name)
+ TTL 30j → 7j (cohérent avec l'architecture documentée)
- 01_ttl_adjustments.sql : supprime le MODIFY ORDER BY impossible, conserve
uniquement les MODIFY TTL (valides pour les déploiements existants)
Résultat : make init-stack sans aucun ⚠ ni ✗
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
