code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: suppression dépendance User-Agent de la détection navigateur

Browse Source 
Changements SQL :
- modern_browser_score : sec-ch-ua→100, Sec-Fetch→70 (plus de UA fallback)
- Ajout has_sec_ch_ua (UInt8) dans agg_header_fingerprint_1h et ml_all_scores
- mss_mobile_mismatch utilise has_sec_ch_ua au lieu de modern_browser_score
- header_order_confidence : PARTITION BY ja4 au lieu de first_ua
- sec_ch_mobile_mismatch : comparaison Client Hints interne (sans UA)
- Migration 03_remove_ua_browser_detection.sql

Changements Python :
- browser.py Axe 3 : Client Hints + Sec-Fetch + is_fake_navigation (PAS de UA)
- Pondération axes : ja4_known 0.30, tls_coherence 0.20 (signaux TLS renforcés)
- preprocessing.py : has_sec_ch_ua ajouté aux features et binary_features

Fichiers modifiés : 8 SQL/Python + 1 migration, 36/36 tests passent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-09 23:06:01 +02:00
parent 00e99e5464
commit 14db3d9040
9 changed files with 101 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
services/correlator/sql/migrations/03_remove_ua_browser_detection.sql Normal file
View File

@ -0,0 +1,45 @@
-- === 03_remove_ua_browser_detection.sql — Suppression dépendance User-Agent ===
--
-- Contexte : l'identification navigateur ne doit PAS se baser sur le User-Agent
-- (trivalement falsifiable). Cette migration :
-- 1. Ajoute has_sec_ch_ua à agg_header_fingerprint_1h
-- 2. Recréé la MV avec modern_browser_score basé sur Client Hints + Sec-Fetch
-- 3. Recréé sec_ch_mobile_mismatch sans UA (Client Hints only)
-- 4. Ajoute has_sec_ch_ua à ml_all_scores
-- ---------------------------------------------------------------------------
-- 1. Nouvelle colonne has_sec_ch_ua
ALTER TABLE ja4_processing.agg_header_fingerprint_1h
ADD COLUMN IF NOT EXISTS has_sec_ch_ua SimpleAggregateFunction(max, UInt8)
AFTER modern_browser_score;
-- 2. Recréer la MV sans dépendance UA
DROP VIEW IF EXISTS ja4_processing.mv_agg_header_fingerprint_1h;
CREATE MATERIALIZED VIEW ja4_processing.mv_agg_header_fingerprint_1h
TO ja4_processing.agg_header_fingerprint_1h AS
SELECT
toStartOfHour(src.time) AS window_start,
toIPv6(src.src_ip) AS src_ip,
any(toString(cityHash64(src.client_headers))) AS header_order_hash,
max(toUInt16(length(src.client_headers) - length(replaceAll(src.client_headers, ',', '')) + 1)) AS header_count,
max(toUInt8(if(position(src.client_headers, 'Accept-Language') > 0, 1, 0))) AS has_accept_language,
max(toUInt8(if(position(src.client_headers, 'Cookie') > 0, 1, 0))) AS has_cookie,
max(toUInt8(if(position(src.client_headers, 'Referer') > 0, 1, 0))) AS has_referer,
-- modern_browser_score : sec-ch-ua → 100, Sec-Fetch → 70, sinon → 0 (PAS de UA)
max(toUInt8(if(length(src.header_sec_ch_ua) > 0, 100, if(length(src.header_sec_fetch_site) > 0, 70, 0)))) AS modern_browser_score,
max(toUInt8(if(length(src.header_sec_ch_ua) > 0, 1, 0))) AS has_sec_ch_ua,
max(toUInt8(if((position(src.header_user_agent, 'Windows') > 0 AND position(src.header_sec_ch_ua_platform, 'Windows') == 0) OR (position(src.header_user_agent, 'iPhone') > 0 AND position(src.header_sec_ch_ua_platform, 'iOS') == 0), 1, 0))) AS ua_ch_mismatch,
-- sec_ch_mobile_mismatch : incohérence interne Client Hints (pas de UA)
max(toUInt8(if(
(src.header_sec_ch_ua_mobile = '?1' AND position(src.header_sec_ch_ua_platform, 'Windows') > 0)
OR (src.header_sec_ch_ua_mobile = '?0' AND position(src.header_sec_ch_ua_platform, 'Android') > 0),
1, 0))) AS sec_ch_mobile_mismatch,
any(src.header_sec_fetch_mode) AS sec_fetch_mode,
any(src.header_sec_fetch_dest) AS sec_fetch_dest
FROM ja4_logs.http_logs AS src
GROUP BY window_start, src.src_ip;
-- 3. Ajouter has_sec_ch_ua à ml_all_scores
ALTER TABLE ja4_processing.ml_all_scores
ADD COLUMN IF NOT EXISTS has_sec_ch_ua UInt8 DEFAULT 0;