2 Commits

Author	SHA1	Message	Date
toto	6d02f21c1e	feat: implement thesis §5 advanced detection techniques as ClickHouse MVs ... New aggregation tables + materialized views: - agg_path_sequences_1h + MV (§5.1 Path Sequence Entropy) - agg_request_timing_1h + MV (§5.3 Request Cadence Fingerprint) - agg_ip_behavior_1h + MV (§5.5 JA4 Drift + §5.8 Cross-Domain) - agg_resource_cascade_1h + MV (§5.4 Resource Dependency Tree) New analytical views: - view_thesis_features_1h: unified view exposing all computable features (path_transition_entropy, cadence_cv, burst_ratio, pause_ratio, ja4_drift_ratio, host_diversity, host_sweep_speed, host_coverage_uniformity) - view_resource_cascade_1h: root_to_first_asset_delay, asset_load_stddev Documented future techniques (not feasible as MV): - §5.2 Bipartite Fleet Graph (needs Python networkx) - §5.6 DNS Shadow Analysis (needs sentinel UDP/53 extension) - §5.7 Compression Ratio Invariant (needs mod_reqin_log extension) Updated: deploy_schema.sh, verify_mvs.py (sections 8-10) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>	2026-04-08 01:42:52 +02:00
toto	51b8eb57a8	feat: port v14 schema fixes, migration, MV verifier, thesis from ja4/ ... deploy_views.sql (v13 → v14): - CRITICAL: ml_detected_anomalies ORDER BY (src_ip) → (src_ip, ja4, host, model_name) ReplacingMergeTree was collapsing all detections to 1 row per IP on merge - Add PARTITION BY toDate + ttl_only_drop_parts on all 4 data tables - ml_all_scores TTL 3d → 7d; ml_detected_anomalies TTL 30d → 7d - agg_host_ip_ja4_1h + agg_header_fingerprint_1h: add partition + TTL 7d - view_ip_recurrence: add WHERE detected_at >= now() - 7 DAY (was full scan) - Remove dead views: summary/timeseries/threat_dist/variability - Add view_dashboard_entities (fixes HTTP 500 in clustering/incidents/fingerprints) - Add view_dashboard_user_agents (fixes HTTP 500 in fingerprints/metrics) - Add view_ai_features_24h (enables ENABLE_MULTIWINDOW in bot_detector) - Mark max_requests_per_sec as DEPRECATED (always 0) New files: - correlator/sql/migrations/01_ttl_adjustments.sql: ALTER TABLE migration - tests/integration/verify_mvs.py: MV pipeline verification assertions - docs/THESIS_HTTP_Traffic_Detection.md: detection techniques thesis All DB references use ja4_processing/ja4_logs (no mabase_prod). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>	2026-04-07 23:51:56 +02:00