code_public/ja4-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sql+api): fix view column mismatches and ClickHouse 24.8 JOIN issue

Browse Source 
- view_form_bruteforce_detected: add post_count, distinct_paths, first_seen, last_seen
- view_host_ip_ja4_rotation: add host, distinct_ja4, ja4_list, window_start
- Replace uniqExact/groupUniqArray with count()/groupArray (no nested-agg error)
- api.py campaigns/graph: move a.src_ip < b.src_ip from JOIN ON to WHERE
  (ClickHouse 24.8 forbids cross-table inequality in JOIN ON condition)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-10 01:05:04 +02:00
parent 040437921c
commit 7a04e47041
2 changed files with 16 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
services/dashboard/backend/routes/api.py
View File

@ -972,8 +972,8 @@ async def campaigns_graph() -> dict[str, Any]:
f"FROM {_DB}.ml_detected_anomalies a " f"FROM {_DB}.ml_detected_anomalies a "
f"INNER JOIN {_DB}.ml_detected_anomalies b " f"INNER JOIN {_DB}.ml_detected_anomalies b "
"ON a.ja4 = b.ja4 AND a.campaign_id = b.campaign_id " "ON a.ja4 = b.ja4 AND a.campaign_id = b.campaign_id "
"AND a.src_ip < b.src_ip "
"WHERE a.campaign_id >= 0 " "WHERE a.campaign_id >= 0 "
"AND a.src_ip < b.src_ip "
"AND a.detected_at >= now() - INTERVAL 7 DAY " "AND a.detected_at >= now() - INTERVAL 7 DAY "
"AND b.detected_at >= now() - INTERVAL 7 DAY " "AND b.detected_at >= now() - INTERVAL 7 DAY "
"LIMIT 2000" "LIMIT 2000"

9
shared/clickhouse/11_views.sql
View File

@ -31,9 +31,8 @@ SELECT
argMax(ja4, ja4_hits) AS ja4, argMax(ja4, ja4_hits) AS ja4,
sum(ja4_hits) AS hits, sum(ja4_hits) AS hits,
sum(ja4_posts) AS post_count, sum(ja4_posts) AS post_count,
-- Alias de compatibilité pour les anciens appels
sum(ja4_posts) AS query_params_count, sum(ja4_posts) AS query_params_count,
uniqExact(ja4) AS distinct_paths, count() AS distinct_paths,
min(w_min) AS first_seen, min(w_min) AS first_seen,
max(w_max) AS last_seen max(w_max) AS last_seen
FROM ( FROM (
@ -65,9 +64,9 @@ CREATE OR REPLACE VIEW ja4_processing.view_host_ip_ja4_rotation AS
SELECT SELECT
src_ip, src_ip,
argMax(host, ja4_hits) AS host, argMax(host, ja4_hits) AS host,
uniqExact(ja4) AS distinct_ja4, count() AS distinct_ja4,
uniqExact(ja4) AS distinct_ja4_count, count() AS distinct_ja4_count,
groupUniqArray(ja4) AS ja4_list, groupArray(ja4) AS ja4_list,
sum(ja4_hits) AS total_hits, sum(ja4_hits) AS total_hits,
max(w_max) AS window_start, max(w_max) AS window_start,
min(w_min) AS first_seen, min(w_min) AS first_seen,