feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

Complete rewrite of the SOC dashboard using FastAPI + Jinja2 + htmx + Chart.js + Tailwind CSS.
Replaces the old React/Vite frontend with server-rendered templates.

Dashboard pages:
- Overview: KPIs, timeline chart, threat distribution, top IPs
- Detections: paginated/filterable anomaly table
- Scores: ml_all_scores with AE error & XGB prob columns
- Traffic: HTTP logs with method/host filters
- IP Investigation: full deep-dive (scores, features, HTTP logs, classify)
- Classification: SOC feedback form + history
- Features: AI + thesis feature stats
- Models: scoring stats + model metadata

API: 9 JSON endpoints with parameterized queries, sort whitelists

SQL fixes:
- 05_aggregation_tables: add deduplicate_merge_projection_mode
- 11_views: fix nested aggregate (argMax inside sum)
- 12_thesis_features: remove invalid 'let' bindings, fix groupArrayIf type

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
2026-04-08 03:21:05 +02:00
parent 228ad7026a
commit b735bab5a5
120 changed files with 1444 additions and 24933 deletions

View File

@ -28,17 +28,20 @@ CREATE OR REPLACE VIEW ja4_processing.view_form_bruteforce_detected AS
SELECT
src_ip,
host,
-- JA4 le plus fréquent pour ce couple IP+hôte
argMax(any(tcp_fp_raw), sum(hits)) AS ja4,
-- Nombre total de requêtes (toutes méthodes)
sum(hits) AS hits,
-- Proxy pour les soumissions de formulaire : nombre de requêtes POST
-- (count_post = SimpleAggregateFunction(sum) dans agg_host_ip_ja4_1h)
sum(count_post) AS query_params_count
FROM ja4_processing.agg_host_ip_ja4_1h
WHERE window_start >= now() - INTERVAL 24 HOUR
-- JA4 avec le plus de hits pour ce couple IP+hôte (from subquery)
argMax(ja4, ja4_hits) AS ja4,
sum(ja4_hits) AS hits,
sum(ja4_posts) AS query_params_count
FROM (
SELECT
src_ip, host, ja4,
sum(hits) AS ja4_hits,
sum(count_post) AS ja4_posts
FROM ja4_processing.agg_host_ip_ja4_1h
WHERE window_start >= now() - INTERVAL 24 HOUR
GROUP BY src_ip, host, ja4
) sub
GROUP BY src_ip, host
-- Seuil : au moins 10 POST pour qualifier d'activité brute-force
HAVING query_params_count >= 10;