|
|
7a04e47041
|
fix(sql+api): fix view column mismatches and ClickHouse 24.8 JOIN issue
- view_form_bruteforce_detected: add post_count, distinct_paths, first_seen, last_seen
- view_host_ip_ja4_rotation: add host, distinct_ja4, ja4_list, window_start
- Replace uniqExact/groupUniqArray with count()/groupArray (no nested-agg error)
- api.py campaigns/graph: move a.src_ip < b.src_ip from JOIN ON to WHERE
(ClickHouse 24.8 forbids cross-table inequality in JOIN ON condition)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
2026-04-10 01:05:04 +02:00 |
|
|
|
b409a70970
|
fix(views): align SQL views with dashboard API expected columns
- view_form_bruteforce_detected: add post_count, distinct_paths, first_seen, last_seen
- view_host_ip_ja4_rotation: add host, distinct_ja4, ja4_list, window_start
- view_ip_recurrence: add worst_threat alias + top_ja4, top_host columns
All three views were missing columns referenced by /api/brute-force,
/api/ja4-rotation and /api/recurrence endpoints, causing 500 errors
on the Tactiques page.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
2026-04-10 00:59:57 +02:00 |
|
|
|
b735bab5a5
|
feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL
Complete rewrite of the SOC dashboard using FastAPI + Jinja2 + htmx + Chart.js + Tailwind CSS.
Replaces the old React/Vite frontend with server-rendered templates.
Dashboard pages:
- Overview: KPIs, timeline chart, threat distribution, top IPs
- Detections: paginated/filterable anomaly table
- Scores: ml_all_scores with AE error & XGB prob columns
- Traffic: HTTP logs with method/host filters
- IP Investigation: full deep-dive (scores, features, HTTP logs, classify)
- Classification: SOC feedback form + history
- Features: AI + thesis feature stats
- Models: scoring stats + model metadata
API: 9 JSON endpoints with parameterized queries, sort whitelists
SQL fixes:
- 05_aggregation_tables: add deduplicate_merge_projection_mode
- 11_views: fix nested aggregate (argMax inside sum)
- 12_thesis_features: remove invalid 'let' bindings, fix groupArrayIf type
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
2026-04-08 03:21:05 +02:00 |
|
|
|
14323f7b05
|
perf(clickhouse): P10 — créer les 4 vues métier manquantes + corriger préfixes DB
Bug de production : view_form_bruteforce_detected, view_host_ip_ja4_rotation,
view_dashboard_entities, view_dashboard_user_agents étaient référencées dans
13 endpoints du dashboard mais n'existaient nulle part dans le schéma.
Tous ces endpoints retournaient HTTP 500 en production.
shared/clickhouse/11_views.sql (nouveau) :
view_form_bruteforce_detected
Source : agg_host_ip_ja4_1h (24h)
Logique : GROUP BY (src_ip, host) HAVING count_post >= 10
Usage : bruteforce.py (3 endpoints), investigation_summary.py
view_host_ip_ja4_rotation
Source : agg_host_ip_ja4_1h (24h)
Logique : uniqExact(ja4) par src_ip, HAVING >= 2 (rotation de fingerprint)
Usage : rotation.py (3 endpoints), investigation_summary.py
view_dashboard_entities
Source : http_logs (7 jours), UNION ALL 5 branches (ip/ja4/country/asn/host)
Colonnes : entity_type, entity_value, src_ip, ja4, host, log_date,
client_headers Array(String), asns Array, countries Array,
user_agents Array
Usage : entities.py (5 endpoints), clustering.py
view_dashboard_user_agents
Source : http_logs (7 jours), GROUP BY (src_ip, ja4, hour)
Colonnes : src_ip, ja4, hour, log_date, user_agents Array(String), requests
Usage : variability.py (4 endpoints), fingerprints.py (5 endpoints)
attributes.py (2 endpoints)
deploy_schema.sh : ajout de 10_perf_indexes.sql et 11_views.sql dans la liste
routes/variability.py + fingerprints.py :
Correction de 9 requêtes utilisant view_dashboard_user_agents sans préfixe
de base de données → remplacé par {settings.CLICKHOUSE_DB_PROCESSING}.view_*
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
2026-04-07 22:30:09 +02:00 |
|
