Files

toto b735bab5a5 feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

Complete rewrite of the SOC dashboard using FastAPI + Jinja2 + htmx + Chart.js + Tailwind CSS.
Replaces the old React/Vite frontend with server-rendered templates.

Dashboard pages:
- Overview: KPIs, timeline chart, threat distribution, top IPs
- Detections: paginated/filterable anomaly table
- Scores: ml_all_scores with AE error & XGB prob columns
- Traffic: HTTP logs with method/host filters
- IP Investigation: full deep-dive (scores, features, HTTP logs, classify)
- Classification: SOC feedback form + history
- Features: AI + thesis feature stats
- Models: scoring stats + model metadata

API: 9 JSON endpoints with parameterized queries, sort whitelists

SQL fixes:
- 05_aggregation_tables: add deduplicate_merge_projection_mode
- 11_views: fix nested aggregate (argMax inside sum)
- 12_thesis_features: remove invalid 'let' bindings, fix groupArrayIf type

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-04-08 03:21:05 +02:00

00_database.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

01_raw_tables.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

02_dictionaries.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

03_anubis_tables.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

04_mv_http_logs.sql

perf(clickhouse): P1 — partition + skipping indexes sur ml_detected_anomalies, http_logs, agg_host_ip_ja4_1h

2026-04-07 22:28:04 +02:00

05_aggregation_tables.sql

feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

2026-04-08 03:21:05 +02:00

06_ml_tables.sql

feat(bot-detector): add XGBoost supervised third voice (#10 )

2026-04-08 02:45:57 +02:00

07_ai_features_view.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

08_users.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

09_audit_table.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

10_perf_indexes.sql

perf(clickhouse): P1 — partition + skipping indexes sur ml_detected_anomalies, http_logs, agg_host_ip_ja4_1h

2026-04-07 22:28:04 +02:00

11_views.sql

feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

2026-04-08 03:21:05 +02:00

12_thesis_features.sql

feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

2026-04-08 03:21:05 +02:00

deploy_schema.sh

feat: implement thesis §5 advanced detection techniques as ClickHouse MVs

2026-04-08 01:42:52 +02:00

README.md

feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized

2026-04-07 16:42:59 +02:00

README.md

ClickHouse Migrations — ja4-platform

Migration Order

Apply these files in numeric order against the ClickHouse server:

clickhouse-client --multiquery < 00_database.sql
clickhouse-client --multiquery < 01_raw_tables.sql
clickhouse-client --multiquery < 02_dictionaries.sql
clickhouse-client --multiquery < 03_anubis_tables.sql
clickhouse-client --multiquery < 04_mv_http_logs.sql
clickhouse-client --multiquery < 05_aggregation_tables.sql
clickhouse-client --multiquery < 06_ml_tables.sql
clickhouse-client --multiquery < 07_ai_features_view.sql
clickhouse-client --multiquery < 08_users.sql
clickhouse-client --multiquery < 09_audit_table.sql

File Descriptions

File	Contents
`00_database.sql`	CREATE DATABASE
`01_raw_tables.sql`	`http_logs_raw` ingest table
`02_dictionaries.sql`	ASN geo dict, bot IP/JA4/network reference tables
`03_anubis_tables.sql`	Anubis crawler rule tables and dictionaries (UA, IP, ASN, country)
`04_mv_http_logs.sql`	Canonical `http_logs` target table + `mv_http_logs` materialized view with full Anubis enrichment
`05_aggregation_tables.sql`	`agg_host_ip_ja4_1h`, `agg_header_fingerprint_1h` + their MVs
`06_ml_tables.sql`	`ml_detected_anomalies`, `ml_all_scores`
`07_ai_features_view.sql`	`view_ai_features_1h` with Anubis enrichment
`08_users.sql`	ClickHouse users and grants
`09_audit_table.sql`	`audit_logs` table for SOC dashboard audit trail

Prerequisites

Place CSV data files in /var/lib/clickhouse/user_files/:

iplocate-ip-to-asn.csv — IP-to-ASN mapping (from IPLocate)
bot_ip.csv — Known bot IP prefixes
bot_ja4.csv — Known bot JA4 fingerprints
asn_reputation.csv — ASN reputation labels

Notes

04_mv_http_logs.sql is the canonical version of the MV, superseding the base version in services/correlator/sql/init.sql. It includes full Anubis enrichment.
All migrations are idempotent (use IF NOT EXISTS / IF EXISTS).
Anubis dictionary passwords in 03_anubis_tables.sql must be changed before production use.