Files

toto 039086a0b3 feat: nouvelles techniques de détection et page tactiques SOC

SQL:
- Ajout 5 colonnes d'agrégation (count_xff, count_unusual_ct,
  count_non_std_port, count_login_post, sec_ch_mobile_mismatch)
- Exposition de 5 features calculées dans view_ai_features_1h
- Migration ALTER TABLE pour déploiements existants

Bot-detector:
- 7 nouvelles features ML (has_xff, unusual_content_type_ratio,
  non_standard_port_ratio, login_post_concentration,
  sec_ch_mobile_mismatch, true_window_size, window_mss_ratio)
- Propagation campaign_id vers ml_all_scores (était toujours -1)
- Escalade campagne : HIGH→CRITICAL si cluster ≥5 membres

Dashboard:
- Page Tactiques SOC : brute-force, rotation JA4, récurrence,
  alertes temps réel — 4 KPIs + 4 panneaux + infobulles doc
- Ajout fmtDate() helper global
- Navigation sidebar mise à jour

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-04-09 14:29:18 +02:00

00_database.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

01_raw_tables.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

02_dictionaries.sql

fix: ASN dictionary pipeline + verbose bot-detector logging

2026-04-08 17:43:55 +02:00

03_anubis_tables.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

04_mv_http_logs.sql

fix: ASN dictionary pipeline + verbose bot-detector logging

2026-04-08 17:43:55 +02:00

05_aggregation_tables.sql

feat: nouvelles techniques de détection et page tactiques SOC

2026-04-09 14:29:18 +02:00

06_ml_tables.sql

fix: P0 audit bugs — bot-detector + dashboard + SQL

2026-04-08 23:33:00 +02:00

07_ai_features_view.sql

feat: nouvelles techniques de détection et page tactiques SOC

2026-04-09 14:29:18 +02:00

08_users.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

09_audit_table.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

10_perf_indexes.sql

perf(clickhouse): P1 — partition + skipping indexes sur ml_detected_anomalies, http_logs, agg_host_ip_ja4_1h

2026-04-07 22:28:04 +02:00

11_views.sql

feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

2026-04-08 03:21:05 +02:00

12_thesis_features.sql

fix: P0 audit bugs — bot-detector + dashboard + SQL

2026-04-08 23:33:00 +02:00

deploy_schema.sh

feat: implement thesis §5 advanced detection techniques as ClickHouse MVs

2026-04-08 01:42:52 +02:00

README.md

feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized

2026-04-07 16:42:59 +02:00

README.md

ClickHouse Migrations — ja4-platform

Migration Order

Apply these files in numeric order against the ClickHouse server:

clickhouse-client --multiquery < 00_database.sql
clickhouse-client --multiquery < 01_raw_tables.sql
clickhouse-client --multiquery < 02_dictionaries.sql
clickhouse-client --multiquery < 03_anubis_tables.sql
clickhouse-client --multiquery < 04_mv_http_logs.sql
clickhouse-client --multiquery < 05_aggregation_tables.sql
clickhouse-client --multiquery < 06_ml_tables.sql
clickhouse-client --multiquery < 07_ai_features_view.sql
clickhouse-client --multiquery < 08_users.sql
clickhouse-client --multiquery < 09_audit_table.sql

File Descriptions

File	Contents
`00_database.sql`	CREATE DATABASE
`01_raw_tables.sql`	`http_logs_raw` ingest table
`02_dictionaries.sql`	ASN geo dict, bot IP/JA4/network reference tables
`03_anubis_tables.sql`	Anubis crawler rule tables and dictionaries (UA, IP, ASN, country)
`04_mv_http_logs.sql`	Canonical `http_logs` target table + `mv_http_logs` materialized view with full Anubis enrichment
`05_aggregation_tables.sql`	`agg_host_ip_ja4_1h`, `agg_header_fingerprint_1h` + their MVs
`06_ml_tables.sql`	`ml_detected_anomalies`, `ml_all_scores`
`07_ai_features_view.sql`	`view_ai_features_1h` with Anubis enrichment
`08_users.sql`	ClickHouse users and grants
`09_audit_table.sql`	`audit_logs` table for SOC dashboard audit trail

Prerequisites

Place CSV data files in /var/lib/clickhouse/user_files/:

iplocate-ip-to-asn.csv — IP-to-ASN mapping (from IPLocate)
bot_ip.csv — Known bot IP prefixes
bot_ja4.csv — Known bot JA4 fingerprints
asn_reputation.csv — ASN reputation labels

Notes

04_mv_http_logs.sql is the canonical version of the MV, superseding the base version in services/correlator/sql/init.sql. It includes full Anubis enrichment.
All migrations are idempotent (use IF NOT EXISTS / IF EXISTS).
Anubis dictionary passwords in 03_anubis_tables.sql must be changed before production use.