Files

toto 7d09c614c3 feat: browser JA4 detection, Anubis bot rules, worldwide ASN data

- Add generate_browser_ja4.py: 1,186 browser JA4 fingerprints from FoxIO + ja4db.com
  covering 11 families (Chromium, Firefox, Safari, Edge, Tor, Opera, Vivaldi...)
- Rewrite generate_bot_ip.py: Anubis YAML rules (Google, Bing, Apple, DuckDuck,
  OpenAI, Perplexity bots) + Tor exit nodes + cloud scanner IPs (3,555 entries)
- Rewrite generate_asn_data.py: worldwide iptoasn.com data (78,049 ASNs, 714K CIDRs)
- Add dict_browser_ja4 ClickHouse dictionary + browser_family in AI features views
- Add /api/browsers dashboard endpoint
- Fix CSV quoting for fields containing commas (User-Agent strings)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-04-08 15:27:37 +02:00

00_database.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

01_raw_tables.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

02_dictionaries.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

03_anubis_tables.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

04_mv_http_logs.sql

perf(clickhouse): P1 — partition + skipping indexes sur ml_detected_anomalies, http_logs, agg_host_ip_ja4_1h

2026-04-07 22:28:04 +02:00

05_aggregation_tables.sql

feat: browser JA4 detection, Anubis bot rules, worldwide ASN data

2026-04-08 15:27:37 +02:00

06_ml_tables.sql

feat(bot-detector): add XGBoost supervised third voice (#10 )

2026-04-08 02:45:57 +02:00

07_ai_features_view.sql

feat: browser JA4 detection, Anubis bot rules, worldwide ASN data

2026-04-08 15:27:37 +02:00

08_users.sql

feat: split ClickHouse into dual configurable databases (ja4_logs / ja4_processing)

2026-04-07 19:10:35 +02:00

09_audit_table.sql

feat: full-stack Docker Compose integration tests

2026-04-07 20:33:25 +02:00

10_perf_indexes.sql

perf(clickhouse): P1 — partition + skipping indexes sur ml_detected_anomalies, http_logs, agg_host_ip_ja4_1h

2026-04-07 22:28:04 +02:00

11_views.sql

feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

2026-04-08 03:21:05 +02:00

12_thesis_features.sql

feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL

2026-04-08 03:21:05 +02:00

deploy_schema.sh

feat: implement thesis §5 advanced detection techniques as ClickHouse MVs

2026-04-08 01:42:52 +02:00

README.md

feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized

2026-04-07 16:42:59 +02:00

README.md

ClickHouse Migrations — ja4-platform

Migration Order

Apply these files in numeric order against the ClickHouse server:

clickhouse-client --multiquery < 00_database.sql
clickhouse-client --multiquery < 01_raw_tables.sql
clickhouse-client --multiquery < 02_dictionaries.sql
clickhouse-client --multiquery < 03_anubis_tables.sql
clickhouse-client --multiquery < 04_mv_http_logs.sql
clickhouse-client --multiquery < 05_aggregation_tables.sql
clickhouse-client --multiquery < 06_ml_tables.sql
clickhouse-client --multiquery < 07_ai_features_view.sql
clickhouse-client --multiquery < 08_users.sql
clickhouse-client --multiquery < 09_audit_table.sql

File Descriptions

File	Contents
`00_database.sql`	CREATE DATABASE
`01_raw_tables.sql`	`http_logs_raw` ingest table
`02_dictionaries.sql`	ASN geo dict, bot IP/JA4/network reference tables
`03_anubis_tables.sql`	Anubis crawler rule tables and dictionaries (UA, IP, ASN, country)
`04_mv_http_logs.sql`	Canonical `http_logs` target table + `mv_http_logs` materialized view with full Anubis enrichment
`05_aggregation_tables.sql`	`agg_host_ip_ja4_1h`, `agg_header_fingerprint_1h` + their MVs
`06_ml_tables.sql`	`ml_detected_anomalies`, `ml_all_scores`
`07_ai_features_view.sql`	`view_ai_features_1h` with Anubis enrichment
`08_users.sql`	ClickHouse users and grants
`09_audit_table.sql`	`audit_logs` table for SOC dashboard audit trail

Prerequisites

Place CSV data files in /var/lib/clickhouse/user_files/:

iplocate-ip-to-asn.csv — IP-to-ASN mapping (from IPLocate)
bot_ip.csv — Known bot IP prefixes
bot_ja4.csv — Known bot JA4 fingerprints
asn_reputation.csv — ASN reputation labels

Notes

04_mv_http_logs.sql is the canonical version of the MV, superseding the base version in services/correlator/sql/init.sql. It includes full Anubis enrichment.
All migrations are idempotent (use IF NOT EXISTS / IF EXISTS).
Anubis dictionary passwords in 03_anubis_tables.sql must be changed before production use.