039086a0b3
SQL: - Ajout 5 colonnes d'agrégation (count_xff, count_unusual_ct, count_non_std_port, count_login_post, sec_ch_mobile_mismatch) - Exposition de 5 features calculées dans view_ai_features_1h - Migration ALTER TABLE pour déploiements existants Bot-detector: - 7 nouvelles features ML (has_xff, unusual_content_type_ratio, non_standard_port_ratio, login_post_concentration, sec_ch_mobile_mismatch, true_window_size, window_mss_ratio) - Propagation campaign_id vers ml_all_scores (était toujours -1) - Escalade campagne : HIGH→CRITICAL si cluster ≥5 membres Dashboard: - Page Tactiques SOC : brute-force, rotation JA4, récurrence, alertes temps réel — 4 KPIs + 4 panneaux + infobulles doc - Ajout fmtDate() helper global - Navigation sidebar mise à jour Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ja4-platform
ja4-platform is a monorepo security pipeline for TLS fingerprinting (JA4/JA3) and bot detection. It captures live network traffic, correlates TLS handshakes with HTTP requests, detects anomalous behavior using machine learning (Isolation Forest), and presents results through a SOC analyst dashboard — all backed by ClickHouse as the central data store.
Pipeline Overview
┌─────────────────────────────────────────────────────────────────────────────┐
│ Linux Server (Apache) │
│ │
│ ┌─────────────────┐ ┌─────────────────────┐ │
│ │ mod-reqin-log │───────▶│ UNIX socket (HTTP) │──┐ │
│ │ (Apache module) │ JSON │ /var/run/logcorr/ │ │ │
│ │ C · httpd DSO │ │ http.socket │ │ │
│ └─────────────────┘ └─────────────────────┘ │ │
│ ▼ │
│ ┌─────────────────┐ ┌─────────────────────┐ ┌──────────────────┐ │
│ │ sentinel │───────▶│ UNIX socket (TLS) │─▶│ correlator │ │
│ │ (TLS capture) │ JSON │ /var/run/logcorr/ │ │ (event join) │ │
│ │ Go · libpcap │ │ network.socket │ │ Go · hex. arch │ │
│ └─────────────────┘ └─────────────────────┘ └────────┬─────────┘ │
│ │ │
└────────────────────────────────────────────────────────────────┼────────────┘
│ INSERT
▼
┌──────────────────┐
│ ClickHouse │
│ ja4_processing │
│ (all tables) │
└────────┬─────────┘
│ SELECT
┌────────────────────┼────────────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ bot-detector │ │ dashboard │
│ (ML anomaly det) │ │ (SOC web UI) │
│ Python · sklearn │ │ FastAPI + React │
└──────────────────┘ └──────────────────┘
Services
| Service | Language | Purpose | Interface |
|---|---|---|---|
| sentinel | Go | Live TLS packet capture, JA4/JA3 fingerprint generation | UNIX socket (network.socket) |
| mod-reqin-log | C | Apache HTTPD module, HTTP request JSON logging | UNIX socket (http.socket) |
| correlator | Go | Joins HTTP + TLS events by src_ip:src_port + time window |
ClickHouse INSERT, file, stdout |
| bot-detector | Python | Isolation Forest ML anomaly detection on aggregated traffic | ClickHouse read/write, HTTP :8080 |
| dashboard | Python/JS | SOC analyst web dashboard (FastAPI + React) | HTTP :8000 |
Shared Libraries
| Library | Language | Description |
|---|---|---|
| go/ja4common | Go | Logger, config loader, shutdown handler, IP filter |
| python/ja4_common | Python | ClickHouse client singleton, settings |
Quickstart
Prerequisites
- Docker (with BuildKit) and Docker Compose
make- No native Go, Python, or C toolchains required — all builds run inside Docker
Build All Services
make build-all
Run All Tests
make test-all
Build RPM Packages
make rpm-all
# RPMs written to services/<service>/dist/
Documentation
| Document | Description |
|---|---|
| Architecture | System architecture, data flow, component interactions |
| Development | Build, test, package, and extend the platform |
| Database Schema | Every ClickHouse table, view, dictionary, and materialized view |
| Database Migrations | Migration order, application, verification, and rollback |
Service Documentation
- Sentinel — TLS capture daemon
- mod-reqin-log — Apache HTTP logging module
- Correlator — HTTP/TLS event correlation engine
- Bot Detector — ML anomaly detection
- Dashboard — SOC web dashboard and API
Shared Library Documentation
- go-ja4common — Go shared library
- python-ja4common — Python shared library
Go Workspace
The repository uses a Go workspace (go.work) to link the Go modules:
go 1.21
use (
./services/sentinel
./services/correlator
./shared/go/ja4common
)
License
See individual service directories for license information.