d469e39da7
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
124 lines
6.9 KiB
Markdown
124 lines
6.9 KiB
Markdown
# ja4-platform
|
|
|
|
**ja4-platform** is a monorepo security pipeline for TLS fingerprinting (JA4/JA3) and bot detection. It captures live network traffic, correlates TLS handshakes with HTTP requests, detects anomalous behavior using machine learning (Isolation Forest), and presents results through a SOC analyst dashboard — all backed by ClickHouse as the central data store.
|
|
|
|
## Pipeline Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ Linux Server (Apache) │
|
|
│ │
|
|
│ ┌─────────────────┐ ┌─────────────────────┐ │
|
|
│ │ mod-reqin-log │───────▶│ UNIX socket (HTTP) │──┐ │
|
|
│ │ (Apache module) │ JSON │ /var/run/logcorr/ │ │ │
|
|
│ │ C · httpd DSO │ │ http.socket │ │ │
|
|
│ └─────────────────┘ └─────────────────────┘ │ │
|
|
│ ▼ │
|
|
│ ┌─────────────────┐ ┌─────────────────────┐ ┌──────────────────┐ │
|
|
│ │ sentinel │───────▶│ UNIX socket (TLS) │─▶│ correlator │ │
|
|
│ │ (TLS capture) │ JSON │ /var/run/logcorr/ │ │ (event join) │ │
|
|
│ │ Go · libpcap │ │ network.socket │ │ Go · hex. arch │ │
|
|
│ └─────────────────┘ └─────────────────────┘ └────────┬─────────┘ │
|
|
│ │ │
|
|
└────────────────────────────────────────────────────────────────┼────────────┘
|
|
│ INSERT
|
|
▼
|
|
┌──────────────────┐
|
|
│ ClickHouse │
|
|
│ mabase_prod │
|
|
│ (all tables) │
|
|
└────────┬─────────┘
|
|
│ SELECT
|
|
┌────────────────────┼────────────────────┐
|
|
▼ ▼
|
|
┌──────────────────┐ ┌──────────────────┐
|
|
│ bot-detector │ │ dashboard │
|
|
│ (ML anomaly det) │ │ (SOC web UI) │
|
|
│ Python · sklearn │ │ FastAPI + React │
|
|
└──────────────────┘ └──────────────────┘
|
|
```
|
|
|
|
## Services
|
|
|
|
| Service | Language | Purpose | Interface |
|
|
|---------|----------|---------|-----------|
|
|
| [sentinel](docs/services/sentinel.md) | Go | Live TLS packet capture, JA4/JA3 fingerprint generation | UNIX socket (`network.socket`) |
|
|
| [mod-reqin-log](docs/services/mod-reqin-log.md) | C | Apache HTTPD module, HTTP request JSON logging | UNIX socket (`http.socket`) |
|
|
| [correlator](docs/services/correlator.md) | Go | Joins HTTP + TLS events by `src_ip:src_port` + time window | ClickHouse INSERT, file, stdout |
|
|
| [bot-detector](docs/services/bot-detector.md) | Python | Isolation Forest ML anomaly detection on aggregated traffic | ClickHouse read/write, HTTP `:8080` |
|
|
| [dashboard](docs/services/dashboard.md) | Python/JS | SOC analyst web dashboard (FastAPI + React) | HTTP `:8000` |
|
|
|
|
## Shared Libraries
|
|
|
|
| Library | Language | Description |
|
|
|---------|----------|-------------|
|
|
| [go/ja4common](docs/shared/go-ja4common.md) | Go | Logger, config loader, shutdown handler, IP filter |
|
|
| [python/ja4_common](docs/shared/python-ja4common.md) | Python | ClickHouse client singleton, settings |
|
|
|
|
## Quickstart
|
|
|
|
### Prerequisites
|
|
|
|
- Docker (with BuildKit) and Docker Compose
|
|
- `make`
|
|
- No native Go, Python, or C toolchains required — all builds run inside Docker
|
|
|
|
### Build All Services
|
|
|
|
```bash
|
|
make build-all
|
|
```
|
|
|
|
### Run All Tests
|
|
|
|
```bash
|
|
make test-all
|
|
```
|
|
|
|
### Build RPM Packages
|
|
|
|
```bash
|
|
make rpm-all
|
|
# RPMs written to services/<service>/dist/
|
|
```
|
|
|
|
## Documentation
|
|
|
|
| Document | Description |
|
|
|----------|-------------|
|
|
| [Architecture](docs/architecture.md) | System architecture, data flow, component interactions |
|
|
| [Development](docs/development.md) | Build, test, package, and extend the platform |
|
|
| [Database Schema](docs/database/schema.md) | Every ClickHouse table, view, dictionary, and materialized view |
|
|
| [Database Migrations](docs/database/migrations.md) | Migration order, application, verification, and rollback |
|
|
|
|
### Service Documentation
|
|
|
|
- [Sentinel](docs/services/sentinel.md) — TLS capture daemon
|
|
- [mod-reqin-log](docs/services/mod-reqin-log.md) — Apache HTTP logging module
|
|
- [Correlator](docs/services/correlator.md) — HTTP/TLS event correlation engine
|
|
- [Bot Detector](docs/services/bot-detector.md) — ML anomaly detection
|
|
- [Dashboard](docs/services/dashboard.md) — SOC web dashboard and API
|
|
|
|
### Shared Library Documentation
|
|
|
|
- [go-ja4common](docs/shared/go-ja4common.md) — Go shared library
|
|
- [python-ja4common](docs/shared/python-ja4common.md) — Python shared library
|
|
|
|
## Go Workspace
|
|
|
|
The repository uses a Go workspace (`go.work`) to link the Go modules:
|
|
|
|
```
|
|
go 1.21
|
|
|
|
use (
|
|
./services/sentinel
|
|
./services/correlator
|
|
./shared/go/ja4common
|
|
)
|
|
```
|
|
|
|
## License
|
|
|
|
See individual service directories for license information.
|