feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
123
README.md
Normal file
123
README.md
Normal file
@ -0,0 +1,123 @@
|
||||
# ja4-platform
|
||||
|
||||
**ja4-platform** is a monorepo security pipeline for TLS fingerprinting (JA4/JA3) and bot detection. It captures live network traffic, correlates TLS handshakes with HTTP requests, detects anomalous behavior using machine learning (Isolation Forest), and presents results through a SOC analyst dashboard — all backed by ClickHouse as the central data store.
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Linux Server (Apache) │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌─────────────────────┐ │
|
||||
│ │ mod-reqin-log │───────▶│ UNIX socket (HTTP) │──┐ │
|
||||
│ │ (Apache module) │ JSON │ /var/run/logcorr/ │ │ │
|
||||
│ │ C · httpd DSO │ │ http.socket │ │ │
|
||||
│ └─────────────────┘ └─────────────────────┘ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────┐ ┌─────────────────────┐ ┌──────────────────┐ │
|
||||
│ │ sentinel │───────▶│ UNIX socket (TLS) │─▶│ correlator │ │
|
||||
│ │ (TLS capture) │ JSON │ /var/run/logcorr/ │ │ (event join) │ │
|
||||
│ │ Go · libpcap │ │ network.socket │ │ Go · hex. arch │ │
|
||||
│ └─────────────────┘ └─────────────────────┘ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
└────────────────────────────────────────────────────────────────┼────────────┘
|
||||
│ INSERT
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ ClickHouse │
|
||||
│ mabase_prod │
|
||||
│ (all tables) │
|
||||
└────────┬─────────┘
|
||||
│ SELECT
|
||||
┌────────────────────┼────────────────────┐
|
||||
▼ ▼
|
||||
┌──────────────────┐ ┌──────────────────┐
|
||||
│ bot-detector │ │ dashboard │
|
||||
│ (ML anomaly det) │ │ (SOC web UI) │
|
||||
│ Python · sklearn │ │ FastAPI + React │
|
||||
└──────────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Language | Purpose | Interface |
|
||||
|---------|----------|---------|-----------|
|
||||
| [sentinel](docs/services/sentinel.md) | Go | Live TLS packet capture, JA4/JA3 fingerprint generation | UNIX socket (`network.socket`) |
|
||||
| [mod-reqin-log](docs/services/mod-reqin-log.md) | C | Apache HTTPD module, HTTP request JSON logging | UNIX socket (`http.socket`) |
|
||||
| [correlator](docs/services/correlator.md) | Go | Joins HTTP + TLS events by `src_ip:src_port` + time window | ClickHouse INSERT, file, stdout |
|
||||
| [bot-detector](docs/services/bot-detector.md) | Python | Isolation Forest ML anomaly detection on aggregated traffic | ClickHouse read/write, HTTP `:8080` |
|
||||
| [dashboard](docs/services/dashboard.md) | Python/JS | SOC analyst web dashboard (FastAPI + React) | HTTP `:8000` |
|
||||
|
||||
## Shared Libraries
|
||||
|
||||
| Library | Language | Description |
|
||||
|---------|----------|-------------|
|
||||
| [go/ja4common](docs/shared/go-ja4common.md) | Go | Logger, config loader, shutdown handler, IP filter |
|
||||
| [python/ja4_common](docs/shared/python-ja4common.md) | Python | ClickHouse client singleton, settings |
|
||||
|
||||
## Quickstart
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker (with BuildKit) and Docker Compose
|
||||
- `make`
|
||||
- No native Go, Python, or C toolchains required — all builds run inside Docker
|
||||
|
||||
### Build All Services
|
||||
|
||||
```bash
|
||||
make build-all
|
||||
```
|
||||
|
||||
### Run All Tests
|
||||
|
||||
```bash
|
||||
make test-all
|
||||
```
|
||||
|
||||
### Build RPM Packages
|
||||
|
||||
```bash
|
||||
make rpm-all
|
||||
# RPMs written to services/<service>/dist/
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Architecture](docs/architecture.md) | System architecture, data flow, component interactions |
|
||||
| [Development](docs/development.md) | Build, test, package, and extend the platform |
|
||||
| [Database Schema](docs/database/schema.md) | Every ClickHouse table, view, dictionary, and materialized view |
|
||||
| [Database Migrations](docs/database/migrations.md) | Migration order, application, verification, and rollback |
|
||||
|
||||
### Service Documentation
|
||||
|
||||
- [Sentinel](docs/services/sentinel.md) — TLS capture daemon
|
||||
- [mod-reqin-log](docs/services/mod-reqin-log.md) — Apache HTTP logging module
|
||||
- [Correlator](docs/services/correlator.md) — HTTP/TLS event correlation engine
|
||||
- [Bot Detector](docs/services/bot-detector.md) — ML anomaly detection
|
||||
- [Dashboard](docs/services/dashboard.md) — SOC web dashboard and API
|
||||
|
||||
### Shared Library Documentation
|
||||
|
||||
- [go-ja4common](docs/shared/go-ja4common.md) — Go shared library
|
||||
- [python-ja4common](docs/shared/python-ja4common.md) — Python shared library
|
||||
|
||||
## Go Workspace
|
||||
|
||||
The repository uses a Go workspace (`go.work`) to link the Go modules:
|
||||
|
||||
```
|
||||
go 1.21
|
||||
|
||||
use (
|
||||
./services/sentinel
|
||||
./services/correlator
|
||||
./shared/go/ja4common
|
||||
)
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
See individual service directories for license information.
|
||||
Reference in New Issue
Block a user