feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
129
Makefile
Normal file
129
Makefile
Normal file
@ -0,0 +1,129 @@
|
||||
# =============================================================================
|
||||
# ja4-platform — Monorepo Makefile
|
||||
# All targets use new service names:
|
||||
# sentinel, correlator, bot-detector, dashboard, mod-reqin-log
|
||||
# =============================================================================
|
||||
|
||||
.PHONY: build-all test-all rpm-all dist \
|
||||
build-sentinel test-sentinel rpm-sentinel \
|
||||
test-mod-reqin-log rpm-mod-reqin-log \
|
||||
build-correlator test-correlator rpm-correlator \
|
||||
build-bot-detector test-bot-detector \
|
||||
build-dashboard test-dashboard \
|
||||
test-ja4common-python
|
||||
|
||||
# --- Root -------------------------------------------------------------------
|
||||
|
||||
build-all: build-sentinel build-correlator build-bot-detector build-dashboard
|
||||
@echo "All services built."
|
||||
|
||||
test-all: test-sentinel test-correlator test-bot-detector test-dashboard test-ja4common-python
|
||||
@echo "All tests completed."
|
||||
|
||||
rpm-all: rpm-sentinel rpm-correlator rpm-mod-reqin-log
|
||||
@echo "All RPMs built."
|
||||
|
||||
dist: rpm-all
|
||||
@echo "Distribution packages ready in services/*/dist/"
|
||||
|
||||
# --- sentinel (was ja4sentinel) ---------------------------------------------
|
||||
|
||||
build-sentinel:
|
||||
docker build \
|
||||
--build-arg VERSION=$$(git -C services/sentinel describe --tags --always 2>/dev/null || echo dev) \
|
||||
--build-arg GIT_COMMIT=$$(git rev-parse --short HEAD 2>/dev/null || echo unknown) \
|
||||
--build-arg BUILD_TIME=$$(date -u +%Y-%m-%dT%H:%M:%SZ) \
|
||||
-f services/sentinel/Dockerfile \
|
||||
-t ja4-platform/sentinel:latest \
|
||||
.
|
||||
|
||||
test-sentinel:
|
||||
# Tests run inside Docker — no native Go required on the host
|
||||
docker build -f services/sentinel/Dockerfile.dev -t ja4-platform/sentinel-tests:latest .
|
||||
docker run --rm --cap-add=NET_RAW --cap-add=NET_ADMIN ja4-platform/sentinel-tests:latest
|
||||
|
||||
rpm-sentinel:
|
||||
# Méthode: Dockerfile.package → builder Go → rpm-builder (rpmbuild ×3) → output alpine
|
||||
docker build \
|
||||
-f services/sentinel/Dockerfile.package \
|
||||
--target output \
|
||||
--output type=local,dest=services/sentinel/dist \
|
||||
--build-arg VERSION=$(shell git -C services/sentinel describe --tags --always 2>/dev/null || echo dev) \
|
||||
.
|
||||
@echo "📦 RPMs sentinel dans services/sentinel/dist/"
|
||||
|
||||
# --- mod-reqin-log (was mod_reqin_log) --------------------------------------
|
||||
|
||||
test-mod-reqin-log:
|
||||
docker build -f services/mod-reqin-log/Dockerfile.tests -t ja4-platform/mod-reqin-log-tests:latest .
|
||||
docker run --rm ja4-platform/mod-reqin-log-tests:latest
|
||||
|
||||
rpm-mod-reqin-log:
|
||||
# Méthode: Dockerfile.package → builder C (×3 distros) → rpm-builder (rpmbuild ×3) → output alpine
|
||||
docker build \
|
||||
-f services/mod-reqin-log/Dockerfile.package \
|
||||
--target output \
|
||||
--output type=local,dest=services/mod-reqin-log/dist \
|
||||
.
|
||||
@echo "📦 RPMs mod-reqin-log dans services/mod-reqin-log/dist/"
|
||||
|
||||
# --- correlator (was logcorrelator) -----------------------------------------
|
||||
|
||||
build-correlator:
|
||||
docker build \
|
||||
-f services/correlator/Dockerfile \
|
||||
-t ja4-platform/correlator:latest \
|
||||
.
|
||||
|
||||
test-correlator:
|
||||
# Tests run inside the Dockerfile builder stage (80% coverage gate enforced)
|
||||
docker build --target builder -f services/correlator/Dockerfile -t ja4-platform/correlator-tests:latest .
|
||||
|
||||
rpm-correlator:
|
||||
# Méthode: Dockerfile.package → builder Go → rpm-builder (rpmbuild ×3) → output alpine
|
||||
docker build \
|
||||
-f services/correlator/Dockerfile.package \
|
||||
--target output \
|
||||
--output type=local,dest=services/correlator/dist \
|
||||
--build-arg VERSION=$(shell git -C services/correlator describe --tags --always 2>/dev/null || echo dev) \
|
||||
.
|
||||
@echo "📦 RPMs correlator dans services/correlator/dist/"
|
||||
|
||||
# --- bot-detector (was bot_detector) ----------------------------------------
|
||||
|
||||
build-bot-detector:
|
||||
docker build \
|
||||
-f services/bot-detector/bot_detector/Dockerfile \
|
||||
-t ja4-platform/bot-detector:latest \
|
||||
.
|
||||
|
||||
test-bot-detector:
|
||||
docker build \
|
||||
-f services/bot-detector/bot_detector/Dockerfile.tests \
|
||||
-t ja4-platform/bot-detector-tests:latest \
|
||||
.
|
||||
docker run --rm ja4-platform/bot-detector-tests:latest
|
||||
|
||||
# --- dashboard --------------------------------------------------------------
|
||||
|
||||
build-dashboard:
|
||||
docker build \
|
||||
-f services/dashboard/Dockerfile \
|
||||
-t ja4-platform/dashboard:latest \
|
||||
.
|
||||
|
||||
test-dashboard:
|
||||
docker build \
|
||||
-f services/dashboard/Dockerfile.tests \
|
||||
-t ja4-platform/dashboard-tests:latest \
|
||||
.
|
||||
docker run --rm ja4-platform/dashboard-tests:latest
|
||||
|
||||
# --- shared/python/ja4_common -----------------------------------------------
|
||||
|
||||
test-ja4common-python:
|
||||
docker build \
|
||||
-f shared/python/ja4_common/Dockerfile.tests \
|
||||
-t ja4-platform/ja4common-python-tests:latest \
|
||||
shared/python/ja4_common/
|
||||
docker run --rm ja4-platform/ja4common-python-tests:latest
|
||||
Reference in New Issue
Block a user