ja4-platform/docs/development.md

Development Guide

This guide covers building, testing, packaging, and extending the ja4-platform monorepo. All build and test operations run inside Docker — no native Go, Python, or C toolchains are required on the host.

Prerequisites

Requirement	Minimum Version	Notes
Docker	20.10+	BuildKit enabled (DOCKER_BUILDKIT=1)
Docker Compose	2.x	For bot-detector and dashboard
make	3.81+	GNU Make
git	2.x	For version tagging

No Go, Python, or C compilers are needed on the host machine.

Building All Services

make build-all

This builds Docker images for:

• ja4-platform/sentinel:latest
• ja4-platform/correlator:latest
• ja4-platform/bot-detector:latest
• ja4-platform/dashboard:latest

mod-reqin-log is an Apache module and is only built as part of the RPM packaging process.

Building Individual Services

make build-sentinel      # Go binary in Docker
make build-correlator    # Go binary in Docker
make build-bot-detector  # Python image
make build-dashboard     # FastAPI + React image

Running Tests

make test-all

Per-Service Testing

Service	Command	Details
sentinel	make test-sentinel	Go tests with -race flag, requires NET_RAW/NET_ADMIN caps
correlator	make test-correlator	Go tests with 80% coverage gate enforced
mod-reqin-log	make test-mod-reqin-log	C unit tests (JSON serialization, config parsing, header handling)
bot-detector	make test-bot-detector	Python pytest suite
dashboard	make test-dashboard	Python pytest for FastAPI routes
ja4_common (Python)	make test-ja4common-python	Shared Python library tests

Building RPM Packages

make rpm-all

Builds RPMs for sentinel, correlator, and mod-reqin-log targeting Rocky Linux 8/9/10:

make rpm-sentinel        # → services/sentinel/dist/rpm/
make rpm-correlator      # → services/correlator/dist/rpm/
make rpm-mod-reqin-log   # → services/mod-reqin-log/dist/rpm/

Each RPM build uses a multi-stage Docker pipeline:

1. Builder stage compiles the binary (Go) or shared object (C)
2. RPM builder stage runs rpmbuild for each target distro (el8, el9, el10)
3. Output stage copies RPMs to the host via --output type=local

Distribution Packages

make dist    # Alias for rpm-all
# RPMs in services/<service>/dist/rpm/el{8,9,10}/

Local Development Workflow

Go Services (sentinel, correlator)

The go.work workspace links Go modules:

go 1.21

use (
    ./services/sentinel
    ./services/correlator
    ./shared/go/ja4common
)

If you have Go 1.21+ installed locally, you can develop without Docker:

# Run sentinel tests locally
cd services/sentinel && go test ./... -race -v

# Run correlator tests locally
cd services/correlator && go test ./... -race -cover -v

# Build sentinel binary locally (requires libpcap-dev)
cd services/sentinel && go build -o ja4sentinel ./cmd/ja4sentinel/

Python Services (bot-detector, dashboard)

# Install shared library in development mode
cd shared/python/ja4_common && pip install -e .

# Run bot-detector locally
cd services/bot-detector && pip install -r bot_detector/requirements.txt
python -m bot_detector.bot_detector

# Run dashboard locally
cd services/dashboard && pip install -r backend/requirements.txt
uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000

C Module (mod-reqin-log)

Requires apxs (Apache extension tool) and development headers:

cd services/mod-reqin-log
make build    # Compiles mod_reqin_log.so
make test     # Runs unit tests
make rpm      # Builds RPM packages

Adding a New Service

Go Service

1. Create the service directory:

   mkdir -p services/my-service/cmd/my-service
   mkdir -p services/my-service/internal
   

2. Initialize the Go module:

   cd services/my-service
   go mod init github.com/antitbone/ja4/my-service
   

3. Add to go.work:

   use (
       ./services/sentinel
       ./services/correlator
       ./services/my-service      # ← add this
       ./shared/go/ja4common
   )
   

4. Import the shared library:

   import (
       "github.com/antitbone/ja4/ja4common/logger"
       "github.com/antitbone/ja4/ja4common/config"
       "github.com/antitbone/ja4/ja4common/shutdown"
   )
   

5. Add Makefile targets:

   build-my-service:
       docker build -f services/my-service/Dockerfile -t ja4-platform/my-service:latest .
   
   test-my-service:
       docker build -f services/my-service/Dockerfile.dev -t ja4-platform/my-service-tests:latest .
       docker run --rm ja4-platform/my-service-tests:latest
   

6. Update build-all and test-all dependencies.

Python Service

1. Create the service directory with a requirements.txt or pyproject.toml.
2. Add ja4-common as a dependency (installed from shared/python/ja4_common).
3. Use from ja4_common.clickhouse import get_client for ClickHouse access.
4. Add Makefile targets following the bot-detector/dashboard pattern.

go.work Workspace

The go.work file at the repository root links all Go modules, allowing cross-module development without publishing:

go 1.21

use (
    ./services/sentinel
    ./services/correlator
    ./shared/go/ja4common
)

When adding a new Go module:

1. go mod init in the service directory
2. Add the path to go.work
3. Reference shared packages via their module path: github.com/antitbone/ja4/ja4common/...
4. Run go work sync to update the workspace

ja4_common Python Package

The shared Python package (shared/python/ja4_common) provides:

• ClickHouseSettings — pydantic-settings model reading from .env
• ClickHouseClient — singleton client with auto-reconnect
• get_client() — module-level singleton accessor

Extending ja4_common

1. Add new modules under shared/python/ja4_common/ja4_common/
2. Export them in __init__.py
3. Add dependencies to pyproject.toml
4. Run tests: make test-ja4common-python

Using in a New Service

Add to requirements.txt:

ja4-common @ file:///app/shared/python/ja4_common

Or in Docker, copy the shared library and install:

COPY shared/python/ja4_common /app/shared/python/ja4_common
RUN pip install /app/shared/python/ja4_common

Environment Variables

Each service reads configuration from environment variables and/or YAML config files. See individual service documentation for the full reference:

• Sentinel configuration
• Correlator configuration
• Bot Detector configuration
• Dashboard configuration