ja4-platform/.github/copilot-instructions.md

Copilot Instructions — ja4-platform

What is this?

A monorepo for a JA4/JA3 TLS fingerprinting security pipeline. Five services capture network traffic, correlate logs, detect bots via ML, and present results in a SOC dashboard. All backed by ClickHouse.

Data flow: mod-reqin-log (Apache HTTP logs) → unix socket → correlator ← unix socket ← sentinel (TLS/TCP capture) → ClickHouse → bot-detector (ML scoring) → dashboard (FastAPI SOC UI)

Build, test, lint

All builds run in Docker — no native Go/Python/C toolchain required on the host.

# Full suite
make test-all        # run all tests (Docker)
make build-all       # build all service images
make rpm-all         # build RPMs (sentinel, correlator, mod-reqin-log) for el8/el9/el10

# Per-service tests
make test-sentinel          # Go tests (needs --cap-add=NET_RAW inside)
make test-correlator        # Go tests with 80% coverage gate
make test-bot-detector      # Python pytest
make test-dashboard         # Python pytest
make test-ja4common-python  # Python pytest (shared lib)
make test-mod-reqin-log     # C cmocka tests

# Single Go test (from service dir, or via Docker):
docker run --rm -v $(pwd):/build -w /build/services/correlator golang:1.24 \
  go test -v -run TestConfigLoad ./internal/config/

# Single Python test (from repo root):
docker build -f services/dashboard/Dockerfile.tests -t dash-tests .
docker run --rm dash-tests pytest backend/tests/test_metrics.py -v -k test_health

# Faster correlator build (skip tests):
docker build --target builder --build-arg SKIP_TESTS=true -f services/correlator/Dockerfile .

# Linting (Go only — no Python linter configured)
cd services/sentinel && go vet ./... && gofmt -l .
cd services/correlator && go vet ./... && gofmt -l .

# Full-stack integration tests (Docker Compose, resets DB each run)
make test-integration          # 8 phases: build → start → schema → traffic → pipeline → dashboard → bot-detector → sentinel
make test-integration-keep     # same but leaves stack running after
make test-integration-down     # tear down integration stack
# run-tests.sh also accepts: --build-only (build images without running tests)

Architecture

Go workspace (go.work, Go 1.24.6)

Three modules in the workspace:

• services/sentinel — TLS/TCP packet capture daemon (gopacket/pcap, systemd)
• services/correlator — log correlation engine, hexagonal architecture
• shared/go/ja4common — shared logger, config, shutdown, ipfilter

Both services have a replace directive in their go.mod pointing to ../../shared/go/ja4common. The workspace takes precedence for local dev; the replace is needed for Docker builds.

Correlator hexagonal architecture

ports/source.go        → EventSource, CorrelatedLogSink, CorrelationProcessor interfaces
adapters/inbound/      → unixsocket (reads from sentinel + mod-reqin-log)
adapters/outbound/     → clickhouse, file, stdout, multi (fan-out wrapper)
domain/                → CorrelationService, CorrelatedLog, NormalizedEvent
app/                   → Orchestrator (wires everything together)
config/                → YAML config loader

Python services

• bot-detector — scikit-learn IsolationForest + DBSCAN + SHAP. Single monolithic module (bot_detector.py). Runs two parallel IF models per cycle:

  • Complet (45 features L3→L7) on correlated traffic (correlated=1, TCP+TLS+HTTP)
  • Applicatif (35 features L7-only) on uncorrelated traffic (correlated=0)
  • Optional 24h variants (Complet_24h/Applicatif_24h) when ENABLE_MULTIWINDOW=true

  model_name is part of the ORDER BY key in both ml_detected_anomalies and ml_all_scores. Uses os.getenv() directly for config, NOT pydantic-settings.

• dashboard — FastAPI + React SPA. 20 route modules in backend/routes/. Uses pydantic-settings (backend/config.py).

• shared/python/ja4_common — ClickHouseClient singleton + ClickHouseSettings (pydantic-settings). Installed as a local package in each Python Dockerfile.

C module

• mod-reqin-log — Apache HTTPD module (C11, built with apxs). Logs HTTP requests as JSON to a Unix socket. Tests use cmocka.

ClickHouse dual-database pattern

Two configurable databases (env vars with defaults):

Env var	Default	Contains
CLICKHOUSE_DB_LOGS	ja4_logs	http_logs_raw, http_logs, mv_http_logs
CLICKHOUSE_DB_PROCESSING	ja4_processing	Aggregations, ML tables, views, dicts, audit

Cross-database references exist — materialized views in one DB read from the other:

• ja4_logs.mv_http_logs references ja4_processing.dict_anubis_* and ja4_processing.dict_iplocate_asn
• ja4_processing.mv_agg_* reads FROM ja4_logs.http_logs

In Python code, always use fully qualified table names:

from ..config import settings
query = f"SELECT ... FROM {settings.CLICKHOUSE_DB_PROCESSING}.ml_detected_anomalies ..."
query = f"SELECT ... FROM {settings.CLICKHOUSE_DB_LOGS}.http_logs ..."

Never hardcode database names in queries.

In Go (correlator), the database is part of the ClickHouse DSN (clickhouse://user:pass@host:9000/ja4_logs). The target table is configurable via YAML (outputs.clickhouse.table).

SQL schema has two deployment paths:

• Base schema: shared/clickhouse/ (10 ordered files). Deploy with shared/clickhouse/deploy_schema.sh which substitutes DB names from env vars.
• Bot-detector views: services/bot-detector/deploy_views.sql — aggregation tables, MVs, ML result tables, dashboard views. Version-controlled separately (currently v14).
• Post-deploy migrations: services/correlator/sql/migrations/ — ALTER TABLE statements for existing deployments (TTL changes, ORDER BY fixes). Run manually: clickhouse-client --multiquery < file.sql.

Data retention (TTL)

Table	TTL	Partition
http_logs_raw	2 hours	toStartOfHour(ingest_time)
http_logs	30 days	toDate(log_date)
agg_host_ip_ja4_1h	7 days	toDate(window_start)
agg_header_fingerprint_1h	7 days	toDate(window_start)
ml_detected_anomalies	7 days	toDate(detected_at)
ml_all_scores	7 days	toDate(window_start)

All aggregation/ML tables use ttl_only_drop_parts=1 for efficient partition-level expiry.

Key conventions

Docker-first builds

Every service has Dockerfile (prod), Dockerfile.dev or Dockerfile.tests (tests), and Go/C services have Dockerfile.package (RPM packaging via 3-stage: builder → rpmbuild × 3 distros → alpine output).

Go config: YAML + env vars

• Sentinel: config.yml, env prefix JA4SENTINEL_
• Correlator: config.yml, env prefix LOGCORRELATOR_
• Both support SIGHUP for log rotation

Python config: pydantic-settings

• Dashboard: backend/config.py → Settings(BaseSettings) with .env file
• ja4_common: ClickHouseSettings(BaseSettings) — singleton at settings
• bot-detector: exception — uses raw os.getenv(), not pydantic-settings

Dashboard route structure

Every route file follows this pattern:

from fastapi import APIRouter, HTTPException, Query
from ..config import settings
from ..database import db

router = APIRouter()

@router.get("/api/something")
async def get_something():
    query = f"SELECT ... FROM {settings.CLICKHOUSE_DB_PROCESSING}.table_name ..."
    result = db.query(query)
    ...

RPM spec files

Located at services/<name>/packaging/rpm/<name>.spec. Version injected via --define "build_version X.Y.Z" at build time.

Inter-service communication

Services communicate via Unix sockets, not HTTP:

• sentinel → /var/run/logcorrelator/network.socket → correlator (source B: TLS/TCP data)
• mod-reqin-log → /var/run/logcorrelator/http.socket → correlator (source A: HTTP data)
• correlator → ClickHouse (batch inserts into ja4_logs.http_logs_raw)

Sentinel requires elevated privileges

Tests need --cap-add=NET_RAW --cap-add=NET_ADMIN for packet capture (pcap).

Comments standard

All code is commented in French (identifiers stay in English). Standard defined in docs/commenting-standard.md:

• Go: godoc // FuncName does X, package-level // Package foo fournit...
• Python: PEP-257 triple-quoted French docstrings on all functions/classes/modules
• C: Doxygen /** @brief ... @param ... @return ... */ before every function, /* ====== Section ====== */ banners
• Bash: standardized header block with Usage: and Variables d'environnement:
• SQL: -- === filename.sql — description === banner + -- --- Table --- section headers

Known gotchas

go.work and Docker build contexts

When building either sentinel or correlator in Docker, the build context must include both service directories because go.work references them both. The root-level Makefiles always use . (repo root) as context — don't change this.

Correlator YAML does not expand env vars

Go's YAML parser reads ${VAR:-default} as a literal string. Use hardcoded values or pass values directly in the YAML file. This is why tests/integration/platform/correlator.yml has a hardcoded DSN.

REGEXP_TREE dictionary requires ≥1 rule

dict_anubis_ua uses LAYOUT(REGEXP_TREE). If anubis_ua_rules is empty, every INSERT into http_logs_raw fails because the materialized view mv_http_logs calls dictGet() on it. The integration test init script seeds a catch-all rule.

TLS/pcap capture needs non-loopback traffic

sentinel listens on a network interface (e.g., eth0), not loopback. Traffic sent to localhost or 127.0.0.1 from the same container is invisible to pcap. In integration tests, traffic must come from a separate container crossing the Docker bridge network.

ClickHouse initialization timing

ClickHouse takes ~15-20s to initialize all 10 SQL files. Integration health checks use a 120s timeout (not the default 60s).

RPM builds must use Rocky Linux

All Dockerfile.package files use rockylinux:9 (or rockylinux:8/almalinux:10) as the build base — never Debian-based images. Reason: Rocky provides libpcap.so.1; Debian provides libpcap.so.0.8. Building sentinel on Debian and running on Rocky produces a missing library error at runtime.

ClickHouse FLAT() layout requires numeric keys

If adding a new dictionary with a String primary key, use COMPLEX_KEY_HASHED() not FLAT().