feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
244
docs/shared/go-ja4common.md
Normal file
244
docs/shared/go-ja4common.md
Normal file
@ -0,0 +1,244 @@
|
||||
# go-ja4common
|
||||
|
||||
`ja4common` is the shared Go library for the ja4-platform, providing unified logging, YAML configuration loading with environment variable overrides, graceful shutdown handling, and IP address filtering. It is used by both [sentinel](../services/sentinel.md) and [correlator](../services/correlator.md).
|
||||
|
||||
**Module path**: `github.com/antitbone/ja4/ja4common`
|
||||
|
||||
**Go version**: 1.21+
|
||||
|
||||
**Dependencies**: `gopkg.in/yaml.v3`
|
||||
|
||||
## Packages
|
||||
|
||||
### logger
|
||||
|
||||
Unified structured logging with two styles:
|
||||
- **Prefix+Fields style** (correlator pattern) — `Logger`
|
||||
- **Component style** (sentinel pattern) — `ComponentLogger`
|
||||
|
||||
#### Types
|
||||
|
||||
```go
|
||||
type LogLevel int
|
||||
|
||||
const (
|
||||
DEBUG LogLevel = iota
|
||||
INFO
|
||||
WARN
|
||||
ERROR
|
||||
)
|
||||
```
|
||||
|
||||
#### Logger API
|
||||
|
||||
| Method | Signature | Description |
|
||||
|--------|-----------|-------------|
|
||||
| `New` | `New(prefix string) *Logger` | Create logger with INFO level |
|
||||
| `NewWithLevel` | `NewWithLevel(prefix, level string) *Logger` | Create logger with specified level |
|
||||
| `SetLevel` | `(l *Logger) SetLevel(level string)` | Change minimum log level at runtime |
|
||||
| `ShouldLog` | `(l *Logger) ShouldLog(level LogLevel) bool` | Check if level would be logged |
|
||||
| `WithFields` | `(l *Logger) WithFields(fields map[string]any) *Logger` | Return new logger with additional fields |
|
||||
| `Info` | `(l *Logger) Info(msg string)` | Log info message |
|
||||
| `Infof` | `(l *Logger) Infof(msg string, args ...any)` | Log formatted info |
|
||||
| `Warn` | `(l *Logger) Warn(msg string)` | Log warning |
|
||||
| `Warnf` | `(l *Logger) Warnf(msg string, args ...any)` | Log formatted warning |
|
||||
| `Error` | `(l *Logger) Error(msg string, err error)` | Log error with optional error value |
|
||||
| `Debug` | `(l *Logger) Debug(msg string)` | Log debug message |
|
||||
| `Debugf` | `(l *Logger) Debugf(msg string, args ...any)` | Log formatted debug |
|
||||
| `ParseLogLevel` | `ParseLogLevel(level string) LogLevel` | Parse string to LogLevel |
|
||||
|
||||
#### ComponentLogger API
|
||||
|
||||
Wraps `Logger` to satisfy sentinel's component-based logging interface:
|
||||
|
||||
| Method | Signature | Description |
|
||||
|--------|-----------|-------------|
|
||||
| `NewComponentLogger` | `NewComponentLogger(level string) *ComponentLogger` | Create component logger |
|
||||
| `Log` | `(c *ComponentLogger) Log(component, level, message string, details map[string]string)` | Log with component context |
|
||||
| `Debug` | `(c *ComponentLogger) Debug(component, message string, details map[string]string)` | Debug with component |
|
||||
| `Info` | `(c *ComponentLogger) Info(component, message string, details map[string]string)` | Info with component |
|
||||
| `Warn` | `(c *ComponentLogger) Warn(component, message string, details map[string]string)` | Warn with component |
|
||||
| `Error` | `(c *ComponentLogger) Error(component, message string, details map[string]string)` | Error with component |
|
||||
|
||||
#### Usage Example
|
||||
|
||||
```go
|
||||
import "github.com/antitbone/ja4/ja4common/logger"
|
||||
|
||||
// Prefix+Fields style
|
||||
log := logger.NewWithLevel("myservice", "DEBUG")
|
||||
log.Info("starting up")
|
||||
log.WithFields(map[string]any{"port": 8080}).Info("listening")
|
||||
|
||||
// Component style (sentinel compatibility)
|
||||
clog := logger.NewComponentLogger("info")
|
||||
clog.Info("capture", "packets received", map[string]string{"count": "1000"})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### config
|
||||
|
||||
Generic YAML configuration loading with environment variable overrides using struct tags.
|
||||
|
||||
#### API
|
||||
|
||||
| Function | Signature | Description |
|
||||
|----------|-----------|-------------|
|
||||
| `LoadYAML` | `LoadYAML[T any](path string, optional bool) (T, error)` | Load and unmarshal YAML file |
|
||||
| `OverrideFromEnv` | `OverrideFromEnv[T any](cfg *T, envPrefix string) error` | Apply env var overrides via `env` struct tags |
|
||||
|
||||
#### Supported Types for Environment Override
|
||||
|
||||
- `string`
|
||||
- `int`, `int8`, `int16`, `int32`, `int64`
|
||||
- `uint`, `uint8`, `uint16`, `uint32`, `uint64`
|
||||
- `bool`
|
||||
- `[]string` (comma-separated)
|
||||
|
||||
#### Usage Example
|
||||
|
||||
```go
|
||||
import "github.com/antitbone/ja4/ja4common/config"
|
||||
|
||||
type MyConfig struct {
|
||||
Host string `yaml:"host" env:"HOST"`
|
||||
Port int `yaml:"port" env:"PORT"`
|
||||
Debug bool `yaml:"debug" env:"DEBUG"`
|
||||
Tags []string `yaml:"tags" env:"TAGS"`
|
||||
}
|
||||
|
||||
// Load YAML (optional=true means missing file returns zero value)
|
||||
cfg, err := config.LoadYAML[MyConfig]("config.yml", true)
|
||||
|
||||
// Override from environment (prefix="" means use tag directly)
|
||||
err = config.OverrideFromEnv(&cfg, "MYAPP")
|
||||
// Reads: MYAPP_HOST, MYAPP_PORT, MYAPP_DEBUG, MYAPP_TAGS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### shutdown
|
||||
|
||||
Graceful shutdown handler that blocks until `SIGTERM`/`SIGINT`, then runs cleanup hooks.
|
||||
|
||||
#### API
|
||||
|
||||
```go
|
||||
type Hook struct {
|
||||
Name string
|
||||
Fn func() error
|
||||
}
|
||||
|
||||
func Handle(ctx context.Context, cancel context.CancelFunc, hooks []Hook, logger simpleLogger)
|
||||
```
|
||||
|
||||
The `Handle` function:
|
||||
1. Blocks until `SIGTERM`, `SIGINT`, or context cancellation
|
||||
2. Calls `cancel()` to propagate shutdown
|
||||
3. Runs all hooks in order, logging errors but not aborting
|
||||
|
||||
#### Usage Example
|
||||
|
||||
```go
|
||||
import "github.com/antitbone/ja4/ja4common/shutdown"
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
|
||||
hooks := []shutdown.Hook{
|
||||
{Name: "close-db", Fn: func() error { return db.Close() }},
|
||||
{Name: "flush-logs", Fn: func() error { return logger.Flush() }},
|
||||
}
|
||||
|
||||
// This blocks until signal received
|
||||
shutdown.Handle(ctx, cancel, hooks, myLogger)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ipfilter
|
||||
|
||||
IP address and CIDR range matching for source IP exclusion.
|
||||
|
||||
#### API
|
||||
|
||||
| Method | Signature | Description |
|
||||
|--------|-----------|-------------|
|
||||
| `New` | `New(excludeList []string) (*Filter, error)` | Create filter from IP/CIDR list |
|
||||
| `ShouldExclude` | `(f *Filter) ShouldExclude(ipStr string) bool` | Check if IP should be excluded |
|
||||
| `Count` | `(f *Filter) Count() (ips int, networks int)` | Return number of loaded entries |
|
||||
|
||||
Accepts: single IPs (`192.168.1.1`), CIDR ranges (`10.0.0.0/8`), IPv6 addresses and ranges.
|
||||
|
||||
#### Usage Example
|
||||
|
||||
```go
|
||||
import "github.com/antitbone/ja4/ja4common/ipfilter"
|
||||
|
||||
filter, err := ipfilter.New([]string{
|
||||
"10.0.0.0/8",
|
||||
"192.168.1.1",
|
||||
"2001:db8::/32",
|
||||
})
|
||||
|
||||
if filter.ShouldExclude("10.0.0.5") {
|
||||
// Skip this IP
|
||||
}
|
||||
|
||||
ips, nets := filter.Count() // 1 IP, 2 networks
|
||||
```
|
||||
|
||||
## Using from a New Service
|
||||
|
||||
### 1. Add to go.mod
|
||||
|
||||
```bash
|
||||
cd services/my-service
|
||||
go mod init github.com/antitbone/ja4/my-service
|
||||
```
|
||||
|
||||
Add the dependency:
|
||||
```
|
||||
require github.com/antitbone/ja4/ja4common v0.0.0
|
||||
```
|
||||
|
||||
### 2. Add to go.work
|
||||
|
||||
In the repository root `go.work`:
|
||||
```
|
||||
use (
|
||||
./services/sentinel
|
||||
./services/correlator
|
||||
./services/my-service // ← add
|
||||
./shared/go/ja4common
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Import and Use
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/antitbone/ja4/ja4common/config"
|
||||
"github.com/antitbone/ja4/ja4common/logger"
|
||||
"github.com/antitbone/ja4/ja4common/shutdown"
|
||||
)
|
||||
|
||||
func main() {
|
||||
log := logger.NewWithLevel("myservice", "INFO")
|
||||
|
||||
cfg, _ := config.LoadYAML[MyConfig]("config.yml", true)
|
||||
config.OverrideFromEnv(&cfg, "MYSERVICE")
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
shutdown.Handle(ctx, cancel, nil, log)
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Sync Workspace
|
||||
|
||||
```bash
|
||||
go work sync
|
||||
```
|
||||
216
docs/shared/python-ja4common.md
Normal file
216
docs/shared/python-ja4common.md
Normal file
@ -0,0 +1,216 @@
|
||||
# python-ja4common
|
||||
|
||||
`ja4_common` is the shared Python library for the ja4-platform, providing a unified ClickHouse client singleton and configuration settings. It is used by [bot-detector](../services/bot-detector.md) and [dashboard](../services/dashboard.md).
|
||||
|
||||
**Package name**: `ja4-common`
|
||||
|
||||
**Python version**: ≥ 3.11
|
||||
|
||||
**Dependencies**:
|
||||
- `clickhouse-connect >= 0.8.0`
|
||||
- `pydantic-settings >= 2.1.0`
|
||||
|
||||
## ClickHouseSettings
|
||||
|
||||
Pydantic-settings model that reads configuration from environment variables and `.env` files.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Default | Env Variable | Description |
|
||||
|-------|------|---------|-------------|-------------|
|
||||
| `CLICKHOUSE_HOST` | str | `"clickhouse"` | `CLICKHOUSE_HOST` | ClickHouse server hostname |
|
||||
| `CLICKHOUSE_PORT` | int | `8123` | `CLICKHOUSE_PORT` | ClickHouse HTTP API port |
|
||||
| `CLICKHOUSE_DB` | str | `"mabase_prod"` | `CLICKHOUSE_DB` | Database name |
|
||||
| `CLICKHOUSE_USER` | str | `"admin"` | `CLICKHOUSE_USER` | Username for authentication |
|
||||
| `CLICKHOUSE_PASSWORD` | str | `""` | `CLICKHOUSE_PASSWORD` | Password for authentication |
|
||||
|
||||
### Configuration Sources
|
||||
|
||||
Settings are loaded in order of precedence:
|
||||
1. **Environment variables** (highest priority)
|
||||
2. **`.env` file** in the current working directory
|
||||
3. **Default values** (lowest priority)
|
||||
|
||||
Environment variable names are **case-sensitive** (e.g., `CLICKHOUSE_HOST`, not `clickhouse_host`).
|
||||
|
||||
### Usage
|
||||
|
||||
```python
|
||||
from ja4_common.settings import settings
|
||||
|
||||
print(settings.CLICKHOUSE_HOST) # "clickhouse" or from env
|
||||
print(settings.CLICKHOUSE_PORT) # 8123 or from env
|
||||
```
|
||||
|
||||
## ClickHouseClient
|
||||
|
||||
Wraps `clickhouse_connect` with auto-reconnection and a clean API.
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Signature | Description |
|
||||
|--------|-----------|-------------|
|
||||
| `connect` | `connect() -> Client` | Returns the underlying `clickhouse_connect` client, creating or reconnecting as needed |
|
||||
| `query` | `query(query: str, params: dict = None)` | Execute a SELECT query, returns result set |
|
||||
| `command` | `command(query: str, params: dict = None)` | Execute a DDL/DML command (CREATE, INSERT, etc.) |
|
||||
| `insert` | `insert(table: str, data, column_names=None)` | Bulk insert data into a table |
|
||||
| `close` | `close()` | Close the connection and release resources |
|
||||
|
||||
### Auto-Reconnection
|
||||
|
||||
The `connect()` method automatically reconnects if the current connection is lost:
|
||||
|
||||
```python
|
||||
def connect(self):
|
||||
if self._client is None or not self._ping():
|
||||
self._client = clickhouse_connect.get_client(
|
||||
host=settings.CLICKHOUSE_HOST,
|
||||
port=settings.CLICKHOUSE_PORT,
|
||||
database=settings.CLICKHOUSE_DB,
|
||||
user=settings.CLICKHOUSE_USER,
|
||||
password=settings.CLICKHOUSE_PASSWORD,
|
||||
connect_timeout=10,
|
||||
)
|
||||
return self._client
|
||||
```
|
||||
|
||||
### Usage Example
|
||||
|
||||
```python
|
||||
from ja4_common.clickhouse import get_client
|
||||
|
||||
client = get_client()
|
||||
|
||||
# SELECT query
|
||||
result = client.query("SELECT count() FROM http_logs WHERE src_ip = {ip:String}", {"ip": "203.0.113.42"})
|
||||
print(result.result_rows)
|
||||
|
||||
# INSERT
|
||||
client.insert("audit_logs", [[datetime.now(), "analyst1", "investigate", "ip", "203.0.113.42"]],
|
||||
column_names=["timestamp", "user_name", "action", "entity_type", "entity_id"])
|
||||
|
||||
# Command
|
||||
client.command("OPTIMIZE TABLE http_logs FINAL")
|
||||
```
|
||||
|
||||
## get_client() Singleton
|
||||
|
||||
The `get_client()` function provides a module-level singleton `ClickHouseClient`:
|
||||
|
||||
```python
|
||||
from ja4_common.clickhouse import get_client
|
||||
|
||||
# First call creates the client
|
||||
client1 = get_client()
|
||||
|
||||
# Subsequent calls return the same instance
|
||||
client2 = get_client()
|
||||
assert client1 is client2
|
||||
```
|
||||
|
||||
### Implementation
|
||||
|
||||
```python
|
||||
_client: Optional[ClickHouseClient] = None
|
||||
|
||||
def get_client() -> ClickHouseClient:
|
||||
global _client
|
||||
if _client is None:
|
||||
_client = ClickHouseClient()
|
||||
return _client
|
||||
```
|
||||
|
||||
## Using from a New Service
|
||||
|
||||
### 1. Add Dependency
|
||||
|
||||
In your service's `requirements.txt`:
|
||||
```
|
||||
ja4-common @ file:///app/shared/python/ja4_common
|
||||
```
|
||||
|
||||
Or in `pyproject.toml`:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"ja4-common",
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Docker Setup
|
||||
|
||||
```dockerfile
|
||||
# Copy shared library
|
||||
COPY shared/python/ja4_common /app/shared/python/ja4_common
|
||||
RUN pip install /app/shared/python/ja4_common
|
||||
|
||||
# Copy service code
|
||||
COPY services/my-service /app/services/my-service
|
||||
```
|
||||
|
||||
### 3. Use in Code
|
||||
|
||||
```python
|
||||
from ja4_common.clickhouse import get_client
|
||||
from ja4_common.settings import settings
|
||||
|
||||
# Access settings
|
||||
print(f"Connecting to {settings.CLICKHOUSE_HOST}:{settings.CLICKHOUSE_PORT}")
|
||||
|
||||
# Use client
|
||||
db = get_client()
|
||||
result = db.query("SELECT count() FROM ml_detected_anomalies")
|
||||
```
|
||||
|
||||
### 4. Environment Configuration
|
||||
|
||||
Create a `.env` file or set environment variables:
|
||||
```bash
|
||||
CLICKHOUSE_HOST=clickhouse.example.com
|
||||
CLICKHOUSE_PORT=8123
|
||||
CLICKHOUSE_DB=mabase_prod
|
||||
CLICKHOUSE_USER=data_writer
|
||||
CLICKHOUSE_PASSWORD=secret
|
||||
```
|
||||
|
||||
## Testing: Mocking the Client
|
||||
|
||||
### Using unittest.mock
|
||||
|
||||
```python
|
||||
from unittest.mock import MagicMock, patch
|
||||
from ja4_common.clickhouse import ClickHouseClient
|
||||
|
||||
def test_my_service():
|
||||
mock_client = MagicMock(spec=ClickHouseClient)
|
||||
mock_client.query.return_value = MagicMock(result_rows=[(42,)])
|
||||
|
||||
with patch("ja4_common.clickhouse._client", mock_client):
|
||||
from ja4_common.clickhouse import get_client
|
||||
client = get_client()
|
||||
result = client.query("SELECT count() FROM http_logs")
|
||||
assert result.result_rows == [(42,)]
|
||||
```
|
||||
|
||||
### Overriding Settings in Tests
|
||||
|
||||
```python
|
||||
from ja4_common.settings import ClickHouseSettings
|
||||
|
||||
# Create custom settings for tests
|
||||
test_settings = ClickHouseSettings(
|
||||
CLICKHOUSE_HOST="localhost",
|
||||
CLICKHOUSE_PORT=8123,
|
||||
CLICKHOUSE_DB="test_db",
|
||||
CLICKHOUSE_USER="test_user",
|
||||
CLICKHOUSE_PASSWORD="test_pass",
|
||||
)
|
||||
```
|
||||
|
||||
## Source Files
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `ja4_common/settings.py` | `ClickHouseSettings` pydantic-settings model |
|
||||
| `ja4_common/clickhouse.py` | `ClickHouseClient` class and `get_client()` singleton |
|
||||
| `pyproject.toml` | Package metadata and dependencies |
|
||||
Reference in New Issue
Block a user