feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
291
services/sentinel/README.md
Normal file
291
services/sentinel/README.md
Normal file
@ -0,0 +1,291 @@
|
||||
# JA4Sentinel
|
||||
|
||||
Outil Go pour capturer le trafic réseau sur un serveur Linux, extraire les handshakes TLS côté client, générer les signatures JA4, enrichir avec des métadonnées IP/TCP, et loguer les résultats vers une ou plusieurs sorties configurables.
|
||||
|
||||
## Fonctionnalités
|
||||
|
||||
- **Capture réseau** : Écoute sur une interface réseau avec filtres BPF configurables
|
||||
- **Parsing TLS** : Extraction des ClientHello TLS depuis les flux TCP
|
||||
- **Fingerprinting** : Génération des empreintes JA4 et JA3 pour chaque client
|
||||
- **Métadonnées** : Enrichissement avec IPMeta (TTL, IP ID, DF) et TCPMeta (window, MSS, options)
|
||||
- **Sorties multiples** : stdout, fichier JSON, socket UNIX (combinables via MultiWriter)
|
||||
- **Logging structuré** : Logs JSON sur stdout/stderr pour intégration avec systemd/journald
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Capture │ ──▶ │ TLSParse │ ──▶ │ Fingerprint │ ──▶ │ Output │
|
||||
│ (pcap) │ │ (ClientHello)│ │ (JA4) │ │ (JSON logs) │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
|
||||
│ │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
api.RawPacket api.TLSClientHello api.Fingerprints api.LogRecord
|
||||
```
|
||||
|
||||
### Modules
|
||||
|
||||
| Module | Responsabilités |
|
||||
|--------|-----------------|
|
||||
| `config` | Chargement et validation de la configuration (YAML, env, CLI) |
|
||||
| `capture` | Capture des paquets réseau via libpcap |
|
||||
| `tlsparse` | Extraction des ClientHello TLS avec suivi d'état de flux |
|
||||
| `fingerprint` | Génération JA4/JA3 via `psanford/tlsfingerprint` |
|
||||
| `output` | Écriture des logs vers stdout, fichier, socket UNIX |
|
||||
| `logging` | Logs structurés JSON pour le diagnostic du service |
|
||||
|
||||
## Installation
|
||||
|
||||
### Prérequis
|
||||
|
||||
- Go 1.24+
|
||||
- libpcap-dev (pour la compilation)
|
||||
- Docker (pour les tests et le déploiement)
|
||||
|
||||
### Note sur libpcap
|
||||
|
||||
**Le binaire est compilé sur Rocky Linux 9** pour une compatibilité maximale avec toutes les distributions RHEL/Rocky/AlmaLinux.
|
||||
libpcap est requis à l'exécution et sera installé automatiquement par le gestionnaire de packages.
|
||||
|
||||
### Packages système
|
||||
|
||||
#### Rocky Linux / RHEL / AlmaLinux (.rpm)
|
||||
|
||||
```bash
|
||||
# Télécharger le package
|
||||
wget https://github.com/your-repo/ja4sentinel/releases/latest/download/ja4sentinel.rpm
|
||||
|
||||
# Installer
|
||||
sudo dnf install ./ja4sentinel.rpm
|
||||
|
||||
# Activer le service
|
||||
sudo systemctl enable ja4sentinel
|
||||
sudo systemctl start ja4sentinel
|
||||
|
||||
# Vérifier le statut
|
||||
sudo systemctl status ja4sentinel
|
||||
```
|
||||
|
||||
#### Distributions supportées
|
||||
|
||||
- Rocky Linux 8, 9, 10
|
||||
- AlmaLinux 8, 9, 10
|
||||
- RHEL 8, 9, 10
|
||||
|
||||
## Configuration
|
||||
|
||||
### Fichier de configuration (YAML)
|
||||
|
||||
```yaml
|
||||
core:
|
||||
interface: eth0
|
||||
listen_ports: [443, 8443]
|
||||
bpf_filter: ""
|
||||
flow_timeout_sec: 30
|
||||
|
||||
outputs:
|
||||
- type: stdout
|
||||
enabled: true
|
||||
- type: file
|
||||
enabled: true
|
||||
params:
|
||||
path: /var/log/ja4sentinel/ja4.log
|
||||
- type: unix_socket
|
||||
enabled: true
|
||||
params:
|
||||
socket_path: /var/run/logcorrelator/network.socket
|
||||
```
|
||||
|
||||
### Variables d'environnement
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `JA4SENTINEL_INTERFACE` | Interface réseau (ex: `eth0`) |
|
||||
| `JA4SENTINEL_PORTS` | Ports à surveiller (ex: `443,8443`) |
|
||||
| `JA4SENTINEL_BPF_FILTER` | Filtre BPF personnalisé |
|
||||
| `JA4SENTINEL_FLOW_TIMEOUT` | Timeout de flux en secondes (défaut: 30) |
|
||||
|
||||
### Ligne de commande
|
||||
|
||||
```bash
|
||||
ja4sentinel --config /etc/ja4sentinel/config.yml
|
||||
ja4sentinel --version
|
||||
```
|
||||
|
||||
## Format des logs
|
||||
|
||||
### Logs de service (stdout/stderr)
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1708876543210000000,
|
||||
"level": "INFO",
|
||||
"component": "capture",
|
||||
"message": "Starting packet capture",
|
||||
"interface": "eth0"
|
||||
}
|
||||
```
|
||||
|
||||
### Logs métier (JA4)
|
||||
|
||||
```json
|
||||
{
|
||||
"src_ip": "192.168.1.100",
|
||||
"src_port": 54321,
|
||||
"dst_ip": "10.0.0.1",
|
||||
"dst_port": 443,
|
||||
"ip_meta_ttl": 64,
|
||||
"ip_meta_total_length": 512,
|
||||
"ip_meta_id": 12345,
|
||||
"ip_meta_df": true,
|
||||
"tcp_meta_window_size": 65535,
|
||||
"tcp_meta_mss": 1460,
|
||||
"tcp_meta_window_scale": 7,
|
||||
"tcp_meta_options": "MSS,WS,SACK,TS",
|
||||
"ja4": "t13d1516h2_8daaf6152771_02cb136f2775",
|
||||
"ja4_hash": "8daaf6152771_02cb136f2775",
|
||||
"ja3": "771,4865-4866-4867,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513,29-23-24,0",
|
||||
"ja3_hash": "a0e6f06c7a6d15e5e3f0f0e6f06c7a6d"
|
||||
}
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
### Tests unitaires
|
||||
|
||||
```bash
|
||||
# En local
|
||||
make test
|
||||
|
||||
# Dans Docker
|
||||
make test-docker
|
||||
|
||||
# Avec détection de race conditions
|
||||
make test-race
|
||||
|
||||
# Avec rapport de couverture
|
||||
make test-coverage
|
||||
```
|
||||
|
||||
### Tests d'intégration
|
||||
|
||||
```bash
|
||||
# Lance les tests bout-à-bout dans Docker
|
||||
make test-integration
|
||||
|
||||
# Nettoyage après tests
|
||||
make test-integration-clean
|
||||
```
|
||||
|
||||
## Déploiement systemd
|
||||
|
||||
Exemple de fichier de service `/etc/systemd/system/ja4sentinel.service` :
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=JA4 client fingerprinting daemon
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ja4sentinel
|
||||
Group=ja4sentinel
|
||||
ExecStart=/usr/local/bin/ja4sentinel --config /etc/ja4sentinel/config.yml
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
Environment=JA4SENTINEL_LOG_LEVEL=info
|
||||
|
||||
# Security
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=full
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN
|
||||
AmbientCapabilities=CAP_NET_RAW CAP_NET_ADMIN
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
## Exemples d'utilisation
|
||||
|
||||
### Surveillance du trafic HTTPS
|
||||
|
||||
```yaml
|
||||
core:
|
||||
interface: eth0
|
||||
listen_ports: [443]
|
||||
outputs:
|
||||
- type: stdout
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Export vers socket UNIX pour traitement externe
|
||||
|
||||
```yaml
|
||||
core:
|
||||
interface: eth0
|
||||
listen_ports: [443, 8443]
|
||||
outputs:
|
||||
- type: unix_socket
|
||||
enabled: true
|
||||
params:
|
||||
socket_path: /var/run/logcorrelator/network.socket
|
||||
# log_level: debug # debug, info, warn, error (défaut: error)
|
||||
```
|
||||
|
||||
### Logging fichier + stdout
|
||||
|
||||
```yaml
|
||||
core:
|
||||
interface: ens192
|
||||
listen_ports: [443]
|
||||
flow_timeout_sec: 60
|
||||
outputs:
|
||||
- type: stdout
|
||||
enabled: true
|
||||
- type: file
|
||||
enabled: true
|
||||
params:
|
||||
path: /var/log/ja4sentinel/ja4.json
|
||||
```
|
||||
|
||||
## Développement
|
||||
|
||||
### Linting
|
||||
|
||||
```bash
|
||||
make lint
|
||||
```
|
||||
|
||||
### Formatage
|
||||
|
||||
```bash
|
||||
make fmt
|
||||
```
|
||||
|
||||
### Nettoyage
|
||||
|
||||
```bash
|
||||
# Supprime les binaires et images Docker
|
||||
make clean
|
||||
|
||||
# Supprime aussi les conteneurs et volumes
|
||||
make clean-all
|
||||
```
|
||||
|
||||
## Licence
|
||||
|
||||
À définir.
|
||||
|
||||
## Contribuer
|
||||
|
||||
1. Fork le projet
|
||||
2. Créer une branche de feature (`git checkout -b feature/amélioration`)
|
||||
3. Commit les changements (`git commit -am 'Ajout fonctionnalité'`)
|
||||
4. Push (`git push origin feature/amélioration`)
|
||||
5. Ouvrir une Pull Request
|
||||
|
||||
---
|
||||
|
||||
**Voir `architecture.yml` pour la documentation complète de l'architecture.**
|
||||
Reference in New Issue
Block a user