feat: implémentation complète du pipeline JA4 + Docker + tests
Nouveaux modules: - cmd/ja4sentinel/main.go : point d'entrée avec pipeline capture→parse→fingerprint→output - internal/config/loader.go : chargement YAML + env (JA4SENTINEL_*) + validation - internal/tlsparse/parser.go : extraction ClientHello avec suivi d'état de flux (NEW/WAIT_CLIENT_HELLO/JA4_DONE) - internal/fingerprint/engine.go : génération JA4/JA3 via psanford/tlsfingerprint - internal/output/writers.go : StdoutWriter, FileWriter, UnixSocketWriter, MultiWriter Infrastructure: - Dockerfile (multi-stage), Dockerfile.dev, Dockerfile.test-server - Makefile (build, test, lint, docker-build-*) - docker-compose.test.yml pour tests d'intégration - README.md (276 lignes) avec architecture, config, exemples API (api/types.go): - Ajout Close() aux interfaces Capture et Parser - Ajout FlowTimeoutSec dans Config (défaut: 30s, env: JA4SENTINEL_FLOW_TIMEOUT) - ServiceLog: +Timestamp, +TraceID, +ConnID - LogRecord: champs flatten (ip_meta_*, tcp_meta_*, ja4*) - Helper NewLogRecord() pour conversion TLSClientHello+Fingerprints→LogRecord Architecture (architecture.yml): - Documentation module logging + interfaces LoggerFactory/Logger - Section service.systemd complète (unit, security, capabilities) - Section logging.strategy (JSON lines, champs, règles) - api.Config: +FlowTimeoutSec documenté Fixes/cleanup: - Suppression internal/api/types.go (consolidé dans api/types.go) - Correction imports logging (ja4sentinel/api) - .dockerignore / .gitignore - config.yml.example Tests: - Tous les modules ont leurs tests (*_test.go) - Tests unitaires : capture, config, fingerprint, output, tlsparse - Tests d'intégration via docker-compose.test.yml Build: - Binaires dans dist/ (make build → dist/ja4sentinel) - Docker runtime avec COPY --from=builder /app/dist/ Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
200
architecture.yml
200
architecture.yml
@ -95,13 +95,30 @@ modules:
|
||||
- "tlsparse"
|
||||
- "fingerprint"
|
||||
|
||||
- name: logging
|
||||
path: "internal/logging"
|
||||
description: "Logs structurés JSON pour le service (stdout/stderr)."
|
||||
responsibilities:
|
||||
- "Fournir une fabrique de loggers (LoggerFactory)."
|
||||
- "Émettre des logs au format JSON lines sur stdout."
|
||||
- "Supporter les niveaux : debug, info, warn, error."
|
||||
- "Inclure timestamp, niveau, composant, message et détails optionnels."
|
||||
allowed_dependencies:
|
||||
- "api"
|
||||
forbidden_dependencies:
|
||||
- "config"
|
||||
- "capture"
|
||||
- "tlsparse"
|
||||
- "fingerprint"
|
||||
- "output"
|
||||
|
||||
- name: cmd_ja4sentinel
|
||||
path: "cmd/ja4sentinel"
|
||||
description: "Point d’entrée de l’application (main)."
|
||||
description: "Point d'entrée de l'application (main)."
|
||||
responsibilities:
|
||||
- "Charger la configuration via le module config."
|
||||
- "Construire les instances des modules (capture, tlsparse, fingerprint, output)."
|
||||
- "Brancher les modules entre eux selon l’architecture pipeline."
|
||||
- "Construire les instances des modules (capture, tlsparse, fingerprint, output, logging)."
|
||||
- "Brancher les modules entre eux selon l'architecture pipeline."
|
||||
- "Gérer les signaux système (arrêt propre)."
|
||||
allowed_dependencies:
|
||||
- "config"
|
||||
@ -110,16 +127,29 @@ modules:
|
||||
- "fingerprint"
|
||||
- "output"
|
||||
- "api"
|
||||
- "logging"
|
||||
forbidden_dependencies: []
|
||||
|
||||
api:
|
||||
types:
|
||||
- name: "api.ServiceLog"
|
||||
description: "Log interne du service ja4sentinel (diagnostic)."
|
||||
fields:
|
||||
- { name: Level, type: "string", description: "niveau: debug, info, warn, error." }
|
||||
- { name: Component, type: "string", description: "module concerné (capture, tlsparse, ...)." }
|
||||
- { name: Message, type: "string", description: "texte du log." }
|
||||
- { name: Details, type: "map[string]string", description: "infos additionnelles (erreurs, IDs...)." }
|
||||
- { name: Timestamp, type: "int64", description: "Timestamp en nanosecondes (auto-rempli par le logger)." }
|
||||
- { name: TraceID, type: "string", description: "ID de tracing distribué (optionnel)." }
|
||||
- { name: ConnID, type: "string", description: "Identifiant de flux TCP (optionnel)." }
|
||||
|
||||
- name: "api.Config"
|
||||
description: "Configuration réseau et TLS de base."
|
||||
fields:
|
||||
- { name: Interface, type: "string", description: "Nom de l’interface réseau (ex: eth0)." }
|
||||
- { name: Interface, type: "string", description: "Nom de l'interface réseau (ex: eth0)." }
|
||||
- { name: ListenPorts, type: "[]uint16", description: "Ports TCP à surveiller (ex: [443, 8443])." }
|
||||
- { name: BPFFilter, type: "string", description: "Filtre BPF optionnel pour la capture." }
|
||||
- { name: FlowTimeoutSec, type: "int", description: "Timeout en secondes pour l'extraction du handshake TLS (défaut: 30)." }
|
||||
|
||||
- name: "api.IPMeta"
|
||||
description: "Métadonnées IP pour fingerprinting de stack."
|
||||
@ -163,15 +193,32 @@ api:
|
||||
- { name: JA3Hash, type: "string", description: "Hash JA3 (optionnel)." }
|
||||
|
||||
- name: "api.LogRecord"
|
||||
description: "Enregistrement de log final envoyé vers les outputs."
|
||||
description: "Enregistrement de log final, sérialisé en JSON objet plat."
|
||||
json_object: true
|
||||
fields:
|
||||
- { name: SrcIP, type: "string", description: "Adresse IP source (client)." }
|
||||
- { name: SrcPort, type: "uint16", description: "Port source (client)." }
|
||||
- { name: DstIP, type: "string", description: "Adresse IP destination (serveur)." }
|
||||
- { name: DstPort, type: "uint16", description: "Port destination (serveur)." }
|
||||
- { name: IPMeta, type: "api.IPMeta", description: "Métadonnées IP." }
|
||||
- { name: TCPMeta, type: "api.TCPMeta", description: "Métadonnées TCP." }
|
||||
- { name: Fingerprints, type: "api.Fingerprints", description: "Empreintes JA4/JA3 associées." }
|
||||
- { name: SrcIP, type: "string", json_key: "src_ip" }
|
||||
- { name: SrcPort, type: "uint16", json_key: "src_port" }
|
||||
- { name: DstIP, type: "string", json_key: "dst_ip" }
|
||||
- { name: DstPort, type: "uint16", json_key: "dst_port" }
|
||||
|
||||
# IPMeta flatten
|
||||
- { name: IPTTL, type: "uint8", json_key: "ip_meta_ttl" }
|
||||
- { name: IPTotalLen, type: "uint16", json_key: "ip_meta_total_length" }
|
||||
- { name: IPID, type: "uint16", json_key: "ip_meta_id" }
|
||||
- { name: IPDF, type: "bool", json_key: "ip_meta_df" }
|
||||
|
||||
# TCPMeta flatten
|
||||
- { name: TCPWindow, type: "uint16", json_key: "tcp_meta_window_size" }
|
||||
- { name: TCPMSS, type: "uint16", json_key: "tcp_meta_mss" }
|
||||
- { name: TCPWScale, type: "uint8", json_key: "tcp_meta_window_scale" }
|
||||
- { name: TCPOptions, type: "string", json_key: "tcp_meta_options" }
|
||||
|
||||
# Fingerprints
|
||||
- { name: JA4, type: "string", json_key: "ja4" }
|
||||
- { name: JA4Hash, type: "string", json_key: "ja4_hash" }
|
||||
- { name: JA3, type: "string", json_key: "ja3" }
|
||||
- { name: JA3Hash, type: "string", json_key: "ja3_hash" }
|
||||
|
||||
|
||||
- name: "api.OutputConfig"
|
||||
description: "Configuration d’une sortie de logs."
|
||||
@ -278,6 +325,49 @@ api:
|
||||
notes:
|
||||
- "Doit supporter plusieurs outputs simultanés via un MultiWriter."
|
||||
|
||||
- name: "logging.LoggerFactory"
|
||||
description: "Fabrique de loggers structurés JSON."
|
||||
module: "logging"
|
||||
methods:
|
||||
- name: "NewLogger"
|
||||
params:
|
||||
- { name: level, type: "string" }
|
||||
returns:
|
||||
- { type: "api.Logger" }
|
||||
- name: "NewDefaultLogger"
|
||||
params: []
|
||||
returns:
|
||||
- { type: "api.Logger" }
|
||||
notes:
|
||||
- "Les logs sont émis en JSON lines sur stdout pour systemd/journald."
|
||||
|
||||
- name: "api.Logger"
|
||||
description: "Interface de logging pour tous les modules."
|
||||
module: "logging"
|
||||
methods:
|
||||
- name: "Debug"
|
||||
params:
|
||||
- { name: component, type: "string" }
|
||||
- { name: message, type: "string" }
|
||||
- { name: details, type: "map[string]string" }
|
||||
- name: "Info"
|
||||
params:
|
||||
- { name: component, type: "string" }
|
||||
- { name: message, type: "string" }
|
||||
- { name: details, type: "map[string]string" }
|
||||
- name: "Warn"
|
||||
params:
|
||||
- { name: component, type: "string" }
|
||||
- { name: message, type: "string" }
|
||||
- { name: details, type: "map[string]string" }
|
||||
- name: "Error"
|
||||
params:
|
||||
- { name: component, type: "string" }
|
||||
- { name: message, type: "string" }
|
||||
- { name: details, type: "map[string]string" }
|
||||
notes:
|
||||
- "Tous les logs passent par stdout/stderr (pas de fichiers directs)."
|
||||
|
||||
architecture:
|
||||
style: "pipeline"
|
||||
flow:
|
||||
@ -479,3 +569,89 @@ dev_tools:
|
||||
- "Génération automatique de Dockerfile.dev et Dockerfile à partir de cette section."
|
||||
- "Génération de fichiers docker-compose.test.yml pour les scénarios d’intégration."
|
||||
|
||||
service:
|
||||
systemd:
|
||||
unit_name: "ja4sentinel.service"
|
||||
description: "JA4 client fingerprinting daemon"
|
||||
wanted_by: "multi-user.target"
|
||||
exec:
|
||||
binary_path: "/usr/local/bin/ja4sentinel"
|
||||
args:
|
||||
- "--config"
|
||||
- "/etc/ja4sentinel/config.yml"
|
||||
user_group:
|
||||
user: "ja4sentinel"
|
||||
group: "ja4sentinel"
|
||||
runtime:
|
||||
working_directory: "/var/lib/ja4sentinel"
|
||||
pid_file: "/run/ja4sentinel.pid"
|
||||
restart: "on-failure"
|
||||
restart_sec: 5
|
||||
environment_prefix: "JA4SENTINEL_"
|
||||
logging:
|
||||
type: "journald"
|
||||
journal_identifier: "ja4sentinel"
|
||||
expectations:
|
||||
- "Le binaire écrit les logs de service sur stdout/stderr."
|
||||
- "Les messages doivent inclure au minimum un niveau (INFO/ERROR) et un composant."
|
||||
security:
|
||||
capabilities:
|
||||
- "CAP_NET_RAW"
|
||||
- "CAP_NET_ADMIN"
|
||||
sandboxing:
|
||||
- "NoNewPrivileges=yes"
|
||||
- "ProtectSystem=full"
|
||||
- "ProtectHome=true"
|
||||
- "PrivateTmp=true"
|
||||
integration_rules:
|
||||
- "Le binaire doit s’arrêter proprement sur SIGTERM (systemd stop)."
|
||||
- "Le module cmd_ja4sentinel gère les signaux et termine la capture proprement."
|
||||
- "Les chemins (config, socket UNIX, logs) doivent être compatibles avec FHS (/etc, /var/run, /var/log)."
|
||||
- "Le module cmd_ja4sentinel capture SIGTERM/SIGINT et déclenche un arrêt propre (stop capture, flush outputs, fermer socket UNIX)."
|
||||
- "Le processus doit retourner un code de sortie non nul en cas d’erreur fatale au démarrage."
|
||||
|
||||
logging:
|
||||
strategy:
|
||||
description: >
|
||||
ja4sentinel écrit ses logs techniques sur stdout/stderr au format JSON lines,
|
||||
afin que systemd/journald puissent les collecter et les filtrer.
|
||||
format: "json_lines"
|
||||
fields:
|
||||
- "timestamp"
|
||||
- "level"
|
||||
- "component" # capture, tlsparse, fingerprint, output, service, ...
|
||||
- "message"
|
||||
- "trace_id" # optionnel
|
||||
- "conn_id" # optionnel (identifiant de flux TCP)
|
||||
rules:
|
||||
- "Pas d’écriture directe dans des fichiers de log techniques depuis le code (stdout/stderr uniquement)."
|
||||
- "Les logs techniques du daemon passent par stdout/stderr (systemd/journald)."
|
||||
- "Les outputs métiers (LogRecord JA4) sont gérés par le module output, vers socket UNIX et/ou fichier JSON."
|
||||
|
||||
json_format:
|
||||
description: >
|
||||
Les LogRecord métiers (JA4 + métadonnées) sont sérialisés en JSON objet plat,
|
||||
avec des champs nommés explicitement pour ingestion dans ClickHouse.
|
||||
rules:
|
||||
- "Pas de tableaux imbriqués ni d’objets deeply nested."
|
||||
- "Toutes les métadonnées IP/TCP sont flatten sous forme de champs scalaires nommés."
|
||||
- "Les noms de champs suivent la convention: ip_meta_*, tcp_meta_*, ja4*."
|
||||
logrecord_schema:
|
||||
# Exemple de mapping pour api.LogRecord (résumé)
|
||||
- "src_ip"
|
||||
- "src_port"
|
||||
- "dst_ip"
|
||||
- "dst_port"
|
||||
- "ip_meta_ttl"
|
||||
- "ip_meta_total_length"
|
||||
- "ip_meta_id"
|
||||
- "ip_meta_df"
|
||||
- "tcp_meta_window_size"
|
||||
- "tcp_meta_mss"
|
||||
- "tcp_meta_window_scale"
|
||||
- "tcp_meta_options" # string joinée, ex: 'MSS,SACK,TS,NOP,WS'
|
||||
- "ja4"
|
||||
- "ja4_hash"
|
||||
- "ja3"
|
||||
- "ja3_hash"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user