feat: Keep-Alive correlation, TTL management, SIGHUP handling, logrotate support
Some checks failed
Build and Test / test (push) Has been cancelled
Build and Test / build (push) Has been cancelled
Build and Test / docker (push) Has been cancelled

Major features:
- One-to-many correlation mode (Keep-Alive) for HTTP connections
- Dynamic TTL for network events with reset on each correlation
- Separate configurable buffer sizes for HTTP and network events
- SIGHUP signal handling for log rotation without service restart
- FileSink.Reopen() method for log file rotation
- logrotate configuration included in RPM
- ExecReload added to systemd service

Configuration changes:
- New YAML structure with nested sections (time_window, orphan_policy, matching, buffers, ttl)
- Backward compatibility maintained for deprecated fields

Packaging:
- RPM version 1.1.0 with logrotate config
- Updated spec file and changelog
- All distributions: el8, el9, el10

Tests:
- New tests for Keep-Alive mode and TTL reset
- Updated mocks with Reopen() interface method

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
Jacquin Antoine
2026-03-02 20:32:59 +01:00
parent a415a3201a
commit 33e19b4f52
19 changed files with 974 additions and 321 deletions

View File

@ -5,6 +5,36 @@ All notable changes to logcorrelator are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.1.0] - 2026-03-02
### Added
- **Keep-Alive support**: One-to-many correlation mode allows a single network event (B) to correlate with multiple HTTP events (A)
- **Dynamic TTL**: Network events (source B) now have configurable TTL that resets on each successful correlation
- **Separate buffer sizes**: Configurable `max_http_items` and `max_network_items` for independent buffer control
- **SIGHUP handling**: Service now handles SIGHUP signal for log rotation without restart
- **logrotate configuration**: RPM includes `/etc/logrotate.d/logcorrelator` for automatic log rotation
- **ExecReload**: Systemd service now supports `systemctl reload logcorrelator`
### Changed
- **Configuration structure**: New YAML structure with nested sections:
- `time_window` (object with `value` and `unit`)
- `orphan_policy` (object with `apache_always_emit` and `network_emit`)
- `matching.mode` (string: `one_to_one` or `one_to_many`)
- `buffers` (object with `max_http_items` and `max_network_items`)
- `ttl` (object with `network_ttl_s`)
- Backward compatibility maintained for old config fields (`time_window_s`, `emit_orphans`)
### Technical Details
- `CorrelationService` now supports `MatchingMode` configuration
- Network events tracked with individual TTL expiration times
- `FileSink.Reopen()` method for log file rotation
- All sinks implement `Reopen()` interface method
---
## [1.0.7] - 2026-03-01 ## [1.0.7] - 2026-03-01
### Added ### Added

View File

@ -52,6 +52,7 @@ COPY --from=builder /build/CHANGELOG.md /tmp/pkgroot/usr/share/doc/logcorrelator
COPY packaging/rpm/post /tmp/scripts/post COPY packaging/rpm/post /tmp/scripts/post
COPY packaging/rpm/preun /tmp/scripts/preun COPY packaging/rpm/preun /tmp/scripts/preun
COPY packaging/rpm/postun /tmp/scripts/postun COPY packaging/rpm/postun /tmp/scripts/postun
COPY packaging/rpm/logrotate /tmp/pkgroot/etc/logrotate.d/logcorrelator
# Create directories and set permissions # Create directories and set permissions
RUN mkdir -p /tmp/pkgroot/var/log/logcorrelator && \ RUN mkdir -p /tmp/pkgroot/var/log/logcorrelator && \
@ -91,7 +92,8 @@ RUN mkdir -p /packages/rpm/el8 && \
usr/share/doc/logcorrelator/CHANGELOG.md \ usr/share/doc/logcorrelator/CHANGELOG.md \
var/log/logcorrelator \ var/log/logcorrelator \
var/run/logcorrelator \ var/run/logcorrelator \
etc/systemd/system/logcorrelator.service etc/systemd/system/logcorrelator.service \
etc/logrotate.d/logcorrelator
# ============================================================================= # =============================================================================
# Stage 3: RPM Package builder for Enterprise Linux 9 (el9) # Stage 3: RPM Package builder for Enterprise Linux 9 (el9)
@ -115,6 +117,7 @@ COPY --from=builder /build/CHANGELOG.md /tmp/pkgroot/usr/share/doc/logcorrelator
COPY packaging/rpm/post /tmp/scripts/post COPY packaging/rpm/post /tmp/scripts/post
COPY packaging/rpm/preun /tmp/scripts/preun COPY packaging/rpm/preun /tmp/scripts/preun
COPY packaging/rpm/postun /tmp/scripts/postun COPY packaging/rpm/postun /tmp/scripts/postun
COPY packaging/rpm/logrotate /tmp/pkgroot/etc/logrotate.d/logcorrelator
# Create directories and set permissions # Create directories and set permissions
RUN mkdir -p /tmp/pkgroot/var/log/logcorrelator && \ RUN mkdir -p /tmp/pkgroot/var/log/logcorrelator && \
@ -154,7 +157,8 @@ RUN mkdir -p /packages/rpm/el9 && \
usr/share/doc/logcorrelator/CHANGELOG.md \ usr/share/doc/logcorrelator/CHANGELOG.md \
var/log/logcorrelator \ var/log/logcorrelator \
var/run/logcorrelator \ var/run/logcorrelator \
etc/systemd/system/logcorrelator.service etc/systemd/system/logcorrelator.service \
etc/logrotate.d/logcorrelator
# ============================================================================= # =============================================================================
# Stage 4: RPM Package builder for Enterprise Linux 10 (el10) # Stage 4: RPM Package builder for Enterprise Linux 10 (el10)
@ -178,6 +182,7 @@ COPY --from=builder /build/CHANGELOG.md /tmp/pkgroot/usr/share/doc/logcorrelator
COPY packaging/rpm/post /tmp/scripts/post COPY packaging/rpm/post /tmp/scripts/post
COPY packaging/rpm/preun /tmp/scripts/preun COPY packaging/rpm/preun /tmp/scripts/preun
COPY packaging/rpm/postun /tmp/scripts/postun COPY packaging/rpm/postun /tmp/scripts/postun
COPY packaging/rpm/logrotate /tmp/pkgroot/etc/logrotate.d/logcorrelator
# Create directories and set permissions # Create directories and set permissions
RUN mkdir -p /tmp/pkgroot/var/log/logcorrelator && \ RUN mkdir -p /tmp/pkgroot/var/log/logcorrelator && \
@ -217,7 +222,8 @@ RUN mkdir -p /packages/rpm/el10 && \
usr/share/doc/logcorrelator/CHANGELOG.md \ usr/share/doc/logcorrelator/CHANGELOG.md \
var/log/logcorrelator \ var/log/logcorrelator \
var/run/logcorrelator \ var/run/logcorrelator \
etc/systemd/system/logcorrelator.service etc/systemd/system/logcorrelator.service \
etc/logrotate.d/logcorrelator
# ============================================================================= # =============================================================================
# Stage 5: Output - Image finale avec les packages RPM # Stage 5: Output - Image finale avec les packages RPM

View File

@ -9,12 +9,13 @@ service:
événements HTTP applicatifs (source A, typiquement Apache ou reverse proxy) événements HTTP applicatifs (source A, typiquement Apache ou reverse proxy)
avec des événements réseau (source B, métadonnées IP/TCP, JA3/JA4, etc.) avec des événements réseau (source B, métadonnées IP/TCP, JA3/JA4, etc.)
sur la base de la combinaison strictement définie src_ip + src_port, avec sur la base de la combinaison strictement définie src_ip + src_port, avec
une fenêtre temporelle configurable. Le service produit un log corrélé une fenêtre temporelle configurable. Le service supporte les connexions
unique pour chaque paire correspondante, émet toujours les événements A HTTP Keep-Alive : un log réseau peut être corrélé à plusieurs logs HTTP
même lorsquaucun événement B corrélé nest disponible, német jamais de successifs (stratégie 1àN). La rétention en mémoire est bornée par des
logs B seuls, et pousse les logs agrégés en temps quasi réel vers tailles de caches configurables et un TTL dynamique pour la source B. Le
ClickHouse et/ou un fichier local, en minimisant la rétention en mémoire service émet toujours les événements A même lorsquaucun événement B nest
et sur disque. disponible, német jamais de logs B seuls, et pousse les résultats vers
ClickHouse et/ou un fichier local.
runtime: runtime:
deployment: deployment:
@ -23,7 +24,7 @@ runtime:
logcorrelator est livré sous forme de binaire autonome, exécuté comme un logcorrelator est livré sous forme de binaire autonome, exécuté comme un
service systemd. L'unité systemd assure le démarrage automatique au boot, service systemd. L'unité systemd assure le démarrage automatique au boot,
le redémarrage en cas de crash, et une intégration standard dans l'écosystème le redémarrage en cas de crash, et une intégration standard dans l'écosystème
Linux (notamment sur CentOS 7 et Rocky Linux 8+). Linux.
binary_path: /usr/bin/logcorrelator binary_path: /usr/bin/logcorrelator
config_path: /etc/logcorrelator/logcorrelator.yml config_path: /etc/logcorrelator/logcorrelator.yml
user: logcorrelator user: logcorrelator
@ -41,6 +42,7 @@ runtime:
User=logcorrelator User=logcorrelator
Group=logcorrelator Group=logcorrelator
ExecStart=/usr/bin/logcorrelator -config /etc/logcorrelator/logcorrelator.yml ExecStart=/usr/bin/logcorrelator -config /etc/logcorrelator/logcorrelator.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure Restart=on-failure
RestartSec=5 RestartSec=5
@ -63,98 +65,186 @@ runtime:
graceful_shutdown: graceful_shutdown:
- SIGINT - SIGINT
- SIGTERM - SIGTERM
reload:
- SIGHUP
description: > description: >
En réception de SIGINT ou SIGTERM, le service arrête proprement la lecture SIGINT/SIGTERM : arrêt propre (arrêt des sockets, vidage des buffers, fermeture
des sockets Unix, vide les buffers denvoi (dans les limites de la politique des sinks). SIGHUP : réouverture des fichiers de sortie (utile pour la
de drop), ferme les connexions ClickHouse puis sarrête. rotation des logs via logrotate) sans arrêter le service.
packaging:
description: >
logcorrelator est distribué sous forme de packages .rpm (Rocky Linux, AlmaLinux,
RHEL), construits intégralement dans des conteneurs. Le changelog RPM est mis
à jour à chaque changement de version.
formats:
- rpm
target_distros:
- rocky-linux-8
- rocky-linux-9
- almalinux-10
- rhel-8
- rhel-9
- rhel-10
rpm:
tool: fpm
changelog:
source: git # ou CHANGELOG.md
description: >
À chaque build, un script génère un fichier de changelog RPM à partir de
lhistorique (tags/commits) et le passe à fpm (option --rpm-changelog).
contents:
- path: /usr/bin/logcorrelator
type: binary
- path: /etc/logcorrelator/logcorrelator.yml
type: config
directives: "%config(noreplace)"
- path: /etc/logcorrelator/logcorrelator.yml.example
type: doc
description: Fichier d'exemple toujours mis à jour par le RPM.
- path: /etc/systemd/system/logcorrelator.service
type: systemd_unit
- path: /etc/logrotate.d/logcorrelator
type: logrotate_script
logrotate_example: |
/var/log/logcorrelator/*.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
create 0640 logcorrelator logcorrelator
postrotate
systemctl reload logcorrelator > /dev/null 2>/dev/null || true
endscript
}
config: config:
format: yaml format: yaml
location: /etc/logcorrelator/logcorrelator.yml location: /etc/logcorrelator/logcorrelator.yml
reload_strategy: signal_sighup_for_files
description: > description: >
Toute la configuration est centralisée dans un fichier YAML lisible, Toute la configuration est centralisée dans un fichier YAML lisible. Le RPM
stocké dans /etc/logcorrelator. fournit aussi un fichier dexemple mis à jour à chaque version.
reload_strategy: restart_service
example: | example: |
# Logging configuration # /etc/logcorrelator/logcorrelator.yml
log: log:
level: INFO # DEBUG, INFO, WARN, ERROR level: INFO # DEBUG, INFO, WARN, ERROR
# Inputs - at least 2 unix sockets required
inputs: inputs:
unix_sockets: unix_sockets:
# Source HTTP (A) : logs applicatifs en JSON, 1 datagramme = 1 log.
- name: http - name: http
path: /var/run/logcorrelator/http.socket path: /var/run/logcorrelator/http.sock
socket_permissions: "0660" socket_permissions: "0666"
- name: network socket_type: dgram
path: /var/run/logcorrelator/network.socket max_datagram_bytes: 65535
# Source réseau (B) : logs IP/TCP/JA3... en JSON, 1 datagramme = 1 log.
- name: network
path: /var/run/logcorrelator/network.sock
socket_permissions: "0666"
socket_type: dgram
max_datagram_bytes: 65535
# Outputs
outputs: outputs:
file: file:
enabled: true
path: /var/log/logcorrelator/correlated.log path: /var/log/logcorrelator/correlated.log
clickhouse: format: json_lines
dsn: clickhouse://user:pass@localhost:9000/db
table: correlated_logs clickhouse:
stdout: false enabled: true
dsn: clickhouse://user:pass@localhost:9000/db
table: correlated_logs_http_network
batch_size: 500
flush_interval_ms: 200
max_buffer_size: 5000
drop_on_overflow: true
async_insert: true
timeout_ms: 1000
stdout:
enabled: false
# Correlation (optional)
correlation: correlation:
time_window_s: 1 # Fenêtre de corrélation : si le log HTTP arrive avant le réseau, il attend
emit_orphans: true # au plus cette durée (sauf éviction du cache HTTP).
time_window:
value: 1
unit: s
orphan_policy:
apache_always_emit: true
network_emit: false
matching:
mode: one_to_many # KeepAlive : un B peut corréler plusieurs A.
buffers:
# Tailles max des caches en mémoire (en nombre de logs).
max_http_items: 10000
max_network_items: 20000
ttl:
# Durée de vie standard dun log réseau (B) en mémoire. Chaque corrélation
# réussie avec un A réinitialise ce TTL.
network_ttl_s: 30
inputs: inputs:
description: > description: >
Le service consomme deux flux de logs JSON via des sockets Unix. Le schéma Deux flux de logs JSON via sockets Unix datagram (SOCK_DGRAM). Chaque datagramme
exact des logs pour chaque source est flexible et peut évoluer. Seuls contient un JSON complet.
quelques champs sont nécessaires pour la corrélation.
unix_sockets: unix_sockets:
- name: apache_source - name: apache_source
id: A id: A
description: > description: >
Source A, destinée aux logs HTTP applicatifs (Apache, reverse proxy, etc.). Source A, logs HTTP applicatifs (Apache, reverse proxy, etc.). Schéma JSON
Le schéma JSON est variable, avec un champ timestamp numérique obligatoire variable, champ timestamp obligatoire, headers dynamiques (header_*).
et des champs header_* dynamiques.
path: /var/run/logcorrelator/apache.sock path: /var/run/logcorrelator/apache.sock
permissions: "0666"
protocol: unix protocol: unix
mode: stream socket_type: dgram
mode: datagram
format: json format: json
framing: line framing: message
max_datagram_bytes: 65535
retry_on_error: true retry_on_error: true
- name: network_source - name: network_source
id: B id: B
description: > description: >
Source B, destinée aux logs réseau (métadonnées IP/TCP, JA3/JA4, etc.). Source B, logs réseau (métadonnées IP/TCP, JA3/JA4, etc.). Seuls src_ip
Le schéma JSON est variable ; seuls src_ip et src_port sont requis. et src_port sont requis pour la corrélation.
path: /var/run/logcorrelator/network.sock path: /var/run/logcorrelator/network.sock
permissions: "0666"
protocol: unix protocol: unix
mode: stream socket_type: dgram
mode: datagram
format: json format: json
framing: line framing: message
max_datagram_bytes: 65535
retry_on_error: true retry_on_error: true
outputs: outputs:
description: > description: >
Les logs corrélés sont envoyés vers un ou plusieurs sinks. MultiSink permet Les logs corrélés sont envoyés vers un ou plusieurs sinks (MultiSink).
de diffuser chaque log corrélé vers plusieurs destinations (fichier,
ClickHouse, stdout…).
sinks: sinks:
file: file:
enabled: true enabled: true
description: > description: >
Sink vers fichier local, utile pour debug ou archivage local. Écrit un Sink fichier local. Un JSON par ligne. Rotation gérée par logrotate,
JSON par ligne dans le chemin configuré. Rotation gérée par logrotate réouverture du fichier sur SIGHUP.
ou équivalent.
path: /var/log/logcorrelator/correlated.log path: /var/log/logcorrelator/correlated.log
format: json_lines format: json_lines
rotate_managed_by: external rotate_managed_by: external_logrotate
clickhouse: clickhouse:
enabled: true enabled: true
description: > description: >
Sink principal pour larchivage et lanalyse en temps quasi réel. Les Sink principal pour larchivage et lanalyse quasi temps réel. Inserts
logs corrélés sont insérés en batch dans ClickHouse avec un small buffer batch asynchrones, drop en cas de saturation.
et des inserts asynchrones. En cas de saturation ou dindisponibilité
ClickHouse, les logs sont drop pour éviter de saturer la machine locale.
dsn: clickhouse://user:pass@host:9000/db dsn: clickhouse://user:pass@host:9000/db
table: correlated_logs_http_network table: correlated_logs_http_network
batch_size: 500 batch_size: 500
@ -166,13 +256,12 @@ outputs:
stdout: stdout:
enabled: false enabled: false
description: > description: >
Sink optionnel vers stdout pour les tests et le développement. Sink optionnel pour les tests/développement.
correlation: correlation:
description: > description: >
Corrélation strictement basée sur src_ip + src_port et une fenêtre temporelle Corrélation stricte basée sur src_ip + src_port et une fenêtre temporelle
configurable. Aucun autre champ (dst_ip, dst_port, JA3/JA4, headers HTTP...) configurable. Aucun autre champ nest utilisé pour la décision de corrélation.
nest utilisé pour la décision de corrélation.
key: key:
- src_ip - src_ip
- src_port - src_port
@ -180,53 +269,50 @@ correlation:
value: 1 value: 1
unit: s unit: s
description: > description: >
Fenêtre de temps symétrique appliquée aux timestamps de A et B. Deux Fenêtre de temps appliquée aux timestamps de A et B. Si B narrive pas dans
événements sont corrélés si |tA - tB| <= time_window. La valeur et l'unité ce délai, A est émis comme orphelin.
sont définies dans le YAML. retention_limits:
max_http_items: 10000
max_network_items: 20000
description: >
Limites des caches. Si max_http_items est atteint, le plus ancien A est
évincé et émis orphelin. Si max_network_items est atteint, le plus ancien B
est supprimé silencieusement.
ttl_management:
network_ttl_s: 30
description: >
TTL des logs réseau. Chaque fois quun B est corrélé à un A (KeepAlive),
son TTL est remis à cette valeur.
timestamp_source: timestamp_source:
apache: field_timestamp apache: field_timestamp
network: reception_time network: reception_time
description: >
Pour A, utilisation du champ numérique "timestamp" (epoch ns). Pour B,
utilisation du temps de réception local.
orphan_policy: orphan_policy:
apache_always_emit: true apache_always_emit: true
network_emit: false network_emit: false
description: >
A est toujours émis (même sans B) avec correlated=false et orphan_side="A".
B nest jamais émis seul.
matching: matching:
mode: one_to_one_first_match mode: one_to_many
description: > description: >
Stratégie 1à1, premier match : lors de larrivée dun événement, on Stratégie 1àN : un log réseau peut être utilisé pour plusieurs logs HTTP
cherche le premier événement compatible dans le buffer de lautre source. successifs tant quil na pas expiré ni été évincé.
Les autres restent en attente ou expirent.
schema: schema:
description: > description: >
Les schémas des sources A et B sont variables. Le service impose seulement Schémas variables pour A et B. Quelques champs seulement sont obligatoires
quelques champs obligatoires nécessaires à la corrélation et accepte des pour la corrélation, les autres sont acceptés sans modification de code.
champs supplémentaires sans modification de code.
source_A: source_A:
description: > description: >
Logs HTTP applicatifs (Apache/reverse proxy) au format JSON. Schéma Logs HTTP applicatifs au format JSON.
variable, avec champs obligatoires pour corrélation (src_ip, src_port,
timestamp) et collecte des autres champs dans des maps.
required_fields: required_fields:
- name: src_ip - name: src_ip
type: string type: string
description: Adresse IP source client.
- name: src_port - name: src_port
type: int type: int
description: Port source client.
- name: timestamp - name: timestamp
type: int64 type: int64
unit: ns unit: ns
description: Timestamp de référence pour la corrélation.
optional_fields: optional_fields:
- name: time - name: time
type: string type: string
format: rfc3339
- name: dst_ip - name: dst_ip
type: string type: string
- name: dst_port - name: dst_port
@ -242,16 +328,10 @@ schema:
dynamic_fields: dynamic_fields:
- pattern: header_* - pattern: header_*
target_map: headers target_map: headers
description: >
Tous les champs header_* sont collectés dans headers[clé] = valeur.
- pattern: "*" - pattern: "*"
target_map: extra target_map: extra
description: >
Tous les champs non reconnus explicitement vont dans extra.
source_B: source_B:
description: > description: Logs réseau JSON (IP/TCP, JA3/JA4...).
Logs réseau JSON (IP/TCP, JA3/JA4...). Schéma variable. src_ip et src_port
sont obligatoires pour la corrélation, le reste est libre.
required_fields: required_fields:
- name: src_ip - name: src_ip
type: string type: string
@ -265,14 +345,10 @@ schema:
dynamic_fields: dynamic_fields:
- pattern: "*" - pattern: "*"
target_map: extra target_map: extra
description: >
Tous les autres champs (ip_meta_*, tcp_meta_*, ja3, ja4, etc.) sont
rangés dans extra.
normalized_event: normalized_event:
description: > description: >
Représentation interne unifiée des événements A/B sur laquelle opère la Représentation interne unifiée des événements A/B.
logique de corrélation.
fields: fields:
- name: source - name: source
type: enum("A","B") type: enum("A","B")
@ -293,13 +369,10 @@ schema:
optional: true optional: true
- name: extra - name: extra
type: map[string]any type: map[string]any
description: Champs additionnels provenant de A ou B.
correlated_log: correlated_log:
description: > description: >
Structure du log corrélé émis vers les sinks (fichier, ClickHouse). Contient Structure du log corrélé émis vers les sinks.
les informations de corrélation et tous les champs des sources A et B fusionnés
dans une structure JSON plate (flat).
fields: fields:
- name: timestamp - name: timestamp
type: time.Time type: time.Time
@ -319,18 +392,13 @@ schema:
type: string type: string
- name: "*" - name: "*"
type: map[string]any type: map[string]any
description: >
Tous les champs additionnels provenant de A et B sont fusionnés
directement à la racine du JSON (structure plate, sans subdivisions).
clickhouse_schema: clickhouse_schema:
strategy: external_ddls strategy: external_ddls
description: > description: >
logcorrelator ne gère pas les ALTER TABLE. La table ClickHouse doit être La table ClickHouse est gérée en dehors du service. logcorrelator remplit
créée/modifiée en dehors du service. logcorrelator remplit les colonnes les colonnes connues et met NULL si un champ manque. Tous les champs fusionnés
existantes qu'il connaît et met NULL si un champ manque. sont exposés dans une colonne JSON (fields).
Depuis la version 1.0.3, les champs apache et network sont remplacés par
une colonne unique fields JSON contenant tous les champs fusionnés.
base_columns: base_columns:
- name: timestamp - name: timestamp
type: DateTime64(9) type: DateTime64(9)
@ -350,31 +418,26 @@ clickhouse_schema:
type: JSON type: JSON
dynamic_fields: dynamic_fields:
mode: map_or_additional_columns mode: map_or_additional_columns
description: >
Les champs dynamiques peuvent être exposés via colonnes dédiées créées par
migration, ou via Map/JSON.
architecture: architecture:
description: > description: >
Architecture hexagonale : domaine de corrélation indépendant, ports Architecture hexagonale : domaine de corrélation indépendant, ports abstraits
abstraits pour les sources/sinks, adaptateurs pour sockets Unix, fichier et pour les sources/sinks, adaptateurs pour sockets Unix, fichier et ClickHouse,
ClickHouse, couche application dorchestration, et modules infra pour couche application dorchestration, et modules infra (config, observabilité).
config/observabilité.
modules: modules:
- name: cmd/logcorrelator - name: cmd/logcorrelator
type: entrypoint type: entrypoint
responsibilities: responsibilities:
- Chargement configuration YAML. - Chargement de la configuration YAML.
- Initialisation des adaptateurs d'entrée/sortie. - Initialisation des adaptateurs d'entrée/sortie.
- Création du CorrelationService. - Création du CorrelationService.
- Démarrage de l'orchestrateur. - Démarrage de lorchestrateur.
- Gestion du cycle de vie (signaux systemd). - Gestion des signaux (SIGINT, SIGTERM, SIGHUP).
- name: internal/domain - name: internal/domain
type: domain type: domain
responsibilities: responsibilities:
- Modèles NormalizedEvent et CorrelatedLog. - Modèles NormalizedEvent et CorrelatedLog.
- Implémentation de CorrelationService (buffers, fenêtre, - CorrelationService (fenêtre, TTL, buffers bornés, 1àN, orphelins).
orphelins).
- name: internal/ports - name: internal/ports
type: ports type: ports
responsibilities: responsibilities:
@ -382,163 +445,70 @@ architecture:
- name: internal/app - name: internal/app
type: application type: application
responsibilities: responsibilities:
- Orchestrator : relier EventSource → CorrelationService → MultiSink. - Orchestrator : EventSource → CorrelationService → MultiSink.
- name: internal/adapters/inbound/unixsocket - name: internal/adapters/inbound/unixsocket
type: adapter_inbound type: adapter_inbound
responsibilities: responsibilities:
- Lecture sockets Unix + parsing JSON → NormalizedEvent. - Lecture Unix datagram (SOCK_DGRAM) et parsing JSON → NormalizedEvent.
- name: internal/adapters/outbound/file - name: internal/adapters/outbound/file
type: adapter_outbound type: adapter_outbound
responsibilities: responsibilities:
- Écriture fichier JSON lines. - Écriture JSON lines.
- Réouverture du fichier sur SIGHUP.
- name: internal/adapters/outbound/clickhouse - name: internal/adapters/outbound/clickhouse
type: adapter_outbound type: adapter_outbound
responsibilities: responsibilities:
- Bufferisation + inserts batch vers ClickHouse. - Bufferisation + inserts batch, gestion du drop_on_overflow.
- Application de drop_on_overflow.
- name: internal/adapters/outbound/multi - name: internal/adapters/outbound/multi
type: adapter_outbound type: adapter_outbound
responsibilities: responsibilities:
- Fan-out vers plusieurs sinks. - Fanout vers plusieurs sinks.
- name: internal/config - name: internal/config
type: infrastructure type: infrastructure
responsibilities: responsibilities:
- Chargement/validation config YAML. - Chargement/validation de la configuration YAML.
- name: internal/observability - name: internal/observability
type: infrastructure type: infrastructure
responsibilities: responsibilities:
- Logging et métriques internes. - Logging interne, métriques (tailles des caches, évictions, erreurs datagram).
testing: testing:
unit: unit:
description: > description: >
Tests unitaires table-driven avec couverture cible ≥ 80 %. Focalisés sur Tests unitaires tabledriven, couverture cible ≥ 80 %, focale sur la logique
la logique de corrélation, parsing et sink ClickHouse.[web:94][web:98][web:102] de corrélation, les caches et les sinks.
coverage_minimum: 0.8 coverage_minimum: 0.8
focus: focus:
- CorrelationService - CorrelationService (fenêtre, TTL, évictions, 1àN)
- Parsing A/B → NormalizedEvent - Parsing A/B → NormalizedEvent (datagrammes)
- ClickHouseSink (batching, overflow) - ClickHouseSink (batching, overflow)
- FileSink (réouverture sur SIGHUP)
- MultiSink - MultiSink
integration: integration:
description: > description: >
Tests dintégration validant le flux complet A+B → corrélation → sinks, Tests dintégration validant le flux complet A+B → corrélation → sinks,
avec sockets simulés et ClickHouse mocké. avec sockets Unix datagram simulées, ClickHouse mocké et scénarios KeepAlive.
docker: docker:
description: > description: >
Build et tests entièrement encapsulés dans Docker, avec multistage build : Build, tests et packaging RPM sont exécutés intégralement dans des conteneurs
un stage builder pour compiler et tester, un stage runtime minimal pour via un multistage build.
exécuter le service.[web:95][web:103]
images:
builder:
base: golang:latest
purpose: build_and_test
runtime:
base: scratch
purpose: run_binary_only
build:
multi_stage: true
steps:
- name: unit_tests
description: >
go test ./... avec génération de couverture. Le build échoue si la
couverture est < 80 %.
- name: compile_binary
description: >
Compilation CGO_ENABLED=0, GOOS=linux, GOARCH=amd64 pour un binaire
statique /usr/bin/logcorrelator.
- name: assemble_runtime_image
description: >
Copie du binaire dans limage runtime et définition de lENTRYPOINT.
packaging:
description: >
logcorrelator est distribué sous forme de packages .rpm (Rocky Linux 8, 9 et AlmaLinux 10),
construits intégralement dans Docker avec fpm.
formats:
- rpm
target_distros:
rpm:
- rocky-linux-8
- rocky-linux-9
- almalinux-10
- rhel-8+
- rhel-9+
- rhel-10+
tool: fpm
build_pipeline: build_pipeline:
dockerfile: Dockerfile.package multi_stage: true
stages: stages:
- name: builder - name: test_and_compile
base: golang:latest
description: > description: >
Compilation du binaire Go avec CGO_ENABLED=0 pour un binaire statique. go test ./... (échec si couverture < 80 %), puis compilation dun binaire
GOOS=linux GOARCH=amd64. statique (CGO_ENABLED=0, GOOS=linux, GOARCH=amd64).
- name: rpm_rocky8_builder - name: rpm_builder
base: ruby:alpine
description: > description: >
Construction du package RPM pour Rocky Linux 8 (el8) avec fpm. Installation de fpm, git et outils RPM. Génération du changelog RPM à
- name: rpm_rocky9_builder partir de lhistorique. Construction des .rpm pour les différentes
distributions.
- name: output_export
base: scratch
description: > description: >
Construction du package RPM pour Rocky Linux 9 (el9) avec fpm. Étape minimale pour exposer les paquets RPM produits (docker build --output).
- name: rpm_almalinux10_builder
description: >
Construction du package RPM pour AlmaLinux 10 (el10) avec fpm.
- name: output
description: >
Image Alpine minimale contenant les packages dans
/packages/rpm/{rocky8,rocky9,almalinux10}.
files:
binary:
source: dist/logcorrelator
dest: /usr/bin/logcorrelator
mode: "0755"
config:
- source: config.example.yml
dest: /etc/logcorrelator/logcorrelator.yml
mode: "0640"
config_file: true
- source: config.example.yml
dest: /usr/share/logcorrelator/logcorrelator.yml.example
mode: "0640"
directories:
- path: /var/log/logcorrelator
mode: "0755"
- path: /var/run/logcorrelator
mode: "0755"
- path: /etc/logcorrelator
mode: "0750"
maintainer_scripts:
rpm:
post: packaging/rpm/post
preun: packaging/rpm/preun
postun: packaging/rpm/postun
dependencies:
rpm:
- systemd
verify:
rpm:
rocky8:
command: docker run --rm -v $(pwd)/dist/rpm/rocky8:/packages rockylinux:8 sh -c "dnf install -y /packages/*.rpm"
rocky9:
command: docker run --rm -v $(pwd)/dist/rpm/rocky9:/packages rockylinux:9 sh -c "dnf install -y /packages/*.rpm"
almalinux10:
command: docker run --rm -v $(pwd)/dist/rpm/almalinux10:/packages almalinux:10 sh -c "dnf install -y /packages/*.rpm"
non_functional:
performance:
target_latency_ms: 1000
description: >
Latence visée < 1 s entre réception et insertion ClickHouse, avec
batching léger.
reliability:
drop_on_clickhouse_failure: true
description: >
En cas de ClickHouse lent/HS, les logs sont drop audelà du buffer pour
protéger la machine.
security:
user_separation: true
privileges: least
description: >
Service sous utilisateur dédié, pas de secrets en clair dans les logs,
principe de moindre privilège.

View File

@ -106,9 +106,12 @@ func main() {
// Create correlation service // Create correlation service
correlationSvc := domain.NewCorrelationService(domain.CorrelationConfig{ correlationSvc := domain.NewCorrelationService(domain.CorrelationConfig{
TimeWindow: cfg.Correlation.GetTimeWindow(), TimeWindow: cfg.Correlation.GetTimeWindow(),
ApacheAlwaysEmit: cfg.Correlation.EmitOrphans, ApacheAlwaysEmit: cfg.Correlation.GetApacheAlwaysEmit(),
NetworkEmit: false, NetworkEmit: false,
MaxBufferSize: domain.DefaultMaxBufferSize, MaxHTTPBufferSize: cfg.Correlation.GetMaxHTTPBufferSize(),
MaxNetworkBufferSize: cfg.Correlation.GetMaxNetworkBufferSize(),
NetworkTTLS: cfg.Correlation.GetNetworkTTLS(),
MatchingMode: cfg.Correlation.GetMatchingMode(),
}, &domain.RealTimeProvider{}) }, &domain.RealTimeProvider{})
// Set logger for correlation service // Set logger for correlation service
@ -134,10 +137,26 @@ func main() {
// Wait for shutdown signal // Wait for shutdown signal
sigChan := make(chan os.Signal, 1) sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM) signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)
for {
sig := <-sigChan sig := <-sigChan
if sig == syscall.SIGHUP {
// Reopen file sinks for log rotation
logger.Info("SIGHUP received, reopening file sinks...")
if err := multiSink.Reopen(); err != nil {
logger.Error("Error reopening file sinks", err)
} else {
logger.Info("File sinks reopened successfully")
}
continue
}
// Shutdown signal received
logger.Info(fmt.Sprintf("Shutdown signal received: %v", sig)) logger.Info(fmt.Sprintf("Shutdown signal received: %v", sig))
break
}
// Graceful shutdown // Graceful shutdown
if err := orchestrator.Stop(); err != nil { if err := orchestrator.Stop(); err != nil {

View File

@ -20,15 +20,44 @@ inputs:
outputs: outputs:
file: file:
enabled: true
path: /var/log/logcorrelator/correlated.log path: /var/log/logcorrelator/correlated.log
clickhouse: clickhouse:
enabled: false
dsn: clickhouse://user:pass@localhost:9000/db dsn: clickhouse://user:pass@localhost:9000/db
table: correlated_logs_http_network table: correlated_logs_http_network
batch_size: 500
flush_interval_ms: 200
max_buffer_size: 5000
drop_on_overflow: true
async_insert: true
timeout_ms: 1000
stdout: stdout:
enabled: false enabled: false
correlation: correlation:
time_window_s: 1 # Time window for correlation (A and B must be within this window)
emit_orphans: true # http toujours émis, network jamais seul time_window:
value: 1
unit: s
# Orphan policy: what to do when no match is found
orphan_policy:
apache_always_emit: true # Always emit A events, even without B match
network_emit: false # Never emit B events alone
# Matching mode: one_to_one or one_to_many (Keep-Alive)
matching:
mode: one_to_many
# Buffer limits (max events in memory)
buffers:
max_http_items: 10000
max_network_items: 20000
# TTL for network events (source B)
ttl:
network_ttl_s: 30

View File

@ -115,6 +115,11 @@ func (s *ClickHouseSink) Name() string {
return "clickhouse" return "clickhouse"
} }
// Reopen is a no-op for ClickHouse (connection is managed internally).
func (s *ClickHouseSink) Reopen() error {
return nil
}
// Write adds a log to the buffer. // Write adds a log to the buffer.
func (s *ClickHouseSink) Write(ctx context.Context, log domain.CorrelatedLog) error { func (s *ClickHouseSink) Write(ctx context.Context, log domain.CorrelatedLog) error {
deadline := time.Now().Add(time.Duration(s.config.TimeoutMs) * time.Millisecond) deadline := time.Now().Add(time.Duration(s.config.TimeoutMs) * time.Millisecond)

View File

@ -38,9 +38,16 @@ func NewFileSink(config Config) (*FileSink, error) {
return nil, fmt.Errorf("invalid file path: %w", err) return nil, fmt.Errorf("invalid file path: %w", err)
} }
return &FileSink{ s := &FileSink{
config: config, config: config,
}, nil }
// Open file on creation
if err := s.openFile(); err != nil {
return nil, err
}
return s, nil
} }
// Name returns the sink name. // Name returns the sink name.
@ -48,6 +55,20 @@ func (s *FileSink) Name() string {
return "file" return "file"
} }
// Reopen closes and reopens the file (for log rotation on SIGHUP).
func (s *FileSink) Reopen() error {
s.mu.Lock()
defer s.mu.Unlock()
if s.file != nil {
if err := s.file.Close(); err != nil {
return fmt.Errorf("failed to close file: %w", err)
}
}
return s.openFile()
}
// Write writes a correlated log to the file. // Write writes a correlated log to the file.
func (s *FileSink) Write(ctx context.Context, log domain.CorrelatedLog) error { func (s *FileSink) Write(ctx context.Context, log domain.CorrelatedLog) error {
s.mu.Lock() s.mu.Lock()

View File

@ -121,3 +121,17 @@ func (s *MultiSink) Close() error {
} }
return firstErr return firstErr
} }
// Reopen reopens all sinks (for log rotation on SIGHUP).
func (s *MultiSink) Reopen() error {
s.mu.RLock()
defer s.mu.RUnlock()
var firstErr error
for _, sink := range s.sinks {
if err := sink.Reopen(); err != nil && firstErr == nil {
firstErr = err
}
}
return firstErr
}

View File

@ -14,6 +14,7 @@ type mockSink struct {
writeFunc func(domain.CorrelatedLog) error writeFunc func(domain.CorrelatedLog) error
flushFunc func() error flushFunc func() error
closeFunc func() error closeFunc func() error
reopenFunc func() error
} }
func (m *mockSink) Name() string { return m.name } func (m *mockSink) Name() string { return m.name }
@ -24,6 +25,12 @@ func (m *mockSink) Write(ctx context.Context, log domain.CorrelatedLog) error {
} }
func (m *mockSink) Flush(ctx context.Context) error { return m.flushFunc() } func (m *mockSink) Flush(ctx context.Context) error { return m.flushFunc() }
func (m *mockSink) Close() error { return m.closeFunc() } func (m *mockSink) Close() error { return m.closeFunc() }
func (m *mockSink) Reopen() error {
if m.reopenFunc != nil {
return m.reopenFunc()
}
return nil
}
func TestMultiSink_Write(t *testing.T) { func TestMultiSink_Write(t *testing.T) {
var mu sync.Mutex var mu sync.Mutex

View File

@ -35,6 +35,11 @@ func (s *StdoutSink) Name() string {
return "stdout" return "stdout"
} }
// Reopen is a no-op for stdout.
func (s *StdoutSink) Reopen() error {
return nil
}
// Write writes a correlated log to stdout. // Write writes a correlated log to stdout.
func (s *StdoutSink) Write(ctx context.Context, log domain.CorrelatedLog) error { func (s *StdoutSink) Write(ctx context.Context, log domain.CorrelatedLog) error {
s.mu.Lock() s.mu.Lock()

View File

@ -58,6 +58,7 @@ func (m *mockSink) Write(ctx context.Context, log domain.CorrelatedLog) error {
} }
func (m *mockSink) Flush(ctx context.Context) error { return nil } func (m *mockSink) Flush(ctx context.Context) error { return nil }
func (m *mockSink) Close() error { return nil } func (m *mockSink) Close() error { return nil }
func (m *mockSink) Reopen() error { return nil }
func (m *mockSink) getWritten() []domain.CorrelatedLog { func (m *mockSink) getWritten() []domain.CorrelatedLog {
m.mu.Lock() m.mu.Lock()

View File

@ -7,6 +7,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/logcorrelator/logcorrelator/internal/domain"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
) )
@ -83,10 +84,61 @@ type StdoutOutputConfig struct {
// CorrelationConfig holds correlation configuration. // CorrelationConfig holds correlation configuration.
type CorrelationConfig struct { type CorrelationConfig struct {
TimeWindow TimeWindowConfig `yaml:"time_window"`
OrphanPolicy OrphanPolicyConfig `yaml:"orphan_policy"`
Matching MatchingConfig `yaml:"matching"`
Buffers BuffersConfig `yaml:"buffers"`
TTL TTLConfig `yaml:"ttl"`
// Deprecated: Use TimeWindow.Value instead
TimeWindowS int `yaml:"time_window_s"` TimeWindowS int `yaml:"time_window_s"`
// Deprecated: Use OrphanPolicy.ApacheAlwaysEmit instead
EmitOrphans bool `yaml:"emit_orphans"` EmitOrphans bool `yaml:"emit_orphans"`
} }
// TimeWindowConfig holds time window configuration.
type TimeWindowConfig struct {
Value int `yaml:"value"`
Unit string `yaml:"unit"` // s, ms, etc.
}
// GetDuration returns the time window as a duration.
func (c *TimeWindowConfig) GetDuration() time.Duration {
value := c.Value
if value <= 0 {
value = 1
}
switch c.Unit {
case "ms", "millisecond", "milliseconds":
return time.Duration(value) * time.Millisecond
case "s", "sec", "second", "seconds":
fallthrough
default:
return time.Duration(value) * time.Second
}
}
// OrphanPolicyConfig holds orphan event policy configuration.
type OrphanPolicyConfig struct {
ApacheAlwaysEmit bool `yaml:"apache_always_emit"`
NetworkEmit bool `yaml:"network_emit"`
}
// MatchingConfig holds matching mode configuration.
type MatchingConfig struct {
Mode string `yaml:"mode"` // one_to_one or one_to_many
}
// BuffersConfig holds buffer size configuration.
type BuffersConfig struct {
MaxHTTPItems int `yaml:"max_http_items"`
MaxNetworkItems int `yaml:"max_network_items"`
}
// TTLConfig holds TTL configuration.
type TTLConfig struct {
NetworkTTLS int `yaml:"network_ttl_s"`
}
// Load loads configuration from a YAML file. // Load loads configuration from a YAML file.
func Load(path string) (*Config, error) { func Load(path string) (*Config, error) {
data, err := os.ReadFile(path) data, err := os.ReadFile(path)
@ -208,7 +260,13 @@ func (c *Config) Validate() error {
} }
// GetTimeWindow returns the time window as a duration. // GetTimeWindow returns the time window as a duration.
// Deprecated: Use TimeWindow.GetDuration() instead.
func (c *CorrelationConfig) GetTimeWindow() time.Duration { func (c *CorrelationConfig) GetTimeWindow() time.Duration {
// New config takes precedence
if c.TimeWindow.Value > 0 {
return c.TimeWindow.GetDuration()
}
// Fallback to deprecated field
value := c.TimeWindowS value := c.TimeWindowS
if value <= 0 { if value <= 0 {
value = 1 value = 1
@ -216,6 +274,47 @@ func (c *CorrelationConfig) GetTimeWindow() time.Duration {
return time.Duration(value) * time.Second return time.Duration(value) * time.Second
} }
// GetApacheAlwaysEmit returns whether to always emit Apache events.
func (c *CorrelationConfig) GetApacheAlwaysEmit() bool {
if c.OrphanPolicy.ApacheAlwaysEmit {
return true
}
// Fallback to deprecated field
return c.EmitOrphans
}
// GetMatchingMode returns the matching mode.
func (c *CorrelationConfig) GetMatchingMode() string {
if c.Matching.Mode != "" {
return c.Matching.Mode
}
return "one_to_many" // Default to Keep-Alive
}
// GetMaxHTTPBufferSize returns the max HTTP buffer size.
func (c *CorrelationConfig) GetMaxHTTPBufferSize() int {
if c.Buffers.MaxHTTPItems > 0 {
return c.Buffers.MaxHTTPItems
}
return domain.DefaultMaxHTTPBufferSize
}
// GetMaxNetworkBufferSize returns the max network buffer size.
func (c *CorrelationConfig) GetMaxNetworkBufferSize() int {
if c.Buffers.MaxNetworkItems > 0 {
return c.Buffers.MaxNetworkItems
}
return domain.DefaultMaxNetworkBufferSize
}
// GetNetworkTTLS returns the network TTL in seconds.
func (c *CorrelationConfig) GetNetworkTTLS() int {
if c.TTL.NetworkTTLS > 0 {
return c.TTL.NetworkTTLS
}
return domain.DefaultNetworkTTLS
}
// GetSocketPermissions returns the socket permissions as os.FileMode. // GetSocketPermissions returns the socket permissions as os.FileMode.
// Default is 0660 (owner + group read/write). // Default is 0660 (owner + group read/write).
func (c *UnixSocketConfig) GetSocketPermissions() os.FileMode { func (c *UnixSocketConfig) GetSocketPermissions() os.FileMode {

View File

@ -9,10 +9,18 @@ import (
) )
const ( const (
// DefaultMaxBufferSize is the default maximum number of events per buffer // DefaultMaxHTTPBufferSize is the default maximum number of HTTP events (source A)
DefaultMaxBufferSize = 10000 DefaultMaxHTTPBufferSize = 10000
// DefaultMaxNetworkBufferSize is the default maximum number of network events (source B)
DefaultMaxNetworkBufferSize = 20000
// DefaultTimeWindow is used when no valid time window is provided // DefaultTimeWindow is used when no valid time window is provided
DefaultTimeWindow = time.Second DefaultTimeWindow = time.Second
// DefaultNetworkTTLS is the default TTL for network events in seconds
DefaultNetworkTTLS = 30
// MatchingModeOneToOne indicates single correlation (consume B after match)
MatchingModeOneToOne = "one_to_one"
// MatchingModeOneToMany indicates Keep-Alive mode (B can match multiple A)
MatchingModeOneToMany = "one_to_many"
) )
// CorrelationConfig holds the correlation configuration. // CorrelationConfig holds the correlation configuration.
@ -20,7 +28,10 @@ type CorrelationConfig struct {
TimeWindow time.Duration TimeWindow time.Duration
ApacheAlwaysEmit bool ApacheAlwaysEmit bool
NetworkEmit bool NetworkEmit bool
MaxBufferSize int // Maximum events to buffer per source MaxHTTPBufferSize int // Maximum events to buffer for source A (HTTP)
MaxNetworkBufferSize int // Maximum events to buffer for source B (Network)
NetworkTTLS int // TTL in seconds for network events (source B)
MatchingMode string // "one_to_one" or "one_to_many" (Keep-Alive)
} }
// CorrelationService handles the correlation logic between source A and B events. // CorrelationService handles the correlation logic between source A and B events.
@ -31,6 +42,7 @@ type CorrelationService struct {
bufferB *eventBuffer bufferB *eventBuffer
pendingA map[string][]*list.Element // key -> ordered elements containing *NormalizedEvent pendingA map[string][]*list.Element // key -> ordered elements containing *NormalizedEvent
pendingB map[string][]*list.Element pendingB map[string][]*list.Element
networkTTLs map[*list.Element]time.Time // TTL expiration time for each B event
timeProvider TimeProvider timeProvider TimeProvider
logger *observability.Logger logger *observability.Logger
} }
@ -62,12 +74,21 @@ func NewCorrelationService(config CorrelationConfig, timeProvider TimeProvider)
if timeProvider == nil { if timeProvider == nil {
timeProvider = &RealTimeProvider{} timeProvider = &RealTimeProvider{}
} }
if config.MaxBufferSize <= 0 { if config.MaxHTTPBufferSize <= 0 {
config.MaxBufferSize = DefaultMaxBufferSize config.MaxHTTPBufferSize = DefaultMaxHTTPBufferSize
}
if config.MaxNetworkBufferSize <= 0 {
config.MaxNetworkBufferSize = DefaultMaxNetworkBufferSize
} }
if config.TimeWindow <= 0 { if config.TimeWindow <= 0 {
config.TimeWindow = DefaultTimeWindow config.TimeWindow = DefaultTimeWindow
} }
if config.NetworkTTLS <= 0 {
config.NetworkTTLS = DefaultNetworkTTLS
}
if config.MatchingMode == "" {
config.MatchingMode = MatchingModeOneToMany // Default to Keep-Alive
}
return &CorrelationService{ return &CorrelationService{
config: config, config: config,
@ -75,6 +96,7 @@ func NewCorrelationService(config CorrelationConfig, timeProvider TimeProvider)
bufferB: newEventBuffer(), bufferB: newEventBuffer(),
pendingA: make(map[string][]*list.Element), pendingA: make(map[string][]*list.Element),
pendingB: make(map[string][]*list.Element), pendingB: make(map[string][]*list.Element),
networkTTLs: make(map[*list.Element]time.Time),
timeProvider: timeProvider, timeProvider: timeProvider,
logger: observability.NewLogger("correlation"), logger: observability.NewLogger("correlation"),
} }
@ -140,9 +162,9 @@ func (s *CorrelationService) getBufferSize(source EventSource) int {
func (s *CorrelationService) isBufferFull(source EventSource) bool { func (s *CorrelationService) isBufferFull(source EventSource) bool {
switch source { switch source {
case SourceA: case SourceA:
return s.bufferA.events.Len() >= s.config.MaxBufferSize return s.bufferA.events.Len() >= s.config.MaxHTTPBufferSize
case SourceB: case SourceB:
return s.bufferB.events.Len() >= s.config.MaxBufferSize return s.bufferB.events.Len() >= s.config.MaxNetworkBufferSize
} }
return false return false
} }
@ -150,14 +172,41 @@ func (s *CorrelationService) isBufferFull(source EventSource) bool {
func (s *CorrelationService) processSourceA(event *NormalizedEvent) ([]CorrelatedLog, bool) { func (s *CorrelationService) processSourceA(event *NormalizedEvent) ([]CorrelatedLog, bool) {
key := event.CorrelationKey() key := event.CorrelationKey()
// Look for the first matching B event (one-to-one first match) // Look for matching B events
if bEvent := s.findAndPopFirstMatch(s.bufferB, s.pendingB, key, func(other *NormalizedEvent) bool { matches := s.findMatches(s.bufferB, s.pendingB, key, func(other *NormalizedEvent) bool {
return s.eventsMatch(event, other) return s.eventsMatch(event, other)
}); bEvent != nil { })
if len(matches) > 0 {
var results []CorrelatedLog
// Correlate with all matching B events (one-to-many)
for _, bEvent := range matches {
correlated := NewCorrelatedLog(event, bEvent) correlated := NewCorrelatedLog(event, bEvent)
s.logger.Debugf("correlation found: A(src_ip=%s src_port=%d) + B(src_ip=%s src_port=%d)", s.logger.Debugf("correlation found: A(src_ip=%s src_port=%d) + B(src_ip=%s src_port=%d)",
event.SrcIP, event.SrcPort, bEvent.SrcIP, bEvent.SrcPort) event.SrcIP, event.SrcPort, bEvent.SrcIP, bEvent.SrcPort)
return []CorrelatedLog{correlated}, false results = append(results, correlated)
// Reset TTL for matched B event (Keep-Alive)
if s.config.MatchingMode == MatchingModeOneToMany {
// Find the element for this B event and reset TTL
bKey := bEvent.CorrelationKey()
if elements, ok := s.pendingB[bKey]; ok {
for _, elem := range elements {
if elem.Value.(*NormalizedEvent) == bEvent {
s.resetNetworkTTL(elem)
break
}
}
}
}
}
// In one-to-one mode, remove the first matching B
if s.config.MatchingMode == MatchingModeOneToOne {
s.removeEvent(s.bufferB, s.pendingB, matches[0])
}
return results, false
} }
// No match found - orphan A event // No match found - orphan A event
@ -206,30 +255,50 @@ func (s *CorrelationService) addEvent(event *NormalizedEvent) {
case SourceB: case SourceB:
elem := s.bufferB.events.PushBack(event) elem := s.bufferB.events.PushBack(event)
s.pendingB[key] = append(s.pendingB[key], elem) s.pendingB[key] = append(s.pendingB[key], elem)
// Set TTL for network event
s.networkTTLs[elem] = s.timeProvider.Now().Add(time.Duration(s.config.NetworkTTLS) * time.Second)
} }
} }
func (s *CorrelationService) cleanExpired() { func (s *CorrelationService) cleanExpired() {
now := s.timeProvider.Now() now := s.timeProvider.Now()
cutoff := now.Add(-s.config.TimeWindow)
// Clean expired events from both buffers using shared logic // Clean expired A events (based on time window)
s.cleanBuffer(s.bufferA, s.pendingA, cutoff) aCutoff := now.Add(-s.config.TimeWindow)
s.cleanBuffer(s.bufferB, s.pendingB, cutoff) s.cleanBuffer(s.bufferA, s.pendingA, aCutoff, nil)
// Clean expired B events (based on TTL)
bCutoff := now.Add(-time.Duration(s.config.NetworkTTLS) * time.Second)
s.cleanBuffer(s.bufferB, s.pendingB, bCutoff, s.networkTTLs)
} }
// cleanBuffer removes expired events from a buffer. // cleanBuffer removes expired events from a buffer.
func (s *CorrelationService) cleanBuffer(buffer *eventBuffer, pending map[string][]*list.Element, cutoff time.Time) { func (s *CorrelationService) cleanBuffer(buffer *eventBuffer, pending map[string][]*list.Element, cutoff time.Time, networkTTLs map[*list.Element]time.Time) {
for elem := buffer.events.Front(); elem != nil; { for elem := buffer.events.Front(); elem != nil; {
next := elem.Next() next := elem.Next()
event := elem.Value.(*NormalizedEvent) event := elem.Value.(*NormalizedEvent)
if event.Timestamp.Before(cutoff) {
// Check if event is expired
isExpired := event.Timestamp.Before(cutoff)
// For B events, also check TTL
if !isExpired && networkTTLs != nil {
if ttl, exists := networkTTLs[elem]; exists {
isExpired = s.timeProvider.Now().After(ttl)
}
}
if isExpired {
key := event.CorrelationKey() key := event.CorrelationKey()
buffer.events.Remove(elem) buffer.events.Remove(elem)
pending[key] = removeElementFromSlice(pending[key], elem) pending[key] = removeElementFromSlice(pending[key], elem)
if len(pending[key]) == 0 { if len(pending[key]) == 0 {
delete(pending, key) delete(pending, key)
} }
// Remove from TTL map
if networkTTLs != nil {
delete(networkTTLs, elem)
}
} }
elem = next elem = next
} }
@ -266,6 +335,76 @@ func (s *CorrelationService) findAndPopFirstMatch(
return nil return nil
} }
// findMatches returns all matching events without removing them (for one-to-many).
func (s *CorrelationService) findMatches(
buffer *eventBuffer,
pending map[string][]*list.Element,
key string,
matcher func(*NormalizedEvent) bool,
) []*NormalizedEvent {
elements, ok := pending[key]
if !ok || len(elements) == 0 {
return nil
}
var matches []*NormalizedEvent
for _, elem := range elements {
other := elem.Value.(*NormalizedEvent)
if matcher(other) {
matches = append(matches, other)
}
}
return matches
}
// getElementByKey finds the list element for a given event in pending map.
func (s *CorrelationService) getElementByKey(pending map[string][]*list.Element, key string, event *NormalizedEvent) *list.Element {
elements, ok := pending[key]
if !ok {
return nil
}
for _, elem := range elements {
if elem.Value.(*NormalizedEvent) == event {
return elem
}
}
return nil
}
// removeEvent removes an event from buffer and pending maps.
func (s *CorrelationService) removeEvent(buffer *eventBuffer, pending map[string][]*list.Element, event *NormalizedEvent) {
key := event.CorrelationKey()
elements, ok := pending[key]
if !ok {
return
}
for idx, elem := range elements {
if elem.Value.(*NormalizedEvent) == event {
buffer.events.Remove(elem)
updated := append(elements[:idx], elements[idx+1:]...)
if len(updated) == 0 {
delete(pending, key)
} else {
pending[key] = updated
}
// Remove from TTL map if present
delete(s.networkTTLs, elem)
break
}
}
}
// resetNetworkTTL resets the TTL for a network event (Keep-Alive).
func (s *CorrelationService) resetNetworkTTL(elem *list.Element) {
if elem == nil {
return
}
s.networkTTLs[elem] = s.timeProvider.Now().Add(time.Duration(s.config.NetworkTTLS) * time.Second)
}
func removeElementFromSlice(elements []*list.Element, target *list.Element) []*list.Element { func removeElementFromSlice(elements []*list.Element, target *list.Element) []*list.Element {
if len(elements) == 0 { if len(elements) == 0 {
return elements return elements
@ -301,6 +440,7 @@ func (s *CorrelationService) Flush() []CorrelatedLog {
s.bufferB.events.Init() s.bufferB.events.Init()
s.pendingA = make(map[string][]*list.Element) s.pendingA = make(map[string][]*list.Element)
s.pendingB = make(map[string][]*list.Element) s.pendingB = make(map[string][]*list.Element)
s.networkTTLs = make(map[*list.Element]time.Time)
return results return results
} }

View File

@ -19,8 +19,12 @@ func TestCorrelationService_Match(t *testing.T) {
config := CorrelationConfig{ config := CorrelationConfig{
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: false, // Don't emit A immediately to test matching ApacheAlwaysEmit: false,
NetworkEmit: false, NetworkEmit: false,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -65,6 +69,10 @@ func TestCorrelationService_NoMatch_DifferentIP(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: true, ApacheAlwaysEmit: true,
NetworkEmit: false, NetworkEmit: false,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -99,6 +107,10 @@ func TestCorrelationService_NoMatch_TimeWindowExceeded(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: true, ApacheAlwaysEmit: true,
NetworkEmit: false, NetworkEmit: false,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -133,6 +145,10 @@ func TestCorrelationService_Flush(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: true, ApacheAlwaysEmit: true,
NetworkEmit: false, NetworkEmit: false,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -164,6 +180,10 @@ func TestCorrelationService_GetBufferSizes(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: false, ApacheAlwaysEmit: false,
NetworkEmit: false, NetworkEmit: false,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -197,6 +217,10 @@ func TestCorrelationService_FlushWithEvents(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: true, ApacheAlwaysEmit: true,
NetworkEmit: true, NetworkEmit: true,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -222,6 +246,7 @@ func TestCorrelationService_FlushWithEvents(t *testing.T) {
elemB := svc.bufferB.events.PushBack(networkEvent) elemB := svc.bufferB.events.PushBack(networkEvent)
svc.pendingB[keyB] = append(svc.pendingB[keyB], elemB) svc.pendingB[keyB] = append(svc.pendingB[keyB], elemB)
svc.networkTTLs[elemB] = now.Add(time.Duration(svc.config.NetworkTTLS) * time.Second)
flushed := svc.Flush() flushed := svc.Flush()
if len(flushed) != 1 { if len(flushed) != 1 {
@ -246,7 +271,8 @@ func TestCorrelationService_BufferOverflow(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: false, ApacheAlwaysEmit: false,
NetworkEmit: false, NetworkEmit: false,
MaxBufferSize: 2, MaxHTTPBufferSize: 2,
MaxNetworkBufferSize: 2,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -282,12 +308,21 @@ func TestCorrelationService_DefaultConfig(t *testing.T) {
config := CorrelationConfig{} config := CorrelationConfig{}
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
if svc.config.MaxBufferSize != DefaultMaxBufferSize { if svc.config.MaxHTTPBufferSize != DefaultMaxHTTPBufferSize {
t.Errorf("expected MaxBufferSize %d, got %d", DefaultMaxBufferSize, svc.config.MaxBufferSize) t.Errorf("expected MaxHTTPBufferSize %d, got %d", DefaultMaxHTTPBufferSize, svc.config.MaxHTTPBufferSize)
}
if svc.config.MaxNetworkBufferSize != DefaultMaxNetworkBufferSize {
t.Errorf("expected MaxNetworkBufferSize %d, got %d", DefaultMaxNetworkBufferSize, svc.config.MaxNetworkBufferSize)
} }
if svc.config.TimeWindow != DefaultTimeWindow { if svc.config.TimeWindow != DefaultTimeWindow {
t.Errorf("expected TimeWindow %v, got %v", DefaultTimeWindow, svc.config.TimeWindow) t.Errorf("expected TimeWindow %v, got %v", DefaultTimeWindow, svc.config.TimeWindow)
} }
if svc.config.NetworkTTLS != DefaultNetworkTTLS {
t.Errorf("expected NetworkTTLS %d, got %d", DefaultNetworkTTLS, svc.config.NetworkTTLS)
}
if svc.config.MatchingMode != MatchingModeOneToMany {
t.Errorf("expected MatchingMode %s, got %s", MatchingModeOneToMany, svc.config.MatchingMode)
}
} }
func TestCorrelationService_NilTimeProvider(t *testing.T) { func TestCorrelationService_NilTimeProvider(t *testing.T) {
@ -310,6 +345,10 @@ func TestCorrelationService_DifferentSourceTypes(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: false, ApacheAlwaysEmit: false,
NetworkEmit: false, NetworkEmit: false,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -333,10 +372,9 @@ func TestCorrelationService_DifferentSourceTypes(t *testing.T) {
SrcPort: 8080, SrcPort: 8080,
} }
results = svc.ProcessEvent(apacheEvent) results = svc.ProcessEvent(apacheEvent)
if len(results) != 1 { if len(results) < 1 {
t.Errorf("expected 1 result (correlated), got %d", len(results)) t.Errorf("expected at least 1 result (correlated), got %d", len(results))
} } else if !results[0].Correlated {
if !results[0].Correlated {
t.Error("expected correlated result") t.Error("expected correlated result")
} }
} }
@ -349,6 +387,10 @@ func TestCorrelationService_NetworkEmitTrue_DoesNotEmitBAlone(t *testing.T) {
TimeWindow: time.Second, TimeWindow: time.Second,
ApacheAlwaysEmit: false, ApacheAlwaysEmit: false,
NetworkEmit: true, NetworkEmit: true,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
MatchingMode: MatchingModeOneToMany,
} }
svc := NewCorrelationService(config, timeProvider) svc := NewCorrelationService(config, timeProvider)
@ -370,3 +412,204 @@ func TestCorrelationService_NetworkEmitTrue_DoesNotEmitBAlone(t *testing.T) {
t.Errorf("expected 0 flushed orphan B events, got %d", len(flushed)) t.Errorf("expected 0 flushed orphan B events, got %d", len(flushed))
} }
} }
func TestCorrelationService_OneToMany_KeepAlive(t *testing.T) {
now := time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC)
timeProvider := &mockTimeProvider{now: now}
config := CorrelationConfig{
TimeWindow: time.Second,
ApacheAlwaysEmit: false,
NetworkEmit: false,
MatchingMode: MatchingModeOneToMany, // Keep-Alive mode
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
}
svc := NewCorrelationService(config, timeProvider)
// Send B event first (network)
networkEvent := &NormalizedEvent{
Source: SourceB,
Timestamp: now,
SrcIP: "192.168.1.1",
SrcPort: 8080,
Raw: map[string]any{"ja3": "abc123"},
}
results := svc.ProcessEvent(networkEvent)
if len(results) != 0 {
t.Fatalf("expected 0 results (B buffered), got %d", len(results))
}
// Send first A event (Apache) - should correlate with B
apacheEvent1 := &NormalizedEvent{
Source: SourceA,
Timestamp: now.Add(500 * time.Millisecond),
SrcIP: "192.168.1.1",
SrcPort: 8080,
Raw: map[string]any{"method": "GET", "path": "/api/first"},
}
results = svc.ProcessEvent(apacheEvent1)
if len(results) != 1 {
t.Errorf("expected 1 correlated result for first A, got %d", len(results))
} else if !results[0].Correlated {
t.Error("expected correlated result for first A")
}
// Send second A event (same connection, Keep-Alive) - should also correlate with same B
apacheEvent2 := &NormalizedEvent{
Source: SourceA,
Timestamp: now.Add(1 * time.Second),
SrcIP: "192.168.1.1",
SrcPort: 8080,
Raw: map[string]any{"method": "GET", "path": "/api/second"},
}
results = svc.ProcessEvent(apacheEvent2)
if len(results) != 1 {
t.Errorf("expected 1 correlated result for second A (Keep-Alive), got %d", len(results))
} else if !results[0].Correlated {
t.Error("expected correlated result for second A (Keep-Alive)")
}
// Verify B is still in buffer (Keep-Alive)
a, b := svc.GetBufferSizes()
if a != 0 {
t.Errorf("expected A buffer empty, got %d", a)
}
if b != 1 {
t.Errorf("expected B buffer size 1 (Keep-Alive), got %d", b)
}
}
func TestCorrelationService_OneToOne_ConsumeB(t *testing.T) {
now := time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC)
timeProvider := &mockTimeProvider{now: now}
config := CorrelationConfig{
TimeWindow: time.Second,
ApacheAlwaysEmit: false,
NetworkEmit: false,
MatchingMode: MatchingModeOneToOne, // Consume B after match
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: DefaultNetworkTTLS,
}
svc := NewCorrelationService(config, timeProvider)
// Send B event first
networkEvent := &NormalizedEvent{
Source: SourceB,
Timestamp: now,
SrcIP: "192.168.1.1",
SrcPort: 8080,
Raw: map[string]any{"ja3": "abc123"},
}
svc.ProcessEvent(networkEvent)
// Send first A event - should correlate and consume B
apacheEvent1 := &NormalizedEvent{
Source: SourceA,
Timestamp: now.Add(500 * time.Millisecond),
SrcIP: "192.168.1.1",
SrcPort: 8080,
}
results := svc.ProcessEvent(apacheEvent1)
if len(results) != 1 {
t.Fatalf("expected 1 correlated result, got %d", len(results))
}
// Send second A event - should NOT correlate (B was consumed)
apacheEvent2 := &NormalizedEvent{
Source: SourceA,
Timestamp: now.Add(1 * time.Second),
SrcIP: "192.168.1.1",
SrcPort: 8080,
}
results = svc.ProcessEvent(apacheEvent2)
if len(results) != 0 {
t.Errorf("expected 0 results (B consumed), got %d", len(results))
}
// Verify both buffers are empty
a, b := svc.GetBufferSizes()
if a != 1 {
t.Errorf("expected A buffer size 1 (second A buffered), got %d", a)
}
if b != 0 {
t.Errorf("expected B buffer empty (consumed), got %d", b)
}
}
func TestCorrelationService_NetworkTTL_ResetOnMatch(t *testing.T) {
now := time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC)
timeProvider := &mockTimeProvider{now: now}
config := CorrelationConfig{
TimeWindow: 5 * time.Second, // 5 seconds time window for correlation
ApacheAlwaysEmit: false,
NetworkEmit: false,
MatchingMode: MatchingModeOneToMany,
MaxHTTPBufferSize: DefaultMaxHTTPBufferSize,
MaxNetworkBufferSize: DefaultMaxNetworkBufferSize,
NetworkTTLS: 10, // 10 seconds TTL for B events
}
svc := NewCorrelationService(config, timeProvider)
// Send B event
networkEvent := &NormalizedEvent{
Source: SourceB,
Timestamp: now,
SrcIP: "192.168.1.1",
SrcPort: 8080,
}
svc.ProcessEvent(networkEvent)
// Verify B is in buffer
_, b := svc.GetBufferSizes()
if b != 1 {
t.Fatalf("expected B in buffer, got %d", b)
}
// Advance time by 3 seconds (before TTL expires)
timeProvider.now = now.Add(3 * time.Second)
// Send A event with timestamp within time window of B
// A's timestamp is t=3s, B's timestamp is t=0s, diff = 3s < 5s (time_window)
apacheEvent := &NormalizedEvent{
Source: SourceA,
Timestamp: timeProvider.now,
SrcIP: "192.168.1.1",
SrcPort: 8080,
}
results := svc.ProcessEvent(apacheEvent)
if len(results) != 1 {
t.Fatalf("expected 1 correlated result, got %d", len(results))
}
// B should still be in buffer (TTL reset)
_, b = svc.GetBufferSizes()
if b != 1 {
t.Errorf("expected B still in buffer after TTL reset, got %d", b)
}
// Advance time by 7 more seconds (total 10s from start, 7s from last match)
timeProvider.now = now.Add(10 * time.Second)
// B should still be alive (TTL was reset to 10s from t=3s, so expires at t=13s)
svc.cleanExpired()
_, b = svc.GetBufferSizes()
if b != 1 {
t.Errorf("expected B still alive after TTL reset, got %d", b)
}
// Advance time past the reset TTL (t=14s > t=13s)
timeProvider.now = now.Add(14 * time.Second)
svc.cleanExpired()
_, b = svc.GetBufferSizes()
if b != 0 {
t.Errorf("expected B expired after reset TTL, got %d", b)
}
}

View File

@ -32,6 +32,10 @@ type CorrelatedLogSink interface {
// Name returns the sink name. // Name returns the sink name.
Name() string Name() string
// Reopen closes and reopens the sink (for log rotation on SIGHUP).
// Optional: only FileSink implements this.
Reopen() error
} }
// CorrelationProcessor defines the interface for the correlation service. // CorrelationProcessor defines the interface for the correlation service.

View File

@ -7,6 +7,7 @@ Type=simple
User=logcorrelator User=logcorrelator
Group=logcorrelator Group=logcorrelator
ExecStart=/usr/bin/logcorrelator -config /etc/logcorrelator/logcorrelator.yml ExecStart=/usr/bin/logcorrelator -config /etc/logcorrelator/logcorrelator.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure Restart=on-failure
RestartSec=5 RestartSec=5

View File

@ -2,7 +2,7 @@
# Compatible with CentOS 7, Rocky Linux 8, 9, 10 # Compatible with CentOS 7, Rocky Linux 8, 9, 10
# Define version before Version: field for RPM macro support # Define version before Version: field for RPM macro support
%global spec_version 1.0.9 %global spec_version 1.1.0
Name: logcorrelator Name: logcorrelator
Version: %{spec_version} Version: %{spec_version}
@ -38,6 +38,7 @@ mkdir -p %{buildroot}/usr/share/logcorrelator
mkdir -p %{buildroot}/var/log/logcorrelator mkdir -p %{buildroot}/var/log/logcorrelator
mkdir -p %{buildroot}/var/run/logcorrelator mkdir -p %{buildroot}/var/run/logcorrelator
mkdir -p %{buildroot}/etc/systemd/system mkdir -p %{buildroot}/etc/systemd/system
mkdir -p %{buildroot}/etc/logrotate.d
# Install binary # Install binary
install -m 0755 %{_sourcedir}/logcorrelator %{buildroot}/usr/bin/logcorrelator install -m 0755 %{_sourcedir}/logcorrelator %{buildroot}/usr/bin/logcorrelator
@ -49,6 +50,9 @@ install -m 0640 %{_sourcedir}/logcorrelator.yml %{buildroot}/usr/share/logcorrel
# Install systemd service # Install systemd service
install -m 0644 %{_sourcedir}/logcorrelator.service %{buildroot}/etc/systemd/system/logcorrelator.service install -m 0644 %{_sourcedir}/logcorrelator.service %{buildroot}/etc/systemd/system/logcorrelator.service
# Install logrotate config
install -m 0644 %{_sourcedir}/logrotate %{buildroot}/etc/logrotate.d/logcorrelator
%post %post
# Create logcorrelator user and group # Create logcorrelator user and group
if ! getent group logcorrelator >/dev/null 2>&1; then if ! getent group logcorrelator >/dev/null 2>&1; then
@ -114,27 +118,64 @@ fi
/var/log/logcorrelator /var/log/logcorrelator
/var/run/logcorrelator /var/run/logcorrelator
/etc/systemd/system/logcorrelator.service /etc/systemd/system/logcorrelator.service
/etc/logrotate.d/logcorrelator
%changelog %changelog
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.3-1 * Mon Mar 02 2026 logcorrelator <dev@example.com> - 1.1.0-1
- Feat: Keep-Alive support (one-to-many correlation mode)
- Feat: Dynamic TTL for network events (source B)
- Feat: Separate buffer sizes for HTTP and network events
- Feat: SIGHUP signal handling for log rotation
- Feat: File sink Reopen() method for log rotation
- Feat: logrotate configuration included
- Feat: ExecReload added to systemd service
- Feat: New YAML config structure (time_window, orphan_policy, matching, buffers, ttl)
- Docs: Updated architecture.yml and config.example.yml
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.7-1
- Added: Log levels DEBUG, INFO, WARN, ERROR configurable via log.level
- Added: Warn and Warnf methods for warning messages
- Added: Debug logs for events received from sockets and correlations
- Added: Warning logs for orphan events and buffer overflow
- Changed: Configuration log.enabled replaced by log.level
- Changed: Orphan events and buffer overflow now logged as WARN instead of DEBUG
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.6-1
- Changed: Configuration YAML simplified, removed service.name, service.language
- Changed: Correlation config simplified, time_window_s instead of nested object
- Changed: Orphan policy simplified to emit_orphans boolean
- Changed: Apache socket renamed to http.socket
- Added: socket_permissions option on unix sockets
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.5-1
- Added: Systemd service auto-start after RPM installation
- Added: Systemd service hardening (TimeoutStartSec, TimeoutStopSec, ReadWritePaths)
- Fixed: Systemd service unit correct config path (.yml instead of .conf)
- Fixed: CI workflow branch name main to master
- Changed: RPM packaging generic el8/el9/el10 directory naming
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.4-1
- Breaking: Flattened JSON output structure - removed apache and network subdivisions - Breaking: Flattened JSON output structure - removed apache and network subdivisions
- All log fields now merged into single-level JSON structure - All log fields now merged into single-level JSON structure
- ClickHouse schema: replaced apache JSON and network JSON columns with fields JSON column - ClickHouse schema: replaced apache JSON and network JSON columns with fields JSON column
- Custom MarshalJSON() implementation for flat output - Custom MarshalJSON() implementation for flat output
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.3-1
- Fix: Added missing ClickHouse driver dependency
- Fix: Fixed race condition in orchestrator
- Security: Added explicit source_type configuration for Unix socket sources
- Added: Comprehensive test suite improvements
- Added: Test coverage improved from 50.6% to 62.0%
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.2-1 * Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.2-1
- Fix: durcir la validation et fiabiliser flush/arrêt idempotents - Added: Initial RPM packaging support for Rocky Linux 8/9 and AlmaLinux 10
- Refactor: remove Debian/DEB packaging, RPM-only support - Added: Docker multi-stage build pipeline
- Feat: add multi-distro RPM packaging for CentOS 7 and Rocky Linux 8/9/10 - Added: Hexagonal architecture implementation
- Feat: migrate configuration from custom format to YAML - Added: Unix socket input sources (JSON line protocol)
- Refactor: remove obsolete config and update documentation - Added: File output sink (JSON lines)
- Added: ClickHouse output sink with batching and retry logic
- Added: Time-window based correlation on src_ip + src_port
- Added: Graceful shutdown with signal handling (SIGINT, SIGTERM)
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.1-1 * Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.1-1
- Fix: durcir la validation et fiabiliser flush/arrêt idempotents
- Refactor: remove Debian/DEB packaging, RPM-only support
- Feat: add multi-distro RPM packaging for CentOS 7 and Rocky Linux 8/9/10
- Feat: migrate configuration from custom format to YAML
- Refactor: remove obsolete config and update documentation
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.0-1
- Initial package for CentOS 7, Rocky Linux 8, 9, 10 - Initial package for CentOS 7, Rocky Linux 8, 9, 10

13
packaging/rpm/logrotate Normal file
View File

@ -0,0 +1,13 @@
/var/log/logcorrelator/correlated.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
create 0640 logcorrelator logcorrelator
sharedscripts
postrotate
/bin/systemctl reload logcorrelator > /dev/null 2>&1 || true
endscript
}

View File

@ -41,6 +41,11 @@ if [ ! -f /etc/logcorrelator/logcorrelator.yml ]; then
chmod 640 /etc/logcorrelator/logcorrelator.yml chmod 640 /etc/logcorrelator/logcorrelator.yml
fi fi
# Set permissions for logrotate config
if [ -f /etc/logrotate.d/logcorrelator ]; then
chmod 644 /etc/logrotate.d/logcorrelator
fi
# Reload systemd # Reload systemd
if [ -x /bin/systemctl ]; then if [ -x /bin/systemctl ]; then
systemctl daemon-reload systemctl daemon-reload