logcorrelator/CHANGELOG.md

Changelog

All notable changes to logcorrelator are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[1.1.0] - 2026-03-02

Added

• Keep-Alive support: One-to-many correlation mode allows a single network event (B) to correlate with multiple HTTP events (A)
• Dynamic TTL: Network events (source B) now have configurable TTL that resets on each successful correlation
• Separate buffer sizes: Configurable max_http_items and max_network_items for independent buffer control
• SIGHUP handling: Service now handles SIGHUP signal for log rotation without restart
• logrotate configuration: RPM includes /etc/logrotate.d/logcorrelator for automatic log rotation
• ExecReload: Systemd service now supports systemctl reload logcorrelator

Changed

• Configuration structure: New YAML structure with nested sections:
  • time_window (object with value and unit)
  • orphan_policy (object with apache_always_emit and network_emit)
  • matching.mode (string: one_to_one or one_to_many)
  • buffers (object with max_http_items and max_network_items)
  • ttl (object with network_ttl_s)
• Backward compatibility maintained for old config fields (time_window_s, emit_orphans)

Technical Details

• CorrelationService now supports MatchingMode configuration
• Network events tracked with individual TTL expiration times
• FileSink.Reopen() method for log file rotation
• All sinks implement Reopen() interface method

---

[1.0.7] - 2026-03-01

Added

• Log levels: DEBUG, INFO, WARN, ERROR configurable via log.level
• Warn and Warnf methods for warning messages
• Debug logs for events received from sockets and correlations
• Warning logs for orphan events and buffer overflow

Changed

• Configuration: debug.enabled replaced by log.level (DEBUG/INFO/WARN/ERROR)
• Orphan events and buffer overflow now logged as WARN instead of DEBUG
• Parse errors logged as WARN

---

[1.0.6] - 2026-03-01

Changed

• Configuration YAML simplified: removed service.name, service.language, enabled flags
• Correlation config simplified: time_window_s (integer) instead of nested time_window object
• Orphan policy simplified: emit_orphans boolean instead of orphan_policy object
• Apache socket renamed to http.socket

Added

• socket_permissions option on unix sockets to configure file permissions (default: 0660)

---

[1.0.4] - 2026-03-01

Added

• Systemd service auto-start after RPM installation
• Systemd service hardening (TimeoutStartSec, TimeoutStopSec, ReadWritePaths)

Fixed

• Systemd service unit: correct config path (.yml instead of .conf)
• CI workflow: branch name main → master
• Go module dependencies cleanup (go mod tidy)

Changed

• RPM packaging: generic el8/el9/el10 directory naming (instead of rocky/almalinux)
• Code cleanup: removed unused CorrelationKeyFull() alias
• Code cleanup: removed duplicate TimeProvider interface from ports package

---

[1.0.3] - 2026-02-28

Changed

• Breaking: Flattened JSON output structure - removed apache and network subdivisions
• All log fields are now merged into a single-level JSON structure for easier parsing
• ClickHouse schema updated: replaced apache JSON and network JSON columns with single fields JSON column

Technical Details

• Custom MarshalJSON() implementation flattens all fields at the root level
• Backward compatibility: existing ClickHouse tables need schema migration to use fields JSON column

---

[1.0.2] - 2026-02-28

Fixed

• Critical: Added missing ClickHouse driver dependency (github.com/ClickHouse/clickhouse-go/v2)
• Critical: Fixed race condition in orchestrator - reduced from two goroutines to one per source
• Security: Added explicit source_type configuration for Unix socket sources to prevent source detection spoofing

Changed

• Unix socket sources now support explicit source_type field in configuration:
  • "A" or "apache" or "http" for Apache/HTTP logs
  • "B" or "network" or "net" for network logs
  • Empty string "" for automatic detection (backward compatible)
• Updated example configuration (config.example.yml) with source_type documentation

Added

• Comprehensive test suite improvements:
  • Added tests for source type detection (explicit + auto-detect fallback)
  • Added tests for config validation (duplicate names/paths, empty fields, ClickHouse settings)
  • Added tests for helper functions (getString, getInt, getInt64)
  • Added tests for port validation in JSON parsing
  • Added tests for MultiSink Flush/Close operations
  • Added tests for FileSink path validation and file operations
  • Added tests for CorrelationService buffer management and flush behavior
• Test coverage improved from 50.6% to 62.0%
• All tests now pass with race detector enabled

Technical Debt

• Fixed unused variable in TestCorrelationService_FlushWithEvents
• Added proper error handling for buffer overflow scenarios
• Improved code documentation in configuration examples

---

[1.0.1] - 2026-02-28

Added

• Initial RPM packaging support for Rocky Linux 8/9 and AlmaLinux 10
• Docker multi-stage build pipeline
• Hexagonal architecture implementation
• Unix socket input sources (JSON line protocol)
• File output sink (JSON lines)
• ClickHouse output sink with batching and retry logic
• MultiSink for fan-out to multiple destinations
• Time-window based correlation on src_ip + src_port
• Graceful shutdown with signal handling (SIGINT, SIGTERM)
• Configuration validation with sensible defaults
• Basic observability (structured logging to stderr)

Configuration

• YAML-based configuration file
• Support for multiple Unix socket inputs
• Configurable time window for correlation
• Orphan event policy (Apache always emit, Network drop)
• ClickHouse batch size, flush interval, and buffer configuration

---

[1.0.0] - 2026-02-27

Added

• Initial release
• Core correlation engine
• Basic HTTP and network log parsing
• File-based output