33e19b4f52
Major features: - One-to-many correlation mode (Keep-Alive) for HTTP connections - Dynamic TTL for network events with reset on each correlation - Separate configurable buffer sizes for HTTP and network events - SIGHUP signal handling for log rotation without service restart - FileSink.Reopen() method for log file rotation - logrotate configuration included in RPM - ExecReload added to systemd service Configuration changes: - New YAML structure with nested sections (time_window, orphan_policy, matching, buffers, ttl) - Backward compatibility maintained for deprecated fields Packaging: - RPM version 1.1.0 with logrotate config - Updated spec file and changelog - All distributions: el8, el9, el10 Tests: - New tests for Keep-Alive mode and TTL reset - Updated mocks with Reopen() interface method Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
177 lines
6.1 KiB
Markdown
177 lines
6.1 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to logcorrelator are documented in this file.
|
|
|
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
|
## [1.1.0] - 2026-03-02
|
|
|
|
### Added
|
|
|
|
- **Keep-Alive support**: One-to-many correlation mode allows a single network event (B) to correlate with multiple HTTP events (A)
|
|
- **Dynamic TTL**: Network events (source B) now have configurable TTL that resets on each successful correlation
|
|
- **Separate buffer sizes**: Configurable `max_http_items` and `max_network_items` for independent buffer control
|
|
- **SIGHUP handling**: Service now handles SIGHUP signal for log rotation without restart
|
|
- **logrotate configuration**: RPM includes `/etc/logrotate.d/logcorrelator` for automatic log rotation
|
|
- **ExecReload**: Systemd service now supports `systemctl reload logcorrelator`
|
|
|
|
### Changed
|
|
|
|
- **Configuration structure**: New YAML structure with nested sections:
|
|
- `time_window` (object with `value` and `unit`)
|
|
- `orphan_policy` (object with `apache_always_emit` and `network_emit`)
|
|
- `matching.mode` (string: `one_to_one` or `one_to_many`)
|
|
- `buffers` (object with `max_http_items` and `max_network_items`)
|
|
- `ttl` (object with `network_ttl_s`)
|
|
- Backward compatibility maintained for old config fields (`time_window_s`, `emit_orphans`)
|
|
|
|
### Technical Details
|
|
|
|
- `CorrelationService` now supports `MatchingMode` configuration
|
|
- Network events tracked with individual TTL expiration times
|
|
- `FileSink.Reopen()` method for log file rotation
|
|
- All sinks implement `Reopen()` interface method
|
|
|
|
---
|
|
|
|
## [1.0.7] - 2026-03-01
|
|
|
|
### Added
|
|
|
|
- Log levels: DEBUG, INFO, WARN, ERROR configurable via `log.level`
|
|
- `Warn` and `Warnf` methods for warning messages
|
|
- Debug logs for events received from sockets and correlations
|
|
- Warning logs for orphan events and buffer overflow
|
|
|
|
### Changed
|
|
|
|
- Configuration: `debug.enabled` replaced by `log.level` (DEBUG/INFO/WARN/ERROR)
|
|
- Orphan events and buffer overflow now logged as WARN instead of DEBUG
|
|
- Parse errors logged as WARN
|
|
|
|
---
|
|
|
|
## [1.0.6] - 2026-03-01
|
|
|
|
### Changed
|
|
|
|
- Configuration YAML simplified: removed `service.name`, `service.language`, `enabled` flags
|
|
- Correlation config simplified: `time_window_s` (integer) instead of nested `time_window` object
|
|
- Orphan policy simplified: `emit_orphans` boolean instead of `orphan_policy` object
|
|
- Apache socket renamed to `http.socket`
|
|
|
|
### Added
|
|
|
|
- `socket_permissions` option on unix sockets to configure file permissions (default: `0660`)
|
|
|
|
---
|
|
|
|
## [1.0.4] - 2026-03-01
|
|
|
|
### Added
|
|
|
|
- Systemd service auto-start after RPM installation
|
|
- Systemd service hardening (TimeoutStartSec, TimeoutStopSec, ReadWritePaths)
|
|
|
|
### Fixed
|
|
|
|
- Systemd service unit: correct config path (.yml instead of .conf)
|
|
- CI workflow: branch name main → master
|
|
- Go module dependencies cleanup (go mod tidy)
|
|
|
|
### Changed
|
|
|
|
- RPM packaging: generic el8/el9/el10 directory naming (instead of rocky/almalinux)
|
|
- Code cleanup: removed unused CorrelationKeyFull() alias
|
|
- Code cleanup: removed duplicate TimeProvider interface from ports package
|
|
|
|
---
|
|
|
|
## [1.0.3] - 2026-02-28
|
|
|
|
### Changed
|
|
|
|
- **Breaking**: Flattened JSON output structure - removed `apache` and `network` subdivisions
|
|
- All log fields are now merged into a single-level JSON structure for easier parsing
|
|
- ClickHouse schema updated: replaced `apache JSON` and `network JSON` columns with single `fields JSON` column
|
|
|
|
### Technical Details
|
|
|
|
- Custom `MarshalJSON()` implementation flattens all fields at the root level
|
|
- Backward compatibility: existing ClickHouse tables need schema migration to use `fields JSON` column
|
|
|
|
---
|
|
|
|
## [1.0.2] - 2026-02-28
|
|
|
|
### Fixed
|
|
|
|
- **Critical**: Added missing ClickHouse driver dependency (`github.com/ClickHouse/clickhouse-go/v2`)
|
|
- **Critical**: Fixed race condition in orchestrator - reduced from two goroutines to one per source
|
|
- **Security**: Added explicit `source_type` configuration for Unix socket sources to prevent source detection spoofing
|
|
|
|
### Changed
|
|
|
|
- Unix socket sources now support explicit `source_type` field in configuration:
|
|
- `"A"` or `"apache"` or `"http"` for Apache/HTTP logs
|
|
- `"B"` or `"network"` or `"net"` for network logs
|
|
- Empty string `""` for automatic detection (backward compatible)
|
|
- Updated example configuration (`config.example.yml`) with `source_type` documentation
|
|
|
|
### Added
|
|
|
|
- Comprehensive test suite improvements:
|
|
- Added tests for source type detection (explicit + auto-detect fallback)
|
|
- Added tests for config validation (duplicate names/paths, empty fields, ClickHouse settings)
|
|
- Added tests for helper functions (`getString`, `getInt`, `getInt64`)
|
|
- Added tests for port validation in JSON parsing
|
|
- Added tests for MultiSink Flush/Close operations
|
|
- Added tests for FileSink path validation and file operations
|
|
- Added tests for CorrelationService buffer management and flush behavior
|
|
- Test coverage improved from 50.6% to 62.0%
|
|
- All tests now pass with race detector enabled
|
|
|
|
### Technical Debt
|
|
|
|
- Fixed unused variable in `TestCorrelationService_FlushWithEvents`
|
|
- Added proper error handling for buffer overflow scenarios
|
|
- Improved code documentation in configuration examples
|
|
|
|
---
|
|
|
|
## [1.0.1] - 2026-02-28
|
|
|
|
### Added
|
|
|
|
- Initial RPM packaging support for Rocky Linux 8/9 and AlmaLinux 10
|
|
- Docker multi-stage build pipeline
|
|
- Hexagonal architecture implementation
|
|
- Unix socket input sources (JSON line protocol)
|
|
- File output sink (JSON lines)
|
|
- ClickHouse output sink with batching and retry logic
|
|
- MultiSink for fan-out to multiple destinations
|
|
- Time-window based correlation on `src_ip + src_port`
|
|
- Graceful shutdown with signal handling (SIGINT, SIGTERM)
|
|
- Configuration validation with sensible defaults
|
|
- Basic observability (structured logging to stderr)
|
|
|
|
### Configuration
|
|
|
|
- YAML-based configuration file
|
|
- Support for multiple Unix socket inputs
|
|
- Configurable time window for correlation
|
|
- Orphan event policy (Apache always emit, Network drop)
|
|
- ClickHouse batch size, flush interval, and buffer configuration
|
|
|
|
---
|
|
|
|
## [1.0.0] - 2026-02-27
|
|
|
|
### Added
|
|
|
|
- Initial release
|
|
- Core correlation engine
|
|
- Basic HTTP and network log parsing
|
|
- File-based output
|