code_public/logcorrelator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
68 Commits 1 Branch 0 Tags
e9dcd8ea512b441da69a8eefb9dbc386467d4b9a
Commit Graph

12 Commits

Author SHA1 Message Date
toto
e9dcd8ea51 feat: observability, IP filtering, stdout/clickhouse fixes (v1.1.11) 
- feat(observability): metrics server with /metrics and /health endpoints
- feat(observability): correlation metrics (events, success/failed, reasons, buffers)
- feat(correlation): IP exclusion filter (exact IPs and CIDR ranges)
- feat(correlation): pending orphan delay for late-arriving B events
- fix(stdout): sink is now a no-op for data; JSON must never appear on stdout
- fix(clickhouse): all flush errors were silently discarded, now properly logged
- fix(clickhouse): buffer overflow with DropOnOverflow now logged at WARN
- fix(clickhouse): retry attempts logged at WARN with attempt/delay/error context
- feat(clickhouse): connection success logged at INFO, batch sends at DEBUG
- feat(clickhouse): SetLogger() for external logger injection
- test(stdout): assert stdout remains empty for correlated and orphan logs
- chore(rpm): bump version to 1.1.11, update changelog
- docs: README and architecture.yml updated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-05 11:40:54 +01:00
toto
e0c622f635 feat(correlation): add configurable delay before emitting orphan A events 
New feature: Apache events can now wait for B events before being emitted as orphans.

Changes:
- Add ApacheEmitDelayMs config (default: 500ms)
- Add pendingOrphans structure for delayed emission
- processSourceA(): add A to pending orphans instead of immediate emission
- processSourceB(): check pending orphans before buffer A
- emitPendingOrphans(): emit orphans after delay expires
- Flush(): emit all pending orphans immediately (shutdown)

Configuration:
correlation:
  orphan_policy:
    apache_always_emit: true
    apache_emit_delay_ms: 500  # Wait 500ms before emitting as orphan

Backward compatibility:
- apache_emit_delay_ms: 0 → immediate emission (legacy mode)
- apache_emit_delay_ms < 0 → default 500ms

Tests added (5 new tests):
- TestCorrelationService_ApacheEmitDelay_BArrivesDuringDelay
- TestCorrelationService_ApacheEmitDelay_NoBArrives
- TestCorrelationService_ApacheEmitDelay_ZeroDelay
- TestCorrelationService_ApacheEmitDelay_MultipleA_SameKey
- TestCorrelationService_ApacheEmitDelay_Flush

All 30 tests pass. Coverage: 75.1%

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 22:03:31 +00:00
Jacquin Antoine
c352e06b88 fix: create socket parent directory on startup
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Create /var/run/logcorrelator/ if missing before binding sockets
- Fixes issue with tmpfs /var/run being cleared on reboot
- Add filepath import for directory handling

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 00:17:01 +01:00
Jacquin Antoine
324b0042f8 fix(rpm): example config in /etc/logcorrelator + socket permissions 0666
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Install logcorrelator.yml.example to /etc/logcorrelator/ instead of /usr/share/logcorrelator/
- Change default socket permissions from 0660 to 0666 (world read/write)
- Bump version to 1.1.2
- Remove CHANGELOG.md

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-02 22:07:50 +01:00
Jacquin Antoine
33e19b4f52 feat: Keep-Alive correlation, TTL management, SIGHUP handling, logrotate support
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Major features:
- One-to-many correlation mode (Keep-Alive) for HTTP connections
- Dynamic TTL for network events with reset on each correlation
- Separate configurable buffer sizes for HTTP and network events
- SIGHUP signal handling for log rotation without service restart
- FileSink.Reopen() method for log file rotation
- logrotate configuration included in RPM
- ExecReload added to systemd service

Configuration changes:
- New YAML structure with nested sections (time_window, orphan_policy, matching, buffers, ttl)
- Backward compatibility maintained for deprecated fields

Packaging:
- RPM version 1.1.0 with logrotate config
- Updated spec file and changelog
- All distributions: el8, el9, el10

Tests:
- New tests for Keep-Alive mode and TTL reset
- Updated mocks with Reopen() interface method

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-02 20:32:59 +01:00
Jacquin Antoine
27c7659397 fix: renforcer corrélation A/B et sorties stdout/fichier
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Co-authored-by: aider (openrouter/openai/gpt-5.3-codex) <aider@aider.chat>
 2026-03-01 12:10:17 +01:00
Jacquin Antoine
a3ae5421cf chore: version 1.0.7 - add log levels
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Add configurable log levels: DEBUG, INFO, WARN, ERROR
- Replace debug.enabled with log.level in configuration
- Add Warn/Warnf methods for warning messages
- Log orphan events and buffer overflow as WARN
- Log parse errors as WARN
- Log raw events and correlations as DEBUG

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-01 02:33:04 +01:00
Jacquin Antoine
56c2923121 chore: version 1.0.6 - simplify YAML configuration
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Remove service.name and service.language (unused)
- Remove enabled flags on outputs (presence = enabled)
- Simplify correlation config: time_window_s (integer) instead of nested object
- Simplify orphan_policy to emit_orphans boolean
- Rename apache socket to http.socket
- Add socket_permissions option for unix sockets (default: 0660)
- Update tests for new configuration format

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-01 01:59:59 +01:00
Jacquin Antoine
180c57c35b chore: release v1.0.2 with critical fixes and test improvements 
- fix: add missing ClickHouse driver dependency
- fix: resolve race condition in orchestrator (single goroutine per source)
- feat: add explicit source_type config for Unix socket sources
- test: improve coverage from 50.6% to 62.0%
- docs: add CHANGELOG.md with release notes
- build: update version to 1.0.2 in build scripts and Dockerfiles

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-28 21:45:00 +01:00
Jacquin Antoine
7e9535122e fix: durcir la validation et fiabiliser flush/arrêt idempotents 
Co-authored-by: aider (openrouter/openai/gpt-5.3-codex) <aider@aider.chat>
 2026-02-28 20:10:28 +01:00
Jacquin Antoine
37f9c21672 feat: migrate configuration from custom format to YAML 
- Replace custom directive-based config parser with YAML using gopkg.in/yaml.v3
- Rename config.example.conf to config.example.yml with YAML syntax
- Update default config path to /etc/logcorrelator/logcorrelator.yml
- Update Dockerfile.package to copy YAML config files
- Update packaging scripts to install logcorrelator.yml
- Update architecture.yml to document YAML configuration
- Add yaml.v3 dependency to go.mod

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-27 15:51:25 +01:00
Jacquin Antoine
8fc14c1e94 Initial commit: logcorrelator with unified packaging (DEB + RPM using fpm) 
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-27 15:31:46 +01:00