code_public/logcorrelator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
59 Commits 1 Branch 0 Tags
97862bb1dc0977b1e02e349d40e58caee70b104a
Commit Graph

19 Commits

Author SHA1 Message Date
toto
24f2d8a3c4 fix(rpm): preserve config on upgrade, set correct ownership/permissions 
RPM packaging improvements:
- Fix %config(noreplace) directive in spec file (logcorrelator.yml)
- Fix post script: use correct path for .yml.example (/etc/logcorrelator/)
- Set /var/run/logcorrelator ownership to logcorrelator:logcorrelator
- Set correct permissions: /var/run (755), /var/log (750), /var/lib (750)
- Add %config(noreplace) for logrotate.d/logcorrelator
- Add comprehensive RPM test script (packaging/test/test-rpm.sh)

Documentation updates:
- Update architecture.yml with filesystem permissions section
- Document socket ownership (logcorrelator:logcorrelator, 0666)
- Document config file policy (%config(noreplace) behavior)
- Add systemd hardening directives (NoNewPrivileges, ProtectSystem)
- Update ClickHouse schema: mark non-implemented fields
- Remove materialized view SQL (managed externally)
- Add stdout sink module documentation

Build pipeline:
- Update Dockerfile.package with comments for config policy
- Add /var/lib/logcorrelator directory creation
- Document fpm %config(noreplace) limitations

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 21:30:27 +00:00
toto
9db6848757 fix: critical Keep-Alive correlation bug - network events evicted prematurely
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Fix cleanExpired() to use TTL map instead of event timestamp for B events
- Increase default correlation time window from 1s to 10s
- Increase default network TTL from 30s to 120s for long sessions
- Use payload timestamp for network events when available (fallback to now)
- Add comprehensive Keep-Alive tests (TTL reset, long session scenarios)
- Bump version to 1.1.7

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 16:32:48 +01:00
toto
eed376d749 docs: update ClickHouse schema with new table structure
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- http_logs_raw: partition by toDate(ingest_time), order by ingest_time
- http_logs: explicit columns (no DEFAULT), extracted by MV
- mv_http_logs: full SELECT with JSONExtract* + coalesce for all fields
- Add 17 HTTP header fields (User-Agent, Accept, Sec-CH-UA, etc.)
- New ORDER BY: (time, src_ip, dst_ip, ja4)
- architecture.yml: match new schema with MV query details

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 14:38:45 +01:00
toto
60cd8d87e4 docs: update ClickHouse schema with TTL, MV and users
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- README.md: add complete DDL with mabase_prod database
- Add TTL (1 day) on http_logs_raw table
- Add materialized view mv_http_logs for automatic data transfer
- Document users (data_writer, analyst) and grants
- Add migration script for existing data
- architecture.yml: add database, TTL settings, MV, users sections

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 13:39:47 +01:00
toto
58b23ccc1e docs: update ClickHouse schema (http_logs_raw + http_logs)
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- README.md: documenter les deux tables (raw + enrichie)
- architecture.yml: décrire le schema complet avec colonnes matérialisées
- Table http_logs_raw: ingestion JSON brut (colonne raw_json unique)
- Table http_logs: extraction des champs via DEFAULT JSONExtract*

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 11:53:13 +01:00
Jacquin Antoine
c352e06b88 fix: create socket parent directory on startup
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Create /var/run/logcorrelator/ if missing before binding sockets
- Fixes issue with tmpfs /var/run being cleared on reboot
- Add filepath import for directory handling

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 00:17:01 +01:00
Jacquin Antoine
eb3cc78170 docs: rename apache.socket to http.socket in examples
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- README.md: use http.socket instead of apache.sock
- architecture.yml: use http.socket instead of apache.sock

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-02 22:35:42 +01:00
Jacquin Antoine
35cb53f0a5 docs: add version consistency requirement in architecture.yml
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Document that version numbers must be consistent across:
- RPM spec file
- Makefile (PKG_VERSION)
- CHANGELOG.md
- git tags

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-02 21:53:21 +01:00
Jacquin Antoine
33e19b4f52 feat: Keep-Alive correlation, TTL management, SIGHUP handling, logrotate support
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Major features:
- One-to-many correlation mode (Keep-Alive) for HTTP connections
- Dynamic TTL for network events with reset on each correlation
- Separate configurable buffer sizes for HTTP and network events
- SIGHUP signal handling for log rotation without service restart
- FileSink.Reopen() method for log file rotation
- logrotate configuration included in RPM
- ExecReload added to systemd service

Configuration changes:
- New YAML structure with nested sections (time_window, orphan_policy, matching, buffers, ttl)
- Backward compatibility maintained for deprecated fields

Packaging:
- RPM version 1.1.0 with logrotate config
- Updated spec file and changelog
- All distributions: el8, el9, el10

Tests:
- New tests for Keep-Alive mode and TTL reset
- Updated mocks with Reopen() interface method

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-02 20:32:59 +01:00
Jacquin Antoine
d3436f6245 1.0.8
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details
2026-03-01 11:30:55 +01:00
Jacquin Antoine
56c2923121 chore: version 1.0.6 - simplify YAML configuration
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Remove service.name and service.language (unused)
- Remove enabled flags on outputs (presence = enabled)
- Simplify correlation config: time_window_s (integer) instead of nested object
- Simplify orphan_policy to emit_orphans boolean
- Rename apache socket to http.socket
- Add socket_permissions option for unix sockets (default: 0660)
- Update tests for new configuration format

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-01 01:59:59 +01:00
Jacquin Antoine
b25dc24ed9 docs: update architecture.yml after code cleanup 
- Remove TimeProvider from internal/ports responsibilities (now only in domain)
- Fix configuration format references: TOML → YAML

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-01 00:28:23 +01:00
Jacquin Antoine
52eebafb0b docs: update README and architecture for v1.0.3 flat JSON structure 
- docs: update README.md with flat JSON output example
- docs: update architecture.yml correlated_log schema
- docs: update ClickHouse schema to use fields JSON column
- breaking: document migration from apache/network to fields

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-28 22:28:45 +01:00
Jacquin Antoine
5f97af3627 docs: update architecture.yml for AlmaLinux 10 support 
- Remove CentOS 7 (EOL, inaccessible repositories)
- Remove Rocky Linux 10 (not available, use AlmaLinux 10 instead)
- Add AlmaLinux 10 to supported distributions
- Update packaging stages and verification commands

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-28 20:59:27 +01:00
Jacquin Antoine
81849b16d8 refactor: remove Debian/DEB packaging, RPM-only support 
- Remove all DEB packaging files (packaging/deb/*)
- Update Dockerfile.package to build RPM packages only
- Update build.sh to remove DEB build steps
- Update architecture.yml to reflect RPM-only distribution
- Update README.md installation instructions for RPM only
- Change runtime image from distroless/base-debian12 to scratch
- Clean up obsolete build artifacts in dist/

Supported distributions now:
- CentOS 7 (el7)
- Rocky Linux 8 (el8)
- Rocky Linux 9 (el9)
- Rocky Linux 10 (el10)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-28 19:55:24 +01:00
Jacquin Antoine
7a9d92a469 feat: add multi-distro RPM packaging for CentOS 7 and Rocky Linux 8/9/10 
- Create RPM maintainer scripts (post, preun, postun)
- Add Docker build stages for each target distribution (el7, el8, el9, el10)
- Update architecture.yml with supported RPM distributions
- Update build.sh to extract distro-specific RPM packages

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-28 17:17:56 +01:00
Jacquin Antoine
37f9c21672 feat: migrate configuration from custom format to YAML 
- Replace custom directive-based config parser with YAML using gopkg.in/yaml.v3
- Rename config.example.conf to config.example.yml with YAML syntax
- Update default config path to /etc/logcorrelator/logcorrelator.yml
- Update Dockerfile.package to copy YAML config files
- Update packaging scripts to install logcorrelator.yml
- Update architecture.yml to document YAML configuration
- Add yaml.v3 dependency to go.mod

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-27 15:51:25 +01:00
Jacquin Antoine
0d84a1284f docs: update architecture.yml with unified packaging section 
- Add packaging section describing DEB and RPM builds with fpm
- Document Dockerfile.package multi-stage build pipeline
- List files, directories, maintainer scripts, and dependencies
- Add verification commands for both package types

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-27 15:41:29 +01:00
Jacquin Antoine
8fc14c1e94 Initial commit: logcorrelator with unified packaging (DEB + RPM using fpm) 
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-27 15:31:46 +01:00