code_public/logcorrelator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
83 Commits 1 Branch 0 Tags
20ebe7240e595b7b19a59a5d52b3dc991bc2a563
Commit Graph

6 Commits

Author SHA1 Message Date
toto
20ebe7240e Feat: Détection menaces HTTP via vues ClickHouse + simplification shutdown
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Nouvelles vues de détection (sql/views.sql) :
- Identification hosts par IP/JA4 (view_host_identification, view_host_ja4_anomalies)
- Détection brute force POST et query params variables
- Header fingerprinting (ordre, headers modernes manquants, Sec-CH-UA)
- ALPN mismatch detection (h2 déclaré mais HTTP/1.1 parlé)
- Rate limiting & burst detection (50 req/min, 20 req/10s)
- Path enumeration/scanning (paths sensibles)
- Payload attacks (SQLi, XSS, path traversal)
- JA4 botnet detection (même fingerprint sur 20+ IPs)
- Correlation quality (orphan ratio >80%)

ClickHouse (sql/init.sql) :
- Compression ZSTD(3) sur champs texte (path, query, headers, ja3/ja4)
- TTL automatique : 1 jour (raw) + 7 jours (http_logs)
- Paramètre ttl_only_drop_parts = 1

Shutdown simplifié (internal/app/orchestrator.go) :
- Suppression ShutdownTimeout et logique de flush/attente
- Stop() = cancel() + Close() uniquement
- systemd TimeoutStopSec gère l'arrêt forcé si besoin

File output toggle (internal/config/*.go) :
- Ajout champ Enabled dans FileOutputConfig
- Le sink fichier n'est créé que si enabled && path != ''
- Tests : TestValidate_FileOutputDisabled, TestLoadConfig_FileOutputDisabled

RPM packaging (packaging/rpm/logcorrelator.spec) :
- Changelog 1.1.18 → 1.1.22
- Suppression logcorrelator-tmpfiles.conf (redondant RuntimeDirectory=)

Nettoyage :
- idees.txt → idees/ (dossier)
- Suppression 91.224.92.185.txt (logs exemple)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-11 18:28:07 +01:00
toto
0fca6e4e93 fix(correlation): Keep-Alive time window + orphan timer + TTL purge (v1.1.14)
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Bug #1 - processSourceA: utilise bEventHasValidTTL en mode one_to_many
  au lieu de eventsMatch qui comparait les timestamps originaux. Apres ~10s
  les requetes A devenaient toutes orphelines alors que la session KA etait active.

Bug #4 - checkPendingOrphansForCorrelation: meme correction, cle identique
  = meme connexion en one_to_many, pas besoin de comparer les timestamps.

Bug #3 - cleanNetworkBufferByTTL: expiration B => emission immediate
  des pending orphans associes (ils ne peuvent plus jamais corréler).

Bug #2 - Orchestrateur: goroutine ticker 250ms appelle EmitPendingOrphans()
  pour drainer les orphans independamment du flux d'evenements entrants.
  EmitPendingOrphans() expose la methode comme publique thread-safe.

Tests: 4 nouveaux tests de non-regression (un par bug).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-05 17:01:37 +01:00
Jacquin Antoine
c352e06b88 fix: create socket parent directory on startup
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
- Create /var/run/logcorrelator/ if missing before binding sockets
- Fixes issue with tmpfs /var/run being cleared on reboot
- Add filepath import for directory handling

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-03 00:17:01 +01:00
Jacquin Antoine
33e19b4f52 feat: Keep-Alive correlation, TTL management, SIGHUP handling, logrotate support
Some checks failed
Build and Test / test (push) Has been cancelled
Details
Build and Test / build (push) Has been cancelled
Details
Build and Test / docker (push) Has been cancelled
Details 
Major features:
- One-to-many correlation mode (Keep-Alive) for HTTP connections
- Dynamic TTL for network events with reset on each correlation
- Separate configurable buffer sizes for HTTP and network events
- SIGHUP signal handling for log rotation without service restart
- FileSink.Reopen() method for log file rotation
- logrotate configuration included in RPM
- ExecReload added to systemd service

Configuration changes:
- New YAML structure with nested sections (time_window, orphan_policy, matching, buffers, ttl)
- Backward compatibility maintained for deprecated fields

Packaging:
- RPM version 1.1.0 with logrotate config
- Updated spec file and changelog
- All distributions: el8, el9, el10

Tests:
- New tests for Keep-Alive mode and TTL reset
- Updated mocks with Reopen() interface method

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-03-02 20:32:59 +01:00
Jacquin Antoine
180c57c35b chore: release v1.0.2 with critical fixes and test improvements 
- fix: add missing ClickHouse driver dependency
- fix: resolve race condition in orchestrator (single goroutine per source)
- feat: add explicit source_type config for Unix socket sources
- test: improve coverage from 50.6% to 62.0%
- docs: add CHANGELOG.md with release notes
- build: update version to 1.0.2 in build scripts and Dockerfiles

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-28 21:45:00 +01:00
Jacquin Antoine
8fc14c1e94 Initial commit: logcorrelator with unified packaging (DEB + RPM using fpm) 
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
 2026-02-27 15:31:46 +01:00