feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
383
services/correlator/packaging/rpm/logcorrelator.spec
Normal file
383
services/correlator/packaging/rpm/logcorrelator.spec
Normal file
@ -0,0 +1,383 @@
|
||||
# logcorrelator RPM spec file
|
||||
# Compatible with CentOS 7, Rocky Linux 8, 9, 10
|
||||
# Built with rpmbuild (not FPM)
|
||||
|
||||
Name: logcorrelator
|
||||
Version: %{version}
|
||||
Release: 1%{?dist}
|
||||
Summary: Log correlation service for HTTP and network events
|
||||
|
||||
License: MIT
|
||||
URL: https://github.com/logcorrelator/logcorrelator
|
||||
Vendor: logcorrelator <dev@example.com>
|
||||
Packager: logcorrelator <dev@example.com>
|
||||
|
||||
BuildArch: x86_64
|
||||
|
||||
# Dependencies
|
||||
Requires: systemd
|
||||
Requires(post): systemd
|
||||
Requires(preun): systemd
|
||||
Requires(postun): systemd
|
||||
|
||||
%description
|
||||
logcorrelator est un service système écrit en Go qui reçoit deux flux de logs JSON
|
||||
via des sockets Unix, corrèle les événements HTTP applicatifs avec des événements
|
||||
réseau, et produit des logs corrélés en temps réel vers ClickHouse et/ou fichier local.
|
||||
|
||||
Notes de sécurité :
|
||||
- Le service s'exécute sous l'utilisateur logcorrelator (non-root)
|
||||
- Les sockets Unix sont créés avec des permissions 0666 (world read/write)
|
||||
- Les répertoires critiques sont protégés : /var/log (750), /var/lib (750), /etc (750)
|
||||
- /var/run/logcorrelator est en 755 pour permettre la création de sockets
|
||||
|
||||
%prep
|
||||
# Files are already in BUILD directory (copied by build-rpm.sh)
|
||||
# No extraction needed
|
||||
echo "Files available in BUILD directory:"
|
||||
ls -la %{_builddir}/
|
||||
|
||||
%install
|
||||
# Create directory structure in buildroot
|
||||
mkdir -p %{buildroot}/usr/bin
|
||||
mkdir -p %{buildroot}/etc/logcorrelator
|
||||
mkdir -p %{buildroot}/var/log/logcorrelator
|
||||
mkdir -p %{buildroot}/var/run/logcorrelator
|
||||
mkdir -p %{buildroot}/var/lib/logcorrelator
|
||||
mkdir -p %{buildroot}/etc/systemd/system
|
||||
mkdir -p %{buildroot}/etc/logrotate.d
|
||||
mkdir -p %{buildroot}/usr/lib/tmpfiles.d
|
||||
|
||||
# Install binary (from BUILD directory)
|
||||
install -m 0755 %{_builddir}/usr/bin/logcorrelator %{buildroot}/usr/bin/logcorrelator
|
||||
|
||||
# Install config files
|
||||
install -m 0640 %{_builddir}/etc/logcorrelator/logcorrelator.yml %{buildroot}/etc/logcorrelator/logcorrelator.yml
|
||||
install -m 0640 %{_builddir}/etc/logcorrelator/logcorrelator.yml.example %{buildroot}/etc/logcorrelator/logcorrelator.yml.example
|
||||
|
||||
# Install systemd service
|
||||
install -m 0644 %{_builddir}/etc/systemd/system/logcorrelator.service %{buildroot}/etc/systemd/system/logcorrelator.service
|
||||
|
||||
# Install logrotate config
|
||||
install -m 0644 %{_builddir}/etc/logrotate.d/logcorrelator %{buildroot}/etc/logrotate.d/logcorrelator
|
||||
|
||||
%post
|
||||
# Create logcorrelator user and group
|
||||
if ! getent group logcorrelator >/dev/null 2>&1; then
|
||||
groupadd --system logcorrelator
|
||||
fi
|
||||
|
||||
if ! getent passwd logcorrelator >/dev/null 2>&1; then
|
||||
useradd --system \
|
||||
--gid logcorrelator \
|
||||
--home-dir /var/lib/logcorrelator \
|
||||
--no-create-home \
|
||||
--shell /usr/sbin/nologin \
|
||||
logcorrelator
|
||||
fi
|
||||
|
||||
# Create directories
|
||||
mkdir -p /var/lib/logcorrelator
|
||||
mkdir -p /var/log/logcorrelator
|
||||
# Note: /var/run/logcorrelator est géré par RuntimeDirectory= (systemd) et tmpfiles.d
|
||||
|
||||
# Set ownership
|
||||
chown -R logcorrelator:logcorrelator /var/lib/logcorrelator
|
||||
chown -R logcorrelator:logcorrelator /var/log/logcorrelator
|
||||
chown -R logcorrelator:logcorrelator /etc/logcorrelator
|
||||
|
||||
# Set permissions
|
||||
chmod 750 /var/lib/logcorrelator
|
||||
chmod 750 /var/log/logcorrelator
|
||||
chmod 750 /etc/logcorrelator
|
||||
|
||||
# Copy default config if not exists
|
||||
if [ ! -f /etc/logcorrelator/logcorrelator.yml ]; then
|
||||
cp /etc/logcorrelator/logcorrelator.yml.example /etc/logcorrelator/logcorrelator.yml
|
||||
chown logcorrelator:logcorrelator /etc/logcorrelator/logcorrelator.yml
|
||||
chmod 640 /etc/logcorrelator/logcorrelator.yml
|
||||
fi
|
||||
|
||||
# Reload systemd and start service
|
||||
if [ -x /bin/systemctl ]; then
|
||||
systemctl daemon-reload
|
||||
systemctl enable logcorrelator.service
|
||||
systemctl start logcorrelator.service
|
||||
fi
|
||||
|
||||
exit 0
|
||||
|
||||
%preun
|
||||
if [ $1 -eq 0 ]; then
|
||||
# Package removal, not upgrade
|
||||
if [ -x /bin/systemctl ]; then
|
||||
systemctl stop logcorrelator.service
|
||||
systemctl disable logcorrelator.service
|
||||
fi
|
||||
fi
|
||||
|
||||
exit 0
|
||||
|
||||
%postun
|
||||
if [ -x /bin/systemctl ]; then
|
||||
systemctl daemon-reload
|
||||
if [ $1 -ge 1 ]; then
|
||||
# Package upgrade, restart service
|
||||
systemctl try-restart logcorrelator.service
|
||||
fi
|
||||
fi
|
||||
|
||||
exit 0
|
||||
|
||||
%files
|
||||
/usr/bin/logcorrelator
|
||||
%config(noreplace) /etc/logcorrelator/logcorrelator.yml
|
||||
/etc/logcorrelator/logcorrelator.yml.example
|
||||
/var/log/logcorrelator
|
||||
/var/lib/logcorrelator
|
||||
/etc/systemd/system/logcorrelator.service
|
||||
%config(noreplace) /etc/logrotate.d/logcorrelator
|
||||
|
||||
%changelog
|
||||
* Wed Mar 11 2026 logcorrelator <dev@example.com> - 1.1.22-1
|
||||
- Feat(outputs): file output enabled/disabled toggle
|
||||
Ajout du champ enabled: true/false dans outputs.file de la configuration.
|
||||
Le sink fichier n'est cree que si enabled: true ET path: defini.
|
||||
Permet de desactiver completement la sortie fichier tout en gardant stdout/clickhouse.
|
||||
Tests: TestValidate_FileOutputDisabled, TestLoadConfig_FileOutputDisabled
|
||||
|
||||
- Fix(systemd): arret immediat sans vidage de queue
|
||||
orchestrator.Stop() ne vide plus les buffers (events en transit perdus).
|
||||
Suppression de ShutdownTimeout et de la logique de flush/attente.
|
||||
systemd TimeoutStopSec=30 gere l'arret force si besoin.
|
||||
Simplification: cancel() + Close() uniquement.
|
||||
|
||||
- Feat(sql): TTL et compression ZSTD sur tables ClickHouse
|
||||
http_logs_raw: TTL 1 jour, compression ZSTD sur raw_json
|
||||
http_logs: TTL 7 jours, compression ZSTD sur champs texte volumineux
|
||||
Parametre ttl_only_drop_parts = 1 pour optimiser les suppressions
|
||||
|
||||
* Mon Mar 09 2026 logcorrelator <dev@example.com> - 1.1.21-1
|
||||
- Update: vues ClickHouse et schema SQL
|
||||
Ajout de bots.sql pour l'identification des bots (User-Agent parsing)
|
||||
Ajout de tables.sql pour les tables de reference
|
||||
Mise a jour de mv1.sql (vue materialisee) avec nouvelle structure de correlation
|
||||
Documentation views.md enrichie avec exemples de requetes et schema complet
|
||||
|
||||
* Mon Mar 09 2026 logcorrelator <dev@example.com> - 1.1.20-1
|
||||
- Fix(rpm): suppression de systemd-tmpfiles.conf redondant
|
||||
RuntimeDirectory=logcorrelator dans le service systemd gere deja /run/logcorrelator
|
||||
automatiquement. La commande systemd-tmpfiles --create causait des erreurs sur
|
||||
les systemes avec /var/lib/mysql existant (fichier au lieu de repertoire).
|
||||
Suppression de /usr/lib/tmpfiles.d/logcorrelator.conf et de systemd-tmpfiles --create.
|
||||
|
||||
* Mon Mar 09 2026 logcorrelator <dev@example.com> - 1.1.19-1
|
||||
- Fix(systemd): stop/restart immediat sans attendre vidage queue
|
||||
L'arret du service ne vide plus les buffers (events en transit perdus).
|
||||
systemd TimeoutStopSec=30 gere deja l'arret force si besoin.
|
||||
Simplification de orchestrator.Stop() : cancel() + Close() uniquement.
|
||||
Suppression de ShutdownTimeout devenu inutile.
|
||||
|
||||
* Mon Mar 09 2026 logcorrelator <dev@example.com> - 1.1.18-1
|
||||
- Fix(outputs): file output enabled: false ne coupait pas l ecriture du fichier
|
||||
Le champ Enabled manquait dans FileOutputConfig. Le sink fichier etait cree
|
||||
meme avec enabled: false tant que path etait defini. Desormais, la condition
|
||||
verifie explicitement enabled && path != "" dans main.go et Validate().
|
||||
Test: TestValidate_FileOutputDisabled et TestLoadConfig_FileOutputDisabled ajoutes.
|
||||
|
||||
* Fri Mar 06 2026 logcorrelator <dev@example.com> - 1.1.17-1
|
||||
- Fix(correlation): champ keepalives non peuple dans ClickHouse
|
||||
Le champ KeepAliveSeq de NormalizedEvent n'etait pas transfere dans les Fields
|
||||
de CorrelatedLog. La vue materialisee ClickHouse extrayait keepalives du JSON
|
||||
mais trouvait toujours 0. Desormais, NewCorrelatedLog et NewCorrelatedLogFromEvent
|
||||
ajoutent explicitement keepalives = KeepAliveSeq dans les Fields.
|
||||
|
||||
* Fri Mar 06 2026 logcorrelator <dev@example.com> - 1.1.16-1
|
||||
- Feat(correlation): emettre les evenements A filtrés par include_dest_ports vers ClickHouse
|
||||
Quand un evenement A (HTTP) etait exclu par le filtre include_dest_ports, il etait
|
||||
silencieusement ignore. Desormais, si ApacheAlwaysEmit=true, l evenement est emis comme
|
||||
non-correle (orphan_side=A) afin d apparaitre dans ClickHouse. Les evenements B restent
|
||||
ignores. Test: TestCorrelationService_IncludeDestPorts_FilteredPort mis a jour +
|
||||
TestCorrelationService_IncludeDestPorts_FilteredPort_NoAlwaysEmit ajoute.
|
||||
|
||||
* Thu Mar 05 2026 logcorrelator <dev@example.com> - 1.1.15-1
|
||||
- Fix(correlation/bug3): perte de donnees quand B expire avec des orphelins en attente
|
||||
cleanNetworkBufferByTTL supprimait les pendingOrphans sans les emettre (perte silencieuse).
|
||||
Desormais, les orphelins A sont retournes immediatement a l'appelant quand B expire,
|
||||
et cleanExpired/ProcessEvent propagent ces resultats vers le sink.
|
||||
Test: TestBTTLExpiry_PurgesPendingOrphans etendu pour verifier l'emission effective.
|
||||
|
||||
* Thu Mar 05 2026 logcorrelator <dev@example.com> - 1.1.14-1
|
||||
- Fix(correlation/bug1): Keep-Alive sessions au-dela de TimeWindow ne correlent plus en orphelins
|
||||
Le matcher dans processSourceA utilisait eventsMatch (comparaison de timestamps) en mode
|
||||
one_to_many. Apres ~10s, B.Timestamp_original depasse la TimeWindow et toutes les requetes
|
||||
suivantes devenaient orphelines. Nouveau matcher bEventHasValidTTL : un B event est valide
|
||||
tant que son TTL n'a pas expire (le TTL est reset a chaque correlation Keep-Alive).
|
||||
- Fix(correlation/bug4): checkPendingOrphansForCorrelation utilisait eventsMatch (meme bug)
|
||||
En mode one_to_many, un B arrivant avec un vieux timestamp ne matchait plus les pending orphans
|
||||
pour la meme cle. Remplace par une verification de cle uniquement (meme cle = meme connexion).
|
||||
- Fix(correlation/bug3): pendingOrphans non purges quand le B expire (cleanNetworkBufferByTTL)
|
||||
Quand un B event expirait (TTL), les pending orphan A associes etaient bloques indefiniment.
|
||||
Ils sont desormais emis immediatement lors de l'expiration du B correspondant.
|
||||
- Fix(correlation/bug2): orphans emis uniquement sur reception d'evenement (pas de timer dedie)
|
||||
EmitPendingOrphans() est maintenant une methode publique thread-safe. L'Orchestrateur
|
||||
demarre un goroutine ticker (250ms) qui appelle EmitPendingOrphans() independamment du flux,
|
||||
garantissant l'emission meme en l'absence de nouveaux evenements.
|
||||
- Feat(ports): ajout de EmitPendingOrphans() dans l'interface CorrelationProcessor
|
||||
- Test: 4 nouveaux tests de non-regression (Bug #1, #2, #3, #4)
|
||||
|
||||
* Thu Mar 05 2026 logcorrelator <dev@example.com> - 1.1.13-1
|
||||
- Fix: Unix sockets ne passent plus en root:root lors des restarts du service
|
||||
- Fix: Ajout de RuntimeDirectory=logcorrelator dans le service systemd (systemd gère /run/logcorrelator avec le bon propriétaire à chaque démarrage/restart)
|
||||
- Fix: Ajout de /usr/lib/tmpfiles.d/logcorrelator.conf pour recréer /run/logcorrelator au boot
|
||||
- Chore: Retrait de /var/run/logcorrelator du RPM %files (géré par tmpfiles.d)
|
||||
- Fix(correlation): emitPendingOrphans - corruption de slice lors de l expiration simultanée de plusieurs orphelins pour la même clé (slice aliasing bug, émissions en double)
|
||||
- Fix(correlation): rotateOldestA - l événement rotaté était perdu silencieusement même avec ApacheAlwaysEmit=true (retourne désormais le CorrelatedLog)
|
||||
- Fix(correlation): Keep-Alive cassé dans le chemin pending-orphan-then-B - le B event n était pas bufferisé en mode one_to_many, bloquant la corrélation des requêtes A2+ du même Keep-Alive
|
||||
- Chore(correlation): suppression du champ mort timer *time.Timer dans pendingOrphan
|
||||
- Feat(correlation): ajout de keepalive_seq dans les logs orphelins pour faciliter le debug (numéro de requête dans la connexion Keep-Alive, 1-based)
|
||||
- Test: 4 nouveaux tests de non-régression pour les bugs de corrélation
|
||||
|
||||
* Thu Mar 05 2026 logcorrelator <dev@example.com> - 1.1.12-1
|
||||
- Feat: New config directive include_dest_ports - restrict correlation to specific destination ports
|
||||
- Feat: If include_dest_ports is non-empty, events on unlisted ports are silently ignored (not correlated, not emitted as orphan)
|
||||
- Feat: New metric failed_dest_port_filtered for monitoring filtered traffic
|
||||
- Feat: Debug log for filtered events: "event excluded by dest port filter: source=A dst_port=22"
|
||||
- Test: New unit tests for include_dest_ports (allowed port, filtered port, empty=all)
|
||||
- Docs: README.md updated with include_dest_ports section and current version references
|
||||
- Docs: architecture.yml updated with include_dest_ports
|
||||
- Fix: config.example.yml - removed obsolete stdout.level field
|
||||
|
||||
* Thu Mar 05 2026 logcorrelator <dev@example.com> - 1.1.11-1
|
||||
- Fix: StdoutSink no longer writes correlated/orphan JSON to stdout
|
||||
- Fix: stdout sink is now a no-op for data; operational logs go to stderr via logger
|
||||
- Fix: ClickHouse sink had no logger - all flush errors were silently discarded
|
||||
- Fix: Periodic, batch and final-close flush errors are now logged at ERROR level
|
||||
- Fix: Buffer overflow with DropOnOverflow=true is now logged at WARN level
|
||||
- Fix: Retry attempts are now logged at WARN level with attempt number, delay and error
|
||||
- Feat: ClickHouse connection success logged at INFO (table, batch_size, flush_interval_ms)
|
||||
- Feat: Successful batch sends logged at DEBUG (rows count, table)
|
||||
- Feat: SetLogger() method added to ClickHouseSink for external logger injection
|
||||
- Test: New unit tests for StdoutSink asserting stdout remains empty for all log types
|
||||
|
||||
* Wed Mar 04 2026 logcorrelator <dev@example.com> - 1.1.10-1
|
||||
- Feat: IP exclusion filter - exclude specific source IPs or CIDR ranges
|
||||
- Feat: Configuration exclude_source_ips supports single IPs and CIDR notation
|
||||
- Feat: Debug logging for excluded IPs
|
||||
- Feat: New metric failed_ip_excluded for monitoring filtered traffic
|
||||
- Feat: Architecture documentation updated with observability section
|
||||
- Use cases: exclude health checks, internal traffic, known bad actors
|
||||
- Docs: README.md updated with IP exclusion documentation
|
||||
- Docs: architecture.yml updated with metrics and troubleshooting guide
|
||||
|
||||
* Wed Mar 04 2026 logcorrelator <dev@example.com> - 1.1.9-1
|
||||
- Feat: Debug logging - detailed DEBUG logs for correlation troubleshooting
|
||||
- Feat: Correlation metrics server (HTTP endpoint /metrics and /health)
|
||||
- Feat: New metrics: events_received, correlations_success/failed, failure reasons
|
||||
- Feat: Failure reason tracking: no_match_key, time_window, buffer_eviction, ttl_expired
|
||||
- Feat: Buffer size monitoring (buffer_a_size, buffer_b_size)
|
||||
- Feat: Orphan tracking (orphans_emitted, orphans_pending, pending_orphan_match)
|
||||
- Feat: Keep-Alive reset counter for connection tracking
|
||||
- Feat: Test scripts added (test-correlation.sh, test-correlation-advanced.py)
|
||||
- Change: Config example updated with metrics section
|
||||
- Docs: README.md updated with debugging guide and troubleshooting table
|
||||
|
||||
* Tue Mar 03 2026 logcorrelator <dev@example.com> - 1.1.8-1
|
||||
- Migrated from FPM to rpmbuild (native RPM build)
|
||||
- Reduced build image size by 200MB (-40%)
|
||||
- Removed FPM gem dependency (use rpmbuild directly)
|
||||
- Scripts post/preun/postun now inline in spec file
|
||||
- Build image: rockylinux:8 instead of ruby:3.2-bookworm
|
||||
|
||||
* Tue Mar 03 2026 logcorrelator <dev@example.com> - 1.1.7-1
|
||||
- Fix: Critical Keep-Alive bug - network events evicted based on original timestamp instead of reset TTL
|
||||
- Fix: Correlation time window increased from 1s to 10s for HTTP Keep-Alive support
|
||||
- Fix: Network source now uses payload timestamp if available (fallback to reception time)
|
||||
- Change: Default network TTL increased from 30s to 120s for long Keep-Alive sessions
|
||||
- Test: Added comprehensive Keep-Alive tests (TTL reset, long session scenarios)
|
||||
|
||||
* Tue Mar 03 2026 logcorrelator <dev@example.com> - 1.1.6-1
|
||||
- Docs: Update ClickHouse schema documentation (http_logs_raw + http_logs tables)
|
||||
- Fix: ClickHouse insertion uses single raw_json column (FORMAT JSONEachRow)
|
||||
- Fix: ClickHouse native API (clickhouse-go/v2 PrepareBatch + Append + Send)
|
||||
|
||||
* Tue Mar 03 2026 logcorrelator <dev@example.com> - 1.1.5-1
|
||||
- Fix: ClickHouse insertion using native clickhouse-go/v2 API (PrepareBatch + Append + Send)
|
||||
- Fix: Replaced database/sql wrapper with clickhouse.Open() and clickhouse.Conn
|
||||
- Fix: Proper batch sending to avoid ATTEMPT_TO_READ_AFTER_EOF errors
|
||||
- Fix: Set correct permissions (755) on /var/run/logcorrelator in RPM post-install
|
||||
|
||||
* Mon Mar 02 2026 logcorrelator <dev@example.com> - 1.1.4-1
|
||||
- Fix: Log raw JSON data on parse errors for debugging
|
||||
|
||||
* Mon Mar 02 2026 logcorrelator <dev@example.com> - 1.1.3-1
|
||||
- Refactor: Switch Unix sockets from STREAM to DGRAM mode (SOCK_DGRAM)
|
||||
- Test: Comprehensive tests added - coverage improved to 74.4%
|
||||
- Fix: Example config file installed to /etc/logcorrelator/logcorrelator.yml.example
|
||||
- Change: Default socket permissions from 0660 to 0666 (world read/write)
|
||||
|
||||
* Mon Mar 02 2026 logcorrelator <dev@example.com> - 1.1.2-1
|
||||
- Fix: Example config file installed to /etc/logcorrelator/logcorrelator.yml.example
|
||||
- Change: Default socket permissions from 0660 to 0666 (world read/write)
|
||||
|
||||
* Mon Mar 02 2026 logcorrelator <dev@example.com> - 1.1.1-1
|
||||
- Fix: Move logcorrelator.yml.example from /usr/share/logcorrelator/ to /etc/logcorrelator/
|
||||
|
||||
* Mon Mar 02 2026 logcorrelator <dev@example.com> - 1.1.0-1
|
||||
- Feat: Keep-Alive support (one-to-many correlation mode)
|
||||
- Feat: Dynamic TTL for network events (source B)
|
||||
- Feat: Separate buffer sizes for HTTP and network events
|
||||
- Feat: SIGHUP signal handling for log rotation
|
||||
- Feat: File sink Reopen() method for log rotation
|
||||
- Feat: logrotate configuration included
|
||||
- Feat: ExecReload added to systemd service
|
||||
- Feat: New YAML config structure (time_window, orphan_policy, matching, buffers, ttl)
|
||||
- Docs: Updated architecture.yml and config.example.yml
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.7-1
|
||||
- Added: Log levels DEBUG, INFO, WARN, ERROR configurable via log.level
|
||||
- Added: Warn and Warnf methods for warning messages
|
||||
- Added: Debug logs for events received from sockets and correlations
|
||||
- Added: Warning logs for orphan events and buffer overflow
|
||||
- Changed: Configuration log.enabled replaced by log.level
|
||||
- Changed: Orphan events and buffer overflow now logged as WARN instead of DEBUG
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.6-1
|
||||
- Changed: Configuration YAML simplified, removed service.name, service.language
|
||||
- Changed: Correlation config simplified, time_window_s instead of nested object
|
||||
- Changed: Orphan policy simplified to emit_orphans boolean
|
||||
- Changed: Apache socket renamed to http.socket
|
||||
- Added: socket_permissions option on unix sockets
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.5-1
|
||||
- Added: Systemd service auto-start after RPM installation
|
||||
- Added: Systemd service hardening (TimeoutStartSec, TimeoutStopSec, ReadWritePaths)
|
||||
- Fixed: Systemd service unit correct config path (.yml instead of .conf)
|
||||
- Fixed: CI workflow branch name main to master
|
||||
- Changed: RPM packaging generic el8/el9/el10 directory naming
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.4-1
|
||||
- Breaking: Flattened JSON output structure - removed apache and network subdivisions
|
||||
- All log fields now merged into single-level JSON structure
|
||||
- ClickHouse schema: replaced apache JSON and network JSON columns with fields JSON column
|
||||
- Custom MarshalJSON() implementation for flat output
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.3-1
|
||||
- Fix: Added missing ClickHouse driver dependency
|
||||
- Fix: Fixed race condition in orchestrator
|
||||
- Security: Added explicit source_type configuration for Unix socket sources
|
||||
- Added: Comprehensive test suite improvements
|
||||
- Added: Test coverage improved from 50.6% to 62.0%
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.2-1
|
||||
- Added: Initial RPM packaging support for Rocky Linux 8/9 and AlmaLinux 10
|
||||
- Added: Docker multi-stage build pipeline
|
||||
- Added: Hexagonal architecture implementation
|
||||
- Added: Unix socket input sources (JSON line protocol)
|
||||
- Added: File output sink (JSON lines)
|
||||
- Added: ClickHouse output sink with batching and retry logic
|
||||
- Added: Time-window based correlation on src_ip + src_port
|
||||
- Added: Graceful shutdown with signal handling (SIGINT, SIGTERM)
|
||||
|
||||
* Sat Feb 28 2026 logcorrelator <dev@example.com> - 1.0.1-1
|
||||
- Initial package for CentOS 7, Rocky Linux 8, 9, 10
|
||||
13
services/correlator/packaging/rpm/logrotate
Normal file
13
services/correlator/packaging/rpm/logrotate
Normal file
@ -0,0 +1,13 @@
|
||||
/var/log/logcorrelator/correlated.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
create 0640 logcorrelator logcorrelator
|
||||
sharedscripts
|
||||
postrotate
|
||||
/bin/systemctl reload logcorrelator > /dev/null 2>&1 || true
|
||||
endscript
|
||||
}
|
||||
258
services/correlator/packaging/test/test-rpm.sh
Executable file
258
services/correlator/packaging/test/test-rpm.sh
Executable file
@ -0,0 +1,258 @@
|
||||
#!/bin/bash
|
||||
# Test script for logcorrelator RPM package
|
||||
# Verifies installation, permissions, and service status
|
||||
#
|
||||
# Usage: ./packaging/test/test-rpm.sh [el8|el9|el10]
|
||||
#
|
||||
# This script tests the RPM package in a Docker container to ensure:
|
||||
# - Installation succeeds
|
||||
# - File permissions are correct
|
||||
# - Service starts properly
|
||||
# - Sockets are created with correct ownership
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$(dirname "$SCRIPT_DIR")")"
|
||||
RPM_DIR="${PROJECT_ROOT}/dist/rpm"
|
||||
|
||||
# Default to el8 if no argument provided
|
||||
DISTRO="${1:-el8}"
|
||||
|
||||
echo "========================================="
|
||||
echo "Testing logcorrelator RPM for ${DISTRO}"
|
||||
echo "========================================="
|
||||
|
||||
# Find the RPM file
|
||||
case "${DISTRO}" in
|
||||
el8|rocky8)
|
||||
RPM_PATH="${RPM_DIR}/el8"
|
||||
BASE_IMAGE="rockylinux:8"
|
||||
;;
|
||||
el9|rocky9)
|
||||
RPM_PATH="${RPM_DIR}/el9"
|
||||
BASE_IMAGE="rockylinux:9"
|
||||
;;
|
||||
el10|alma10)
|
||||
RPM_PATH="${RPM_DIR}/el10"
|
||||
BASE_IMAGE="almalinux:10"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown distribution: ${DISTRO}"
|
||||
echo "Valid options: el8, el9, el10"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Find the latest RPM file
|
||||
RPM_FILE=$(ls -t "${RPM_PATH}"/logcorrelator-*.rpm 2>/dev/null | head -n 1)
|
||||
|
||||
if [ -z "${RPM_FILE}" ]; then
|
||||
echo "ERROR: No RPM file found in ${RPM_PATH}"
|
||||
echo "Please run 'make package-rpm' first"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Testing RPM: ${RPM_FILE}"
|
||||
echo "Base image: ${BASE_IMAGE}"
|
||||
echo ""
|
||||
|
||||
# Create test script
|
||||
TEST_SCRIPT=$(cat <<'EOF'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "=== Installing logcorrelator RPM ==="
|
||||
rpm -ivh /tmp/logcorrelator.rpm
|
||||
|
||||
echo ""
|
||||
echo "=== Checking user and group ==="
|
||||
if ! getent group logcorrelator >/dev/null; then
|
||||
echo "FAIL: logcorrelator group not created"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: logcorrelator group exists"
|
||||
|
||||
if ! getent passwd logcorrelator >/dev/null; then
|
||||
echo "FAIL: logcorrelator user not created"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: logcorrelator user exists"
|
||||
|
||||
echo ""
|
||||
echo "=== Checking directory permissions ==="
|
||||
|
||||
# Check /var/run/logcorrelator
|
||||
DIR="/var/run/logcorrelator"
|
||||
if [ ! -d "$DIR" ]; then
|
||||
echo "FAIL: $DIR does not exist"
|
||||
exit 1
|
||||
fi
|
||||
OWNER=$(stat -c '%U:%G' "$DIR")
|
||||
PERMS=$(stat -c '%a' "$DIR")
|
||||
if [ "$OWNER" != "logcorrelator:logcorrelator" ]; then
|
||||
echo "FAIL: $DIR owner is $OWNER (expected logcorrelator:logcorrelator)"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$PERMS" != "755" ]; then
|
||||
echo "FAIL: $DIR permissions are $PERMS (expected 755)"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $DIR - owner=$OWNER, permissions=$PERMS"
|
||||
|
||||
# Check /var/log/logcorrelator
|
||||
DIR="/var/log/logcorrelator"
|
||||
if [ ! -d "$DIR" ]; then
|
||||
echo "FAIL: $DIR does not exist"
|
||||
exit 1
|
||||
fi
|
||||
OWNER=$(stat -c '%U:%G' "$DIR")
|
||||
PERMS=$(stat -c '%a' "$DIR")
|
||||
if [ "$OWNER" != "logcorrelator:logcorrelator" ]; then
|
||||
echo "FAIL: $DIR owner is $OWNER (expected logcorrelator:logcorrelator)"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$PERMS" != "750" ]; then
|
||||
echo "FAIL: $DIR permissions are $PERMS (expected 750)"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $DIR - owner=$OWNER, permissions=$PERMS"
|
||||
|
||||
# Check /var/lib/logcorrelator
|
||||
DIR="/var/lib/logcorrelator"
|
||||
if [ ! -d "$DIR" ]; then
|
||||
echo "FAIL: $DIR does not exist"
|
||||
exit 1
|
||||
fi
|
||||
OWNER=$(stat -c '%U:%G' "$DIR")
|
||||
PERMS=$(stat -c '%a' "$DIR")
|
||||
if [ "$OWNER" != "logcorrelator:logcorrelator" ]; then
|
||||
echo "FAIL: $DIR owner is $OWNER (expected logcorrelator:logcorrelator)"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$PERMS" != "750" ]; then
|
||||
echo "FAIL: $DIR permissions are $PERMS (expected 750)"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $DIR - owner=$OWNER, permissions=$PERMS"
|
||||
|
||||
echo ""
|
||||
echo "=== Checking config files ==="
|
||||
|
||||
# Check config file exists and has correct permissions
|
||||
CONFIG="/etc/logcorrelator/logcorrelator.yml"
|
||||
if [ ! -f "$CONFIG" ]; then
|
||||
echo "FAIL: $CONFIG does not exist"
|
||||
exit 1
|
||||
fi
|
||||
OWNER=$(stat -c '%U:%G' "$CONFIG")
|
||||
PERMS=$(stat -c '%a' "$CONFIG")
|
||||
if [ "$OWNER" != "logcorrelator:logcorrelator" ]; then
|
||||
echo "FAIL: $CONFIG owner is $OWNER (expected logcorrelator:logcorrelator)"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$PERMS" != "640" ]; then
|
||||
echo "FAIL: $CONFIG permissions are $PERMS (expected 640)"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $CONFIG - owner=$OWNER, permissions=$PERMS"
|
||||
|
||||
# Check example config file
|
||||
EXAMPLE_CONFIG="/etc/logcorrelator/logcorrelator.yml.example"
|
||||
if [ ! -f "$EXAMPLE_CONFIG" ]; then
|
||||
echo "FAIL: $EXAMPLE_CONFIG does not exist"
|
||||
exit 1
|
||||
fi
|
||||
OWNER=$(stat -c '%U:%G' "$EXAMPLE_CONFIG")
|
||||
PERMS=$(stat -c '%a' "$EXAMPLE_CONFIG")
|
||||
if [ "$OWNER" != "logcorrelator:logcorrelator" ]; then
|
||||
echo "FAIL: $EXAMPLE_CONFIG owner is $OWNER (expected logcorrelator:logcorrelator)"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$PERMS" != "640" ]; then
|
||||
echo "FAIL: $EXAMPLE_CONFIG permissions are $PERMS (expected 640)"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $EXAMPLE_CONFIG - owner=$OWNER, permissions=$PERMS"
|
||||
|
||||
echo ""
|
||||
echo "=== Checking systemd service ==="
|
||||
if [ ! -f /etc/systemd/system/logcorrelator.service ]; then
|
||||
echo "FAIL: systemd service file not found"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: systemd service file exists"
|
||||
|
||||
echo ""
|
||||
echo "=== Checking logrotate config ==="
|
||||
if [ ! -f /etc/logrotate.d/logcorrelator ]; then
|
||||
echo "FAIL: logrotate config not found"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: logrotate config exists"
|
||||
|
||||
echo ""
|
||||
echo "=== Testing service start ==="
|
||||
# Try to start the service (may fail in container without full systemd)
|
||||
if command -v systemctl >/dev/null 2>&1; then
|
||||
systemctl daemon-reload || true
|
||||
if systemctl start logcorrelator.service 2>/dev/null; then
|
||||
echo "OK: service started successfully"
|
||||
|
||||
# Wait for sockets to be created
|
||||
sleep 2
|
||||
|
||||
echo ""
|
||||
echo "=== Checking sockets ==="
|
||||
HTTP_SOCKET="/var/run/logcorrelator/http.socket"
|
||||
NETWORK_SOCKET="/var/run/logcorrelator/network.socket"
|
||||
|
||||
if [ -S "$HTTP_SOCKET" ]; then
|
||||
OWNER=$(stat -c '%U:%G' "$HTTP_SOCKET")
|
||||
PERMS=$(stat -c '%a' "$HTTP_SOCKET")
|
||||
echo "OK: $HTTP_SOCKET exists - owner=$OWNER, permissions=$PERMS"
|
||||
if [ "$PERMS" != "666" ]; then
|
||||
echo "WARN: socket permissions are $PERMS (expected 666)"
|
||||
fi
|
||||
else
|
||||
echo "WARN: $HTTP_SOCKET not found (service may not have started)"
|
||||
fi
|
||||
|
||||
if [ -S "$NETWORK_SOCKET" ]; then
|
||||
OWNER=$(stat -c '%U:%G' "$NETWORK_SOCKET")
|
||||
PERMS=$(stat -c '%a' "$NETWORK_SOCKET")
|
||||
echo "OK: $NETWORK_SOCKET exists - owner=$OWNER, permissions=$PERMS"
|
||||
if [ "$PERMS" != "666" ]; then
|
||||
echo "WARN: socket permissions are $PERMS (expected 666)"
|
||||
fi
|
||||
else
|
||||
echo "WARN: $NETWORK_SOCKET not found (service may not have started)"
|
||||
fi
|
||||
|
||||
systemctl stop logcorrelator.service || true
|
||||
else
|
||||
echo "WARN: service failed to start (expected in minimal container)"
|
||||
fi
|
||||
else
|
||||
echo "WARN: systemctl not available (minimal container)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "========================================="
|
||||
echo "All tests passed!"
|
||||
echo "========================================="
|
||||
EOF
|
||||
)
|
||||
|
||||
# Run test in Docker container
|
||||
echo "Running tests in Docker container..."
|
||||
echo ""
|
||||
|
||||
docker run --rm \
|
||||
-v "${RPM_FILE}:/tmp/logcorrelator.rpm:ro" \
|
||||
-v "${TEST_SCRIPT}:/test.sh:ro" \
|
||||
"${BASE_IMAGE}" \
|
||||
bash /test.sh
|
||||
|
||||
echo ""
|
||||
echo "Test completed successfully for ${DISTRO}"
|
||||
Reference in New Issue
Block a user