25d4943714
- Update RPM version numbers to 1.1.6 - Fix config file name (.yml not .conf) - Add complete configuration example with current schema - Add ClickHouse DSN format documentation - Add Troubleshooting section (ClickHouse, MV, sockets, systemd) - Update project structure with accurate file names Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
582 lines
18 KiB
Markdown
582 lines
18 KiB
Markdown
# logcorrelator
|
|
|
|
Service de corrélation de logs HTTP et réseau écrit en Go.
|
|
|
|
## Description
|
|
|
|
**logcorrelator** reçoit deux flux de logs JSON via des sockets Unix :
|
|
- **Source A** : logs HTTP applicatifs (Apache, reverse proxy)
|
|
- **Source B** : logs réseau (métadonnées IP/TCP, JA3/JA4, etc.)
|
|
|
|
Il corrèle les événements sur la base de `src_ip + src_port` avec une fenêtre temporelle configurable, et produit des logs corrélés vers :
|
|
- Un fichier local (JSON lines)
|
|
- ClickHouse (pour analyse et archivage)
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
|
│ Apache Source │────▶│ │────▶│ File Sink │
|
|
│ (Unix Socket) │ │ Correlation │ │ (JSON lines) │
|
|
└─────────────────┘ │ Service │ └─────────────────┘
|
|
│ │
|
|
┌─────────────────┐ │ - Buffers │ ┌─────────────────┐
|
|
│ Network Source │────▶│ - Time Window │────▶│ ClickHouse │
|
|
│ (Unix Socket) │ │ - Orphan Policy │ │ Sink │
|
|
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
## Build (100% Docker)
|
|
|
|
Tout le build et les tests s'exécutent dans des containers Docker :
|
|
|
|
```bash
|
|
# Build complet (binaire + tests + RPM)
|
|
make package-rpm
|
|
|
|
# Uniquement les tests
|
|
make test
|
|
|
|
# Build manuel avec Docker
|
|
docker build --target builder -t logcorrelator-builder .
|
|
docker build --target runtime -t logcorrelator:latest .
|
|
```
|
|
|
|
### Prérequis
|
|
|
|
- Docker 20.10+
|
|
- Bash
|
|
|
|
## Installation
|
|
|
|
### Depuis Docker
|
|
|
|
```bash
|
|
# Build de l'image
|
|
docker build --target runtime -t logcorrelator:latest .
|
|
|
|
# Exécuter
|
|
docker run -d \
|
|
--name logcorrelator \
|
|
-v /var/run/logcorrelator:/var/run/logcorrelator \
|
|
-v /var/log/logcorrelator:/var/log/logcorrelator \
|
|
-v ./config.example.yml:/etc/logcorrelator/logcorrelator.yml \
|
|
logcorrelator:latest
|
|
```
|
|
|
|
### Depuis les packages RPM
|
|
|
|
```bash
|
|
# Générer les packages
|
|
make package-rpm
|
|
|
|
# Installer le package RPM (Rocky Linux 8/9/10)
|
|
sudo dnf install -y dist/rpm/el8/logcorrelator-1.1.6-1.el8.x86_64.rpm
|
|
sudo dnf install -y dist/rpm/el9/logcorrelator-1.1.6-1.el9.x86_64.rpm
|
|
sudo dnf install -y dist/rpm/el10/logcorrelator-1.1.6-1.el10.x86_64.rpm
|
|
|
|
# Activer et démarrer le service
|
|
sudo systemctl enable logcorrelator
|
|
sudo systemctl start logcorrelator
|
|
|
|
# Vérifier le statut
|
|
sudo systemctl status logcorrelator
|
|
```
|
|
|
|
### Build manuel (sans Docker)
|
|
|
|
```bash
|
|
# Prérequis: Go 1.21+
|
|
go build -o logcorrelator ./cmd/logcorrelator
|
|
|
|
# Exécuter
|
|
./logcorrelator -config config.example.yml
|
|
```
|
|
|
|
## Configuration
|
|
|
|
La configuration utilise un fichier YAML. Voir `config.example.yml` pour un exemple complet.
|
|
|
|
```yaml
|
|
# /etc/logcorrelator/logcorrelator.yml
|
|
|
|
log:
|
|
level: INFO # DEBUG, INFO, WARN, ERROR
|
|
|
|
inputs:
|
|
unix_sockets:
|
|
# Source HTTP (A) : logs applicatifs en JSON
|
|
- name: http
|
|
path: /var/run/logcorrelator/http.sock
|
|
socket_permissions: "0666"
|
|
socket_type: dgram
|
|
|
|
# Source réseau (B) : logs IP/TCP/JA3... en JSON
|
|
- name: network
|
|
path: /var/run/logcorrelator/network.sock
|
|
socket_permissions: "0666"
|
|
socket_type: dgram
|
|
|
|
outputs:
|
|
file:
|
|
enabled: true
|
|
path: /var/log/logcorrelator/correlated.log
|
|
format: json_lines
|
|
|
|
clickhouse:
|
|
enabled: true
|
|
dsn: clickhouse://data_writer:password@localhost:9000/mabase_prod
|
|
table: http_logs_raw
|
|
batch_size: 500
|
|
flush_interval_ms: 200
|
|
max_buffer_size: 5000
|
|
drop_on_overflow: true
|
|
|
|
correlation:
|
|
time_window:
|
|
value: 1
|
|
unit: s
|
|
orphan_policy:
|
|
apache_always_emit: true
|
|
network_emit: false
|
|
matching:
|
|
mode: one_to_many # Keep-Alive : un B peut corréler plusieurs A
|
|
buffers:
|
|
max_http_items: 10000
|
|
max_network_items: 20000
|
|
ttl:
|
|
network_ttl_s: 30
|
|
```
|
|
|
|
### Format du DSN ClickHouse
|
|
|
|
```
|
|
clickhouse://username:password@host:port/database
|
|
```
|
|
|
|
Exemple : `clickhouse://data_writer:MonMotDePasse@127.0.0.1:9000/mabase_prod`
|
|
|
|
Ports courants :
|
|
- `9000` : port natif (recommandé pour le driver Go)
|
|
- `8123` : port HTTP (alternative)
|
|
|
|
## Format des logs
|
|
|
|
### Source A (HTTP)
|
|
|
|
```json
|
|
{
|
|
"src_ip": "192.168.1.1",
|
|
"src_port": 8080,
|
|
"dst_ip": "10.0.0.1",
|
|
"dst_port": 80,
|
|
"timestamp": 1704110400000000000,
|
|
"method": "GET",
|
|
"path": "/api/test",
|
|
"header_host": "example.com"
|
|
}
|
|
```
|
|
|
|
### Source B (Réseau)
|
|
|
|
```json
|
|
{
|
|
"src_ip": "192.168.1.1",
|
|
"src_port": 8080,
|
|
"dst_ip": "10.0.0.1",
|
|
"dst_port": 443,
|
|
"ja3": "abc123def456",
|
|
"ja4": "xyz789"
|
|
}
|
|
```
|
|
|
|
### Log corrélé (sortie)
|
|
|
|
**Version 1.0.3+** - Structure JSON plate (flat) :
|
|
|
|
```json
|
|
{
|
|
"timestamp": "2024-01-01T12:00:00Z",
|
|
"src_ip": "192.168.1.1",
|
|
"src_port": 8080,
|
|
"dst_ip": "10.0.0.1",
|
|
"dst_port": 80,
|
|
"correlated": true,
|
|
"method": "GET",
|
|
"path": "/api/test",
|
|
"ja3": "abc123def456",
|
|
"ja4": "xyz789"
|
|
}
|
|
```
|
|
|
|
Tous les champs des sources A et B sont fusionnés au même niveau. Les champs de corrélation (`src_ip`, `src_port`, `dst_ip`, `dst_port`, `correlated`, `orphan_side`) sont toujours présents, et tous les autres champs des logs sources sont ajoutés directement à la racine.
|
|
|
|
## Schema ClickHouse
|
|
|
|
Le service utilise deux tables ClickHouse dans la base `mabase_prod` :
|
|
|
|
### Setup complet
|
|
|
|
```sql
|
|
-- 1. Créer la base de données
|
|
CREATE DATABASE IF NOT EXISTS mabase_prod;
|
|
|
|
-- 2. Table brute avec TTL (1 jour de rétention)
|
|
DROP TABLE IF EXISTS mabase_prod.http_logs_raw;
|
|
|
|
CREATE TABLE mabase_prod.http_logs_raw
|
|
(
|
|
raw_json String,
|
|
ingest_time DateTime DEFAULT now()
|
|
)
|
|
ENGINE = MergeTree
|
|
PARTITION BY toDate(ingest_time)
|
|
ORDER BY ingest_time;
|
|
|
|
-- 3. Table parsée
|
|
DROP TABLE IF EXISTS mabase_prod.http_logs;
|
|
|
|
CREATE TABLE mabase_prod.http_logs
|
|
(
|
|
time DateTime,
|
|
log_date Date DEFAULT toDate(time),
|
|
|
|
src_ip IPv4,
|
|
src_port UInt16,
|
|
dst_ip IPv4,
|
|
dst_port UInt16,
|
|
|
|
method LowCardinality(String),
|
|
scheme LowCardinality(String),
|
|
host LowCardinality(String),
|
|
path String,
|
|
query String,
|
|
http_version LowCardinality(String),
|
|
orphan_side LowCardinality(String),
|
|
|
|
correlated UInt8,
|
|
keepalives UInt16,
|
|
a_timestamp UInt64,
|
|
b_timestamp UInt64,
|
|
conn_id String,
|
|
|
|
ip_meta_df UInt8,
|
|
ip_meta_id UInt32,
|
|
ip_meta_total_length UInt32,
|
|
ip_meta_ttl UInt8,
|
|
tcp_meta_options LowCardinality(String),
|
|
tcp_meta_window_size UInt32,
|
|
syn_to_clienthello_ms Int32,
|
|
|
|
tls_version LowCardinality(String),
|
|
tls_sni LowCardinality(String),
|
|
ja3 String,
|
|
ja3_hash String,
|
|
ja4 String,
|
|
|
|
header_user_agent String,
|
|
header_accept String,
|
|
header_accept_encoding String,
|
|
header_accept_language String,
|
|
header_x_request_id String,
|
|
header_x_trace_id String,
|
|
header_x_forwarded_for String,
|
|
|
|
header_sec_ch_ua String,
|
|
header_sec_ch_ua_mobile String,
|
|
header_sec_ch_ua_platform String,
|
|
header_sec_fetch_dest String,
|
|
header_sec_fetch_mode String,
|
|
header_sec_fetch_site String
|
|
)
|
|
ENGINE = MergeTree
|
|
PARTITION BY log_date
|
|
ORDER BY (time, src_ip, dst_ip, ja4);
|
|
|
|
-- 4. Vue matérialisée (RAW → logs)
|
|
DROP VIEW IF EXISTS mabase_prod.mv_http_logs;
|
|
|
|
CREATE MATERIALIZED VIEW mabase_prod.mv_http_logs
|
|
TO mabase_prod.http_logs
|
|
AS
|
|
SELECT
|
|
-- 1. Temps
|
|
parseDateTimeBestEffort(
|
|
coalesce(JSONExtractString(raw_json, 'time'), '1970-01-01T00:00:00Z')
|
|
) AS time,
|
|
toDate(time) AS log_date,
|
|
|
|
-- 2. Réseau L3/L4
|
|
toIPv4(coalesce(JSONExtractString(raw_json, 'src_ip'), '0.0.0.0')) AS src_ip,
|
|
toUInt16(coalesce(JSONExtractUInt(raw_json, 'src_port'), 0)) AS src_port,
|
|
toIPv4(coalesce(JSONExtractString(raw_json, 'dst_ip'), '0.0.0.0')) AS dst_ip,
|
|
toUInt16(coalesce(JSONExtractUInt(raw_json, 'dst_port'), 0)) AS dst_port,
|
|
|
|
-- 3. HTTP de base
|
|
coalesce(JSONExtractString(raw_json, 'method'), '') AS method,
|
|
coalesce(JSONExtractString(raw_json, 'scheme'), '') AS scheme,
|
|
coalesce(JSONExtractString(raw_json, 'host'), '') AS host,
|
|
coalesce(JSONExtractString(raw_json, 'path'), '') AS path,
|
|
coalesce(JSONExtractString(raw_json, 'query'), '') AS query,
|
|
coalesce(JSONExtractString(raw_json, 'http_version'), '') AS http_version,
|
|
coalesce(JSONExtractString(raw_json, 'orphan_side'), '') AS orphan_side,
|
|
|
|
-- 4. Connexion / corrélation
|
|
toUInt8(coalesce(JSONExtractBool(raw_json, 'correlated'), 0)) AS correlated,
|
|
toUInt16(coalesce(JSONExtractUInt(raw_json, 'keepalives'), 0)) AS keepalives,
|
|
coalesce(JSONExtractUInt(raw_json, 'a_timestamp'), 0) AS a_timestamp,
|
|
coalesce(JSONExtractUInt(raw_json, 'b_timestamp'), 0) AS b_timestamp,
|
|
coalesce(JSONExtractString(raw_json, 'conn_id'), '') AS conn_id,
|
|
|
|
-- 5. IP/TCP
|
|
toUInt8(coalesce(JSONExtractBool(raw_json, 'ip_meta_df'), 0)) AS ip_meta_df,
|
|
coalesce(JSONExtractUInt(raw_json, 'ip_meta_id'), 0) AS ip_meta_id,
|
|
coalesce(JSONExtractUInt(raw_json, 'ip_meta_total_length'), 0) AS ip_meta_total_length,
|
|
coalesce(JSONExtractUInt(raw_json, 'ip_meta_ttl'), 0) AS ip_meta_ttl,
|
|
coalesce(JSONExtractString(raw_json, 'tcp_meta_options'), '') AS tcp_meta_options,
|
|
coalesce(JSONExtractUInt(raw_json, 'tcp_meta_window_size'), 0) AS tcp_meta_window_size,
|
|
toInt32(coalesce(JSONExtractInt(raw_json, 'syn_to_clienthello_ms'), 0)) AS syn_to_clienthello_ms,
|
|
|
|
-- 6. TLS / JA3/JA4
|
|
coalesce(JSONExtractString(raw_json, 'tls_version'), '') AS tls_version,
|
|
coalesce(JSONExtractString(raw_json, 'tls_sni'), '') AS tls_sni,
|
|
coalesce(JSONExtractString(raw_json, 'ja3'), '') AS ja3,
|
|
coalesce(JSONExtractString(raw_json, 'ja3_hash'), '') AS ja3_hash,
|
|
coalesce(JSONExtractString(raw_json, 'ja4'), '') AS ja4,
|
|
|
|
-- 7. Headers HTTP
|
|
coalesce(JSONExtractString(raw_json, 'header_User-Agent'), '') AS header_user_agent,
|
|
coalesce(JSONExtractString(raw_json, 'header_Accept'), '') AS header_accept,
|
|
coalesce(JSONExtractString(raw_json, 'header_Accept-Encoding'), '') AS header_accept_encoding,
|
|
coalesce(JSONExtractString(raw_json, 'header_Accept-Language'), '') AS header_accept_language,
|
|
coalesce(JSONExtractString(raw_json, 'header_X-Request-Id'), '') AS header_x_request_id,
|
|
coalesce(JSONExtractString(raw_json, 'header_X-Trace-Id'), '') AS header_x_trace_id,
|
|
coalesce(JSONExtractString(raw_json, 'header_X-Forwarded-For'), '') AS header_x_forwarded_for,
|
|
|
|
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA'), '') AS header_sec_ch_ua,
|
|
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA-Mobile'), '') AS header_sec_ch_ua_mobile,
|
|
coalesce(JSONExtractString(raw_json, 'header_Sec-CH-UA-Platform'), '') AS header_sec_ch_ua_platform,
|
|
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Dest'), '') AS header_sec_fetch_dest,
|
|
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Mode'), '') AS header_sec_fetch_mode,
|
|
coalesce(JSONExtractString(raw_json, 'header_Sec-Fetch-Site'), '') AS header_sec_fetch_site
|
|
|
|
FROM mabase_prod.http_logs_raw;
|
|
```
|
|
|
|
### Utilisateurs et permissions
|
|
|
|
```sql
|
|
-- Créer les utilisateurs
|
|
CREATE USER IF NOT EXISTS data_writer IDENTIFIED WITH TonMotDePasseInsert;
|
|
CREATE USER IF NOT EXISTS analyst IDENTIFIED WITH TonMotDePasseAnalyst;
|
|
|
|
-- Droits pour data_writer (INSERT + SELECT pour la MV)
|
|
GRANT INSERT(raw_json) ON mabase_prod.http_logs_raw TO data_writer;
|
|
GRANT SELECT(raw_json) ON mabase_prod.http_logs_raw TO data_writer;
|
|
|
|
-- Droits pour analyst (lecture seule sur les logs parsés)
|
|
GRANT SELECT ON mabase_prod.http_logs TO analyst;
|
|
```
|
|
|
|
### Format d'insertion
|
|
|
|
Le service envoie chaque log corrélé sérialisé en JSON dans la colonne `raw_json` :
|
|
|
|
```sql
|
|
INSERT INTO mabase_prod.http_logs_raw (raw_json) FORMAT JSONEachRow
|
|
{"raw_json":"{\"timestamp\":\"2024-01-01T12:00:00Z\",\"src_ip\":\"192.168.1.1\",\"correlated\":true,...}"}
|
|
```
|
|
|
|
### Migration des données existantes
|
|
|
|
Si vous avez déjà des données dans l'ancienne table `http_logs_raw` :
|
|
|
|
```sql
|
|
INSERT INTO mabase_prod.http_logs (raw_json)
|
|
SELECT raw_json
|
|
FROM mabase_prod.http_logs_raw;
|
|
```
|
|
|
|
### Sanity checks - Vérification de l'ingestion
|
|
|
|
Après avoir déployé le service, vérifiez que les données circulent correctement :
|
|
|
|
```sql
|
|
-- 1. Tables présentes
|
|
SELECT
|
|
database,
|
|
table,
|
|
engine
|
|
FROM system.tables
|
|
WHERE database = currentDatabase()
|
|
AND table IN ('http_logs_raw', 'http_logs', 'mv_http_logs');
|
|
|
|
-- 2. Définition de la vue matérialisée
|
|
SHOW CREATE TABLE mv_http_logs;
|
|
|
|
-- 3. Vérifier que les inserts bruts arrivent
|
|
SELECT
|
|
count(*) AS rows_raw,
|
|
min(ingest_time) AS min_ingest,
|
|
max(ingest_time) AS max_ingest
|
|
FROM http_logs_raw;
|
|
|
|
-- 4. Voir les derniers logs bruts
|
|
SELECT
|
|
ingest_time,
|
|
raw_json
|
|
FROM http_logs_raw
|
|
ORDER BY ingest_time DESC
|
|
LIMIT 5;
|
|
|
|
-- 5. Vérifier que la MV alimente http_logs
|
|
SELECT
|
|
count(*) AS rows_flat,
|
|
min(time) AS min_time,
|
|
max(time) AS max_time
|
|
FROM http_logs;
|
|
|
|
-- 6. Voir les derniers logs parsés
|
|
SELECT
|
|
time,
|
|
src_ip,
|
|
dst_ip,
|
|
method,
|
|
host,
|
|
path,
|
|
header_user_agent,
|
|
tls_version,
|
|
ja4
|
|
FROM http_logs
|
|
ORDER BY time DESC
|
|
LIMIT 10;
|
|
```
|
|
|
|
**Interprétation :**
|
|
- Si `rows_raw` > 0 mais `rows_flat` = 0 : la vue matérialisée ne fonctionne pas (vérifiez les droits SELECT sur `http_logs_raw`)
|
|
- Si les deux comptes sont > 0 : l'ingestion et le parsing fonctionnent correctement
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
# Via Docker
|
|
./test.sh
|
|
|
|
# Local
|
|
go test ./...
|
|
go test -cover ./...
|
|
go test -coverprofile=coverage.out ./...
|
|
go tool cover -html=coverage.out
|
|
```
|
|
|
|
## Signaux
|
|
|
|
| Signal | Comportement |
|
|
|--------|--------------|
|
|
| `SIGINT` | Arrêt gracieux |
|
|
| `SIGTERM` | Arrêt gracieux |
|
|
|
|
Lors de l'arrêt gracieux :
|
|
1. Fermeture des sockets Unix
|
|
2. Flush des buffers
|
|
3. Émission des événements en attente
|
|
4. Fermeture des connexions ClickHouse
|
|
|
|
## Logs internes
|
|
|
|
Les logs internes sont envoyés vers stderr :
|
|
|
|
```bash
|
|
# Docker
|
|
docker logs -f logcorrelator
|
|
|
|
# Systemd
|
|
journalctl -u logcorrelator -f
|
|
```
|
|
|
|
## Structure du projet
|
|
|
|
```
|
|
.
|
|
├── cmd/logcorrelator/ # Point d'entrée
|
|
├── internal/
|
|
│ ├── adapters/
|
|
│ │ ├── inbound/unixsocket/ # Sources HTTP et réseau
|
|
│ │ └── outbound/
|
|
│ │ ├── clickhouse/ # Sink ClickHouse
|
|
│ │ ├── file/ # Sink fichier
|
|
│ │ └── multi/ # Multi-sink
|
|
│ ├── app/ # Orchestration
|
|
│ ├── config/ # Configuration YAML
|
|
│ ├── domain/ # Domaine (corrélation)
|
|
│ ├── observability/ # Logging interne
|
|
│ └── ports/ # Interfaces
|
|
├── config.example.yml # Exemple de config
|
|
├── Dockerfile # Build multi-stage
|
|
├── Dockerfile.package # Packaging RPM
|
|
├── Makefile # Commandes de build
|
|
├── architecture.yml # Spécification architecture
|
|
└── logcorrelator.service # Unité systemd
|
|
```
|
|
|
|
## License
|
|
|
|
MIT
|
|
|
|
## Troubleshooting
|
|
|
|
### ClickHouse : erreurs d'insertion
|
|
|
|
**Erreur : `No such column timestamp`**
|
|
- Vérifiez que la table de destination est bien `http_logs_raw` (colonne unique `raw_json`)
|
|
- Le service envoie un JSON sérialisé dans `raw_json`, pas des colonnes séparées
|
|
|
|
**Erreur : `ACCESS_DENIED`**
|
|
- Vérifiez les droits de l'utilisateur `data_writer` :
|
|
```sql
|
|
GRANT INSERT(raw_json) ON mabase_prod.http_logs_raw TO data_writer;
|
|
GRANT SELECT(raw_json) ON mabase_prod.http_logs_raw TO data_writer;
|
|
```
|
|
|
|
### Vue matérialisée ne fonctionne pas
|
|
|
|
**Symptôme :** `http_logs_raw` a des données, mais `http_logs` est vide
|
|
|
|
1. Vérifiez que la MV existe :
|
|
```sql
|
|
SHOW CREATE TABLE mv_http_logs;
|
|
```
|
|
|
|
2. Vérifiez les droits SELECT pour `data_writer` sur `http_logs_raw`
|
|
|
|
3. Testez manuellement :
|
|
```sql
|
|
INSERT INTO mabase_prod.http_logs
|
|
SELECT * FROM mabase_prod.mv_http_logs
|
|
WHERE time > now() - INTERVAL 1 HOUR;
|
|
```
|
|
|
|
### Sockets Unix : permission denied
|
|
|
|
**Erreur :** `permission denied` sur `/var/run/logcorrelator/*.sock`
|
|
|
|
- Vérifiez que les sockets ont les permissions `0666`
|
|
- Vérifiez que l'utilisateur `logcorrelator` peut lire/écrire
|
|
|
|
### Service systemd ne démarre pas
|
|
|
|
1. Vérifiez les logs :
|
|
```bash
|
|
journalctl -u logcorrelator -n 50 --no-pager
|
|
```
|
|
|
|
2. Vérifiez la configuration :
|
|
```bash
|
|
cat /etc/logcorrelator/logcorrelator.yml
|
|
```
|
|
|
|
3. Testez manuellement :
|
|
```bash
|
|
/usr/bin/logcorrelator -config /etc/logcorrelator/logcorrelator.yml
|
|
```
|