Initial commit: logcorrelator with unified packaging (DEB + RPM using fpm)
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
Jacquin Antoine
278
README.md
Normal file
278
README.md
Normal file
@ -0,0 +1,278 @@
|
||||
# logcorrelator
|
||||
|
||||
Service de corrélation de logs HTTP et réseau écrit en Go.
|
||||
|
||||
## Description
|
||||
|
||||
**logcorrelator** reçoit deux flux de logs JSON via des sockets Unix :
|
||||
- **Source A** : logs HTTP applicatifs (Apache, reverse proxy)
|
||||
- **Source B** : logs réseau (métadonnées IP/TCP, JA3/JA4, etc.)
|
||||
|
||||
Il corrèle les événements sur la base de `src_ip + src_port` avec une fenêtre temporelle configurable, et produit des logs corrélés vers :
|
||||
- Un fichier local (JSON lines)
|
||||
- ClickHouse (pour analyse et archivage)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||
│ Apache Source │────▶│ │────▶│ File Sink │
|
||||
│ (Unix Socket) │ │ Correlation │ │ (JSON lines) │
|
||||
└─────────────────┘ │ Service │ └─────────────────┘
|
||||
│ │
|
||||
┌─────────────────┐ │ - Buffers │ ┌─────────────────┐
|
||||
│ Network Source │────▶│ - Time Window │────▶│ ClickHouse │
|
||||
│ (Unix Socket) │ │ - Orphan Policy │ │ Sink │
|
||||
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Build (100% Docker)
|
||||
|
||||
Tout le build et les tests s'exécutent dans des containers Docker :
|
||||
|
||||
```bash
|
||||
# Build complet (binaire + tests + RPM)
|
||||
./build.sh
|
||||
|
||||
# Uniquement les tests
|
||||
./test.sh
|
||||
|
||||
# Build manuel avec Docker
|
||||
docker build --target builder -t logcorrelator-builder .
|
||||
docker build --target runtime -t logcorrelator:latest .
|
||||
```
|
||||
|
||||
### Prérequis
|
||||
|
||||
- Docker 20.10+
|
||||
- Bash
|
||||
|
||||
## Installation
|
||||
|
||||
### Depuis Docker
|
||||
|
||||
```bash
|
||||
# Build de l'image
|
||||
./build.sh
|
||||
|
||||
# Exécuter
|
||||
docker run -d \
|
||||
--name logcorrelator \
|
||||
-v /var/run/logcorrelator:/var/run/logcorrelator \
|
||||
-v /var/log/logcorrelator:/var/log/logcorrelator \
|
||||
-v ./config.conf:/etc/logcorrelator/logcorrelator.conf \
|
||||
logcorrelator:latest
|
||||
```
|
||||
|
||||
### Depuis le package RPM (Rocky Linux 8+)
|
||||
|
||||
```bash
|
||||
# Générer le RPM
|
||||
./build.sh
|
||||
|
||||
# Installer le package
|
||||
sudo rpm -ivh dist/logcorrelator-1.0.0.rpm
|
||||
|
||||
# Activer et démarrer le service
|
||||
sudo systemctl enable logcorrelator
|
||||
sudo systemctl start logcorrelator
|
||||
|
||||
# Vérifier le statut
|
||||
sudo systemctl status logcorrelator
|
||||
```
|
||||
|
||||
### Build manuel (sans Docker)
|
||||
|
||||
```bash
|
||||
# Prérequis: Go 1.21+
|
||||
go build -o logcorrelator ./cmd/logcorrelator
|
||||
|
||||
# Exécuter
|
||||
./logcorrelator -config config.example.conf
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
La configuration utilise un fichier texte simple avec des directives :
|
||||
|
||||
```bash
|
||||
# Format: directive value [value...]
|
||||
# Lignes starting with # sont des commentaires
|
||||
|
||||
service.name logcorrelator
|
||||
service.language go
|
||||
|
||||
# Inputs (au moins 2 requis)
|
||||
input.unix_socket apache_source /var/run/logcorrelator/apache.sock json
|
||||
input.unix_socket network_source /var/run/logcorrelator/network.sock json
|
||||
|
||||
# Outputs
|
||||
output.file.enabled true
|
||||
output.file.path /var/log/logcorrelator/correlated.log
|
||||
|
||||
output.clickhouse.enabled false
|
||||
output.clickhouse.dsn clickhouse://user:pass@localhost:9000/db
|
||||
output.clickhouse.table correlated_logs_http_network
|
||||
output.clickhouse.batch_size 500
|
||||
output.clickhouse.flush_interval_ms 200
|
||||
|
||||
# Corrélation
|
||||
correlation.key src_ip,src_port
|
||||
correlation.time_window.value 1
|
||||
correlation.time_window.unit s
|
||||
|
||||
# Politique des orphelins
|
||||
correlation.orphan_policy.apache_always_emit true
|
||||
correlation.orphan_policy.network_emit false
|
||||
```
|
||||
|
||||
### Directives disponibles
|
||||
|
||||
| Directive | Description | Défaut |
|
||||
|-----------|-------------|--------|
|
||||
| `service.name` | Nom du service | `logcorrelator` |
|
||||
| `service.language` | Langage | `go` |
|
||||
| `input.unix_socket` | Socket Unix (name path [format]) | Requis |
|
||||
| `output.file.enabled` | Activer sortie fichier | `true` |
|
||||
| `output.file.path` | Chemin fichier | `/var/log/logcorrelator/correlated.log` |
|
||||
| `output.clickhouse.enabled` | Activer ClickHouse | `false` |
|
||||
| `output.clickhouse.dsn` | DSN ClickHouse | - |
|
||||
| `output.clickhouse.table` | Table ClickHouse | - |
|
||||
| `output.clickhouse.batch_size` | Taille batch | `500` |
|
||||
| `output.clickhouse.flush_interval_ms` | Intervalle flush | `200` |
|
||||
| `output.clickhouse.max_buffer_size` | Buffer max | `5000` |
|
||||
| `output.clickhouse.drop_on_overflow` | Drop si overflow | `true` |
|
||||
| `output.stdout.enabled` | Sortie stdout (debug) | `false` |
|
||||
| `correlation.key` | Clés de corrélation | `src_ip,src_port` |
|
||||
| `correlation.time_window.value` | Valeur fenêtre | `1` |
|
||||
| `correlation.time_window.unit` | Unité (ms/s/m) | `s` |
|
||||
| `correlation.orphan_policy.apache_always_emit` | Émettre A seul | `true` |
|
||||
| `correlation.orphan_policy.network_emit` | Émettre B seul | `false` |
|
||||
|
||||
## Format des logs
|
||||
|
||||
### Source A (HTTP)
|
||||
|
||||
```json
|
||||
{
|
||||
"src_ip": "192.168.1.1",
|
||||
"src_port": 8080,
|
||||
"dst_ip": "10.0.0.1",
|
||||
"dst_port": 80,
|
||||
"timestamp": 1704110400000000000,
|
||||
"method": "GET",
|
||||
"path": "/api/test",
|
||||
"header_host": "example.com"
|
||||
}
|
||||
```
|
||||
|
||||
### Source B (Réseau)
|
||||
|
||||
```json
|
||||
{
|
||||
"src_ip": "192.168.1.1",
|
||||
"src_port": 8080,
|
||||
"dst_ip": "10.0.0.1",
|
||||
"dst_port": 443,
|
||||
"ja3": "abc123def456",
|
||||
"ja4": "xyz789"
|
||||
}
|
||||
```
|
||||
|
||||
### Log corrélé (sortie)
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2024-01-01T12:00:00Z",
|
||||
"src_ip": "192.168.1.1",
|
||||
"src_port": 8080,
|
||||
"dst_ip": "10.0.0.1",
|
||||
"dst_port": 80,
|
||||
"correlated": true,
|
||||
"apache": {"method": "GET", "path": "/api/test"},
|
||||
"network": {"ja3": "abc123def456"}
|
||||
}
|
||||
```
|
||||
|
||||
## Schema ClickHouse
|
||||
|
||||
```sql
|
||||
CREATE TABLE correlated_logs_http_network (
|
||||
timestamp DateTime64(9),
|
||||
src_ip String,
|
||||
src_port UInt32,
|
||||
dst_ip String,
|
||||
dst_port UInt32,
|
||||
correlated UInt8,
|
||||
orphan_side String,
|
||||
apache JSON,
|
||||
network JSON
|
||||
) ENGINE = MergeTree()
|
||||
ORDER BY (timestamp, src_ip, src_port);
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
# Via Docker
|
||||
./test.sh
|
||||
|
||||
# Local
|
||||
go test ./...
|
||||
go test -cover ./...
|
||||
go test -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
## Signaux
|
||||
|
||||
| Signal | Comportement |
|
||||
|--------|--------------|
|
||||
| `SIGINT` | Arrêt gracieux |
|
||||
| `SIGTERM` | Arrêt gracieux |
|
||||
|
||||
Lors de l'arrêt gracieux :
|
||||
1. Fermeture des sockets Unix
|
||||
2. Flush des buffers
|
||||
3. Émission des événements en attente
|
||||
4. Fermeture des connexions ClickHouse
|
||||
|
||||
## Logs internes
|
||||
|
||||
Les logs internes sont envoyés vers stderr :
|
||||
|
||||
```bash
|
||||
# Docker
|
||||
docker logs -f logcorrelator
|
||||
|
||||
# Systemd
|
||||
journalctl -u logcorrelator -f
|
||||
```
|
||||
|
||||
## Structure du projet
|
||||
|
||||
```
|
||||
.
|
||||
├── cmd/logcorrelator/ # Point d'entrée
|
||||
├── internal/
|
||||
│ ├── adapters/
|
||||
│ │ ├── inbound/unixsocket/
|
||||
│ │ └── outbound/
|
||||
│ │ ├── clickhouse/
|
||||
│ │ ├── file/
|
||||
│ │ └── multi/
|
||||
│ ├── app/ # Orchestration
|
||||
│ ├── config/ # Configuration
|
||||
│ ├── domain/ # Domaine (corrélation)
|
||||
│ ├── observability/ # Logging
|
||||
│ └── ports/ # Interfaces
|
||||
├── config.example.conf # Exemple de config
|
||||
├── Dockerfile # Build multi-stage
|
||||
├── build.sh # Script de build
|
||||
├── test.sh # Script de tests
|
||||
└── logcorrelator.service # Unité systemd
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user