docs: README improvements - config, troubleshooting, structure
- Update RPM version numbers to 1.1.6 - Fix config file name (.yml not .conf) - Add complete configuration example with current schema - Add ClickHouse DSN format documentation - Add Troubleshooting section (ClickHouse, MV, sockets, systemd) - Update project structure with accurate file names Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
147
README.md
147
README.md
@ -71,9 +71,9 @@ docker run -d \
|
|||||||
make package-rpm
|
make package-rpm
|
||||||
|
|
||||||
# Installer le package RPM (Rocky Linux 8/9/10)
|
# Installer le package RPM (Rocky Linux 8/9/10)
|
||||||
sudo dnf install -y dist/rpm/rocky8/logcorrelator-1.0.7-1.el8.x86_64.rpm
|
sudo dnf install -y dist/rpm/el8/logcorrelator-1.1.6-1.el8.x86_64.rpm
|
||||||
sudo dnf install -y dist/rpm/rocky9/logcorrelator-1.0.7-1.el9.x86_64.rpm
|
sudo dnf install -y dist/rpm/el9/logcorrelator-1.1.6-1.el9.x86_64.rpm
|
||||||
sudo dnf install -y dist/rpm/almalinux10/logcorrelator-1.0.7-1.el10.x86_64.rpm
|
sudo dnf install -y dist/rpm/el10/logcorrelator-1.1.6-1.el10.x86_64.rpm
|
||||||
|
|
||||||
# Activer et démarrer le service
|
# Activer et démarrer le service
|
||||||
sudo systemctl enable logcorrelator
|
sudo systemctl enable logcorrelator
|
||||||
@ -95,51 +95,70 @@ go build -o logcorrelator ./cmd/logcorrelator
|
|||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
La configuration utilise un fichier YAML :
|
La configuration utilise un fichier YAML. Voir `config.example.yml` pour un exemple complet.
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# Service configuration
|
# /etc/logcorrelator/logcorrelator.yml
|
||||||
service:
|
|
||||||
name: logcorrelator
|
log:
|
||||||
language: go
|
level: INFO # DEBUG, INFO, WARN, ERROR
|
||||||
|
|
||||||
# Input sources (at least 2 required)
|
|
||||||
inputs:
|
inputs:
|
||||||
unix_sockets:
|
unix_sockets:
|
||||||
- name: http_source
|
# Source HTTP (A) : logs applicatifs en JSON
|
||||||
path: /var/run/logcorrelator/http.socket
|
- name: http
|
||||||
format: json
|
path: /var/run/logcorrelator/http.sock
|
||||||
- name: network_source
|
socket_permissions: "0666"
|
||||||
path: /var/run/logcorrelator/network.socket
|
socket_type: dgram
|
||||||
format: json
|
|
||||||
|
# Source réseau (B) : logs IP/TCP/JA3... en JSON
|
||||||
|
- name: network
|
||||||
|
path: /var/run/logcorrelator/network.sock
|
||||||
|
socket_permissions: "0666"
|
||||||
|
socket_type: dgram
|
||||||
|
|
||||||
# File output
|
|
||||||
outputs:
|
outputs:
|
||||||
file:
|
file:
|
||||||
enabled: true
|
enabled: true
|
||||||
path: /var/log/logcorrelator/correlated.log
|
path: /var/log/logcorrelator/correlated.log
|
||||||
|
format: json_lines
|
||||||
|
|
||||||
# ClickHouse output
|
|
||||||
outputs:
|
|
||||||
clickhouse:
|
clickhouse:
|
||||||
enabled: false
|
enabled: true
|
||||||
dsn: clickhouse://user:pass@localhost:9000/db
|
dsn: clickhouse://data_writer:password@localhost:9000/mabase_prod
|
||||||
table: correlated_logs_http_network
|
table: http_logs_raw
|
||||||
|
batch_size: 500
|
||||||
|
flush_interval_ms: 200
|
||||||
|
max_buffer_size: 5000
|
||||||
|
drop_on_overflow: true
|
||||||
|
|
||||||
# Correlation configuration
|
|
||||||
correlation:
|
correlation:
|
||||||
key:
|
|
||||||
- src_ip
|
|
||||||
- src_port
|
|
||||||
time_window:
|
time_window:
|
||||||
value: 1
|
value: 1
|
||||||
unit: s
|
unit: s
|
||||||
orphan_policy:
|
orphan_policy:
|
||||||
apache_always_emit: true
|
apache_always_emit: true
|
||||||
network_emit: false
|
network_emit: false
|
||||||
|
matching:
|
||||||
|
mode: one_to_many # Keep-Alive : un B peut corréler plusieurs A
|
||||||
|
buffers:
|
||||||
|
max_http_items: 10000
|
||||||
|
max_network_items: 20000
|
||||||
|
ttl:
|
||||||
|
network_ttl_s: 30
|
||||||
```
|
```
|
||||||
|
|
||||||
Exemple complet dans `config.example.yml`.
|
### Format du DSN ClickHouse
|
||||||
|
|
||||||
|
```
|
||||||
|
clickhouse://username:password@host:port/database
|
||||||
|
```
|
||||||
|
|
||||||
|
Exemple : `clickhouse://data_writer:MonMotDePasse@127.0.0.1:9000/mabase_prod`
|
||||||
|
|
||||||
|
Ports courants :
|
||||||
|
- `9000` : port natif (recommandé pour le driver Go)
|
||||||
|
- `8123` : port HTTP (alternative)
|
||||||
|
|
||||||
## Format des logs
|
## Format des logs
|
||||||
|
|
||||||
@ -482,23 +501,81 @@ journalctl -u logcorrelator -f
|
|||||||
├── cmd/logcorrelator/ # Point d'entrée
|
├── cmd/logcorrelator/ # Point d'entrée
|
||||||
├── internal/
|
├── internal/
|
||||||
│ ├── adapters/
|
│ ├── adapters/
|
||||||
│ │ ├── inbound/unixsocket/
|
│ │ ├── inbound/unixsocket/ # Sources HTTP et réseau
|
||||||
│ │ └── outbound/
|
│ │ └── outbound/
|
||||||
│ │ ├── clickhouse/
|
│ │ ├── clickhouse/ # Sink ClickHouse
|
||||||
│ │ ├── file/
|
│ │ ├── file/ # Sink fichier
|
||||||
│ │ └── multi/
|
│ │ └── multi/ # Multi-sink
|
||||||
│ ├── app/ # Orchestration
|
│ ├── app/ # Orchestration
|
||||||
│ ├── config/ # Configuration
|
│ ├── config/ # Configuration YAML
|
||||||
│ ├── domain/ # Domaine (corrélation)
|
│ ├── domain/ # Domaine (corrélation)
|
||||||
│ ├── observability/ # Logging
|
│ ├── observability/ # Logging interne
|
||||||
│ └── ports/ # Interfaces
|
│ └── ports/ # Interfaces
|
||||||
├── config.example.conf # Exemple de config
|
├── config.example.yml # Exemple de config
|
||||||
├── Dockerfile # Build multi-stage
|
├── Dockerfile # Build multi-stage
|
||||||
├── build.sh # Script de build
|
├── Dockerfile.package # Packaging RPM
|
||||||
├── test.sh # Script de tests
|
├── Makefile # Commandes de build
|
||||||
|
├── architecture.yml # Spécification architecture
|
||||||
└── logcorrelator.service # Unité systemd
|
└── logcorrelator.service # Unité systemd
|
||||||
```
|
```
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
MIT
|
MIT
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### ClickHouse : erreurs d'insertion
|
||||||
|
|
||||||
|
**Erreur : `No such column timestamp`**
|
||||||
|
- Vérifiez que la table de destination est bien `http_logs_raw` (colonne unique `raw_json`)
|
||||||
|
- Le service envoie un JSON sérialisé dans `raw_json`, pas des colonnes séparées
|
||||||
|
|
||||||
|
**Erreur : `ACCESS_DENIED`**
|
||||||
|
- Vérifiez les droits de l'utilisateur `data_writer` :
|
||||||
|
```sql
|
||||||
|
GRANT INSERT(raw_json) ON mabase_prod.http_logs_raw TO data_writer;
|
||||||
|
GRANT SELECT(raw_json) ON mabase_prod.http_logs_raw TO data_writer;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vue matérialisée ne fonctionne pas
|
||||||
|
|
||||||
|
**Symptôme :** `http_logs_raw` a des données, mais `http_logs` est vide
|
||||||
|
|
||||||
|
1. Vérifiez que la MV existe :
|
||||||
|
```sql
|
||||||
|
SHOW CREATE TABLE mv_http_logs;
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Vérifiez les droits SELECT pour `data_writer` sur `http_logs_raw`
|
||||||
|
|
||||||
|
3. Testez manuellement :
|
||||||
|
```sql
|
||||||
|
INSERT INTO mabase_prod.http_logs
|
||||||
|
SELECT * FROM mabase_prod.mv_http_logs
|
||||||
|
WHERE time > now() - INTERVAL 1 HOUR;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sockets Unix : permission denied
|
||||||
|
|
||||||
|
**Erreur :** `permission denied` sur `/var/run/logcorrelator/*.sock`
|
||||||
|
|
||||||
|
- Vérifiez que les sockets ont les permissions `0666`
|
||||||
|
- Vérifiez que l'utilisateur `logcorrelator` peut lire/écrire
|
||||||
|
|
||||||
|
### Service systemd ne démarre pas
|
||||||
|
|
||||||
|
1. Vérifiez les logs :
|
||||||
|
```bash
|
||||||
|
journalctl -u logcorrelator -n 50 --no-pager
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Vérifiez la configuration :
|
||||||
|
```bash
|
||||||
|
cat /etc/logcorrelator/logcorrelator.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Testez manuellement :
|
||||||
|
```bash
|
||||||
|
/usr/bin/logcorrelator -config /etc/logcorrelator/logcorrelator.yml
|
||||||
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user