fix(ja4ebpf): split bpf2go generate into Ja4Tc + Ja4Ssl, fix RPM systemd-rpm-macros
- Use two separate //go:generate directives (Ja4Tc for tc_capture.c, Ja4Ssl
for uprobe_ssl.c) to avoid duplicate LICENSE symbol and multi-file clang issue
- Update loader.go to hold tcObjs/sslObjs separately with correct field names:
UprobeSslSetFd, UprobeSslReadEntry, UretprobeSslReadExit,
KprobeAccept4Entry, KretprobeAccept4Exit
- Add systemd-rpm-macros to all three RPM build stages (el8/el9/el10)
so that %{_unitdir} macro resolves correctly
- RPMs now build successfully for el8, el9, el10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
@ -13,11 +13,9 @@
|
||||
|
||||
| Service | Runtime | Port par défaut |
|
||||
|---------|---------|-----------------|
|
||||
| sentinel | Go binary + libpcap (RPM) | — (capture réseau passive) |
|
||||
| correlator | Go binary (RPM) | 8080 (metrics, optionnel) |
|
||||
| mod-reqin-log | Apache module .so (RPM) | — (intégré à httpd) |
|
||||
| ja4ebpf | Go binary + eBPF CO-RE (RPM) | — (capture réseau passive) |
|
||||
| bot-detector | Python 3.11 (Docker) | 8080 (health check) |
|
||||
| dashboard | Python 3.11 / FastAPI (Docker) | 8000 (API) + 3000 (frontend) |
|
||||
| dashboard | Python 3.11 / FastAPI (Docker) | 8000 (API) |
|
||||
|
||||
---
|
||||
|
||||
@ -206,103 +204,64 @@ sudo chown -R clickhouse:clickhouse /var/lib/clickhouse/user_files/
|
||||
|
||||
---
|
||||
|
||||
## Étape 5 — Installation des services Go (RPM)
|
||||
## Étape 5 — Installation de ja4ebpf (RPM)
|
||||
|
||||
### Build des RPMs
|
||||
|
||||
```bash
|
||||
# Tous les RPMs (sentinel + correlator + mod-reqin-log) × 3 distros
|
||||
make rpm-all
|
||||
|
||||
# Ou individuellement
|
||||
make rpm-sentinel # → services/sentinel/dist/
|
||||
make rpm-correlator # → services/correlator/dist/
|
||||
make rpm-mod-reqin-log # → services/mod-reqin-log/dist/
|
||||
# RPM ja4ebpf × 3 distros (el8, el9, el10)
|
||||
make rpm-ja4ebpf
|
||||
# → services/ja4ebpf/dist/rpm/el{8,9,10}/
|
||||
```
|
||||
|
||||
Les RPMs sont générés dans `services/<nom>/dist/` avec un sous-dossier par distro (el8, el9, el10).
|
||||
Le build utilise un pipeline Docker multi-étapes Rocky Linux :
|
||||
1. L’étape `go-builder` compile le bytecode eBPF (clang/llvm) puis le binaire Go statique
|
||||
2. Les étapes `rpm-el8`, `rpm-el9`, `rpm-el10` exécutent `rpmbuild` pour chaque distro cible
|
||||
3. L’étape `alpine` collecte les RPMs via `--output type=local`
|
||||
|
||||
### Installation des RPMs
|
||||
### Installation du RPM
|
||||
|
||||
```bash
|
||||
# Sur le serveur cible (Rocky 9 par exemple)
|
||||
sudo yum install -y ./ja4sentinel-*.el9.x86_64.rpm
|
||||
sudo yum install -y ./logcorrelator-*.el9.x86_64.rpm
|
||||
sudo yum install -y ./mod_reqin_log-*.el9.x86_64.rpm
|
||||
sudo yum install -y ./ja4ebpf-*.el9.x86_64.rpm
|
||||
```
|
||||
|
||||
### Configuration du sentinel
|
||||
### Configuration de ja4ebpf
|
||||
|
||||
```bash
|
||||
# Fichier de configuration principal
|
||||
sudo cp /etc/ja4sentinel/config.yml.default /etc/ja4sentinel/config.yml
|
||||
sudo vi /etc/ja4sentinel/config.yml
|
||||
sudo cp /etc/ja4ebpf/config.yml.example /etc/ja4ebpf/config.yml
|
||||
sudo vi /etc/ja4ebpf/config.yml
|
||||
```
|
||||
|
||||
Variables d'environnement (dans `/etc/sysconfig/ja4sentinel` ou `.env`) :
|
||||
Variables d’environnement clés (dans `/etc/sysconfig/ja4ebpf`) :
|
||||
|
||||
| Variable | Défaut | Description |
|
||||
|----------|--------|-------------|
|
||||
| `JA4EBPF_INTERFACE` | `eth0` | Interface réseau à observer |
|
||||
| `JA4EBPF_CLICKHOUSE_DSN` | — | DSN ClickHouse (ex: `clickhouse://data_writer:pwd@host:9000/ja4_logs`) |
|
||||
| `JA4EBPF_TARGET_BINARY` | `/usr/sbin/httpd` | Binaire OpenSSL à hooker (uprobe SSL_read) |
|
||||
| `JA4EBPF_BATCH_SIZE` | `500` | Taille des batchs d’insertion ClickHouse |
|
||||
| `JA4EBPF_FLUSH_INTERVAL_MS` | `200` | Intervalle de flush (ms) |
|
||||
| `JA4EBPF_SESSION_TIMEOUT_MS` | `500` | Timeout orphelin (ms) |
|
||||
|
||||
```bash
|
||||
JA4SENTINEL_INTERFACE=eth0 # Interface réseau à capturer
|
||||
JA4SENTINEL_PORTS=443,8443 # Ports TLS à surveiller
|
||||
sudo systemctl enable --now ja4ebpf
|
||||
sudo systemctl status ja4ebpf
|
||||
journalctl -u ja4ebpf -f
|
||||
```
|
||||
|
||||
Le sentinel écrit vers le socket Unix du correlator : `/var/run/logcorrelator/network.socket`
|
||||
### Capabilities Linux requises (SELinux Enforcing)
|
||||
|
||||
```bash
|
||||
sudo systemctl enable --now ja4sentinel
|
||||
sudo systemctl status ja4sentinel
|
||||
journalctl -u ja4sentinel -f
|
||||
```
|
||||
`ja4ebpf` ne tourne **pas** en root absolu. Le fichier systemd utilise les capabilities minimales :
|
||||
|
||||
### Configuration du correlator
|
||||
| Capability | Raison |
|
||||
|------------|--------|
|
||||
| `CAP_BPF` | Chargement des programmes eBPF (kernel 5.8+) |
|
||||
| `CAP_SYS_ADMIN` | Requis pour les uprobes et RHEL 8 (kernel 4.18 <5.8) |
|
||||
| `CAP_NET_ADMIN` | Attachement des hooks TC ingress |
|
||||
| `CAP_PERFMON` | Accès aux perf events pour les uprobes |
|
||||
|
||||
```bash
|
||||
sudo cp /etc/logcorrelator/config.yml.default /etc/logcorrelator/config.yml
|
||||
sudo vi /etc/logcorrelator/config.yml
|
||||
```
|
||||
|
||||
Configuration minimale pour activer ClickHouse (`config.yml`) :
|
||||
|
||||
```yaml
|
||||
outputs:
|
||||
clickhouse:
|
||||
enabled: true
|
||||
dsn: clickhouse://data_writer:VotreMotDePasse@localhost:9000/ja4_logs
|
||||
table: http_logs_raw
|
||||
batch_size: 500
|
||||
flush_interval_ms: 200
|
||||
|
||||
file:
|
||||
enabled: true
|
||||
path: /var/log/logcorrelator/correlated.log
|
||||
```
|
||||
|
||||
Variable d'environnement alternative (dans `/etc/sysconfig/logcorrelator`) :
|
||||
|
||||
```bash
|
||||
LOGCORRELATOR_CLICKHOUSE_DSN=clickhouse://data_writer:VotreMotDePasse@localhost:9000/ja4_logs
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl enable --now logcorrelator
|
||||
sudo systemctl status logcorrelator
|
||||
```
|
||||
|
||||
### Configuration de mod-reqin-log
|
||||
|
||||
Le module Apache écrit les requêtes HTTP en JSON vers le socket Unix du correlator.
|
||||
|
||||
```bash
|
||||
# Le RPM installe automatiquement le module dans Apache
|
||||
# Vérifier le chargement
|
||||
httpd -M | grep reqin
|
||||
|
||||
# La configuration est dans /etc/httpd/conf.d/mod_reqin_log.conf
|
||||
# Le socket par défaut : /var/run/logcorrelator/http.socket
|
||||
sudo systemctl restart httpd
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Étape 6 — Installation des services Python (Docker)
|
||||
|
||||
@ -374,7 +333,7 @@ docker compose up -d
|
||||
|
||||
```bash
|
||||
# Services systemd (Go)
|
||||
sudo systemctl status ja4sentinel logcorrelator httpd
|
||||
sudo systemctl status ja4ebpf
|
||||
|
||||
# Services Docker (Python)
|
||||
docker compose -f services/bot-detector/docker-compose.yml ps
|
||||
@ -384,7 +343,7 @@ docker compose -f services/dashboard/docker-compose.yaml ps
|
||||
### 2. Vérifier l'ingestion des logs
|
||||
|
||||
```bash
|
||||
# Logs bruts ingérés par le correlator
|
||||
# Logs bruts ingérés par ja4ebpf
|
||||
clickhouse-client --query "SELECT count() FROM ja4_logs.http_logs_raw"
|
||||
|
||||
# Logs parsés par la vue matérialisée
|
||||
@ -480,27 +439,34 @@ Télécharge et génère tous les fichiers CSV de référence (bot IPs, JA4, ASN
|
||||
## Schéma réseau récapitulatif
|
||||
|
||||
```
|
||||
┌──────────────┐ Unix socket ┌──────────────┐ ClickHouse ┌──────────────────┐
|
||||
│ mod-reqin-log│──── http.socket ─────→│ │ INSERT INTO │ │
|
||||
│ (Apache C11) │ (source A) │ correlator │───→ ja4_logs. │ ClickHouse │
|
||||
└──────────────┘ │ │ http_logs_raw │ │
|
||||
│ │ │ ┌──────────────┐ │
|
||||
┌──────────────┐ Unix socket │ │ MV parse JSON │ │ ja4_logs │ │
|
||||
│ sentinel │──── network.socket ──→│ │ ↓ │ │ _raw → _logs│ │
|
||||
│ (Go+libpcap) │ (source B) └──────────────┘ ja4_logs. │ └──────────────┘ │
|
||||
└──────────────┘ http_logs │ │
|
||||
│ │ ┌──────────────┐ │
|
||||
6 MVs agrégation│ │ ja4_processing│ │
|
||||
↓ │ │ agg_* (×6) │ │
|
||||
┌──────────────┐ SELECT features ┌──────────────┐ ja4_processing. │ │ ml_* (×2) │ │
|
||||
│ bot-detector │←─── view_ai_features ─│ │ agg_*, view_* │ │ views, dicts│ │
|
||||
│ (Python 3.11)│ view_thesis_feat │ ClickHouse │ │ │ audit_logs │ │
|
||||
│ EIF+AE+XGB │ │ │ ml_all_scores │ └──────────────┘ │
|
||||
│ │───→ INSERT scores ────→│ │ ml_detected_* │ │
|
||||
└──────────────┘ └──────────────┘ └──────────────────┘
|
||||
↑
|
||||
┌──────────────┐ SELECT * │
|
||||
│ dashboard │←─── ja4_processing.ml_*, agg_*, views ────────────────────────────┘
|
||||
│ (FastAPI) │←─── ja4_logs.http_logs ───────────────────────────────────────────┘
|
||||
└──────────────┘
|
||||
Trafic HTTPS/HTTP (port 80/443)
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| ja4ebpf | TC ingress hook -- L3/L4/L5 (SYN, TLS ClientHello)
|
||||
| (eBPF CO-RE) | uprobe SSL_read -- L7 HTTPS (flux déchiffré)
|
||||
| | kprobe tcp_recvmsg - L7 HTTP (port 80/8080)
|
||||
+-----------------+
|
||||
|
|
||||
| INSERT batch (HTTP bulk)
|
||||
v
|
||||
+-----------------+ +------------------+
|
||||
| ClickHouse |-- MV parse JSON -> | ja4_logs. |
|
||||
| | | http_logs_raw |
|
||||
| | | http_logs |
|
||||
| |-- 6 MVs agg -----> | ja4_processing. |
|
||||
| | | agg_*(x6) |
|
||||
| | | ml_*(x2) |
|
||||
+-----------------+ +------------------+
|
||||
^
|
||||
|
|
||||
+-----------------+ SELECT view_ai_features
|
||||
| bot-detector |<-- view_thesis_features
|
||||
| (EIF+AE+XGB) |
|
||||
| |--> INSERT ml_all_scores, ml_detected_anomalies
|
||||
+-----------------+
|
||||
+-----------------+
|
||||
| dashboard |<-- ja4_processing.ml_*, agg_*, views
|
||||
| (FastAPI) |<-- ja4_logs.http_logs
|
||||
+-----------------+
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user