docs: réécriture complète de la documentation des services en français
- bot-detector.md : architecture 11 modules, 77/65 features, ensemble triple voix (EIF+AE+XGBoost), browser 5 axes, HDBSCAN, toutes les variables d'environnement vérifiées depuis le code source - dashboard.md : corrigé stack (Jinja2+htmx, pas React+Vite), 14 pages + 35 API routes + health, dual-database, IPv4/IPv6 - python-ja4common.md : ajouté CLICKHOUSE_DB_PROCESSING/LOGS, schéma dual-database, note dashboard n'utilise pas ja4_common Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@ -1,307 +1,343 @@
|
||||
# Dashboard
|
||||
|
||||
The dashboard is a SOC (Security Operations Center) web application built with FastAPI (backend) and React (frontend) that provides real-time visualization, investigation, and analysis of bot detections generated by the [bot-detector](bot-detector.md). It queries ClickHouse (`ja4_processing`) for all data.
|
||||
Application web SOC (Security Operations Center) construite avec **FastAPI + Jinja2 + htmx**,
|
||||
offrant la visualisation en temps réel, l'investigation et l'analyse des détections de bots
|
||||
générées par le [bot-detector](bot-detector.md). Interroge ClickHouse sur deux bases de données
|
||||
(`ja4_processing` et `ja4_logs`).
|
||||
|
||||
## Technology Stack
|
||||
---
|
||||
|
||||
| Component | Technology |
|
||||
|-----------|-----------|
|
||||
## Pile technologique
|
||||
|
||||
| Composant | Technologie |
|
||||
|-----------|-------------|
|
||||
| Backend | Python 3.11 + FastAPI |
|
||||
| Frontend | React + Vite |
|
||||
| Database | ClickHouse (via `ja4_common` shared client) |
|
||||
| API Docs | Swagger UI (`/docs`) and ReDoc (`/redoc`) |
|
||||
| Templates | Jinja2 (rendu côté serveur) |
|
||||
| Interactions dynamiques | htmx (mises à jour partielles via JSON API) |
|
||||
| Graphiques | Chart.js + ECharts |
|
||||
| Style | Tailwind CSS (CDN) |
|
||||
| Base de données | ClickHouse via `clickhouse-connect` (client propre, **PAS** `ja4_common`) |
|
||||
| Documentation API | Swagger UI (`/docs`) + OpenAPI JSON (`/openapi.json`) |
|
||||
|
||||
> **Note :** le dashboard n'utilise **pas** la bibliothèque partagée `ja4_common`. Il possède
|
||||
> son propre client ClickHouse léger dans `backend/database.py`.
|
||||
|
||||
---
|
||||
|
||||
## Structure des fichiers
|
||||
|
||||
```
|
||||
services/dashboard/
|
||||
├── backend/
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py Application FastAPI, montage templates + static, CORS, health
|
||||
│ ├── config.py Variables d'environnement, safe_identifier()
|
||||
│ ├── database.py Client clickhouse-connect singleton, query(), query_scalar(), execute()
|
||||
│ └── routes/
|
||||
│ ├── __init__.py
|
||||
│ ├── api.py 35 routes JSON (1635 lignes)
|
||||
│ └── pages.py 14 routes de pages HTML (83 lignes)
|
||||
├── templates/
|
||||
│ ├── base.html Template de base (layout, navigation, Tailwind CDN)
|
||||
│ ├── overview.html Vue d'ensemble du dashboard
|
||||
│ ├── detections.html Tableau paginé des détections d'anomalies
|
||||
│ ├── scores.html Tableau paginé de tous les scores ML
|
||||
│ ├── traffic.html Tableau paginé du trafic HTTP brut
|
||||
│ ├── ip_detail.html Investigation détaillée d'une IP
|
||||
│ ├── ja4_detail.html Investigation détaillée d'une empreinte JA4
|
||||
│ ├── cluster_detail.html Investigation détaillée d'un cluster/campagne
|
||||
│ ├── campaigns.html Vue des campagnes HDBSCAN
|
||||
│ ├── features.html Analyse des features ML
|
||||
│ ├── models.html Métadonnées et performances des modèles
|
||||
│ ├── classify.html Interface de classification SOC
|
||||
│ ├── tactics.html Tactiques de détection (brute-force, rotation, récurrence)
|
||||
│ ├── reflists.html Listes de référence et dictionnaires ClickHouse
|
||||
│ └── network.html Graphe réseau des campagnes
|
||||
└── static/ Fichiers statiques (JS, CSS)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `CLICKHOUSE_HOST` | string | `clickhouse` | ClickHouse hostname |
|
||||
| `CLICKHOUSE_PORT` | int | `8123` | ClickHouse HTTP port |
|
||||
| `CLICKHOUSE_DB` | string | `ja4_processing` | Database name |
|
||||
| `CLICKHOUSE_USER` | string | `admin` | ClickHouse user |
|
||||
| `CLICKHOUSE_PASSWORD` | string | `""` | ClickHouse password |
|
||||
| `API_HOST` | string | `0.0.0.0` | API listen address |
|
||||
| `API_PORT` | int | `8000` | API listen port |
|
||||
| `CORS_ORIGINS` | list | `["http://localhost:3000", "http://127.0.0.1:3000"]` | Allowed CORS origins |
|
||||
Toute la configuration est lue via `os.getenv()` dans `backend/config.py`.
|
||||
|
||||
## API Reference
|
||||
| Variable | Type | Défaut | Description |
|
||||
|----------|------|--------|-------------|
|
||||
| `CLICKHOUSE_HOST` | str | `localhost` | Nom d'hôte du serveur ClickHouse |
|
||||
| `CLICKHOUSE_PORT` | int | `8123` | Port HTTP ClickHouse |
|
||||
| `CLICKHOUSE_USER` | str | `default` | Utilisateur ClickHouse |
|
||||
| `CLICKHOUSE_PASSWORD` | str | `""` | Mot de passe ClickHouse |
|
||||
| `CLICKHOUSE_DB_PROCESSING` | str | `ja4_processing` | Base de données ML/agrégations (fallback : `CLICKHOUSE_DB`) |
|
||||
| `CLICKHOUSE_DB_LOGS` | str | `ja4_logs` | Base de données des logs HTTP |
|
||||
| `API_HOST` | str | `0.0.0.0` | Adresse d'écoute du serveur |
|
||||
| `API_PORT` | int | `8000` | Port d'écoute du serveur |
|
||||
|
||||
All endpoints are prefixed with `/api/`. The dashboard exposes **74+ endpoints** across 20 routers.
|
||||
### Schéma dual-database
|
||||
|
||||
### Health
|
||||
Le dashboard utilise deux bases de données ClickHouse :
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/health` | Health check — returns ClickHouse connection status |
|
||||
- **`CLICKHOUSE_DB_PROCESSING`** (`ja4_processing`) : tables ML, agrégations, dictionnaires, feedback SOC
|
||||
- **`CLICKHOUSE_DB_LOGS`** (`ja4_logs`) : `http_logs`, trafic brut
|
||||
|
||||
Les noms de bases sont validés par `safe_identifier()` (regex alphanumérique + underscore) avant injection dans les requêtes SQL.
|
||||
|
||||
### Dualité IPv4/IPv6
|
||||
|
||||
- `http_logs.src_ip` est de type `IPv4`
|
||||
- Les tables ML (`ml_all_scores`, `ml_detected_anomalies`) stockent `IPv6` (mappé `::ffff:x.x.x.x`)
|
||||
- Utiliser `toIPv6()` pour les requêtes sur les tables ML
|
||||
- Utiliser `toIPv4OrZero()` pour les requêtes sur `http_logs`
|
||||
- Retirer le préfixe `::ffff:` à l'affichage
|
||||
|
||||
---
|
||||
|
||||
### Metrics (`/api/metrics`)
|
||||
## Couche base de données (`backend/database.py`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/metrics` | Global dashboard metrics: detection counts by threat level, unique IPs, time series |
|
||||
| GET | `/api/metrics/threats` | Threat distribution summary |
|
||||
| GET | `/api/metrics/baseline` | Human baseline statistics |
|
||||
Client `clickhouse-connect` avec singleton paresseux :
|
||||
|
||||
| Fonction | Signature | Description |
|
||||
|----------|-----------|-------------|
|
||||
| `get_client()` | `→ Client` | Singleton `clickhouse_connect.get_client()` |
|
||||
| `query(sql, params)` | `→ list[dict]` | Exécute un SELECT, retourne une liste de dicts |
|
||||
| `query_scalar(sql, params)` | `→ Any` | Exécute un SELECT, retourne une valeur scalaire |
|
||||
| `execute(sql, params)` | `→ None` | Exécute un DDL/DML sans retour |
|
||||
|
||||
Conversion automatique des types : `IPv4Address`, `IPv6Address`, `bytes` → chaînes JSON-friendly.
|
||||
|
||||
### Patron de requêtes paramétrées
|
||||
|
||||
```python
|
||||
from backend.config import DB_PROCESSING, DB_LOGS, safe_identifier
|
||||
from backend.database import query, query_scalar
|
||||
|
||||
_DB = safe_identifier(DB_PROCESSING)
|
||||
rows = query(
|
||||
f"SELECT src_ip, anomaly_score FROM {_DB}.ml_detected_anomalies "
|
||||
"WHERE src_ip = toIPv6({ip:String}) ORDER BY detected_at DESC LIMIT 10",
|
||||
{"ip": ip_value},
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Detections (`/api/detections`)
|
||||
## Routes de pages (14)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/detections` | Paginated detection list with filtering, sorting, and text search |
|
||||
| GET | `/api/detections/{detection_id}` | Single detection details |
|
||||
Toutes les pages sont rendues côté serveur via Jinja2 et utilisent htmx pour les mises à jour dynamiques.
|
||||
|
||||
**Query Parameters** (GET `/api/detections`):
|
||||
| # | Chemin | Template | Description |
|
||||
|---|--------|----------|-------------|
|
||||
| 1 | `/` | `overview.html` | Vue d'ensemble : statistiques 24h, distribution des menaces, top IPs, timeline |
|
||||
| 2 | `/detections` | `detections.html` | Tableau paginé des détections d'anomalies avec filtres |
|
||||
| 3 | `/scores` | `scores.html` | Tableau paginé de tous les scores ML avec filtres |
|
||||
| 4 | `/traffic` | `traffic.html` | Tableau paginé des logs HTTP bruts avec filtres |
|
||||
| 5 | `/ip/{ip}` | `ip_detail.html` | Investigation détaillée d'une IP (détections, scores, logs, features, récurrence) |
|
||||
| 6 | `/classify` | `classify.html` | Interface de classification SOC (feedback analyste) |
|
||||
| 7 | `/features` | `features.html` | Analyse des features ML (profils humain/bot, importance) |
|
||||
| 8 | `/models` | `models.html` | Métadonnées des modèles, statistiques de scoring |
|
||||
| 9 | `/network` | `network.html` | Graphe réseau des campagnes (IP ↔ JA4 partagés) |
|
||||
| 10 | `/campaigns` | `campaigns.html` | Campagnes HDBSCAN (clusters de bots agrégés) |
|
||||
| 11 | `/ja4/{fingerprint:path}` | `ja4_detail.html` | Investigation détaillée d'une empreinte JA4 |
|
||||
| 12 | `/cluster/{cid}` | `cluster_detail.html` | Investigation détaillée d'un cluster |
|
||||
| 13 | `/tactics` | `tactics.html` | Tactiques de détection (brute-force, rotation JA4/UA, récurrence) |
|
||||
| 14 | `/reflists` | `reflists.html` | Listes de référence et dictionnaires ClickHouse |
|
||||
|
||||
| Parameter | Type | Description |
|
||||
---
|
||||
|
||||
## Routes API (35)
|
||||
|
||||
Toutes les routes sont préfixées par `/api` et retournent du JSON. Utilisées par htmx depuis les templates et consommables par des clients externes.
|
||||
|
||||
### Vue d'ensemble et statistiques
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 1 | GET | `/api/overview` | Résumé du dashboard : détections/scores/trafic 24h, distribution des menaces, top IPs, timeline, modèles, statistiques navigateur |
|
||||
| 2 | GET | `/api/heatmap` | Heatmap temporelle (heure × jour de semaine) depuis `http_logs` |
|
||||
| 3 | GET | `/api/geo` | Répartition géographique et ASN des sessions |
|
||||
| 4 | GET | `/api/alerts` | Flux d'alertes en direct (détections récentes HIGH/CRITICAL) |
|
||||
| 5 | GET | `/api/timeline-detail` | Détail horaire de la timeline par niveau de menace |
|
||||
|
||||
### Détections et scores
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 6 | GET | `/api/detections` | Liste paginée des détections d'anomalies avec filtres (threat_level, search, asn, country, ja4, bot, browser) et tri |
|
||||
| 7 | GET | `/api/scores` | Liste paginée des scores ML avec filtres similaires |
|
||||
| 8 | GET | `/api/traffic` | Liste paginée des logs HTTP bruts avec filtres |
|
||||
|
||||
### Investigation IP
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 9 | GET | `/api/ip/{ip}` | Détail IP : détections, scores, logs HTTP, features AI, récurrence |
|
||||
| 10 | GET | `/api/ip/{ip}/radar` | Données pour graphique radar : IP vs baseline ISP vs baseline bot |
|
||||
|
||||
### Features et modèles
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 11 | GET | `/api/features` | Statistiques des features AI/thèse, profils humain/bot, importance (variance) |
|
||||
| 12 | GET | `/api/models` | Métadonnées des modèles depuis fichiers JSON + statistiques de scoring ClickHouse |
|
||||
| 13 | GET | `/api/models/timeline` | Volume horaire de scoring par modèle (7 jours) |
|
||||
| 14 | GET | `/api/models/threats` | Répartition des niveaux de menace par modèle |
|
||||
|
||||
### Empreintes et navigateurs
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 15 | GET | `/api/fingerprints` | Analyse des empreintes JA4 avec mapping de bots |
|
||||
| 16 | GET | `/api/browsers` | Distribution des familles de navigateurs via JA4 |
|
||||
| 17 | GET | `/api/behavior` | Données scatter des features comportementales + distributions |
|
||||
|
||||
### Classification SOC
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 18 | GET | `/api/classify/stats` | Résumé statistique des classifications SOC |
|
||||
| 19 | GET | `/api/classify/suggested` | Top des IPs non classifiées triées par sévérité |
|
||||
| 20 | POST | `/api/classify` | Soumettre un feedback analyste SOC (bot/legitimate/suspicious) |
|
||||
| 21 | GET | `/api/classifications` | Entrées de feedback SOC récentes |
|
||||
|
||||
### Campagnes et clusters
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 22 | GET | `/api/campaigns` | Clusters de campagnes HDBSCAN (agrégés) |
|
||||
| 23 | GET | `/api/campaigns/graph` | Données de graphe réseau (nœuds IP + arêtes JA4 partagés) |
|
||||
| 24 | GET | `/api/campaigns/scatter` | Scatter plot : score d'anomalie vs vélocité par IP et campagne |
|
||||
| 25 | GET | `/api/campaigns/{cid}` | Détail campagne : membres, profil, timeline |
|
||||
|
||||
### Investigation JA4 et cluster
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 26 | GET | `/api/ja4/{fingerprint:path}` | Investigation d'empreinte JA4 (IPs, scores, logs, features) |
|
||||
| 27 | GET | `/api/cluster/{cid}` | Investigation enrichie de cluster (profil, membres, graphe, répartitions) |
|
||||
|
||||
### Tactiques de détection
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 28 | GET | `/api/brute-force` | Détection de brute-force / credential stuffing |
|
||||
| 29 | GET | `/api/ja4-rotation` | Détection de rotation d'empreintes JA4 (évasion) |
|
||||
| 30 | GET | `/api/recurrence` | IPs de menaces persistantes/récurrentes |
|
||||
| 31 | GET | `/api/cascade/{ip}` | Cascade de ressources pour détection de navigateur headless |
|
||||
| 32 | GET | `/api/ua-rotation` | Détection de rotation de User-Agent |
|
||||
|
||||
### Dictionnaires et listes de référence
|
||||
|
||||
| # | Méthode | Chemin | Description |
|
||||
|---|---------|--------|-------------|
|
||||
| 33 | GET | `/api/dictionaries` | Métadonnées des dictionnaires ClickHouse |
|
||||
| 34 | GET | `/api/reflist/{name}` | Contenu paginé d'une liste de référence / dictionnaire |
|
||||
| 35 | GET | `/api/reflist/{name}/stats` | Statistiques agrégées d'une liste de référence |
|
||||
|
||||
### Autres routes
|
||||
|
||||
| Méthode | Chemin | Description |
|
||||
|---------|--------|-------------|
|
||||
| GET | `/health` | Point de santé — retourne l'état de connexion ClickHouse |
|
||||
| — | `/static` | Montage de fichiers statiques (`backend/static/`) |
|
||||
| GET | `/docs` | Documentation Swagger UI (auto-généré par FastAPI) |
|
||||
| GET | `/openapi.json` | Schéma OpenAPI (auto-généré par FastAPI) |
|
||||
|
||||
**Total : 14 pages + 35 API + 1 health = 50 routes applicatives**
|
||||
|
||||
---
|
||||
|
||||
## Paramètres de requête courants
|
||||
|
||||
Les endpoints paginés (`/api/detections`, `/api/scores`, `/api/traffic`) acceptent :
|
||||
|
||||
| Paramètre | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `page` | int | Page number (default: 1) |
|
||||
| `page_size` | int | Items per page (default: 20) |
|
||||
| `threat_level` | string | Filter by threat level |
|
||||
| `model_name` | string | Filter by model name |
|
||||
| `search` | string | Full-text search across IP, JA4, host, bot_name |
|
||||
| `sort_by` | string | Sort field |
|
||||
| `sort_order` | string | `asc` or `desc` |
|
||||
| `page` | int | Numéro de page (défaut : 1) |
|
||||
| `page_size` | int | Éléments par page (défaut : 20) |
|
||||
| `threat_level` | str | Filtre par niveau de menace |
|
||||
| `model_name` | str | Filtre par nom de modèle |
|
||||
| `search` | str | Recherche plein texte (IP, JA4, host, bot_name) |
|
||||
| `sort_by` | str | Champ de tri (validé contre une whitelist) |
|
||||
| `sort_order` | str | `asc` ou `desc` |
|
||||
|
||||
---
|
||||
|
||||
### Investigation (`/api/investigation`)
|
||||
## Workflow SOC
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/investigation/{ip}/summary` | **Primary investigation endpoint.** Aggregates ML score, brute-force, TCP spoofing, JA4 rotation, persistence, and 24h timeline into a single response with a `risk_score` (0–100) |
|
||||
Le dashboard est conçu pour le workflow d'un analyste SOC :
|
||||
|
||||
1. **Vue d'ensemble** (`/`) — identifier les tendances de menaces 24h, les pics d'anomalies
|
||||
2. **Détections** (`/detections`) — filtrer les anomalies par sévérité, rechercher des IPs spécifiques
|
||||
3. **Investigation IP** (`/ip/{ip}`) — examiner les détections, scores, logs HTTP, features ML, radar de comparaison
|
||||
4. **Analyse JA4** (`/ja4/{fp}`) — investiguer une empreinte TLS, voir toutes les IPs associées
|
||||
5. **Campagnes** (`/campaigns`) — visualiser les clusters de bots, graphe réseau, scatter plot
|
||||
6. **Cluster** (`/cluster/{cid}`) — plonger dans un cluster spécifique, examiner les membres et profils
|
||||
7. **Tactiques** (`/tactics`) — surveiller le brute-force, la rotation JA4/UA, les menaces récurrentes
|
||||
8. **Classification** (`/classify`) — soumettre un feedback analyste (bot/légitime/suspect) pour alimenter XGBoost
|
||||
9. **Features** (`/features`) — comparer les profils de features humain vs bot
|
||||
10. **Modèles** (`/models`) — surveiller les performances des modèles ML
|
||||
|
||||
---
|
||||
|
||||
### Reputation (`/api/reputation`)
|
||||
## Architecture des templates
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/reputation/ip/{ip_address}` | Full IP reputation from IP-API.com and IPinfo.io (proxy, VPN, Tor, hosting detection) |
|
||||
| GET | `/api/reputation/ip/{ip_address}/summary` | Simplified reputation summary |
|
||||
### Template de base (`base.html`)
|
||||
|
||||
Fournit le layout commun :
|
||||
|
||||
- Navigation latérale avec liens vers toutes les pages
|
||||
- Chargement CDN de Tailwind CSS, htmx, Chart.js, ECharts
|
||||
- Bloc `{% block content %}` pour le contenu de chaque page
|
||||
|
||||
### Rendu côté serveur + htmx
|
||||
|
||||
Chaque page est rendue côté serveur par Jinja2. Les interactions dynamiques utilisent htmx :
|
||||
|
||||
```html
|
||||
<!-- Exemple : chargement dynamique des détections -->
|
||||
<div hx-get="/api/detections?page=1&page_size=20"
|
||||
hx-trigger="load"
|
||||
hx-target="#detections-table">
|
||||
</div>
|
||||
```
|
||||
|
||||
Les graphiques sont rendus côté client par Chart.js et ECharts à partir des données JSON de l'API.
|
||||
|
||||
---
|
||||
|
||||
### Analysis (`/api/analysis`)
|
||||
## CORS
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/analysis/{ip}/subnet` | Subnet analysis for an IP (related IPs in same /24) |
|
||||
| GET | `/api/analysis/{ip}/country` | Country-level analysis for an IP |
|
||||
| GET | `/api/analysis/country` | Global country analysis across all detections |
|
||||
| GET | `/api/analysis/{ip}/ja4` | JA4 fingerprint analysis for an IP |
|
||||
| GET | `/api/analysis/{ip}/user-agents` | User-agent analysis for an IP |
|
||||
| GET | `/api/analysis/{ip}/recommendation` | SOC classification recommendation |
|
||||
| POST | `/api/analysis/classifications` | Create a classification (legitimate/suspicious/malicious) |
|
||||
| GET | `/api/analysis/classifications` | List all classifications |
|
||||
| GET | `/api/analysis/classifications/stats` | Classification statistics |
|
||||
Le middleware CORS est configuré de manière permissive dans `main.py` :
|
||||
|
||||
```python
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
```
|
||||
|
||||
L'application est principalement SSR mais l'API JSON est aussi consommable par des clients externes.
|
||||
|
||||
---
|
||||
|
||||
### Entities (`/api/entities`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/entities/types` | List available entity types |
|
||||
| GET | `/api/entities/subnet/{subnet}` | Investigate a subnet |
|
||||
| GET | `/api/entities/{entity_type}/{entity_value}` | Investigate any entity (IP, JA4, subnet, UA, host) |
|
||||
| GET | `/api/entities/{entity_type}/{entity_value}/related` | Related entities |
|
||||
| GET | `/api/entities/{entity_type}/{entity_value}/user_agents` | User-agents for entity |
|
||||
| GET | `/api/entities/{entity_type}/{entity_value}/client_headers` | Client headers for entity |
|
||||
| GET | `/api/entities/{entity_type}/{entity_value}/paths` | URL paths for entity |
|
||||
| GET | `/api/entities/{entity_type}/{entity_value}/query_params` | Query parameters for entity |
|
||||
|
||||
---
|
||||
|
||||
### Incidents (`/api/incidents`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/incidents` | List all incidents |
|
||||
| GET | `/api/incidents/clusters` | Active incident clusters (behavioral similarity grouping) |
|
||||
| GET | `/api/incidents/{cluster_id}` | Incident cluster details |
|
||||
| POST | `/api/incidents/{cluster_id}/classify` | Classify an incident cluster |
|
||||
|
||||
---
|
||||
|
||||
### Fingerprints (`/api/fingerprints`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/fingerprints/spoofing` | TLS fingerprint spoofing detection |
|
||||
| GET | `/api/fingerprints/ja4-ua-matrix` | JA4 ↔ User-Agent correlation matrix |
|
||||
| GET | `/api/fingerprints/ua-analysis` | Suspicious user-agent analysis |
|
||||
| GET | `/api/fingerprints/ip/{ip}/coherence` | Fingerprint coherence analysis per IP |
|
||||
| GET | `/api/fingerprints/legitimate-ja4` | Known legitimate JA4 fingerprints |
|
||||
| GET | `/api/fingerprints/asn-correlation` | JA4-ASN correlation analysis |
|
||||
|
||||
---
|
||||
|
||||
### Brute Force (`/api/bruteforce`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/bruteforce/targets` | Brute-force target hosts |
|
||||
| GET | `/api/bruteforce/attackers` | Brute-force attacker IPs |
|
||||
| GET | `/api/bruteforce/timeline` | Brute-force attack timeline |
|
||||
| GET | `/api/bruteforce/host/{host}/attackers` | Attackers for a specific host |
|
||||
|
||||
---
|
||||
|
||||
### TCP Spoofing (`/api/tcp-spoofing`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/tcp-spoofing/overview` | TCP/OS fingerprint spoofing overview |
|
||||
| GET | `/api/tcp-spoofing/list` | Spoofing detection list |
|
||||
| GET | `/api/tcp-spoofing/matrix` | TTL × MSS anomaly matrix |
|
||||
|
||||
---
|
||||
|
||||
### Header Fingerprint (`/api/headers`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/headers/clusters` | Header fingerprint clusters (suspicious patterns) |
|
||||
| GET | `/api/headers/cluster/{hash}/ips` | IPs sharing a header fingerprint |
|
||||
|
||||
---
|
||||
|
||||
### Heatmap (`/api/heatmap`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/heatmap/hourly` | Hourly traffic heatmap |
|
||||
| GET | `/api/heatmap/top-hosts` | Top hosts by traffic volume |
|
||||
| GET | `/api/heatmap/matrix` | Activity/hour matrix |
|
||||
|
||||
---
|
||||
|
||||
### Botnets (`/api/botnets`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/botnets/ja4-spread` | JA4 geographic spread (botnet indicator) |
|
||||
| GET | `/api/botnets/ja4/{ja4}/countries` | Country distribution for a JA4 fingerprint |
|
||||
| GET | `/api/botnets/summary` | Global botnet detection summary |
|
||||
|
||||
---
|
||||
|
||||
### Rotation (`/api/rotation`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/rotation/ja4-rotators` | IPs rotating JA4 fingerprints (evasion detection) |
|
||||
| GET | `/api/rotation/persistent-threats` | Persistent threats across time windows |
|
||||
| GET | `/api/rotation/ip/{ip}/ja4-history` | JA4 fingerprint history for an IP |
|
||||
| GET | `/api/rotation/sophistication` | Sophistication score analysis |
|
||||
| GET | `/api/rotation/proactive-hunt` | Proactive threat hunting suggestions |
|
||||
|
||||
---
|
||||
|
||||
### ML Features (`/api/ml`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/ml/top-anomalies` | Top anomalies with feature details |
|
||||
| GET | `/api/ml/ip/{ip}/radar` | Feature radar chart data for an IP |
|
||||
| GET | `/api/ml/score-distribution` | Anomaly score distribution histogram |
|
||||
| GET | `/api/ml/score-trends` | Score trends over time |
|
||||
| GET | `/api/ml/b-features` | Source B (TCP/TLS) feature analysis |
|
||||
| GET | `/api/ml/campaigns` | ML-detected campaign analysis |
|
||||
| GET | `/api/ml/scatter` | Feature scatter plot data |
|
||||
|
||||
---
|
||||
|
||||
### Attributes (`/api/attributes`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/attributes/{attr_type}` | List distinct values for an attribute (ja4, user_agent, asn, country, host) with counts |
|
||||
|
||||
---
|
||||
|
||||
### Variability (`/api/variability`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/variability/{attr_type}/{value}` | Behavioral variability analysis for an attribute value |
|
||||
| GET | `/api/variability/{attr_type}/{value}/ips` | IPs associated with an attribute value |
|
||||
| GET | `/api/variability/{attr_type}/{value}/attributes` | Attribute breakdown for a value |
|
||||
| GET | `/api/variability/{attr_type}/{value}/user_agents` | User-agents for an attribute value |
|
||||
|
||||
---
|
||||
|
||||
### Clustering (`/api/clustering`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/clustering/status` | Clustering cache status |
|
||||
| GET | `/api/clustering/clusters` | K-Means cluster list |
|
||||
| GET | `/api/clustering/cluster/{cluster_id}/points` | Data points in a cluster |
|
||||
| GET | `/api/clustering/cluster/{cluster_id}/ips` | IPs in a cluster |
|
||||
|
||||
---
|
||||
|
||||
### Search (`/api/search`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/search/quick` | Cross-entity search (IP, JA4, host, UA, country, ASN) |
|
||||
|
||||
---
|
||||
|
||||
### Audit (`/api/audit`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| POST | `/api/audit/logs` | Create an audit log entry |
|
||||
| GET | `/api/audit/logs` | Query audit logs (filtered, paginated) |
|
||||
| GET | `/api/audit/stats` | Audit statistics |
|
||||
| GET | `/api/audit/users/activity` | Per-user activity summary |
|
||||
|
||||
## Frontend Structure
|
||||
|
||||
The React frontend is built with Vite and served as static assets:
|
||||
|
||||
- **Entry point**: `/` → `frontend/dist/index.html`
|
||||
- **Static assets**: `/assets/*` → `frontend/dist/assets/`
|
||||
- **SPA routing**: All non-`/api/` paths fall through to `index.html` (React Router)
|
||||
- **API proxy**: Frontend calls `/api/*` which is handled by FastAPI routers
|
||||
|
||||
## Services
|
||||
|
||||
### IPReputationService
|
||||
|
||||
Queries public IP reputation databases (IP-API.com, IPinfo.io) without API keys:
|
||||
- Proxy/VPN/Tor detection
|
||||
- ASN, country, ISP information
|
||||
- Hosting provider identification
|
||||
|
||||
### ClusteringEngine
|
||||
|
||||
K-Means clustering on ML features with caching:
|
||||
- Automatic cluster count selection
|
||||
- Feature normalization via StandardScaler
|
||||
- In-memory cache with TTL
|
||||
|
||||
## Deployment
|
||||
## Déploiement
|
||||
|
||||
```bash
|
||||
# Build Docker image
|
||||
# Construction de l'image Docker
|
||||
make build-dashboard
|
||||
|
||||
# Run tests
|
||||
# Tests
|
||||
make test-dashboard
|
||||
|
||||
# Run locally (development)
|
||||
# Exécution locale (développement)
|
||||
cd services/dashboard
|
||||
uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Health Check
|
||||
### Point de santé
|
||||
|
||||
```
|
||||
GET /health → {"status": "healthy", "clickhouse": "connected"}
|
||||
|
||||
Reference in New Issue
Block a user