feat: roadmap détection bots §2-9 — HTTP/2, cohérence, drift, flotte, Jaccard, ExIFFI, méta-learner, métriques
Étape 2 — Fingerprinting HTTP/2 dans le pipeline ML : - Ajout du dictionnaire dict_browser_h2 (11 familles de navigateurs) dans 05_aggregation_tables.sql - Ajout du CTE h2_agg et 4 features HTTP/2 dans 07_ai_features_view.sql : h2_settings_known, h2_pseudo_order_match, h2_ja4_coherence, h2_settings_rare - Calcul du fingerprint_coherence_score (5 axes pondérés) dans la vue - Ajout du 6e axe axis_h2_coherence dans browser.py (poids rééquilibrés) - browser_h2.csv : 11 fingerprints Akamai → famille navigateur Étape 3 — Pré-filtre de cohérence sur la baseline humaine : - pipeline.py exclut les sessions avec fingerprint_coherence_score < seuil de la baseline d'entraînement - FINGERPRINT_COHERENCE_THRESHOLD configurable via env (défaut 0.25) - Log des sessions exclues pour analyse SOC Étape 4 — Détection de drift améliorée : - scoring.py : passage de 5 à 9 quantiles (p5…p95) - Ajout de la divergence KL en complément du test KS - Détection de drift adversarial (≥80% des features dérivent dans la même direction) - Split temporel strict pour la validation Étape 5 — Graphe bipartite JA4×ASN (§5.2) : - fleet.py : détection de flottes via NetworkX + Louvain (imports optionnels) - enrich_with_fleet_score() : ajout fleet_score + fleet_campaign_flag au DataFrame - cycle.py : appel après preprocess_df avec log du nombre de sessions en flotte - SQL migration 05_fleet_metrics_tables.sql : table fleet_detections (TTL 7j) - Dashboard : /fleet + /api/fleet (communautés détectées) + template fleet.html Étape 6 — Cross-domain Jaccard §5.8 : - 12_thesis_features.sql : CTE jaccard_paths → cross_domain_path_similarity - Signal : même chemins (/admin, /wp-login) sur plusieurs hosts = scanner Étape 7 — ExIFFI + erreurs AE par feature : - scoring.py : compute_exiffi_importance() par permutation, compute_ae_feature_errors() - pipeline.py : calcul ExIFFI sur X_test, mapping index → dict pour anomalies - build_reason() enrichi avec exiffi_top quand SHAP inactif Étape 8 — Méta-learner pour la pondération de l'ensemble : - scoring.py : classe MetaLearner (LogisticRegression, fallback poids fixes <1000 labels) - Collecte des labels depuis le cycle courant (known_bots, légitimes, Anubis) - pipeline.py : remplacement des poids fixes par MetaLearner.predict() Étape 9 — Métriques de performance et monitoring : - metrics.py : record_cycle_metrics() — taux anomalie, drift, corrélation, latence - SQL migration 05_fleet_metrics_tables.sql : table ml_performance_metrics (TTL 90j) - Dashboard : /health + /api/health + template health.html - cycle.py : appel record_cycle_metrics en fin de cycle (Complet + Applicatif) Tests : 36/36 bot-detector tests passent Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@ -155,6 +155,10 @@
|
||||
<svg class="w-4 h-4 shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 10V3L4 14h7v7l9-11h-7z"/></svg>
|
||||
<span class="nav-text">Tactiques</span>
|
||||
</a>
|
||||
<a href="/fleet" class="nav-item {% if active_page == 'fleet' %}active{% endif %}" title="Flottes JA4×ASN (§5.2)">
|
||||
<svg class="w-4 h-4 shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M3.055 11H5a2 2 0 012 2v1a2 2 0 002 2 2 2 0 012 2v2.945M8 3.935V5.5A2.5 2.5 0 0010.5 8h.5a2 2 0 012 2 2 2 0 104 0 2 2 0 012-2h1.064M15 20.488V18a2 2 0 012-2h3.064M21 12a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
|
||||
<span class="nav-text">Flottes</span>
|
||||
</a>
|
||||
|
||||
<div class="nav-group-title">Investigation</div>
|
||||
<a href="/traffic" class="nav-item {% if active_page == 'traffic' %}active{% endif %}" title="Logs HTTP bruts">
|
||||
@ -183,6 +187,10 @@
|
||||
<svg class="w-4 h-4 shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 7v10c0 2.21 3.58 4 8 4s8-1.79 8-4V7M4 7c0 2.21 3.58 4 8 4s8-1.79 8-4M4 7c0-2.21 3.58-4 8-4s8 1.79 8 4m0 5c0 2.21-3.58 4-8 4s-8-1.79-8-4"/></svg>
|
||||
<span class="nav-text">Listes réf.</span>
|
||||
</a>
|
||||
<a href="/health" class="nav-item {% if active_page == 'health' %}active{% endif %}" title="Santé du pipeline ML (§9)">
|
||||
<svg class="w-4 h-4 shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 19v-6a2 2 0 00-2-2H5a2 2 0 00-2 2v6a2 2 0 002 2h2a2 2 0 002-2zm0 0V9a2 2 0 012-2h2a2 2 0 012 2v10m-6 0a2 2 0 002 2h2a2 2 0 002-2m0 0V5a2 2 0 012-2h2a2 2 0 012 2v14a2 2 0 01-2 2h-2a2 2 0 01-2-2z"/></svg>
|
||||
<span class="nav-text">Santé ML</span>
|
||||
</a>
|
||||
</nav>
|
||||
<!-- Footer -->
|
||||
<div class="px-3 py-3 border-t border-gray-800 shrink-0">
|
||||
|
||||
226
services/dashboard/backend/templates/fleet.html
Normal file
226
services/dashboard/backend/templates/fleet.html
Normal file
@ -0,0 +1,226 @@
|
||||
{% extends "base.html" %}
|
||||
{% block page_title %}Flottes JA4×ASN — §5.2{% endblock %}
|
||||
{% block content %}
|
||||
<div class="p-4 lg:p-6 space-y-4 max-w-[1920px] mx-auto">
|
||||
|
||||
<!-- ═══ Header KPIs ═══ -->
|
||||
<div class="flex flex-wrap items-center gap-4 mb-2">
|
||||
<h1 class="text-xl font-bold text-white flex items-center gap-2">
|
||||
<svg class="w-6 h-6 text-cyan-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M3.055 11H5a2 2 0 012 2v1a2 2 0 002 2 2 2 0 012 2v2.945M8 3.935V5.5A2.5 2.5 0 0010.5 8h.5a2 2 0 012 2 2 2 0 104 0 2 2 0 012-2h1.064M15 20.488V18a2 2 0 012-2h3.064M21 12a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
|
||||
Flottes JA4×ASN
|
||||
</h1>
|
||||
<div class="ml-auto flex items-center gap-3">
|
||||
<div class="text-center px-3">
|
||||
<div class="text-2xl font-bold text-cyan-400" id="kpi-total">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">Flottes</div>
|
||||
</div>
|
||||
<div class="text-center px-3 border-l border-gray-700">
|
||||
<div class="text-2xl font-bold text-red-400" id="kpi-ips">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">IPs impliquées</div>
|
||||
</div>
|
||||
<div class="text-center px-3 border-l border-gray-700">
|
||||
<div class="text-2xl font-bold text-amber-400" id="kpi-maxscore">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">Score max</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ Doc banner ═══ -->
|
||||
<div class="bg-gray-900/50 border border-gray-800 rounded-lg px-4 py-3 text-xs text-gray-400 leading-relaxed">
|
||||
<strong class="text-cyan-300">Détection de flottes §5.2</strong> — Analyse du graphe bipartite
|
||||
<strong>G = (JA4 ∪ ASN, E)</strong> pour identifier les flottes de bots coordonnées qui
|
||||
font tourner leurs fingerprints JA4 et ASN. Les communautés suspectes sont détectées via
|
||||
l'algorithme Louvain (ou composantes connexes en fallback).
|
||||
<br><strong>fleet_score = taille × densité / log₂(n_asn + 2).</strong>
|
||||
Un score ≥ 2.0 indique une communauté coordonnée. Chaque ligne représente une communauté
|
||||
avec les fingerprints JA4 et ASNs impliqués.
|
||||
<br><strong>Action SOC :</strong> Bloquer les ASNs ou les JA4 d'une flotte peut neutraliser
|
||||
des milliers de bots en une seule règle.
|
||||
</div>
|
||||
|
||||
<!-- ═══ Table des flottes détectées ═══ -->
|
||||
<div class="section-card overflow-hidden">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-cyan-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M3.055 11H5a2 2 0 012 2v1a2 2 0 002 2 2 2 0 012 2v2.945M8 3.935V5.5A2.5 2.5 0 0010.5 8h.5a2 2 0 012 2 2 2 0 104 0 2 2 0 012-2h1.064"/></svg>
|
||||
Communautés suspectes (7 jours)
|
||||
<span class="relative inline-block"><button onclick="docToggle(this)" class="doc-btn">ⓘ</button><div class="doc-panel">
|
||||
<h4>Flottes JA4×ASN</h4>
|
||||
<p>Chaque ligne = une communauté du graphe bipartite. Les flottes avec un
|
||||
<strong>fleet_score</strong> élevé sont les plus coordonnées.</p>
|
||||
<p><strong>ja4_set :</strong> Fingerprints TLS utilisés par la flotte.</p>
|
||||
<p><strong>asn_set :</strong> Réseaux (ASN) de la flotte.</p>
|
||||
<p><strong>n_ips :</strong> Nombre d'IPs distinctes dans la communauté.</p>
|
||||
<p class="doc-source">Source : fleet_detections (7j)</p>
|
||||
</div></span>
|
||||
</span>
|
||||
<span id="load-status" class="text-[10px] text-gray-500">Chargement…</span>
|
||||
</div>
|
||||
<div class="overflow-x-auto">
|
||||
<table class="data-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Détecté le</th>
|
||||
<th>Comm.</th>
|
||||
<th>Score</th>
|
||||
<th>IPs</th>
|
||||
<th>JA4 impliqués</th>
|
||||
<th>ASNs impliqués</th>
|
||||
<th>Échantillon IPs</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="fleet-body">
|
||||
<tr><td colspan="7" class="text-center text-gray-500 py-8">Chargement…</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ Graphique score distribution ═══ -->
|
||||
<div class="grid grid-cols-1 lg:grid-cols-2 gap-4">
|
||||
<div class="section-card">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-amber-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 19v-6a2 2 0 00-2-2H5a2 2 0 00-2 2v6a2 2 0 002 2h2a2 2 0 002-2zm0 0V9a2 2 0 012-2h2a2 2 0 012 2v10m-6 0a2 2 0 002 2h2a2 2 0 002-2m0 0V5a2 2 0 012-2h2a2 2 0 012 2v14a2 2 0 01-2 2h-2a2 2 0 01-2-2z"/></svg>
|
||||
Distribution des fleet_scores
|
||||
</span>
|
||||
</div>
|
||||
<div class="section-body">
|
||||
<div id="score-chart" style="height:260px"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section-card">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-purple-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><circle cx="9" cy="12" r="1"/><circle cx="15" cy="12" r="1"/><circle cx="12" cy="7" r="1"/><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12l3-5 3 5"/></svg>
|
||||
IPs par flotte
|
||||
</span>
|
||||
</div>
|
||||
<div class="section-body">
|
||||
<div id="ips-chart" style="height:260px"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* Page Flottes JA4×ASN — chargement et rendu
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
function fmtTs(ts) {
|
||||
if (!ts) return '—';
|
||||
return new Date(ts).toLocaleString('fr-FR', {dateStyle:'short', timeStyle:'short'});
|
||||
}
|
||||
|
||||
function scoreColor(score) {
|
||||
if (score >= 5) return 'badge-critical';
|
||||
if (score >= 3) return 'badge-high';
|
||||
return 'badge-medium';
|
||||
}
|
||||
|
||||
function renderTags(arr, maxShow = 3) {
|
||||
if (!arr || arr.length === 0) return '<span class="text-gray-600">—</span>';
|
||||
const shown = arr.slice(0, maxShow);
|
||||
const rest = arr.length - shown.length;
|
||||
const tags = shown.map(v =>
|
||||
`<span class="inline-block bg-gray-800 text-gray-300 text-[10px] px-1.5 py-0.5 rounded mr-1 mb-1 font-mono">${v}</span>`
|
||||
).join('');
|
||||
const more = rest > 0 ? `<span class="text-gray-500 text-[10px]">+${rest}</span>` : '';
|
||||
return tags + more;
|
||||
}
|
||||
|
||||
async function loadFleet() {
|
||||
try {
|
||||
const res = await fetch('/api/fleet');
|
||||
const data = await res.json();
|
||||
const fleets = data.fleets || [];
|
||||
|
||||
// KPIs
|
||||
const totalIps = fleets.reduce((s, f) => s + (f.n_ips || 0), 0);
|
||||
const maxScore = fleets.length ? Math.max(...fleets.map(f => f.fleet_score || 0)) : 0;
|
||||
document.getElementById('kpi-total').textContent = fleets.length;
|
||||
document.getElementById('kpi-ips').textContent = totalIps.toLocaleString('fr-FR');
|
||||
document.getElementById('kpi-maxscore').textContent = maxScore.toFixed(2);
|
||||
document.getElementById('load-status').textContent =
|
||||
fleets.length ? `${fleets.length} communauté(s)` : 'Aucune flotte détectée';
|
||||
|
||||
// Table
|
||||
const tbody = document.getElementById('fleet-body');
|
||||
if (fleets.length === 0) {
|
||||
tbody.innerHTML = '<tr><td colspan="7" class="text-center text-gray-500 py-8">Aucune flotte détectée sur les 7 derniers jours</td></tr>';
|
||||
} else {
|
||||
tbody.innerHTML = fleets.map(f => `
|
||||
<tr>
|
||||
<td class="text-gray-400 text-xs">${fmtTs(f.detected_at)}</td>
|
||||
<td class="font-mono text-xs text-gray-300">#${f.community_id}</td>
|
||||
<td><span class="badge ${scoreColor(f.fleet_score)}">${(f.fleet_score || 0).toFixed(3)}</span></td>
|
||||
<td class="font-bold text-white">${(f.n_ips || 0).toLocaleString('fr-FR')}</td>
|
||||
<td class="max-w-xs">${renderTags(f.ja4_set, 2)}</td>
|
||||
<td class="max-w-xs">${renderTags(f.asn_set, 3)}</td>
|
||||
<td class="max-w-xs">${renderTags(f.ip_sample, 3)}</td>
|
||||
</tr>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
// Graphique distribution des scores
|
||||
renderScoreChart(fleets);
|
||||
renderIpsChart(fleets);
|
||||
|
||||
} catch (err) {
|
||||
console.error('Erreur chargement flottes :', err);
|
||||
document.getElementById('load-status').textContent = 'Erreur de chargement';
|
||||
document.getElementById('fleet-body').innerHTML =
|
||||
'<tr><td colspan="7" class="text-center text-red-500 py-8">Erreur de chargement</td></tr>';
|
||||
}
|
||||
}
|
||||
|
||||
function renderScoreChart(fleets) {
|
||||
const el = document.getElementById('score-chart');
|
||||
if (!el || !fleets.length) return;
|
||||
const chart = echarts.init(el, 'dark');
|
||||
// Histogramme des fleet_scores
|
||||
const scores = fleets.map(f => parseFloat((f.fleet_score || 0).toFixed(2)));
|
||||
chart.setOption({
|
||||
backgroundColor: 'transparent',
|
||||
tooltip: { trigger: 'axis' },
|
||||
xAxis: { type:'category', data: fleets.map((f,i) => `#${f.community_id}`),
|
||||
axisLabel:{color:'#6b7280', fontSize:10, rotate:45} },
|
||||
yAxis: { type:'value', name:'Fleet Score', nameTextStyle:{color:'#6b7280',fontSize:10},
|
||||
axisLabel:{color:'#6b7280', fontSize:10} },
|
||||
series: [{
|
||||
type: 'bar',
|
||||
data: scores.map(s => ({
|
||||
value: s,
|
||||
itemStyle: { color: s >= 5 ? '#ef4444' : s >= 3 ? '#f97316' : '#eab308' }
|
||||
})),
|
||||
barMaxWidth: 40,
|
||||
}],
|
||||
});
|
||||
window.addEventListener('resize', () => chart.resize());
|
||||
}
|
||||
|
||||
function renderIpsChart(fleets) {
|
||||
const el = document.getElementById('ips-chart');
|
||||
if (!el || !fleets.length) return;
|
||||
const chart = echarts.init(el, 'dark');
|
||||
chart.setOption({
|
||||
backgroundColor: 'transparent',
|
||||
tooltip: { trigger: 'item', formatter: '{b}: {c} IPs' },
|
||||
series: [{
|
||||
type: 'pie',
|
||||
radius: ['40%', '70%'],
|
||||
data: fleets.slice(0, 15).map(f => ({
|
||||
name: `#${f.community_id}`,
|
||||
value: f.n_ips || 0,
|
||||
})),
|
||||
label: { color: '#9ca3af', fontSize: 10 },
|
||||
emphasis: { itemStyle: { shadowBlur: 10, shadowOffsetX: 0, shadowColor: 'rgba(0,0,0,0.5)' } },
|
||||
}],
|
||||
});
|
||||
window.addEventListener('resize', () => chart.resize());
|
||||
}
|
||||
|
||||
loadFleet();
|
||||
</script>
|
||||
{% endblock %}
|
||||
371
services/dashboard/backend/templates/health.html
Normal file
371
services/dashboard/backend/templates/health.html
Normal file
@ -0,0 +1,371 @@
|
||||
{% extends "base.html" %}
|
||||
{% block page_title %}Santé du Pipeline ML — §9{% endblock %}
|
||||
{% block content %}
|
||||
<div class="p-4 lg:p-6 space-y-4 max-w-[1920px] mx-auto">
|
||||
|
||||
<!-- ═══ Header KPIs ═══ -->
|
||||
<div class="flex flex-wrap items-center gap-4 mb-2">
|
||||
<h1 class="text-xl font-bold text-white flex items-center gap-2">
|
||||
<svg class="w-6 h-6 text-green-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 19v-6a2 2 0 00-2-2H5a2 2 0 00-2 2v6a2 2 0 002 2h2a2 2 0 002-2zm0 0V9a2 2 0 012-2h2a2 2 0 012 2v10m-6 0a2 2 0 002 2h2a2 2 0 002-2m0 0V5a2 2 0 012-2h2a2 2 0 012 2v14a2 2 0 01-2 2h-2a2 2 0 01-2-2z"/></svg>
|
||||
Santé du Pipeline ML
|
||||
</h1>
|
||||
<div class="ml-auto flex items-center gap-3 flex-wrap">
|
||||
<div class="text-center px-3">
|
||||
<div class="text-2xl font-bold text-brand-500" id="kpi-cycles">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">Cycles (7j)</div>
|
||||
</div>
|
||||
<div class="text-center px-3 border-l border-gray-700">
|
||||
<div class="text-2xl font-bold text-orange-400" id="kpi-anomaly">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">Taux anomalie moy.</div>
|
||||
</div>
|
||||
<div class="text-center px-3 border-l border-gray-700">
|
||||
<div class="text-2xl font-bold text-yellow-400" id="kpi-latency">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">Latence moy. (ms)</div>
|
||||
</div>
|
||||
<div class="text-center px-3 border-l border-gray-700">
|
||||
<div class="text-2xl font-bold text-red-400" id="kpi-drifts">—</div>
|
||||
<div class="text-[10px] text-gray-500 uppercase tracking-wider">Alertes drift</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ Doc banner ═══ -->
|
||||
<div class="bg-gray-900/50 border border-gray-800 rounded-lg px-4 py-3 text-xs text-gray-400 leading-relaxed">
|
||||
<strong class="text-green-300">Métriques de performance — Étape 9</strong> — Chaque cycle du
|
||||
bot-detector (~5 min) enregistre son taux d'anomalie, sa latence, son taux de corrélation
|
||||
réseau, le nombre de features valides et les alertes de dérive.
|
||||
<br><strong>Alertes :</strong>
|
||||
<span class="text-red-400">drift_alert</span> = dérive de features > 30% —
|
||||
<span class="text-orange-400">anomaly_rate > 10%</span> = risque de sur-détection —
|
||||
<span class="text-yellow-400">latence > 300s</span> = problème de performance.
|
||||
<br><strong>Action SOC :</strong> Un drift prolongé nécessite un re-entraînement manuel.
|
||||
Un taux de corrélation bas (< 50%) indique un problème sentinel/correlator.
|
||||
</div>
|
||||
|
||||
<!-- ═══ Graphiques temporels ═══ -->
|
||||
<div class="grid grid-cols-1 lg:grid-cols-2 gap-4">
|
||||
<!-- Taux d'anomalie -->
|
||||
<div class="section-card">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-orange-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 17h8m0 0V9m0 8l-8-8-4 4-6-6"/></svg>
|
||||
Taux d'anomalie par cycle
|
||||
<span class="relative inline-block"><button onclick="docToggle(this)" class="doc-btn">ⓘ</button><div class="doc-panel">
|
||||
<h4>Taux d'anomalie</h4>
|
||||
<p>Pourcentage de sessions classées HIGH/CRITICAL/MEDIUM/LOW par cycle, par modèle.</p>
|
||||
<p><strong>Seuil d'alerte :</strong> > 10% → sur-détection probable. < 0.5% → sous-détection.</p>
|
||||
<p class="doc-source">Source : ml_performance_metrics</p>
|
||||
</div></span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="section-body"><div id="anomaly-chart" style="height:240px"></div></div>
|
||||
</div>
|
||||
<!-- Latence -->
|
||||
<div class="section-card">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-yellow-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
|
||||
Latence des cycles (ms)
|
||||
<span class="relative inline-block"><button onclick="docToggle(this)" class="doc-btn">ⓘ</button><div class="doc-panel">
|
||||
<h4>Latence de traitement</h4>
|
||||
<p>Durée totale du cycle bot-detector en millisecondes (fetch ClickHouse + scoring ML + insert).</p>
|
||||
<p><strong>Seuil d'alerte :</strong> > 300 000ms (5 min) → le cycle dépasse l'intervalle planifié.</p>
|
||||
<p class="doc-source">Source : ml_performance_metrics.cycle_latency_ms</p>
|
||||
</div></span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="section-body"><div id="latency-chart" style="height:240px"></div></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Drift rate et taux de corrélation -->
|
||||
<div class="grid grid-cols-1 lg:grid-cols-2 gap-4">
|
||||
<div class="section-card">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-red-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-2.5L13.732 4c-.77-.833-1.964-.833-2.732 0L4.082 16.5c-.77.833.192 2.5 1.732 2.5z"/></svg>
|
||||
Dérive des features
|
||||
</span>
|
||||
</div>
|
||||
<div class="section-body"><div id="drift-chart" style="height:200px"></div></div>
|
||||
</div>
|
||||
<div class="section-card">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-blue-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13.828 10.172a4 4 0 00-5.656 0l-4 4a4 4 0 105.656 5.656l1.102-1.101m-.758-4.899a4 4 0 005.656 0l4-4a4 4 0 00-5.656-5.656l-1.1 1.1"/></svg>
|
||||
Taux de corrélation réseau
|
||||
<span class="relative inline-block"><button onclick="docToggle(this)" class="doc-btn">ⓘ</button><div class="doc-panel">
|
||||
<h4>Taux de corrélation</h4>
|
||||
<p>Proportion de sessions avec <code>correlated=1</code> (JA4 TLS corrélé par sentinel).</p>
|
||||
<p><strong>Valeur attendue :</strong> > 50%. En dessous, vérifier sentinel et logcorrelator.</p>
|
||||
<p class="doc-source">Source : ml_performance_metrics.correlated_rate</p>
|
||||
</div></span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="section-body"><div id="corr-chart" style="height:200px"></div></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ═══ Table des cycles récents ═══ -->
|
||||
<div class="section-card overflow-hidden">
|
||||
<div class="section-header">
|
||||
<span class="section-title">
|
||||
<svg class="w-4 h-4 text-green-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"/></svg>
|
||||
Cycles récents
|
||||
</span>
|
||||
<div class="flex items-center gap-2">
|
||||
<select id="model-filter" class="bg-gray-800 text-gray-300 text-xs border border-gray-700 rounded px-2 py-1">
|
||||
<option value="">Tous les modèles</option>
|
||||
</select>
|
||||
<span id="table-status" class="text-[10px] text-gray-500">Chargement…</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="overflow-x-auto">
|
||||
<table class="data-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Cycle</th>
|
||||
<th>Modèle</th>
|
||||
<th>Sessions</th>
|
||||
<th>Corrélation</th>
|
||||
<th>Anomalies</th>
|
||||
<th>CRITICAL</th>
|
||||
<th>HIGH</th>
|
||||
<th>Drift</th>
|
||||
<th>Latence (ms)</th>
|
||||
<th>Features</th>
|
||||
<th>Baseline</th>
|
||||
<th>Meta</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="metrics-body">
|
||||
<tr><td colspan="12" class="text-center text-gray-500 py-8">Chargement…</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
/* ════════════════════════════════════════════════════════════════════════════
|
||||
* Page Santé du Pipeline ML — chargement et rendu
|
||||
* ════════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
let _allMetrics = [];
|
||||
|
||||
function fmtTs(ts) {
|
||||
if (!ts) return '—';
|
||||
return new Date(ts).toLocaleString('fr-FR', {dateStyle:'short', timeStyle:'short'});
|
||||
}
|
||||
|
||||
function pct(v) {
|
||||
return (v * 100).toFixed(1) + '%';
|
||||
}
|
||||
|
||||
function driftBadge(rate, alert) {
|
||||
if (alert) return `<span class="badge badge-critical">${pct(rate)} ⚠</span>`;
|
||||
if (rate > 0.15) return `<span class="badge badge-high">${pct(rate)}</span>`;
|
||||
return `<span class="badge badge-low">${pct(rate)}</span>`;
|
||||
}
|
||||
|
||||
function anomalyBadge(rate) {
|
||||
if (rate > 0.10) return `<span class="badge badge-critical">${pct(rate)}</span>`;
|
||||
if (rate > 0.05) return `<span class="badge badge-high">${pct(rate)}</span>`;
|
||||
if (rate > 0.01) return `<span class="badge badge-medium">${pct(rate)}</span>`;
|
||||
return `<span class="badge badge-low">${pct(rate)}</span>`;
|
||||
}
|
||||
|
||||
function corrBadge(rate) {
|
||||
if (rate < 0.30) return `<span class="badge badge-critical">${pct(rate)}</span>`;
|
||||
if (rate < 0.50) return `<span class="badge badge-high">${pct(rate)}</span>`;
|
||||
return `<span class="badge badge-low">${pct(rate)}</span>`;
|
||||
}
|
||||
|
||||
function latencyBadge(ms) {
|
||||
if (ms > 300000) return `<span class="badge badge-critical">${ms.toLocaleString()}</span>`;
|
||||
if (ms > 120000) return `<span class="badge badge-high">${ms.toLocaleString()}</span>`;
|
||||
return `<span class="text-gray-300 text-xs">${ms.toLocaleString()}</span>`;
|
||||
}
|
||||
|
||||
function renderTable(metrics) {
|
||||
const tbody = document.getElementById('metrics-body');
|
||||
if (!metrics.length) {
|
||||
tbody.innerHTML = '<tr><td colspan="12" class="text-center text-gray-500 py-8">Aucune donnée sur les 7 derniers jours</td></tr>';
|
||||
return;
|
||||
}
|
||||
tbody.innerHTML = metrics.map(m => `
|
||||
<tr class="${m.drift_alert ? 'bg-red-900/10' : ''}">
|
||||
<td class="text-gray-400 text-xs">${fmtTs(m.cycle_at)}</td>
|
||||
<td><span class="badge badge-known">${m.model_name || '—'}</span></td>
|
||||
<td class="text-right font-mono text-xs">${(m.total_sessions || 0).toLocaleString('fr-FR')}</td>
|
||||
<td class="text-right">${corrBadge(m.correlated_rate || 0)}</td>
|
||||
<td class="text-right">${anomalyBadge(m.anomaly_rate || 0)}</td>
|
||||
<td class="text-right font-bold text-red-400">${m.critical_count || 0}</td>
|
||||
<td class="text-right font-bold text-orange-400">${m.high_count || 0}</td>
|
||||
<td class="text-right">${driftBadge(m.drift_rate || 0, m.drift_alert)}</td>
|
||||
<td class="text-right">${latencyBadge(m.cycle_latency_ms || 0)}</td>
|
||||
<td class="text-right text-xs text-gray-400">${m.features_valid || 0}/${m.features_total || 0}</td>
|
||||
<td class="text-right text-xs text-gray-400">${(m.baseline_size || 0).toLocaleString()}</td>
|
||||
<td class="text-center">${m.meta_learner_active ? '✓' : '—'}</td>
|
||||
</tr>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
function filterAndRender() {
|
||||
const modelFilter = document.getElementById('model-filter').value;
|
||||
const filtered = modelFilter
|
||||
? _allMetrics.filter(m => m.model_name === modelFilter)
|
||||
: _allMetrics;
|
||||
renderTable(filtered);
|
||||
}
|
||||
|
||||
async function loadHealth() {
|
||||
try {
|
||||
const res = await fetch('/api/health');
|
||||
const data = await res.json();
|
||||
_allMetrics = data.metrics || [];
|
||||
|
||||
// KPIs globaux
|
||||
const driftAlerts = _allMetrics.filter(m => m.drift_alert).length;
|
||||
document.getElementById('kpi-cycles').textContent = _allMetrics.length;
|
||||
document.getElementById('kpi-anomaly').textContent = (data.avg_anomaly_rate * 100).toFixed(2) + '%';
|
||||
document.getElementById('kpi-latency').textContent = Math.round(data.avg_latency_ms || 0).toLocaleString('fr-FR');
|
||||
document.getElementById('kpi-drifts').textContent = driftAlerts;
|
||||
document.getElementById('table-status').textContent = `${_allMetrics.length} cycle(s)`;
|
||||
|
||||
// Filtre modèles
|
||||
const models = [...new Set(_allMetrics.map(m => m.model_name).filter(Boolean))];
|
||||
const sel = document.getElementById('model-filter');
|
||||
models.forEach(name => {
|
||||
const opt = document.createElement('option');
|
||||
opt.value = name;
|
||||
opt.textContent = name;
|
||||
sel.appendChild(opt);
|
||||
});
|
||||
sel.addEventListener('change', filterAndRender);
|
||||
|
||||
renderTable(_allMetrics);
|
||||
renderCharts(_allMetrics);
|
||||
|
||||
} catch (err) {
|
||||
console.error('Erreur chargement métriques :', err);
|
||||
document.getElementById('table-status').textContent = 'Erreur de chargement';
|
||||
document.getElementById('metrics-body').innerHTML =
|
||||
'<tr><td colspan="12" class="text-center text-red-500 py-8">Erreur de chargement</td></tr>';
|
||||
}
|
||||
}
|
||||
|
||||
function renderCharts(metrics) {
|
||||
if (!metrics.length) return;
|
||||
|
||||
// Ordre chronologique pour les graphes
|
||||
const ordered = [...metrics].reverse();
|
||||
const labels = ordered.map(m => fmtTs(m.cycle_at));
|
||||
|
||||
// Couleurs par modèle
|
||||
const modelColors = {'Complet':'#6366f1', 'Applicatif':'#22d3ee'};
|
||||
const colorFor = name => modelColors[name] || '#9ca3af';
|
||||
|
||||
// --- Graphe taux d'anomalie ---
|
||||
const anomalyEl = document.getElementById('anomaly-chart');
|
||||
if (anomalyEl) {
|
||||
const byModel = {};
|
||||
ordered.forEach(m => {
|
||||
(byModel[m.model_name] = byModel[m.model_name] || []).push(
|
||||
+(m.anomaly_rate * 100).toFixed(2)
|
||||
);
|
||||
});
|
||||
const anomalyChart = echarts.init(anomalyEl, 'dark');
|
||||
anomalyChart.setOption({
|
||||
backgroundColor: 'transparent',
|
||||
tooltip: { trigger:'axis', valueFormatter: v => v + '%' },
|
||||
legend: { bottom: 0, textStyle:{color:'#9ca3af', fontSize:10} },
|
||||
xAxis: { type:'category', data: labels, axisLabel:{color:'#6b7280', fontSize:9, rotate:30, interval:'auto'} },
|
||||
yAxis: { type:'value', name:'%', axisLabel:{color:'#6b7280', fontSize:10},
|
||||
axisLine:{lineStyle:{color:'#374151'}} },
|
||||
series: Object.entries(byModel).map(([name, vals]) => ({
|
||||
name, type:'line', data: vals, smooth: true,
|
||||
lineStyle:{color: colorFor(name), width:2},
|
||||
itemStyle:{color: colorFor(name)},
|
||||
areaStyle:{color: colorFor(name), opacity:0.08},
|
||||
symbol:'none',
|
||||
})),
|
||||
markLine: { data:[{yAxis:10, label:{formatter:'Seuil 10%', fontSize:9}, lineStyle:{color:'#ef4444', type:'dashed'}}] },
|
||||
});
|
||||
window.addEventListener('resize', () => anomalyChart.resize());
|
||||
}
|
||||
|
||||
// --- Graphe latence ---
|
||||
const latencyEl = document.getElementById('latency-chart');
|
||||
if (latencyEl) {
|
||||
const byModel = {};
|
||||
ordered.forEach(m => {
|
||||
(byModel[m.model_name] = byModel[m.model_name] || []).push(m.cycle_latency_ms || 0);
|
||||
});
|
||||
const latencyChart = echarts.init(latencyEl, 'dark');
|
||||
latencyChart.setOption({
|
||||
backgroundColor: 'transparent',
|
||||
tooltip: { trigger:'axis', valueFormatter: v => v.toLocaleString() + ' ms' },
|
||||
legend: { bottom: 0, textStyle:{color:'#9ca3af', fontSize:10} },
|
||||
xAxis: { type:'category', data: labels, axisLabel:{color:'#6b7280', fontSize:9, rotate:30, interval:'auto'} },
|
||||
yAxis: { type:'value', name:'ms', axisLabel:{color:'#6b7280', fontSize:10} },
|
||||
series: Object.entries(byModel).map(([name, vals]) => ({
|
||||
name, type:'bar', data: vals, stack:'total',
|
||||
itemStyle:{color: colorFor(name)}, barMaxWidth:20,
|
||||
})),
|
||||
});
|
||||
window.addEventListener('resize', () => latencyChart.resize());
|
||||
}
|
||||
|
||||
// --- Graphe drift ---
|
||||
const driftEl = document.getElementById('drift-chart');
|
||||
if (driftEl) {
|
||||
const driftChart = echarts.init(driftEl, 'dark');
|
||||
driftChart.setOption({
|
||||
backgroundColor: 'transparent',
|
||||
tooltip: { trigger:'axis', valueFormatter: v => (v * 100).toFixed(1) + '%' },
|
||||
xAxis: { type:'category', data: labels, axisLabel:{color:'#6b7280', fontSize:9, rotate:30, interval:'auto'} },
|
||||
yAxis: { type:'value', max:1, axisLabel:{color:'#6b7280', fontSize:10,
|
||||
formatter: v => (v*100).toFixed(0) + '%'} },
|
||||
series: [{
|
||||
type: 'bar',
|
||||
data: ordered.map(m => ({
|
||||
value: m.drift_rate || 0,
|
||||
itemStyle: { color: m.drift_alert ? '#ef4444' : '#6366f1' },
|
||||
})),
|
||||
barMaxWidth: 20,
|
||||
}],
|
||||
});
|
||||
window.addEventListener('resize', () => driftChart.resize());
|
||||
}
|
||||
|
||||
// --- Graphe corrélation réseau ---
|
||||
const corrEl = document.getElementById('corr-chart');
|
||||
if (corrEl) {
|
||||
const byModel = {};
|
||||
ordered.forEach(m => {
|
||||
(byModel[m.model_name] = byModel[m.model_name] || []).push(
|
||||
+(m.correlated_rate * 100).toFixed(1)
|
||||
);
|
||||
});
|
||||
const corrChart = echarts.init(corrEl, 'dark');
|
||||
corrChart.setOption({
|
||||
backgroundColor: 'transparent',
|
||||
tooltip: { trigger:'axis', valueFormatter: v => v + '%' },
|
||||
legend: { bottom: 0, textStyle:{color:'#9ca3af', fontSize:10} },
|
||||
xAxis: { type:'category', data: labels, axisLabel:{color:'#6b7280', fontSize:9, rotate:30, interval:'auto'} },
|
||||
yAxis: { type:'value', max:100, axisLabel:{color:'#6b7280', fontSize:10,
|
||||
formatter: v => v + '%'} },
|
||||
series: Object.entries(byModel).map(([name, vals]) => ({
|
||||
name, type:'line', data: vals, smooth:true,
|
||||
lineStyle:{color: colorFor(name), width:2},
|
||||
itemStyle:{color: colorFor(name)}, symbol:'none',
|
||||
})),
|
||||
});
|
||||
window.addEventListener('resize', () => corrChart.resize());
|
||||
}
|
||||
}
|
||||
|
||||
loadHealth();
|
||||
</script>
|
||||
{% endblock %}
|
||||
Reference in New Issue
Block a user