- §2.1.3: Simplifié Anubis à 2 dictionnaires (dict_anubis_ip, dict_anubis_asn) avec priorité COALESCE
- §2.4.2: Ajouté bibliothèque isotree, formule de calibration, ntrees=300, sérialisation joblib
- §2.4.2b/§2.4.4: Remplacé DBSCAN par HDBSCAN partout
- §2.4.2c: Remplacé régression logistique par pondération linéaire fixe, ajouté formule et poids
- §2.4.3: Clarifié approximation par 5 quantiles pour la détection de dérive
- §3.1: Mis à jour le diagramme ASCII (dual-database, 3×EIF+AE+XGB, HDBSCAN, 55 routes)
- §3.8: Mis à jour la trifurcation + ajouté détection multifactorielle navigateur (5 axes)
- §4: Élargi taxonomie de 51 à 65+ features sur 8 familles
- §5: Ajouté statut d'implémentation (✅/❌) à chaque technique
- §6: Ajouté §6.6 résultats de déploiement (3M+ logs, 34K sessions/cycle)
- §7: Mis à jour conclusion (65+ features, 5/8 techniques, refactorisation modulaire)
- §8: Ajouté références isotree, PyTorch, HDBSCAN, XGBoost, SHAP
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exports http_logs from prod ClickHouse via HTTP API, imports into dev
with dynamic date shifting (max(time) → now() by default).
Features:
- Batch export in Native format (200K rows/batch, ~10s each)
- Auto date shift: prod max(time) aligned to current time
- --shift N: manual override (seconds)
- --days N: filter to last N days only
- --cron: silent mode for scheduled runs
- Staging table approach: export → staging → INSERT SELECT with shift → cleanup
Tested: 3,054,122 rows imported in ~3 minutes, dates 2026-04-03→2026-04-09.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove UA regex extraction (extract_ua_regex, _extract_ua_from_all/any)
- Remove Country rule collection from parse_bot_policies_inline
- Simplify fetch_rules.py: collect_all_rules returns (ip_rules, asn_rules)
- Remove insert_ua_rules and insert_country_rules functions
- reload_dicts now only reloads dict_anubis_ip + dict_anubis_asn
- Simplify CASE blocks in 04_mv_http_logs.sql, 07_ai_features_view.sql,
view_ai_features_anubis.sql, mv_http_logs.sql: IP > ASN (was 5-level
UA+IP > UA > IP > ASN > Country cascade)
- Remove dict_anubis_country + dict_anubis_ua from 03_anubis_tables.sql
(UA table kept as stub for REGEXP_TREE catch-all compatibility)
- Remove anubis_country_rules table from schema
- Remove Anubis UA and Country tabs from dashboard reflists page
- Remove anubis_ua_rules/country_rules from API reflist queries
- deploy_schema.sql simplified from 339 to 122 lines
- 764 lines removed across 9 files
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix doc tooltips: split CSS into <style type='text/tailwindcss'> for
@apply directives + raw CSS for reliable doc panel rendering
- Convert doc panels from click-toggle to hover-based infobulles with
arrow pointer, fade-in animation, and auto-dismiss on mobile
- Replace '?' icons with 'ⓘ' across all 11 templates (51 tooltips)
- Full-width layout: reduce padding on mobile (px-3), scale up on
desktop (lg:px-5, xl:px-6) for maximum screen utilization
- Auto-collapse sidebar on narrow screens (<1024px)
- Keyboard shortcuts: Alt+1–9 for page navigation, Alt+B toggle sidebar
- Add LEGITIMATE_BROWSER filter button to detections page
- Sticky header with stronger blur (backdrop-blur-md)
- All 46 routes pass tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Suppression de bot_detector.py (1982 lignes) remplacé par 11 modules
- Tests navigateur mis à jour pour le système multifactoriel (browser_confidence)
- 36/36 tests passent avec la nouvelle structure modulaire
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
XGB: query was selecting features from ml_all_scores which doesn't
store them. Now joins ml_all_scores (labels) with view_ai_features_1h
(features). Dynamically discovers available columns to skip thesis §5
features not present in the view. Returns (model, features) tuple.
SHAP: TreeExplainer doesn't support isotree. Fall back to permutation-
based Explainer(model.decision_function, X_sample) for isotree.
Verified: XGB trained on 50000 labels (18436 positives), triple-voice
ensemble scoring active (EIF+AE+XGB), SHAP silent.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bot-detector:
- B1.1: campaign_id and raw_anomaly_score now inserted into ml_detected_anomalies
- B1.4/B1.5: log_decision argument order fixed (cycle_id, name)
- B1.7: AE broadcast error — model now returns features list, scoring
uses model's features instead of current cycle's (prevents dim mismatch)
- B1.8: Anubis ALLOW bots now get bot_name from anubis_bot_name
Dashboard:
- C1.1: XSS in ip_detail.html — {{ ip | tojson }} instead of raw string
- C1.2: Stored XSS via innerHTML — added escapeHtml() helper, all user-facing
formatters (fmtIP, fmtASN, fmtCountry, fmtJA4, fmtBotName, fmtLabel) sanitized
- C2.1: status filter now correctly filters http_version column
- C2.2: heatmap toDayOfWeek() - 1 for 0-indexed JS days
SQL:
- B1.3: view_ip_recurrence worst_score uses max() not min() (0=normal, 1=anomal)
- B1.6: view_resource_cascade_1h joined into view_thesis_features_1h (§5.4)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- 350K browser rows (14K IPs) using real JA4s from browser_ja4.csv
- 100K scanner rows (3K IPs) with vuln/cred/scraper/DDoS sub-categories
- 30K legit bot rows (2K IPs) from real bot_ip.csv CIDRs
- 20K AI bot rows (1K IPs) for GPTBot, ClaudeBot, etc.
Key improvements:
- Load browser_ja4.csv at startup, match JA4 to browser family
- Load bot_ip.csv to generate IPs from real Googlebot/Bingbot CIDRs
- Hard-coded ISP /24 prefixes from real ASNs (Comcast, Orange, DT, etc.)
- Realistic navigation patterns with Referer chains and cookies
- Sec-CH-UA headers for Chromium browsers (modern_browser_score >= 50)
- Batch size increased to 2000, progress reporting every 10K rows
- New CLI args: --rows, --ips, --seed, --data-dir
- Bot JA4s are synthetic hashes guaranteed NOT in browser_ja4.csv
Also updated:
- Dockerfile: COPY *.py (was missing seed_clickhouse.py)
- docker-compose.yml: mount scripts/data as /app/data for CSV access
- run-tests.sh: updated seeder description comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add navigation helpers (fmtASN, fmtCountry, fmtJA4, fmtBotName,
fmtThreatLink, fmtLabel) to base.html for SOC analyst drill-down.
Update all templates:
- overview.html: clickable table cells + ECharts click handlers for
ASN, country, JA4, bot, and threat charts
- detections.html: URL param pre-filters, active filter bar with
clear buttons, clickable ASN/country/JA4/threat in table
- scores.html: URL param pre-filters, clickable threat/JA4/country
- traffic.html: clickable JA4 and country columns
- ip_detail.html: clickable threat/JA4 in detections, clickable
asn_org/country_code/asn_label in AI features grid
- network.html: click handlers on ASN treemap and country sunburst,
fmtJA4Full/fmtLabel/fmtBotName/fmtASN in tables
- features.html: scatter plot click navigates to /ip/{ip}
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add BuildRequires: systemd-rpm-macros to sentinel and correlator specs
- Replace manual systemctl calls with %systemd_post, %systemd_preun,
%systemd_postun_with_restart macros (handles daemon-reload, stop/disable,
try-restart on upgrade correctly and is a no-op in containers)
- ja4sentinel.spec: use %{_unitdir} macro instead of hardcoded path
(/usr/lib/systemd/system); remove cross-service /var/run/logcorrelator
from %files and %post (owned by logcorrelator package, not sentinel)
- logcorrelator.spec: move unit from /etc/systemd/system (admin namespace)
to %{_unitdir} (/usr/lib/systemd/system) — correct packaging location;
move user/group creation from %post to %pre so file ownership is valid
during RPM install phase; add Requires(pre): shadow-utils; fix bare
directory entries in %files with %dir macro; add version fallback macro
so spec is buildable without --define version
- test-rpm.sh: auto-build RPM via Dockerfile.package if dist/rpm/ is
empty; update service file path check to /usr/lib/systemd/system/
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3 SQL files were missing from the docker-compose.yml volume mounts:
- 10_perf_indexes.sql (performance indexes)
- 11_views.sql (dashboard views)
- 12_thesis_features.sql (thesis §5 MVs and views)
Also make 10_perf_indexes.sql non-fatal in init script since ALTER TABLE
ADD INDEX may fail if index already exists.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- EIF: Extended Isolation Forest via isotree (fallback to sklearn IF)
- Benford's Law deviation feature on inter-request timing
- Lag-1 autocorrelation feature for cadence analysis
- Validation gate: reject model if val_anomaly_rate > 20%
- Feature pruning: remove variance < 1e-6 features before training
- Quantile drift: replace N(μ,σ) synthetic with quantile interpolation
- Thread safety: Lock for _service_healthy/_consecutive_failures
- Score normalization: inverted to [0,1] where 1=most anomalous
SQL: add lag1_autocorrelation + benford_deviation to view_thesis_features_1h
Tests: 10 new test functions covering all improvements
Integration: verify_mvs.py checks new thesis feature columns
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Analyse exhaustive feature-par-feature des techniques de détection
implémentées vs ce que décrit la thèse. Score: 97% base, 6% techniques
avancées, 72% global pondéré.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
deploy_views.sql (v13 → v14):
- CRITICAL: ml_detected_anomalies ORDER BY (src_ip) → (src_ip, ja4, host, model_name)
ReplacingMergeTree was collapsing all detections to 1 row per IP on merge
- Add PARTITION BY toDate + ttl_only_drop_parts on all 4 data tables
- ml_all_scores TTL 3d → 7d; ml_detected_anomalies TTL 30d → 7d
- agg_host_ip_ja4_1h + agg_header_fingerprint_1h: add partition + TTL 7d
- view_ip_recurrence: add WHERE detected_at >= now() - 7 DAY (was full scan)
- Remove dead views: summary/timeseries/threat_dist/variability
- Add view_dashboard_entities (fixes HTTP 500 in clustering/incidents/fingerprints)
- Add view_dashboard_user_agents (fixes HTTP 500 in fingerprints/metrics)
- Add view_ai_features_24h (enables ENABLE_MULTIWINDOW in bot_detector)
- Mark max_requests_per_sec as DEPRECATED (always 0)
New files:
- correlator/sql/migrations/01_ttl_adjustments.sql: ALTER TABLE migration
- tests/integration/verify_mvs.py: MV pipeline verification assertions
- docs/THESIS_HTTP_Traffic_Detection.md: detection techniques thesis
All DB references use ja4_processing/ja4_logs (no mabase_prod).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
view_ip_recurrence :
Ajout de WHERE detected_at >= now() - INTERVAL 30 DAY
→ Avec PARTITION BY (P1), ClickHouse élagage les partitions hors de cette
plage avant même de lire les données. La vue ne scanne que les partitions
actives (au lieu des 30 partitions journalières complètes).
→ ORDER BY (src_ip) garantit que le GROUP BY src_ip lit des données
contiguës (aucune réorganisation mémoire).
rotation.py — supprimer FINAL sur ml_detected_anomalies :
FINAL force une déduplication complète du ReplacingMergeTree en mémoire
(équivalent à un DISTINCT sur toute la table) — une des opérations les plus
coûteuses dans ClickHouse.
Fix : remplacer le sous-SELECT FINAL par view_ip_recurrence (déjà aggrégée
par src_ip, retourne recurrence directement sans FINAL).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Problème : 8 clauses WHERE appliquaient une fonction sur la colonne src_ip :
WHERE replaceRegexpAll(toString(src_ip), '^::ffff:', '') = %(ip)s
→ ClickHouse ne peut pas utiliser l'index de tri ou les skipping indexes
quand une fonction est appliquée à la colonne filtrée.
Fix : transformer l'INPUT (le paramètre) plutôt que la colonne :
WHERE src_ip = IPv4MappedToIPv6(toIPv4(%(ip)s))
→ src_ip reste intact → ClickHouse utilise les indexes (P1) et la
projection proj_by_ip (P1) pour ces requêtes.
Fichiers modifiés :
investigation_summary.py — 6 WHERE (ml_detected_anomalies, agg_host_ip_ja4_1h,
view_form_bruteforce_detected, view_host_ip_ja4_rotation,
view_ip_recurrence)
ml_features.py — 1 WHERE (view_ai_features_1h)
rotation.py — 1 WHERE (agg_host_ip_ja4_1h)
Note : les 27 autres occurrences de replaceRegexpAll dans les SELECT sont des
transformations d'affichage (IPv6→IPv4 pour l'UI) et ne bloquent pas les indexes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Problème : toutes les requêtes du dashboard WHERE detected_at >= now() - INTERVAL N
faisaient un full scan car ml_detected_anomalies avait ORDER BY (src_ip) sans
partition ni index temporel.
Changements :
- 06_ml_tables.sql :
* ml_detected_anomalies : PARTITION BY toYYYYMMDD(detected_at)
→ élagage de partitions journalières sur toutes les requêtes temporelles
* INDEX idx_detected_at (minmax) → skip des granules hors plage
* INDEX idx_threat_level set(8) → skip pour countIf(threat_level = ...)
* INDEX idx_bot_name bloom_filter → skip pour bot_name != ''
* ttl_only_drop_parts = 1 → TTL par suppression de partition entière
* ml_all_scores : même traitement (PARTITION BY + 2 indexes)
- 04_mv_http_logs.sql :
* http_logs : INDEX idx_src_ip bloom_filter(0.01)
→ les requêtes WHERE src_ip = X (analysis.py, variability.py) sautent
~90% des granules sans scanner toute la plage temporelle
* INDEX idx_ja4 bloom_filter(0.01) → idem pour filtres JA4
- 05_aggregation_tables.sql :
* agg_host_ip_ja4_1h : PROJECTION proj_by_ip ORDER BY (src_ip, window_start, ...)
→ investigation_summary.py et rotation.py (WHERE src_ip = X) utilisent
automatiquement la projection au lieu de scanner tous les window_start
- 10_perf_indexes.sql (nouveau) :
* Migration ALTER TABLE pour instances existantes
* ADD INDEX + MATERIALIZE INDEX pour les 4 tables
* ADD PROJECTION + MATERIALIZE PROJECTION pour agg_host_ip_ja4_1h
* Note : PARTITION BY sur table existante nécessite recréation (documenté)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add make test-integration* commands
- Add SKIP_TESTS=true fast build flag
- Add 'Comments standard' section referencing docs/commenting-standard.md
- Add 'Known gotchas' section with 6 non-obvious issues:
* go.work build context must include both sentinel + correlator
* YAML does not expand env vars in Go (hardcode DSN)
* REGEXP_TREE dict requires >=1 rule or inserts fail
* pcap only captures non-loopback traffic
* ClickHouse init needs 120s timeout
* RPM builds must use Rocky Linux (libpcap.so.1 vs .so.0.8)
* FLAT() layout requires numeric keys (use COMPLEX_KEY_HASHED for strings)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
