Commit Graph

5 Commits

Author SHA1 Message Date
1fa6aec784 fix: SQL view ordering, purge-db flag, ctest directory
- 12_thesis_features.sql: move view_resource_cascade_1h before view_thesis_features_1h
- Makefile: purge-db uses --reset (not --clean)
- mod-reqin-log: ctest --test-dir build/tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-09 22:39:25 +02:00
db306fb9da fix: P0 audit bugs — bot-detector + dashboard + SQL
Bot-detector:
- B1.1: campaign_id and raw_anomaly_score now inserted into ml_detected_anomalies
- B1.4/B1.5: log_decision argument order fixed (cycle_id, name)
- B1.7: AE broadcast error — model now returns features list, scoring
  uses model's features instead of current cycle's (prevents dim mismatch)
- B1.8: Anubis ALLOW bots now get bot_name from anubis_bot_name

Dashboard:
- C1.1: XSS in ip_detail.html — {{ ip | tojson }} instead of raw string
- C1.2: Stored XSS via innerHTML — added escapeHtml() helper, all user-facing
  formatters (fmtIP, fmtASN, fmtCountry, fmtJA4, fmtBotName, fmtLabel) sanitized
- C2.1: status filter now correctly filters http_version column
- C2.2: heatmap toDayOfWeek() - 1 for 0-indexed JS days

SQL:
- B1.3: view_ip_recurrence worst_score uses max() not min() (0=normal, 1=anomal)
- B1.6: view_resource_cascade_1h joined into view_thesis_features_1h (§5.4)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-08 23:33:00 +02:00
b735bab5a5 feat(dashboard): rebuild SOC dashboard + fix ClickHouse SQL
Complete rewrite of the SOC dashboard using FastAPI + Jinja2 + htmx + Chart.js + Tailwind CSS.
Replaces the old React/Vite frontend with server-rendered templates.

Dashboard pages:
- Overview: KPIs, timeline chart, threat distribution, top IPs
- Detections: paginated/filterable anomaly table
- Scores: ml_all_scores with AE error & XGB prob columns
- Traffic: HTTP logs with method/host filters
- IP Investigation: full deep-dive (scores, features, HTTP logs, classify)
- Classification: SOC feedback form + history
- Features: AI + thesis feature stats
- Models: scoring stats + model metadata

API: 9 JSON endpoints with parameterized queries, sort whitelists

SQL fixes:
- 05_aggregation_tables: add deduplicate_merge_projection_mode
- 11_views: fix nested aggregate (argMax inside sum)
- 12_thesis_features: remove invalid 'let' bindings, fix groupArrayIf type

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-08 03:21:05 +02:00
f6e2d3c0ca feat(bot-detector): implement 8 state-of-art improvements
- EIF: Extended Isolation Forest via isotree (fallback to sklearn IF)
- Benford's Law deviation feature on inter-request timing
- Lag-1 autocorrelation feature for cadence analysis
- Validation gate: reject model if val_anomaly_rate > 20%
- Feature pruning: remove variance < 1e-6 features before training
- Quantile drift: replace N(μ,σ) synthetic with quantile interpolation
- Thread safety: Lock for _service_healthy/_consecutive_failures
- Score normalization: inverted to [0,1] where 1=most anomalous

SQL: add lag1_autocorrelation + benford_deviation to view_thesis_features_1h
Tests: 10 new test functions covering all improvements
Integration: verify_mvs.py checks new thesis feature columns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-08 02:31:26 +02:00
6d02f21c1e feat: implement thesis §5 advanced detection techniques as ClickHouse MVs
New aggregation tables + materialized views:
- agg_path_sequences_1h + MV (§5.1 Path Sequence Entropy)
- agg_request_timing_1h + MV (§5.3 Request Cadence Fingerprint)
- agg_ip_behavior_1h + MV (§5.5 JA4 Drift + §5.8 Cross-Domain)
- agg_resource_cascade_1h + MV (§5.4 Resource Dependency Tree)

New analytical views:
- view_thesis_features_1h: unified view exposing all computable features
  (path_transition_entropy, cadence_cv, burst_ratio, pause_ratio,
   ja4_drift_ratio, host_diversity, host_sweep_speed,
   host_coverage_uniformity)
- view_resource_cascade_1h: root_to_first_asset_delay, asset_load_stddev

Documented future techniques (not feasible as MV):
- §5.2 Bipartite Fleet Graph (needs Python networkx)
- §5.6 DNS Shadow Analysis (needs sentinel UDP/53 extension)
- §5.7 Compression Ratio Invariant (needs mod_reqin_log extension)

Updated: deploy_schema.sh, verify_mvs.py (sections 8-10)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-08 01:42:52 +02:00