Add .gitignore rules for generated CSV data, eBPF compiled objects,
and vmlinux.h header. Remove 19 tracked files (~175 MB) that can be
regenerated from scripts (generate_*.py), bpftool, or bpf2go.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The prod data export was made before http/2 columns were added to
http_logs (h2_fingerprint, h2_settings_fp, h2_window_update,
h2_pseudo_order). The INSERT SELECT now provides empty/zero literals
for those 4 columns so the 56-col Native export imports into the
60-col table without a column count mismatch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ensure h2 columns are always included on fresh init. Also add migration
loop for fleet_detections and ml_performance_metrics tables.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root cause: CREATE TABLE IF NOT EXISTS is a no-op on existing tables,
so stale schemas miss new columns. Views (07+) then fail with
UNKNOWN_IDENTIFIER errors.
Fix: split SQL execution into 3 phases:
Phase 1: databases, raw tables, dictionaries (00-04)
Phase 2: DROP all derived tables (agg_*, ml_*) — safe, repopulated by MVs
Phase 3: recreate derived tables + views with full current schema (05-12)
This removes the incomplete inline migrations and makes the script
truly idempotent regardless of prior schema version.
Tested: fresh --reset, existing stale DB, idempotent re-run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exports http_logs from prod ClickHouse via HTTP API, imports into dev
with dynamic date shifting (max(time) → now() by default).
Features:
- Batch export in Native format (200K rows/batch, ~10s each)
- Auto date shift: prod max(time) aligned to current time
- --shift N: manual override (seconds)
- --days N: filter to last N days only
- --cron: silent mode for scheduled runs
- Staging table approach: export → staging → INSERT SELECT with shift → cleanup
Tested: 3,054,122 rows imported in ~3 minutes, dates 2026-04-03→2026-04-09.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
