From 78a2c328c3caad031fcbc1a1d2f0bead5a84d32f Mon Sep 17 00:00:00 2001 From: Antoine Jacquin Date: Mon, 7 Sep 2026 21:30:59 +0200 Subject: [PATCH] Document code audit findings and architecture notes in AGENTS.md --- AGENTS.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 712b8b9..01c32ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,6 +29,34 @@ - **`_`-prefixed names are critical private**: `_create_ground_pipeline`, `_fallback_to_smrf`, `_fill_nans`, `_init_gpu`, `_process_file_standalone` — do not call from outside their module. - **`build_index()` writes 3 files**: `output/index.html` (data shell, `const TILES` embedded), `output/assets/app.css` and `output/assets/app.js` (source: `_APP_CSS`/`_APP_JS` constants in `index.py`). `webapp.py` serves `/assets` with no-cache headers. Each tile carries `meta` — ground method read from `DTM/*_dtm{_rXpY}_method.txt` (falls back to the primary-resolution sidecar) + per-viz dates/sizes. +## Architecture Notes (from code audit 2025-09) + +### Module structure & data flow +- `cli.py` → `pipeline.py` (LidarArchaeoPipeline) → per-file: `dtm.py` (classify + rasterize) → `visualizations.py` (17 products) → `rendering.py` (GeoTIFF→AVIF) → `index.py` (Leaflet map) +- `gpu.py` provides CuPy/NumPy proxy (`xp`), lazy init, OOM fallback. `safe_gpu_call` wraps all non-IGN viz calls. +- `webapp.py` (FastAPI) serves the map + `/api/generate` launches the pipeline as a subprocess. Two-machine mode: `LIDAR_GENERATION_URL` delegates to remote worker. +- `progress.py` writes JSONL events (O_APPEND, atomic) read by webapp for live progress. + +### Key design decisions (intentional, do not "fix") +- **`_res_suffix` hardcodes 0.5 as "no suffix"**: coupled to `index.py` parsing (`_strip_res_suffix` defaults to 0.5 when no suffix). Changing requires sidecar metadata. +- **GPU scoring** (`major*1000 + minor*100 + mem_mi`): compute capability priority is intentional — a newer GPU with less VRAM is preferred. +- **`webapp.py` reads env at import time**: deployment-focused single-purpose server, env is set once in docker-compose. +- **Repeated try/except in visualizations** (14× same pattern): intentional convention for uniform `return None` behavior. +- **`_d8_accumulate_numba` defines `@njit` inside the function**: `cache=True` makes subsequent calls fast; the Python function object creation is negligible. +- **`pkill -9 -f "pdal pipeline"`** in cli.py signal handler: belt-and-suspenders alongside `os.killpg`. Scoped to "pdal pipeline" to avoid killing unrelated PDAL processes. + +### Performance characteristics +- `_priority_flood` uses numba JIT binary heap (single int64 array, flat view for elevation). Python heapq fallback if numba unavailable. +- `_d8_accumulate_numba` uses numba with `argsort` top-down sweep. Python fallback exists. +- Ray-tracing (SVF, openness): processes one direction at a time to limit VRAM. Auto-falls back to CPU on OOM via `_ray_trace_horizons`. +- Multi-resolution: primary res (default 0.5) has no filename suffix; additional resolutions use `_r0p2` style suffix. Ground classification done once, shared across resolutions. +- `ProcessPoolExecutor` has a 2-hour wall-clock safety timeout (prevents indefinite hang from stuck workers). + +### Numba usage pattern +- Defined at function scope with `@njit(cache=True)` — first call compiles (~2-3s), subsequent calls hit disk cache. +- Must use flat 1D array views (`arr.ravel()`) for integer indexing — 2D arrays with a single int index return a row slice in nopython mode. +- Pattern: try numba → return None on ImportError → caller falls back to pure Python. + ## Commit & Pull Request Guidelines Commits use imperative tense, short single-line subjects (~60–80 chars), no prefixes or scopes. Compound commits are common — multiple related changes joined by commas or "and". Examples: ``Fix multi-GPU with lazy CuPy init + rendering improvements``, ``Add multi-resolution support and remove PDF generation``, ``Fix corrupted COPC detection, add CSF→SMRF fallback, improve MSRM colormap, add SVF and anisotropic openness``.