lidar_rendu/AGENTS.md

Workflow

• install: docker build -t lidar-lidar . (deps baked into image)
• build: docker build -t lidar-lidar .
• build webapp légère (Raspberry Pi, déploiement 2 machines — cf. docs/DEPLOY_WEBAPP.md): docker compose -f docker-compose.webapp.yml up -d --build (image Dockerfile.webapp, sans PDAL/GPU)
• build générateur de tuiles (machine de traitement): docker compose -f docker-compose.worker.yml up -d --build (service worker, API pour les webapp distantes)
• simulation locale du mode deux machines : docker compose -f docker-compose.local-2m.yml up -d --build — worker GPU sur :8974 + webapp légère sur :8973 avec son PROPRE cache output-webapp/ peuplé à la demande depuis le worker. Permet de rebuild l'interface sans toucher au worker, et réciproquement. Résolution 0,2 m uniquement (GENERATE_RESOLUTIONS).
• stack webapp (machine légère): ./serve-webapp.sh [start|stop|restart|status|sync|logs|update], config dans webapp.env (modèle webapp.env.example, ignoré par git)
• test all: ./run.sh --test (rebuild automatique de l'image avant les tests ; en docker run direct, rebuild manuellement d'abord)
• test file: docker run --rm lidar-lidar python3 -m pytest -v --pyargs lidar_pipeline.tests.<module>
• test case: docker run --rm lidar-lidar python3 -m pytest -v --pyargs lidar_pipeline.tests.<module>::<TestClass>::<test_method>
• lint: not configured
• format: not configured
• after every edit: ./run.sh --test
• RÈGLE 1 — toujours lancer via docker compose (jamais docker run direct) : carte/API → docker compose up -d --build serve (port 8973) ; traitement ponctuel → docker compose run --rm --build process [options] ; logs → docker compose logs -f serve ; arrêt → docker compose down.
• RÈGLE 2 — TOUJOURS --build : le code est baké dans l'image (jamais monté). Sans --build, up/run réutilisent l'image existante et l'ANCIEN code tourne. --build est quasi instantané grâce au cache (le .dockerignore exclut input/ et output/ du contexte). Après édition : docker compose up -d --build serve recrée le conteneur sur du neuf.
• test rapide sans rebuild (code monté par-dessus l'image): docker run --rm -e PYTHONPATH=/app -v $(pwd)/lidar_pipeline:/app/lidar_pipeline lidar-lidar python3 -m pytest --pyargs lidar_pipeline.tests
• debug: ./run.sh --debug (file:line logging); container shell: docker run --rm -it -v $(pwd)/input:/data/input -v $(pwd)/output:/data/output --entrypoint bash lidar-lidar

Conventions

• Generation is 0.2 m only (policy): /api/generate (GENERATE_RESOLUTIONS in webapp.py), the compose process command and the CLI -r default all produce 0.2 m exclusively; 0.5 m stays available via explicit -r 0.5. Completeness detection (complete_cells) requires the viz at 0.2 m only.
• Sub-tuilage intégral : _CARTO_SUBTILED_VIZ (vide dans index.py) découpe TOUTES les couches en quadrants 500 m à 0,2 m ; ortho/topo sont encodées en AVIF q75 (_SUBTILE_DETAIL_VIZ) contre q55 pour les rampes de couleur. Une couche qui échoue à la découpe retombe en dalle entière (_fallback_full_dalle) sans pénaliser les autres.
• Bilingual naming: all code identifiers are English; every user-facing string, log message, argparse help, and comment is French.
• Adding a visualization requires 4 edits: (1) generate_X() in visualizations.py, (2) entry in VIZ_STEPS in pipeline.py, (3) entry in COLORMAPS in rendering.py, (4) entry in VIZ_LEGENDS in index.py (title/legend/description + sampled cmap gradient — single text source merged into COLORMAPS at import, also used by the export mosaic legend in export.py). Missing any one breaks the pipeline.
• generate_* signature is strict: (dem_file, basename, vis_dir, resolution, shared=None) returning Path on success, None on failure. IGN overlays (ortho, topo) omit shared.
• Return None on failure, never raise: dtm.py, visualizations.py, and ign.py all return None to let the pipeline continue. Raising aborts the entire file.
• Logger is always logging.getLogger("lidar"), never __name__. All modules route through this single logger so worker processes can configure it.
• Filename special-cases in _expected_output_path(): pos_open → positive_openness, neg_open → negative_openness, hillshade → hillshade_multi.
• Default output is AVIF, not WebP. Use --format webp for WebP. Quality default is 98.
• Tests use lazy imports inside each test function, never at module top, to avoid importing CuPy/GDAL at import time.
• _-prefixed names are critical private: _create_ground_pipeline, _fallback_to_smrf, _fill_nans, _init_gpu, _process_file_standalone — do not call from outside their module.
• build_index() writes 3 files: output/index.html (data shell, const TILES embedded), output/assets/app.css and output/assets/app.js (source: _APP_CSS/_APP_JS constants in index.py). webapp.py serves /assets with no-cache headers. Each tile carries meta — ground method read from DTM/*_dtm{_rXpY}_method.txt (falls back to the primary-resolution sidecar) + per-viz dates/sizes.

Architecture Notes (from code audit 2025-09)

Module structure & data flow

• cli.py → pipeline.py (LidarArchaeoPipeline) → per-file: dtm.py (classify + rasterize) → visualizations.py (17 products) → rendering.py (GeoTIFF→AVIF) → index.py (Leaflet map)
• gpu.py provides CuPy/NumPy proxy (xp), lazy init, OOM fallback. safe_gpu_call wraps all non-IGN viz calls.
• webapp.py (FastAPI) serves the map + /api/generate launches the pipeline as a subprocess. Two-machine mode: LIDAR_GENERATION_URL delegates to remote worker. /api/export assembles adjacent tiles into an image/PDF via export.py (local cache, no delegation).
• progress.py writes JSONL events (O_APPEND, atomic) read by webapp for live progress.
• export.py stitches adjacent tile visualizations into a seamless mosaic (PNG/JPEG/WebP) or multi-page PDF, Pillow-only for the lightweight webapp.

Key design decisions (intentional, do not "fix")

• _res_suffix hardcodes 0.5 as "no suffix": coupled to index.py parsing (_strip_res_suffix defaults to 0.5 when no suffix). Changing requires sidecar metadata.
• GPU scoring (major*1000 + minor*100 + mem_mi): compute capability priority is intentional — a newer GPU with less VRAM is preferred.
• webapp.py reads env at import time: deployment-focused single-purpose server, env is set once in docker-compose.
• Repeated try/except in visualizations (14× same pattern): intentional convention for uniform return None behavior.
• _d8_accumulate_numba defines @njit inside the function: cache=True makes subsequent calls fast; the Python function object creation is negligible.
• pkill -9 -f "pdal pipeline" in cli.py signal handler: belt-and-suspenders alongside os.killpg. Scoped to "pdal pipeline" to avoid killing unrelated PDAL processes.

Performance characteristics

• _priority_flood uses numba JIT binary heap (single int64 array, flat view for elevation). Python heapq fallback if numba unavailable.
• _d8_accumulate_numba uses numba with argsort top-down sweep. Python fallback exists.
• Ray-tracing (SVF, openness): processes one direction at a time to limit VRAM. Auto-falls back to CPU on OOM via _ray_trace_horizons.
• Multi-resolution: primary res (default 0.5) has no filename suffix; additional resolutions use _r0p2 style suffix. Ground classification done once, shared across resolutions.
• ProcessPoolExecutor has a 2-hour wall-clock safety timeout (prevents indefinite hang from stuck workers).

Numba usage pattern

• Defined at function scope with @njit(cache=True) — first call compiles (~2-3s), subsequent calls hit disk cache.
• Must use flat 1D array views (arr.ravel()) for integer indexing — 2D arrays with a single int index return a row slice in nopython mode.
• Pattern: try numba → return None on ImportError → caller falls back to pure Python.

Commit & Pull Request Guidelines

Commits use imperative tense, short single-line subjects (~60–80 chars), no prefixes or scopes. Compound commits are common — multiple related changes joined by commas or "and". Examples: Fix multi-GPU with lazy CuPy init + rendering improvements, Add multi-resolution support and remove PDF generation, Fix corrupted COPC detection, add CSF→SMRF fallback, improve MSRM colormap, add SVF and anisotropic openness.

No PR template, no CI pipeline, no issue tracker. This is a standalone Docker project with no formal PR process.