e2533bac48
Le bouton ⌖ recentre la carte sur la position GPS du téléphone. L'API Geolocation exige un contexte sécurisé : la webapp sert désormais du HTTP et le TLS est porté par le reverse proxy (Traefik, labels dans l'override local) ; le webapp est transparent au proxy (URLs relatives, X-Forwarded-For pour la restriction LIDAR_REGEN_CIDR). Ajoute le modèle d'override docker-compose.webapp.override.yml.example, un rebuild d'index à la demande et met à jour doc/déploiement ; les tuiles sont servies à la demande.
69 lines
8.5 KiB
Markdown
69 lines
8.5 KiB
Markdown
## Workflow
|
||
|
||
- install: `docker build -t lidar-lidar .` (deps baked into image)
|
||
- build: `docker build -t lidar-lidar .`
|
||
- build webapp légère (Raspberry Pi, déploiement 2 machines — cf. `docs/DEPLOY_WEBAPP.md`): `docker compose -f docker-compose.webapp.yml up -d --build` (image `Dockerfile.webapp`, sans PDAL/GPU)
|
||
- build générateur de tuiles (machine de traitement): `docker compose -f docker-compose.worker.yml up -d --build` (service `worker`, API pour les webapp distantes)
|
||
- simulation locale du mode deux machines : `docker compose -f docker-compose.local-2m.yml up -d --build` — worker GPU sur :8974 + webapp légère sur :8973 avec son PROPRE cache `output-webapp/` peuplé à la demande depuis le worker. Permet de rebuild l'interface sans toucher au worker, et réciproquement. Résolution 0,2 m uniquement (GENERATE_RESOLUTIONS).
|
||
- stack webapp (machine légère): `./serve-webapp.sh [start|stop|restart|status|sync|logs|update]`, config dans `webapp.env` (modèle `webapp.env.example`, ignoré par git)
|
||
- test all: `./run.sh --test` (rebuild automatique de l'image avant les tests ; en `docker run` direct, rebuild manuellement d'abord)
|
||
- test file: `docker run --rm lidar-lidar python3 -m pytest -v --pyargs lidar_pipeline.tests.<module>`
|
||
- test case: `docker run --rm lidar-lidar python3 -m pytest -v --pyargs lidar_pipeline.tests.<module>::<TestClass>::<test_method>`
|
||
- lint: not configured
|
||
- format: not configured
|
||
- after every edit: `./run.sh --test`
|
||
- **RÈGLE 1 — toujours lancer via docker compose** (jamais `docker run` direct) : carte/API → `docker compose up -d --build serve` (port 8973) ; traitement ponctuel → `docker compose run --rm --build process [options]` ; logs → `docker compose logs -f serve` ; arrêt → `docker compose down`.
|
||
- **RÈGLE 2 — TOUJOURS `--build` : le code est baké dans l'image (jamais monté).** Sans `--build`, `up`/`run` réutilisent l'image existante et l'ANCIEN code tourne. `--build` est quasi instantané grâce au cache (le .dockerignore exclut input/ et output/ du contexte). Après édition : `docker compose up -d --build serve` recrée le conteneur sur du neuf.
|
||
- test rapide sans rebuild (code monté par-dessus l'image): `docker run --rm -e PYTHONPATH=/app -v $(pwd)/lidar_pipeline:/app/lidar_pipeline lidar-lidar python3 -m pytest --pyargs lidar_pipeline.tests`
|
||
- debug: `./run.sh --debug` (file:line logging); container shell: `docker run --rm -it -v $(pwd)/input:/data/input -v $(pwd)/output:/data/output --entrypoint bash lidar-lidar`
|
||
|
||
## Conventions
|
||
|
||
- **Generation is 0.2 m only** (policy): `/api/generate` (`GENERATE_RESOLUTIONS` in `webapp.py`), the compose `process` command and the CLI `-r` default all produce 0.2 m exclusively; 0.5 m stays available via explicit `-r 0.5`. Completeness detection (`complete_cells`) requires the viz at 0.2 m only.
|
||
- **Sub-tuilage intégral** : `_CARTO_SUBTILED_VIZ` (vide dans `index.py`) découpe TOUTES les couches en quadrants 500 m à 0,2 m ; ortho/topo sont encodées en AVIF q75 (`_SUBTILE_DETAIL_VIZ`) contre q55 pour les rampes de couleur. Une couche qui échoue à la découpe retombe en dalle entière (`_fallback_full_dalle`) sans pénaliser les autres.
|
||
- **Bilingual naming**: all code identifiers are English; every user-facing string, log message, argparse help, and comment is French.
|
||
- **Adding a visualization requires 4 edits**: (1) `generate_X()` in `visualizations.py`, (2) entry in `VIZ_STEPS` in `pipeline.py`, (3) entry in `COLORMAPS` in `rendering.py`, (4) entry in `VIZ_LEGENDS` in `index.py` (title/legend/description + sampled cmap gradient — single text source merged into `COLORMAPS` at import, also used by the export mosaic legend in `export.py`). Missing any one breaks the pipeline.
|
||
- **`generate_*` signature is strict**: `(dem_file, basename, vis_dir, resolution, shared=None)` returning `Path` on success, `None` on failure. IGN overlays (`ortho`, `topo`) omit `shared`.
|
||
- **Return `None` on failure, never raise**: `dtm.py`, `visualizations.py`, and `ign.py` all return `None` to let the pipeline continue. Raising aborts the entire file.
|
||
- **Logger is always `logging.getLogger("lidar")`**, never `__name__`. All modules route through this single logger so worker processes can configure it.
|
||
- **Filename special-cases** in `_expected_output_path()`: `pos_open` → `positive_openness`, `neg_open` → `negative_openness`, `hillshade` → `hillshade_multi`.
|
||
- **Default output is AVIF**, not WebP. Use `--format webp` for WebP. Quality default is 98.
|
||
- **Tests use lazy imports inside each test function**, never at module top, to avoid importing CuPy/GDAL at import time.
|
||
- **`_`-prefixed names are critical private**: `_create_ground_pipeline`, `_fallback_to_smrf`, `_fill_nans`, `_init_gpu`, `_process_file_standalone` — do not call from outside their module.
|
||
- **`build_index()` writes 3 files**: `output/index.html` (data shell, `const TILES` embedded), `output/assets/app.css` and `output/assets/app.js` (source: `_APP_CSS`/`_APP_JS` constants in `index.py`). `webapp.py` serves `/assets` with no-cache headers. Each tile carries `meta` — ground method read from `DTM/*_dtm{_rXpY}_method.txt` (falls back to the primary-resolution sidecar) + per-viz dates/sizes.
|
||
|
||
## Architecture Notes (from code audit 2025-09)
|
||
|
||
### Module structure & data flow
|
||
- `cli.py` → `pipeline.py` (LidarArchaeoPipeline) → per-file: `dtm.py` (classify + rasterize) → `visualizations.py` (17 products) → `rendering.py` (GeoTIFF→AVIF) → `index.py` (Leaflet map)
|
||
- `gpu.py` provides CuPy/NumPy proxy (`xp`), lazy init, OOM fallback. `safe_gpu_call` wraps all non-IGN viz calls.
|
||
- `webapp.py` (FastAPI) serves the map + `/api/generate` launches the pipeline as a subprocess. Two-machine mode: `LIDAR_GENERATION_URL` delegates to remote worker. `/api/export` assembles adjacent tiles into an image/PDF via `export.py` (local cache, no delegation).
|
||
- `progress.py` writes JSONL events (O_APPEND, atomic) read by webapp for live progress.
|
||
- `export.py` stitches adjacent tile visualizations into a seamless mosaic (PNG/JPEG/WebP) or multi-page PDF, Pillow-only for the lightweight webapp.
|
||
|
||
### Key design decisions (intentional, do not "fix")
|
||
- **`_res_suffix` hardcodes 0.5 as "no suffix"**: coupled to `index.py` parsing (`_strip_res_suffix` defaults to 0.5 when no suffix). Changing requires sidecar metadata.
|
||
- **GPU scoring** (`major*1000 + minor*100 + mem_mi`): compute capability priority is intentional — a newer GPU with less VRAM is preferred.
|
||
- **`webapp.py` reads env at import time**: deployment-focused single-purpose server, env is set once in docker-compose.
|
||
- **Repeated try/except in visualizations** (14× same pattern): intentional convention for uniform `return None` behavior.
|
||
- **`_d8_accumulate_numba` defines `@njit` inside the function**: `cache=True` makes subsequent calls fast; the Python function object creation is negligible.
|
||
- **`pkill -9 -f "pdal pipeline"`** in cli.py signal handler: belt-and-suspenders alongside `os.killpg`. Scoped to "pdal pipeline" to avoid killing unrelated PDAL processes.
|
||
|
||
### Performance characteristics
|
||
- `_priority_flood` uses numba JIT binary heap (single int64 array, flat view for elevation). Python heapq fallback if numba unavailable.
|
||
- `_d8_accumulate_numba` uses numba with `argsort` top-down sweep. Python fallback exists.
|
||
- Ray-tracing (SVF, openness): processes one direction at a time to limit VRAM. Auto-falls back to CPU on OOM via `_ray_trace_horizons`.
|
||
- Multi-resolution: primary res (default 0.5) has no filename suffix; additional resolutions use `_r0p2` style suffix. Ground classification done once, shared across resolutions.
|
||
- `ProcessPoolExecutor` has a 2-hour wall-clock safety timeout (prevents indefinite hang from stuck workers).
|
||
|
||
### Numba usage pattern
|
||
- Defined at function scope with `@njit(cache=True)` — first call compiles (~2-3s), subsequent calls hit disk cache.
|
||
- Must use flat 1D array views (`arr.ravel()`) for integer indexing — 2D arrays with a single int index return a row slice in nopython mode.
|
||
- Pattern: try numba → return None on ImportError → caller falls back to pure Python.
|
||
|
||
## Commit & Pull Request Guidelines
|
||
|
||
Commits use imperative tense, short single-line subjects (~60–80 chars), no prefixes or scopes. Compound commits are common — multiple related changes joined by commas or "and". Examples: ``Fix multi-GPU with lazy CuPy init + rendering improvements``, ``Add multi-resolution support and remove PDF generation``, ``Fix corrupted COPC detection, add CSF→SMRF fallback, improve MSRM colormap, add SVF and anisotropic openness``.
|
||
|
||
No PR template, no CI pipeline, no issue tracker. This is a standalone Docker project with no formal PR process.
|