Skip ground classification when DTM already exists
If the DTM .tif exists and --force is not set, skip both ground classification and DTM generation entirely. Previously, the pipeline would spend 3+ minutes reclassifying ground even when the DTM was already present and would be reused anyway. Also includes: SharedDEM cache, enhanced WebP cartouche (compass rose, adaptive scale bar, enriched info bar), removed COG/viewer, UTF-8 fix for parallel workers, skip logic for DTM and PDF. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
33
CLAUDE.md
33
CLAUDE.md
@ -18,8 +18,6 @@ All commands run inside Docker. Use `./run.sh` as the primary interface.
|
||||
./run.sh -g --file LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc # Single file
|
||||
./run.sh --ground-classification pmf # Force PMF ground classification
|
||||
./run.sh -g --keep-tif # Keep intermediate TIFF files
|
||||
./run.sh -g --no-viewer # Skip web viewer generation
|
||||
./run.sh serve # Start web map server
|
||||
./run.sh # Print help (no args)
|
||||
```
|
||||
|
||||
@ -34,19 +32,27 @@ docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data
|
||||
### Module responsibilities
|
||||
|
||||
- **`cli.py`** — argparse + logging setup. Entry point via `python -m lidar_pipeline`.
|
||||
- **`pipeline.py`** — `LidarArchaeoPipeline` orchestrator. `VIZ_STEPS` registry maps names to generate functions. `FilePrefixFilter` for parallel logging.
|
||||
- **`pipeline.py`** — `LidarArchaeoPipeline` orchestrator. `VIZ_STEPS` registry maps names to generate functions. `FilePrefixFilter` for parallel logging. Creates `SharedDEM` once per file and passes it to all visualizations.
|
||||
- **`dtm.py`** — PDAL ground classification (SMRF/PMF/CSF + auto-detection) and DTM generation via scipy `binned_statistic_2d`.
|
||||
- **`visualizations.py`** — 15 `generate_*` functions + 2 IGN overlay lambdas. All take `(dem_file, basename, vis_dir, resolution)` and return a TIF path or None.
|
||||
- **`visualizations.py`** — 15 `generate_*` functions + 2 IGN overlay lambdas. All take `(dem_file, basename, vis_dir, resolution, shared=None)` and return a TIF path or None. `SharedDEM` class pre-computes gradient, NaN mask, LRM to avoid redundant I/O and computation.
|
||||
- **`gpu.py`** — CuPy/numpy abstraction: `HAS_GPU`, `to_gpu()`, `to_cpu()`, `xp_gaussian_filter()`, `xp_uniform_filter()`, `xp_minimum_filter()`, `gpu_cleanup()`. Falls back to CPU gracefully.
|
||||
- **`ign.py`** — IGN WMTS tile download + overlay generation for orthophoto and topographic maps.
|
||||
- **`rendering.py`** — `COLORMAPS` dict maps filename keywords to (cmap, title, legend, description). `tif_to_png()` converts TIF→WebP with legend/scale/north arrow. `convert_to_cog()` converts TIF→Cloud Optimized GeoTIFF. `generate_cog_metadata()` creates metadata JSON for web viewer. `generate_pdf_report()` creates A3 PDF.
|
||||
- **`viewer.py`** — Generates MapLibre GL JS HTML viewer with layer controls, opacity sliders, and IGN/OSM basemaps.
|
||||
- **`server.py`** — TiTiler-based Starlette server for serving COG tiles and viewer HTML. Entry point via `python -m lidar_pipeline.server`.
|
||||
- **`rendering.py`** — `COLORMAPS` dict maps filename keywords to (cmap, title, legend, description). `tif_to_png()` converts TIF→WebP with legend/scale/north arrow. `generate_pdf_report()` creates A3 PDF.
|
||||
|
||||
### SharedDEM optimization
|
||||
|
||||
`SharedDEM` pre-computes once per file:
|
||||
- DEM data (single I/O read)
|
||||
- NaN mask + filled DEM (single `_fill_nans` call, avoiding ~20 redundant calls)
|
||||
- Gradient components (dy, dx, slope, aspect) shared by hillshade, slope, aspect, curvature
|
||||
- LRM at 15m kernel (shared by lrm + anomalies)
|
||||
|
||||
`_filter_nanaware_from_filled()` applies filters on the pre-filled DEM, skipping the expensive `_fill_nans` interpolation.
|
||||
|
||||
### Adding a visualization
|
||||
|
||||
Three places must be updated:
|
||||
1. `visualizations.py` — add `generate_X(dem_file, basename, vis_dir, resolution)` function
|
||||
1. `visualizations.py` — add `generate_X(dem_file, basename, vis_dir, resolution, shared=None)` function
|
||||
2. `pipeline.py` `VIZ_STEPS` — add `('name', generate_X)` entry
|
||||
3. `rendering.py` `COLORMAPS` — add entry keyed by the output filename keyword
|
||||
|
||||
@ -61,7 +67,11 @@ Override with `--ground-classification {auto,smrf,pmf,csf}`.
|
||||
|
||||
### NaN handling
|
||||
|
||||
DTM small gaps (< 1m from existing data) are filled using `rasterio.fill.fillnodata`. Large gaps remain as NaN. Visualization functions use `_fill_nans()` and `_filter_nanaware()` to avoid NaN propagation through filters.
|
||||
DTM small gaps (< 1m from existing data) are filled using `rasterio.fill.fillnodata`. Large gaps remain as NaN. `SharedDEM` fills NaN once; `_filter_nanaware_from_filled()` applies filters on the pre-filled array and restores the NaN mask.
|
||||
|
||||
### Flow accumulation
|
||||
|
||||
Uses priority-flood algorithm (Wang & Liu 2006) for sink filling, which is O(n log n) instead of iterative minimum_filter. D8 accumulation uses numba JIT; falls back to pure Python if numba unavailable.
|
||||
|
||||
### Parallel processing
|
||||
|
||||
@ -72,7 +82,6 @@ Uses `ProcessPoolExecutor` with `'spawn'` start method (required for CUDA). Each
|
||||
- **Language**: UI messages and comments in French. Code identifiers in English.
|
||||
- **Logging**: Use `logger = logging.getLogger("lidar")`. Prefix per-file logs via `_file_filter.basename`.
|
||||
- **GPU pattern**: `arr_gpu = to_gpu(arr)` → compute → `result = to_cpu(arr_gpu)` → `gpu_cleanup()` between visualizations.
|
||||
- **Output format**: Visualizations saved as WebP (not PNG). TIFF intermediates deleted unless `--keep-tif` or viewer enabled. COGs generated for web viewer by default. PDF reports use `PILImage.open().convert('RGB')`.
|
||||
- **Web viewer**: MapLibre GL JS + TiTiler. COGs served as raster tiles. `./run.sh serve` starts server on port 8000.
|
||||
- **Flow accumulation**: Uses numba JIT for D8 accumulation loop. Falls back to pure Python if numba unavailable.
|
||||
- **Output format**: Visualizations saved as WebP. TIFF intermediates deleted after conversion unless `--keep-tif`. No COGs or viewer — only WebP + PDF report remain.
|
||||
- **Compression**: TIF intermediates use `deflate` compression (faster than LZW for float32 data).
|
||||
- **Tests**: Run only inside Docker via `./run.sh --test`. Synthetic DEM fixture in `tests/conftest.py`.
|
||||
Reference in New Issue
Block a user