- Scope pkill to 'pdal pipeline' (avoids killing unrelated PDAL jobs)
- Inline _strip_lidar_ext in dtm.py to remove circular import from pipeline
- _res_suffix_str now delegates to LidarArchaeoPipeline._res_suffix
Prevents indefinite hang if a worker gets stuck (deadlocked PDAL,
I/O stall). 2-hour safety net for the entire batch; remaining
futures are cancelled on timeout.
Replaces the pure-Python heapq implementation with a compiled binary
min-heap (~200x faster for large grids). Falls back to Python when
numba is unavailable. Uses a flat array view for heap elevation
comparisons to avoid 2D indexing issues in nopython mode.
index.py rewritten as a continuous Leaflet map: rotated L93 tiles, stackable
visualization layers (per-layer opacity, drag-reorder persisted in
localStorage), tile info panel, live rebuild after each tile during a run.
Leaflet is vendored in assets/vendor/ so the map works fully offline;
georeferencing falls back rasterio -> pyproj -> affine so the lightweight
webapp (no GDAL) is supported.
docker-compose.worker.yml adds the tile generator service (full image + GPU)
that remote webapps call via LIDAR_GENERATION_URL, plus a one-shot process
profile. webapp.py gains LIDAR_AUTO_SYNC_SECONDS periodic cache refresh and
LIDAR_REGEN_CIDR restricting generation to the local network. run.sh
--serve-webapp now mounts ~/.ssh read-only so the rsync sync works.
docs/DEPLOY_WEBAPP.md completed for Raspberry Pi deployment: prerequisites,
git clone install, SSH key setup, first sync, update procedure and
troubleshooting.
La webapp (carte + vignettes) et la génération de tuiles se déploient sur
deux machines : image légère Dockerfile.webapp (FastAPI + Pillow AVIF
natif + pyproj) sur Raspberry Pi, pipeline complet sur la machine de
traitement. LIDAR_GENERATION_URL délègue /api/generate, /api/preview et
/api/status ; /api/sync ramène les tuiles par rsync puis régénère
vignettes et index localement. Token partagé optionnel
(LIDAR_API_TOKEN/LIDAR_REMOTE_TOKEN). Retire du dépôt les journaux
internes (.swival, audit-findings) et les données (data/, notebooks/).
Doc : docs/DEPLOY_WEBAPP.md.
- Web map: multi-select picker in the generation bar (aspect, slope,
positive openness, anisotropic openness, wavelet) passed to the API;
the layer panel is restricted to the same shortlist (PANEL_VIZ) and
a refresh button rebuilds the index when new layers appear on disk;
jobs started outside the UI are now adopted into the visible queue
- Uniform colors across tiles: openness/anisotropic/sailore now store
local z-scores, and all renderers use fixed ranges (0-3 sigma, SVF
0-1 physical, slope 0-30 deg) instead of per-tile percentile stretches
- Ray-tracing falls back to CPU when VRAM is exhausted so openness and
SVF no longer fail silently on shared GPUs
- build_index merges visualizations available at only one resolution
into the displayed tile so in-progress layers stay visible
- 11 new tests (142 passing)
- webapp.py: FastAPI serving the continuous map (port 8973) with
/api/preview, /api/generate and /api/status; tiles are downloaded
from IGN and processed in a logged subprocess, tracked live in a
side "File de génération" panel that survives page reloads
- fetch_ign.py: download missing 1 km LiDAR HD tiles from the IGN
geoplateforme before processing
- index.py: tile thumbnails and 500 m subtiles are now invalidated by
mtime so regenerating a tile refreshes its cached images; progress
logging per tile
- dtm.py: back to the historical gap handling (small gaps filled by
fillnodata only, larger holes left as nodata rendered black);
lowest-return floor only via --bare-earth, IGN class selection via
--ign-classes
- cli.py: positional input now optional (--rebuild-index works alone)
- docker-compose.yml: serve (GPU, port 8973) and process services;
launch via docker compose only (documented in AGENTS.md/AGENTS.md)
- tests: 131 passing, incl. regressions for thumbnail staleness,
--rebuild-index without input, and nodata rendering
_gpu_candidates was never populated (typo: _candidate_gpus), breaking
num_gpus()/available_gpu_ids() and the multi-GPU round-robin. available_gpu_ids
also indexed tuples with dict syntax (TypeError). disable_gpu() now frees the
memory pool before clearing refs (was leaking VRAM). to_cpu() and safe_gpu_call()
survive disable_gpu() via a persistent _cupy_ndarray type reference so orphaned
GPU arrays are recovered to CPU. log_gpu_status() now uses Device(0) instead of
the host index (CuPy renumbers to 0 after CUDA_VISIBLE_DEVICES).
Small archaeological features (ditches, walls, post-holes) were drowned out
by large-scale topography (100-200m). Fix: replace 100/200m with finer
scales (3, 8, 15m), weight 5-10m heaviest, clip absolute z-score to 3.0
before combination to prevent large-scale outliers from dominating.
GPU multi-processing fix:
- gpu.py: revert to CUDA_VISIBLE_DEVICES approach with lazy CuPy init
(Device.use() caused CUDA_ERROR_NO_BINARY_FOR_GPU on GPU 1)
- CuPy is imported lazily on first to_gpu() call, allowing
CUDA_VISIBLE_DEVICES to be set before CUDA context creation
- nvidia-smi used for GPU count detection (no CUDA import needed)
- pipeline.py: add tip message suggesting -w N when multiple GPUs detected
Rendering improvements:
- Title: split into bold title (14pt) + italic description (10pt)
- North arrow: moved inside data area (top-right) with transparent
background — no longer overlaps title
- Colorbar: full height (compass gap removed), ScalarFormatter with
useOffset=False to prevent scientific notation on small values
Performance:
- rendering.py: save matplotlib figure to BytesIO instead of temp PNG
file — eliminates disk I/O between matplotlib and PIL
- visualizations.py: cap max_dist at 300 for ray-tracing (SVF,
openness, aniso_open) — avoids 500+ iterations at 0.2m resolution
- pipeline.py: deduplicate n_gpus calculation in parallel path
GPU fix:
- Revert to CUDA_VISIBLE_DEVICES approach but with lazy CuPy init
- gpu.py: CuPy is no longer imported at module level; _init_gpu()
imports it lazily on first to_gpu() call. This allows workers to
set CUDA_VISIBLE_DEVICES before CuPy creates a CUDA context.
- gpu.py: detect GPU count via nvidia-smi (no CUDA context needed)
- pipeline.py: each worker sets CUDA_VISIBLE_DEVICES=N before CuPy
init, so each process uses only its assigned GPU
Rendering improvements:
- Title: split into bold title (14pt) + italic description (10pt)
instead of single 15pt bold block
- North arrow: moved inside data area (top-right corner) with
semi-transparent white background for readability over data
- Colorbar: full height (no gap for compass rose), added
ScalarFormatter(useOffset=False) to avoid scientific notation
- Colorbar compass rose gap removed since north arrow is now
inside the data area
The 80km context made the red LiDAR zone rectangle too small to see.
30km (15km radius) at zoom 12 gives a clear view where the 1km data
zone is visible as a distinct red rectangle in the inset map.
Multi-GPU:
- gpu.py: lazy CuPy initialization so CUDA_VISIBLE_DEVICES takes
effect before context creation in worker processes
- gpu.py: detect GPU count via nvidia-smi (no CUDA import needed)
- gpu.py: add set_active_gpu() to assign workers to specific GPUs
- pipeline.py: distribute files across GPUs (file % num_gpus) in
parallel mode so both GPUs are used simultaneously
- pipeline.py: log GPU count when multiple GPUs detected
Layout fixes:
- rendering.py: move scale bar left of location map to avoid overlap
(scale bar ends at fig_x=0.78, map starts at 0.82)
- rendering.py: expand location map inset to 0.16x0.13 fig coords
- rendering.py: return bounds from _download_location_map so imshow
extent matches the actual IGN tile coverage (80km context)
- ign.py: add min_zoom parameter to download_ign_tiles, fixing the
location map that was broken (zoom 10 blocked by hardcoded min_zoom=15)
- Replace RdBu_r with Crameri roma for all 6 relief-family colormaps
(curvature, mslrm, lrm, tpi, sailore, aniso_open): perceptually
uniform, CVD-friendly, dark center makes near-zero values visible
- Replace hsv with twilight for aspect: perceptually uniform cyclic
colormap with no hue discontinuity
- Add cmcrameri dependency (falls back to RdBu_r if unavailable)
- Fix IGN location map: pass min_zoom=8 to download_ign_tiles so zoom
10 context maps work (was hardcoded to min_zoom=15, blocking the
loop entirely for zoom_level=10)
- Expand location map context from 3x extent to 80km fixed radius for
better regional context at zoom 10
- Remove dead code: flow colormap entry, _nice_scale dead loop,
aspect='auto' in location map imshow, outdated PDF report order list
- Fix SVF docstring: 8 directions → 16 directions
- validate_laz: verify point data accessibility (not just headers) to detect
corrupted COPC files that pass header checks but fail on data reads
- classify_ground: fallback from CSF to SMRF when CSF produces no ground
points or PDAL errors (fixes 2/9 failing tiles)
- MSRM: preserve sign in weighted combination so RdBu_r colormap shows
both red (elevated) and blue (depressed) instead of red only
- Add Sky-View Factor (SVF) visualization: cos²(horizon angle) over 16
directions, excellent for archaeological earthwork detection
- Add Anisotropic Openness: directional weighting (NW-SE/NE-SW) enhances
linear feature detection aligned with common settlement patterns
- Remove anomalies and flow visualizations (replaced by SVF + aniso_open)
- Location inset: use IGN topographic map at zoom 10 instead of simplified
France outline, with red rectangle marker and fallback
- Remove flow (hydrological accumulation) from VIZ_STEPS
- MSRM/TPI/roughness/anomalies: revert z-score (x-mean)/std to std normalization x/std
to preserve contrast and visibility of linear features (paths, ditches, trenches)
- MSRM: adaptive scales based on resolution, archaeological weight combination
- TPI: extend from 2 to 4 scales (3m/15m/50m/200m) with weighted combination
- Hillshade: 8 directions instead of 4, altitude 35° instead of 30°
- LRM: adaptive sigma based on resolution
- Openness: doubled radius (100m instead of 50m)
- Roughness: multi-scale (3m fine + 15m broad) instead of single 5x5 window
- Anomalies: uses MSRM multi-scale relief instead of single LRM 15m
- Wavelet: 8 adaptive scales, std normalization, archaeological weights
- Remove svf (Sky-View Factor) and local_dominance visualizations
- Add AVIF format support (default), quality 98
- Add multi-resolution support (-r 0.5,0.2)
- Improve Ctrl+C handling for immediate process termination
- Update rendering.py descriptions for all modified visualizations
- --resolution now accepts comma-separated values (e.g. 0.5,0.2)
- Additional resolutions get suffixed output dirs: basename_r0p2/
- DTM files are named basename_dtm_r0p2.tif for extra resolutions
- Ground classification is done once and shared across resolutions
- PDF report generation removed per user request
- Fix --file argument to accept full filenames with extensions
Previously --file LHD_FXX_...copc.laz would fail because it appended
extensions. Now tries exact filename match first, then falls back to
adding extensions.
- WebP output now uses quality=85 by default (down from lossless),
reducing file size by ~75% (35MB → 5-8MB per visualization)
- Added --quality N (1-100) and --lossless flags in CLI and run.sh
- Added --only and --skip to select/exclude specific visualizations
(e.g., --only hillshade,svf,lrm or --skip ortho,topo)
- VIZ_STEPS filtering is done in LidarArchaeoPipeline.__init__
- SharedDEM is skipped when all selected visualizations already exist
- Invalid visualization names are validated at startup with clear error
Fixes "zero-size array to reduction operation" crash on corrupt/incomplete
LAZ files. Added checks at each step:
- validate_laz(): check point_count > 0 via laspy header, parse PDAL
info JSON for point count when using PDAL fallback
- detect_ground_method(): return 'smrf' default if point cloud is empty
after PDAL conversion instead of crashing on np.max(empty_array)
- _read_with_pdal(): log warning and return None if converted file has
0 points
- create_dtm_fast(): fail gracefully if ground file has 0 points
- classify_ground(): check output file size after PDAL pipeline to
catch empty ground classifications early
Two optimizations to avoid ~2min wasted per file on re-runs:
1. pipeline.py: Check which visualizations need regeneration before
computing SharedDEM. If all WebP outputs exist, skip SharedDEM
entirely. If only IGN overlays need updating, also skip SharedDEM.
2. visualizations.py: Make SharedDEM attributes lazy (filled, gradient,
lrm_15) so only the data actually needed is computed. For example,
if only hillshade is regenerated, LRM at 15m is never calculated.
- Dockerfile: install PDAL 2.10.1 from conda-forge (was 2.3 from apt)
Ubuntu 22.04's PDAL 2.3 cannot read COPC v1.1 files from IGN LiDAR HD
- dtm.py: add _read_with_pdal() fallback for COPC files that laspy can't read
- dtm.py: validate_laz() now tries PDAL when laspy fails
- dtm.py: create_dtm_fast() and detect_ground_method() use PDAL fallback
- ign.py: auto-retry at lower zoom on 404 errors
- pipeline.py: check DTM resolution mismatch and regenerate if needed
- pipeline.py: propagate actual DTM resolution to visualizations
- pipeline.py: add --init to docker run for proper Ctrl+C signal handling
- Remove RRIM and Multi-Hillshade RGB visualizations
