Layout uniforme WebP: axes fixes + aspect='equal' pour superposition géolocalisée
- Positions d'axes fixes (data_left/bottom/width/height_frac) pour alignement pixel-parfait entre terrain et ortho/topo - aspect='equal' au lieu de 'auto' pour conserver les proportions géographiques - Colorbar descriptive pour les visualisations RGB (ortho/topo) - Comblage des petits trous DTM (< 1m) via rasterio.fill.fillnodata - Suppression de la visualisation "dépressions" - Hillshade composite: 0.7*hillshade + 0.3*cos(slope) - D8 flow accumulation accéléré par numba JIT (fallback Python) - Flag --keep-tif pour conserver les TIFF intermédiaires - --force supprime aussi les TIF existants avant régénération - ETA affiché pendant la génération des visualisations - Répertoires temp dans temp/ pour traitement parallèle Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
10
CLAUDE.md
10
CLAUDE.md
@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||||||
|
|
||||||
## Project Overview
|
## Project Overview
|
||||||
|
|
||||||
LiDAR archaeological processing pipeline that generates 18 terrain visualizations from LAZ/LAS point clouds. Runs in Docker with optional NVIDIA GPU acceleration (CuPy). Designed for French LiDAR HD data in Lambert 93 (EPSG:2154).
|
LiDAR archaeological processing pipeline that generates 17 terrain visualizations from LAZ/LAS point clouds. Runs in Docker with optional NVIDIA GPU acceleration (CuPy). Designed for French LiDAR HD data in Lambert 93 (EPSG:2154).
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
@ -17,6 +17,7 @@ All commands run inside Docker. Use `./run.sh` as the primary interface.
|
|||||||
./run.sh --test # Run unit tests
|
./run.sh --test # Run unit tests
|
||||||
./run.sh -g --file LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc # Single file
|
./run.sh -g --file LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc # Single file
|
||||||
./run.sh --ground-classification pmf # Force PMF ground classification
|
./run.sh --ground-classification pmf # Force PMF ground classification
|
||||||
|
./run.sh -g --keep-tif # Keep intermediate TIFF files
|
||||||
./run.sh # Print help (no args)
|
./run.sh # Print help (no args)
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -33,7 +34,7 @@ docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data
|
|||||||
- **`cli.py`** — argparse + logging setup. Entry point via `python -m lidar_pipeline`.
|
- **`cli.py`** — argparse + logging setup. Entry point via `python -m lidar_pipeline`.
|
||||||
- **`pipeline.py`** — `LidarArchaeoPipeline` orchestrator. `VIZ_STEPS` registry maps names to generate functions. `FilePrefixFilter` for parallel logging.
|
- **`pipeline.py`** — `LidarArchaeoPipeline` orchestrator. `VIZ_STEPS` registry maps names to generate functions. `FilePrefixFilter` for parallel logging.
|
||||||
- **`dtm.py`** — PDAL ground classification (SMRF/PMF/CSF + auto-detection) and DTM generation via scipy `binned_statistic_2d`.
|
- **`dtm.py`** — PDAL ground classification (SMRF/PMF/CSF + auto-detection) and DTM generation via scipy `binned_statistic_2d`.
|
||||||
- **`visualizations.py`** — 16 `generate_*` functions + 2 IGN overlay lambdas. All take `(dem_file, basename, vis_dir, resolution)` and return a TIF path or None.
|
- **`visualizations.py`** — 15 `generate_*` functions + 2 IGN overlay lambdas. All take `(dem_file, basename, vis_dir, resolution)` and return a TIF path or None.
|
||||||
- **`gpu.py`** — CuPy/numpy abstraction: `HAS_GPU`, `to_gpu()`, `to_cpu()`, `xp_gaussian_filter()`, `xp_uniform_filter()`, `xp_minimum_filter()`, `gpu_cleanup()`. Falls back to CPU gracefully.
|
- **`gpu.py`** — CuPy/numpy abstraction: `HAS_GPU`, `to_gpu()`, `to_cpu()`, `xp_gaussian_filter()`, `xp_uniform_filter()`, `xp_minimum_filter()`, `gpu_cleanup()`. Falls back to CPU gracefully.
|
||||||
- **`ign.py`** — IGN WMTS tile download + overlay generation for orthophoto and topographic maps.
|
- **`ign.py`** — IGN WMTS tile download + overlay generation for orthophoto and topographic maps.
|
||||||
- **`rendering.py`** — `COLORMAPS` dict maps filename keywords to (cmap, title, legend, description). `tif_to_png()` converts TIF→WebP with legend/scale/north arrow. `generate_pdf_report()` creates A3 PDF.
|
- **`rendering.py`** — `COLORMAPS` dict maps filename keywords to (cmap, title, legend, description). `tif_to_png()` converts TIF→WebP with legend/scale/north arrow. `generate_pdf_report()` creates A3 PDF.
|
||||||
@ -56,7 +57,7 @@ Override with `--ground-classification {auto,smrf,pmf,csf}`.
|
|||||||
|
|
||||||
### NaN handling
|
### NaN handling
|
||||||
|
|
||||||
DTM zones without LiDAR data are kept as NaN (no interpolation). Visualization functions use `_fill_nans()` and `_filter_nanaware()` to avoid NaN propagation through filters.
|
DTM small gaps (< 1m from existing data) are filled using `rasterio.fill.fillnodata`. Large gaps remain as NaN. Visualization functions use `_fill_nans()` and `_filter_nanaware()` to avoid NaN propagation through filters.
|
||||||
|
|
||||||
### Parallel processing
|
### Parallel processing
|
||||||
|
|
||||||
@ -67,5 +68,6 @@ Uses `ProcessPoolExecutor` with `'spawn'` start method (required for CUDA). Each
|
|||||||
- **Language**: UI messages and comments in French. Code identifiers in English.
|
- **Language**: UI messages and comments in French. Code identifiers in English.
|
||||||
- **Logging**: Use `logger = logging.getLogger("lidar")`. Prefix per-file logs via `_file_filter.basename`.
|
- **Logging**: Use `logger = logging.getLogger("lidar")`. Prefix per-file logs via `_file_filter.basename`.
|
||||||
- **GPU pattern**: `arr_gpu = to_gpu(arr)` → compute → `result = to_cpu(arr_gpu)` → `gpu_cleanup()` between visualizations.
|
- **GPU pattern**: `arr_gpu = to_gpu(arr)` → compute → `result = to_cpu(arr_gpu)` → `gpu_cleanup()` between visualizations.
|
||||||
- **Output format**: Visualizations saved as WebP (not PNG). PDF reports use `PILImage.open().convert('RGB')`.
|
- **Output format**: Visualizations saved as WebP (not PNG). TIFF intermediates deleted unless `--keep-tif`. PDF reports use `PILImage.open().convert('RGB')`.
|
||||||
|
- **Flow accumulation**: Uses numba JIT for D8 accumulation loop. Falls back to pure Python if numba unavailable.
|
||||||
- **Tests**: Run only inside Docker via `./run.sh --test`. Synthetic DEM fixture in `tests/conftest.py`.
|
- **Tests**: Run only inside Docker via `./run.sh --test`. Synthetic DEM fixture in `tests/conftest.py`.
|
||||||
@ -31,7 +31,8 @@ RUN pip3 install --no-cache-dir \
|
|||||||
scipy \
|
scipy \
|
||||||
tqdm \
|
tqdm \
|
||||||
Pillow \
|
Pillow \
|
||||||
pytest
|
pytest \
|
||||||
|
numba
|
||||||
|
|
||||||
# Install CuPy for GPU acceleration (optional - will fallback to numpy if not available)
|
# Install CuPy for GPU acceleration (optional - will fallback to numpy if not available)
|
||||||
RUN pip3 install --no-cache-dir cupy-cuda12x || echo "CuPy not available - GPU acceleration disabled"
|
RUN pip3 install --no-cache-dir cupy-cuda12x || echo "CuPy not available - GPU acceleration disabled"
|
||||||
|
|||||||
@ -6,6 +6,7 @@ Handles argument parsing, logging configuration, and entry point.
|
|||||||
import argparse
|
import argparse
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
|
import shutil
|
||||||
import signal
|
import signal
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
@ -110,6 +111,11 @@ Exemples:
|
|||||||
action="store_true",
|
action="store_true",
|
||||||
help="Reclassifier le sol même si le fichier .las existe déjà"
|
help="Reclassifier le sol même si le fichier .las existe déjà"
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--keep-tif",
|
||||||
|
action="store_true",
|
||||||
|
help="Conserver les fichiers TIFF intermédiaires (sinon supprimés après conversion WebP)"
|
||||||
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--ground-classification",
|
"--ground-classification",
|
||||||
choices=["auto", "smrf", "pmf", "csf"],
|
choices=["auto", "smrf", "pmf", "csf"],
|
||||||
@ -163,7 +169,8 @@ Exemples:
|
|||||||
workers=args.workers,
|
workers=args.workers,
|
||||||
force=args.force,
|
force=args.force,
|
||||||
ground_method=args.ground_classification,
|
ground_method=args.ground_classification,
|
||||||
force_classify=args.force_classification
|
force_classify=args.force_classification,
|
||||||
|
keep_tif=args.keep_tif
|
||||||
)
|
)
|
||||||
|
|
||||||
# If --file is specified, process only matching files
|
# If --file is specified, process only matching files
|
||||||
@ -199,6 +206,18 @@ Exemples:
|
|||||||
logger.info(f" → {laz_file.name}")
|
logger.info(f" → {laz_file.name}")
|
||||||
for laz_file in unique_files:
|
for laz_file in unique_files:
|
||||||
pipeline.process_file(laz_file)
|
pipeline.process_file(laz_file)
|
||||||
|
|
||||||
|
# Clean up temporary files
|
||||||
|
logger.info("Nettoyage des fichiers temporaires...")
|
||||||
|
try:
|
||||||
|
if pipeline.temp_dir.exists():
|
||||||
|
shutil.rmtree(pipeline.temp_dir)
|
||||||
|
temp_base = pipeline.output_dir / "temp"
|
||||||
|
if temp_base.exists():
|
||||||
|
shutil.rmtree(temp_base)
|
||||||
|
logger.info(" ✓ Fichiers temporaires supprimés")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f" Note: Impossible de supprimer les fichiers temporaires: {e}")
|
||||||
else:
|
else:
|
||||||
pipeline.process_all()
|
pipeline.process_all()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|||||||
@ -289,13 +289,25 @@ def create_dtm_fast(las_file, basename, dtm_dir, resolution):
|
|||||||
dtm = stat.statistic.T
|
dtm = stat.statistic.T
|
||||||
dtm = dtm[::-1, :] # Flip Y so north is at top
|
dtm = dtm[::-1, :] # Flip Y so north is at top
|
||||||
|
|
||||||
# Keep NaN for zones without LiDAR data — no interpolation
|
# Fill small gaps (< 1m from existing data) while keeping large gaps as NaN
|
||||||
nan_count = np.count_nonzero(np.isnan(dtm))
|
nan_count = np.count_nonzero(np.isnan(dtm))
|
||||||
if nan_count > 0:
|
if nan_count > 0:
|
||||||
total = dtm.size
|
total = dtm.size
|
||||||
nan_pct = 100.0 * nan_count / total
|
nan_pct = 100.0 * nan_count / total
|
||||||
logger.info(f" {nan_count:,} pixels sans données ({nan_pct:.1f}%)")
|
logger.info(f" {nan_count:,} pixels sans données ({nan_pct:.1f}%)")
|
||||||
|
|
||||||
|
max_gap_pixels = max(1, int(1.0 / resolution))
|
||||||
|
from rasterio.fill import fillnodata
|
||||||
|
valid_mask = ~np.isnan(dtm)
|
||||||
|
dtm_filled = fillnodata(dtm, mask=valid_mask, max_search_distance=max_gap_pixels)
|
||||||
|
small_gap_mask = np.isnan(dtm) & ~np.isnan(dtm_filled)
|
||||||
|
filled_count = np.count_nonzero(small_gap_mask)
|
||||||
|
if filled_count > 0:
|
||||||
|
dtm = np.where(small_gap_mask, dtm_filled, dtm)
|
||||||
|
logger.info(f" {filled_count:,} petits trous comblés (< {max_gap_pixels}px)")
|
||||||
|
remaining = np.count_nonzero(np.isnan(dtm))
|
||||||
|
logger.info(f" {remaining:,} pixels restent sans données (grands écarts)")
|
||||||
|
|
||||||
# Save as GeoTIFF
|
# Save as GeoTIFF
|
||||||
output_tif = dtm_dir / f"{basename}_dtm.tif"
|
output_tif = dtm_dir / f"{basename}_dtm.tif"
|
||||||
transform = from_bounds(min_x, min_y, max_x, max_y, width, height)
|
transform = from_bounds(min_x, min_y, max_x, max_y, width, height)
|
||||||
|
|||||||
@ -3,7 +3,7 @@
|
|||||||
LidarArchaeoPipeline coordinates the full processing chain:
|
LidarArchaeoPipeline coordinates the full processing chain:
|
||||||
1. Ground classification (PDAL/SMRF)
|
1. Ground classification (PDAL/SMRF)
|
||||||
2. DTM generation
|
2. DTM generation
|
||||||
3. Visualization generation (18 products)
|
3. Visualization generation (17 products)
|
||||||
4. Rendering (WebP + PDF report)
|
4. Rendering (WebP + PDF report)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@ -57,7 +57,7 @@ from .dtm import classify_ground, create_dtm_fast
|
|||||||
from .visualizations import (
|
from .visualizations import (
|
||||||
generate_hillshade, generate_slope, generate_aspect, generate_curvature,
|
generate_hillshade, generate_slope, generate_aspect, generate_curvature,
|
||||||
generate_lrm, generate_svf, generate_openness,
|
generate_lrm, generate_svf, generate_openness,
|
||||||
generate_mslrm, generate_tpi, generate_depressions, generate_sailore,
|
generate_mslrm, generate_tpi, generate_sailore,
|
||||||
generate_roughness, generate_anomalies, generate_wavelet,
|
generate_roughness, generate_anomalies, generate_wavelet,
|
||||||
generate_flow,
|
generate_flow,
|
||||||
)
|
)
|
||||||
@ -80,7 +80,6 @@ VIZ_STEPS = [
|
|||||||
('neg_open', lambda d, b, v, r: generate_openness(d, b, v, r, positive=False)),
|
('neg_open', lambda d, b, v, r: generate_openness(d, b, v, r, positive=False)),
|
||||||
('mslrm', generate_mslrm),
|
('mslrm', generate_mslrm),
|
||||||
('tpi', generate_tpi),
|
('tpi', generate_tpi),
|
||||||
('depressions', generate_depressions),
|
|
||||||
('sailore', generate_sailore),
|
('sailore', generate_sailore),
|
||||||
('roughness', generate_roughness),
|
('roughness', generate_roughness),
|
||||||
('anomalies', generate_anomalies),
|
('anomalies', generate_anomalies),
|
||||||
@ -106,7 +105,7 @@ VIZ_STEPS = [
|
|||||||
class LidarArchaeoPipeline:
|
class LidarArchaeoPipeline:
|
||||||
"""Orchestrates the LiDAR archaeological analysis pipeline."""
|
"""Orchestrates the LiDAR archaeological analysis pipeline."""
|
||||||
|
|
||||||
def __init__(self, input_dir, output_dir, resolution=0.5, workers=1, force=False, ground_method='auto', force_classify=False):
|
def __init__(self, input_dir, output_dir, resolution=0.5, workers=1, force=False, ground_method='auto', force_classify=False, keep_tif=False):
|
||||||
self.input_dir = Path(input_dir)
|
self.input_dir = Path(input_dir)
|
||||||
self.output_dir = Path(output_dir)
|
self.output_dir = Path(output_dir)
|
||||||
self.resolution = resolution
|
self.resolution = resolution
|
||||||
@ -114,6 +113,7 @@ class LidarArchaeoPipeline:
|
|||||||
self.force = force
|
self.force = force
|
||||||
self.ground_method = ground_method
|
self.ground_method = ground_method
|
||||||
self.force_classify = force_classify
|
self.force_classify = force_classify
|
||||||
|
self.keep_tif = keep_tif
|
||||||
self.temp_dir = self.output_dir / "temp"
|
self.temp_dir = self.output_dir / "temp"
|
||||||
|
|
||||||
if not self.input_dir.exists():
|
if not self.input_dir.exists():
|
||||||
@ -137,6 +137,7 @@ class LidarArchaeoPipeline:
|
|||||||
logger.info(f" Force : {'OUI' if self.force else 'non (skip existing)'}")
|
logger.info(f" Force : {'OUI' if self.force else 'non (skip existing)'}")
|
||||||
logger.info(f" Classification sol : {self.ground_method}")
|
logger.info(f" Classification sol : {self.ground_method}")
|
||||||
logger.info(f" Force classif.: {'OUI' if self.force_classify else 'non'}")
|
logger.info(f" Force classif.: {'OUI' if self.force_classify else 'non'}")
|
||||||
|
logger.info(f" Keep TIFF : {'OUI' if self.keep_tif else 'non'}")
|
||||||
|
|
||||||
def find_laz_files(self):
|
def find_laz_files(self):
|
||||||
"""Find all LAZ/LAS files in input directory."""
|
"""Find all LAZ/LAS files in input directory."""
|
||||||
@ -171,8 +172,24 @@ class LidarArchaeoPipeline:
|
|||||||
|
|
||||||
vis_results = {}
|
vis_results = {}
|
||||||
total = len(VIZ_STEPS)
|
total = len(VIZ_STEPS)
|
||||||
|
elapsed_times = []
|
||||||
|
|
||||||
for idx, (name, func) in enumerate(VIZ_STEPS, 1):
|
for idx, (name, func) in enumerate(VIZ_STEPS, 1):
|
||||||
|
# When --force, delete existing TIF to ensure clean regeneration
|
||||||
|
if self.force:
|
||||||
|
for tif in file_vis_dir.glob(f"{basename}_{name}.tif"):
|
||||||
|
tif.unlink(missing_ok=True)
|
||||||
|
# Special cases for differently-named TIFs
|
||||||
|
if name == 'pos_open':
|
||||||
|
for tif in file_vis_dir.glob(f"{basename}_positive_openness.tif"):
|
||||||
|
tif.unlink(missing_ok=True)
|
||||||
|
elif name == 'neg_open':
|
||||||
|
for tif in file_vis_dir.glob(f"{basename}_negative_openness.tif"):
|
||||||
|
tif.unlink(missing_ok=True)
|
||||||
|
elif name == 'hillshade':
|
||||||
|
for tif in file_vis_dir.glob(f"{basename}_hillshade_multi.tif"):
|
||||||
|
tif.unlink(missing_ok=True)
|
||||||
|
|
||||||
# Check if output WebP already exists (skip unless --force)
|
# Check if output WebP already exists (skip unless --force)
|
||||||
if not self.force:
|
if not self.force:
|
||||||
# Determine expected WebP filename from the viz name
|
# Determine expected WebP filename from the viz name
|
||||||
@ -199,8 +216,14 @@ class LidarArchaeoPipeline:
|
|||||||
result = func(dtm_file, basename, file_vis_dir, self.resolution)
|
result = func(dtm_file, basename, file_vis_dir, self.resolution)
|
||||||
vis_results[name] = result
|
vis_results[name] = result
|
||||||
elapsed = time.time() - t0
|
elapsed = time.time() - t0
|
||||||
|
elapsed_times.append(elapsed)
|
||||||
if result:
|
if result:
|
||||||
logger.info(f" [{idx}/{total}] ✓ {name} ({elapsed:.1f}s)")
|
eta = ""
|
||||||
|
if len(elapsed_times) > 1:
|
||||||
|
avg_time = sum(elapsed_times) / len(elapsed_times)
|
||||||
|
remaining = (total - idx) * avg_time
|
||||||
|
eta = f" — ETA: {remaining:.0f}s"
|
||||||
|
logger.info(f" [{idx}/{total}] ✓ {name} ({elapsed:.1f}s){eta}")
|
||||||
else:
|
else:
|
||||||
logger.warning(f" [{idx}/{total}] ✗ {name} — no output ({elapsed:.1f}s)")
|
logger.warning(f" [{idx}/{total}] ✗ {name} — no output ({elapsed:.1f}s)")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@ -214,7 +237,7 @@ class LidarArchaeoPipeline:
|
|||||||
logger.info(" Conversion images WebP:")
|
logger.info(" Conversion images WebP:")
|
||||||
for name, tif_file in vis_results.items():
|
for name, tif_file in vis_results.items():
|
||||||
if tif_file and isinstance(tif_file, Path) and tif_file.suffix == '.tif' and tif_file.exists():
|
if tif_file and isinstance(tif_file, Path) and tif_file.suffix == '.tif' and tif_file.exists():
|
||||||
webp_file = tif_to_png(tif_file, file_vis_dir, self.resolution)
|
webp_file = tif_to_png(tif_file, file_vis_dir, self.resolution, keep_tif=self.keep_tif)
|
||||||
if webp_file:
|
if webp_file:
|
||||||
logger.info(f" ✓ {webp_file.name}")
|
logger.info(f" ✓ {webp_file.name}")
|
||||||
|
|
||||||
@ -293,7 +316,7 @@ class LidarArchaeoPipeline:
|
|||||||
logger.info(f"Fichiers: {len(files)}")
|
logger.info(f"Fichiers: {len(files)}")
|
||||||
with ProcessPoolExecutor(max_workers=self.workers) as executor:
|
with ProcessPoolExecutor(max_workers=self.workers) as executor:
|
||||||
future_to_file = {
|
future_to_file = {
|
||||||
executor.submit(_process_file_standalone, str(laz_file), str(self.input_dir), str(self.output_dir), self.resolution, self.force, self.ground_method, self.force_classify): laz_file
|
executor.submit(_process_file_standalone, str(laz_file), str(self.input_dir), str(self.output_dir), self.resolution, self.force, self.ground_method, self.force_classify, self.keep_tif): laz_file
|
||||||
for laz_file in files
|
for laz_file in files
|
||||||
}
|
}
|
||||||
done = 0
|
done = 0
|
||||||
@ -346,16 +369,16 @@ class LidarArchaeoPipeline:
|
|||||||
try:
|
try:
|
||||||
if self.temp_dir.exists():
|
if self.temp_dir.exists():
|
||||||
shutil.rmtree(self.temp_dir)
|
shutil.rmtree(self.temp_dir)
|
||||||
# Also clean up per-file temp directories from parallel workers
|
# Also clean up any subdirectories inside temp/
|
||||||
for d in self.output_dir.glob("temp_*"):
|
temp_base = self.output_dir / "temp"
|
||||||
if d.is_dir():
|
if temp_base.exists():
|
||||||
shutil.rmtree(d, ignore_errors=True)
|
shutil.rmtree(temp_base)
|
||||||
logger.info(" ✓ Fichiers temporaires supprimés")
|
logger.info(" ✓ Fichiers temporaires supprimés")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(f" Note: Impossible de supprimer les fichiers temporaires: {e}")
|
logger.warning(f" Note: Impossible de supprimer les fichiers temporaires: {e}")
|
||||||
|
|
||||||
|
|
||||||
def _process_file_standalone(laz_file_str, input_dir, output_dir, resolution, force=False, ground_method='auto', force_classify=False):
|
def _process_file_standalone(laz_file_str, input_dir, output_dir, resolution, force=False, ground_method='auto', force_classify=False, keep_tif=False):
|
||||||
"""Standalone function for multiprocessing — creates its own pipeline instance.
|
"""Standalone function for multiprocessing — creates its own pipeline instance.
|
||||||
|
|
||||||
Each worker gets its own temp directory to avoid file conflicts.
|
Each worker gets its own temp directory to avoid file conflicts.
|
||||||
@ -371,9 +394,9 @@ def _process_file_standalone(laz_file_str, input_dir, output_dir, resolution, fo
|
|||||||
worker_logger.addHandler(handler)
|
worker_logger.addHandler(handler)
|
||||||
worker_logger.addFilter(_file_filter)
|
worker_logger.addFilter(_file_filter)
|
||||||
|
|
||||||
pipeline = LidarArchaeoPipeline(input_dir, output_dir, resolution=resolution, workers=1, force=force, ground_method=ground_method, force_classify=force_classify)
|
pipeline = LidarArchaeoPipeline(input_dir, output_dir, resolution=resolution, workers=1, force=force, ground_method=ground_method, force_classify=force_classify, keep_tif=keep_tif)
|
||||||
basename = _file_basename(laz_file_str)
|
basename = _file_basename(laz_file_str)
|
||||||
pipeline.temp_dir = pipeline.output_dir / f"temp_{basename}"
|
pipeline.temp_dir = pipeline.output_dir / "temp" / basename
|
||||||
pipeline.temp_dir.mkdir(exist_ok=True)
|
pipeline.temp_dir.mkdir(exist_ok=True)
|
||||||
laz_file = Path(laz_file_str)
|
laz_file = Path(laz_file_str)
|
||||||
result = pipeline.process_file(laz_file)
|
result = pipeline.process_file(laz_file)
|
||||||
|
|||||||
@ -24,7 +24,6 @@ import matplotlib
|
|||||||
matplotlib.use('Agg')
|
matplotlib.use('Agg')
|
||||||
import matplotlib.pyplot as plt
|
import matplotlib.pyplot as plt
|
||||||
from matplotlib import rcParams
|
from matplotlib import rcParams
|
||||||
from matplotlib.gridspec import GridSpec
|
|
||||||
from matplotlib.patches import Polygon as MplPolygon, Rectangle as RectPatch
|
from matplotlib.patches import Polygon as MplPolygon, Rectangle as RectPatch
|
||||||
from mpl_toolkits.axes_grid1.inset_locator import inset_axes
|
from mpl_toolkits.axes_grid1.inset_locator import inset_axes
|
||||||
|
|
||||||
@ -119,14 +118,6 @@ COLORMAPS = {
|
|||||||
'description': 'Identifie la position topographique — utile pour repérer crêtes vs vallées à grande échelle',
|
'description': 'Identifie la position topographique — utile pour repérer crêtes vs vallées à grande échelle',
|
||||||
'vmin_mode': 'symmetric', 'sym_pct': (2, 98),
|
'vmin_mode': 'symmetric', 'sym_pct': (2, 98),
|
||||||
},
|
},
|
||||||
'depressions': {
|
|
||||||
'cmap': 'YlOrRd',
|
|
||||||
'title': 'Dépressions (Remplissage hydrologique)',
|
|
||||||
'legend': 'Profondeur des cuvettes détectées (m)\nTransparent = Pas de dépression\nJaune = Dépression légère | Rouge = Dépression profonde\nMax: {vmax:.2f}m',
|
|
||||||
'description': 'Simule le remplissage d\'eau — détecte dolines, sinkholes, cuvettes et zones inondables',
|
|
||||||
'vmin_mode': 'fixed', 'vmin_val': 0,
|
|
||||||
'vmax_mode': 'percentile', 'vmax_pct': 99,
|
|
||||||
},
|
|
||||||
'sailore': {
|
'sailore': {
|
||||||
'cmap': 'seismic',
|
'cmap': 'seismic',
|
||||||
'title': 'SAILORE - LRM Auto-Adaptatif',
|
'title': 'SAILORE - LRM Auto-Adaptatif',
|
||||||
@ -194,8 +185,9 @@ def _apply_colormap(data, tif_file):
|
|||||||
info = RGB_LEGENDS[key]
|
info = RGB_LEGENDS[key]
|
||||||
return data, None, info['title'], info['legend'], info['description'], True
|
return data, None, info['title'], info['legend'], info['description'], True
|
||||||
|
|
||||||
# Find matching colormap
|
# Find matching colormap — sort by key length descending so 'mslrm' matches before 'lrm'
|
||||||
for key, info in COLORMAPS.items():
|
for key in sorted(COLORMAPS.keys(), key=len, reverse=True):
|
||||||
|
info = COLORMAPS[key]
|
||||||
if key in name:
|
if key in name:
|
||||||
valid_data = np.asarray(data.compressed() if hasattr(data, 'compressed') else data.flatten())
|
valid_data = np.asarray(data.compressed() if hasattr(data, 'compressed') else data.flatten())
|
||||||
valid_data = valid_data[~np.isnan(valid_data)]
|
valid_data = valid_data[~np.isnan(valid_data)]
|
||||||
@ -246,13 +238,14 @@ def _apply_colormap(data, tif_file):
|
|||||||
return data, 'terrain', title, 'Altitude normalisée', '', False
|
return data, 'terrain', title, 'Altitude normalisée', '', False
|
||||||
|
|
||||||
|
|
||||||
def tif_to_png(tif_file, vis_dir, resolution):
|
def tif_to_png(tif_file, vis_dir, resolution, keep_tif=False):
|
||||||
"""Convert GeoTIFF to visualization WebP with GPS coordinates, legend, and scale bar.
|
"""Convert GeoTIFF to visualization WebP with GPS coordinates, legend, and scale bar.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
tif_file: Path to input GeoTIFF.
|
tif_file: Path to input GeoTIFF.
|
||||||
vis_dir: Output directory for the WebP file.
|
vis_dir: Output directory for the WebP file.
|
||||||
resolution: Grid resolution in m/px.
|
resolution: Grid resolution in m/px.
|
||||||
|
keep_tif: If True, keep the source TIFF after conversion.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Path to output WebP file, or None on failure.
|
Path to output WebP file, or None on failure.
|
||||||
@ -359,18 +352,21 @@ def tif_to_png(tif_file, vis_dir, resolution):
|
|||||||
saved_vmin = None
|
saved_vmin = None
|
||||||
saved_vmax = None
|
saved_vmax = None
|
||||||
|
|
||||||
# Create figure — adapt width to data resolution for sharp rendering
|
# Create figure with FIXED layout for consistent data area position
|
||||||
# At high res (5000+px wide), we need a larger figure to avoid downsampling artifacts
|
# All visualizations use the same axes positions so they can be overlaid
|
||||||
fig_width = max(20, width / 150)
|
fig_width = max(20, width / 150)
|
||||||
fig_width = min(fig_width, 40) # cap at 40 inches
|
fig_width = min(fig_width, 40)
|
||||||
map_aspect = height / width
|
fig_height = fig_width * 0.7 + 2.0 # Fixed header + footer space
|
||||||
fig = plt.figure(figsize=(fig_width, fig_width * map_aspect * 0.7 + 2.5),
|
fig = plt.figure(figsize=(fig_width, fig_height), facecolor='white')
|
||||||
facecolor='white')
|
|
||||||
gs = GridSpec(2, 1, height_ratios=[1.0, 0.06],
|
|
||||||
hspace=0.04, figure=fig,
|
|
||||||
left=0.06, right=0.88, top=0.93, bottom=0.08)
|
|
||||||
|
|
||||||
ax = fig.add_subplot(gs[0])
|
# Fixed data area position — identical for ALL visualization types
|
||||||
|
# This ensures overlay/superposition works across all WebP images
|
||||||
|
data_left = 0.08
|
||||||
|
data_bottom = 0.12
|
||||||
|
data_width_frac = 0.74
|
||||||
|
data_height_frac = 0.78
|
||||||
|
|
||||||
|
ax = fig.add_axes([data_left, data_bottom, data_width_frac, data_height_frac])
|
||||||
if is_rgba or is_rgb:
|
if is_rgba or is_rgb:
|
||||||
im = ax.imshow(data, aspect='equal', origin='upper',
|
im = ax.imshow(data, aspect='equal', origin='upper',
|
||||||
interpolation='bilinear')
|
interpolation='bilinear')
|
||||||
@ -380,22 +376,34 @@ def tif_to_png(tif_file, vis_dir, resolution):
|
|||||||
|
|
||||||
ax.set_title(f"{title}\n{description}", fontsize=15, fontweight='bold', pad=10)
|
ax.set_title(f"{title}\n{description}", fontsize=15, fontweight='bold', pad=10)
|
||||||
|
|
||||||
if not is_rgb:
|
# Colorbar/legend area — always at the same position for consistent layout
|
||||||
if is_rgba and saved_cmap is not None:
|
cbar_left = data_left + data_width_frac + 0.02
|
||||||
# Create a ScalarMappable for the colorbar from the saved colormap
|
cbar_width = 0.04
|
||||||
|
if is_rgb:
|
||||||
|
# RGB: descriptive text label instead of gradient colorbar
|
||||||
|
cbar_ax = fig.add_axes([cbar_left, data_bottom, cbar_width, data_height_frac])
|
||||||
|
cbar_ax.set_xticks([])
|
||||||
|
cbar_ax.set_yticks([])
|
||||||
|
cbar_ax.text(0.5, 0.5, legend_label, transform=cbar_ax.transAxes,
|
||||||
|
fontsize=9, fontweight='bold', rotation=90,
|
||||||
|
verticalalignment='center', horizontalalignment='center',
|
||||||
|
wrap=True)
|
||||||
|
cbar_ax.set_frame_on(False)
|
||||||
|
elif is_rgba and saved_cmap is not None:
|
||||||
|
cbar_ax = fig.add_axes([cbar_left, data_bottom, cbar_width, data_height_frac])
|
||||||
sm = plt.cm.ScalarMappable(cmap=saved_cmap,
|
sm = plt.cm.ScalarMappable(cmap=saved_cmap,
|
||||||
norm=plt.Normalize(vmin=saved_vmin, vmax=saved_vmax))
|
norm=plt.Normalize(vmin=saved_vmin, vmax=saved_vmax))
|
||||||
sm.set_array([])
|
sm.set_array([])
|
||||||
cbar = plt.colorbar(sm, ax=ax, pad=0.02, shrink=0.85, aspect=30)
|
cbar = plt.colorbar(sm, cax=cbar_ax)
|
||||||
else:
|
|
||||||
cbar = plt.colorbar(im, ax=ax, pad=0.02, shrink=0.85, aspect=30)
|
|
||||||
cbar.ax.tick_params(labelsize=9, width=1.5)
|
cbar.ax.tick_params(labelsize=9, width=1.5)
|
||||||
cbar.outline.set_linewidth(1.5)
|
cbar.outline.set_linewidth(1.5)
|
||||||
cbar.set_label(legend_label, fontsize=10, fontweight='bold')
|
cbar.set_label(legend_label, fontsize=10, fontweight='bold')
|
||||||
else:
|
else:
|
||||||
ax.text(1.02, 0.5, legend_label, transform=ax.transAxes,
|
cbar_ax = fig.add_axes([cbar_left, data_bottom, cbar_width, data_height_frac])
|
||||||
fontsize=10, fontweight='bold', rotation=90,
|
cbar = plt.colorbar(im, cax=cbar_ax)
|
||||||
verticalalignment='center', horizontalalignment='left')
|
cbar.ax.tick_params(labelsize=9, width=1.5)
|
||||||
|
cbar.outline.set_linewidth(1.5)
|
||||||
|
cbar.set_label(legend_label, fontsize=10, fontweight='bold')
|
||||||
|
|
||||||
# GPS coordinate ticks
|
# GPS coordinate ticks
|
||||||
if gps_coords and 'x_ticks' in gps_coords:
|
if gps_coords and 'x_ticks' in gps_coords:
|
||||||
@ -444,8 +452,8 @@ def tif_to_png(tif_file, vis_dir, resolution):
|
|||||||
north_ax.text(0.5, 0.95, 'N', ha='center', va='top',
|
north_ax.text(0.5, 0.95, 'N', ha='center', va='top',
|
||||||
fontsize=13, fontweight='bold', color='black', zorder=11)
|
fontsize=13, fontweight='bold', color='black', zorder=11)
|
||||||
|
|
||||||
# Bottom info bar
|
# Bottom info bar — fixed position
|
||||||
info_ax = fig.add_subplot(gs[1])
|
info_ax = fig.add_axes([data_left, 0.02, data_width_frac + cbar_width + 0.02, 0.07])
|
||||||
info_ax.axis('off')
|
info_ax.axis('off')
|
||||||
|
|
||||||
extent_km_x = (max_x - min_x) / 1000
|
extent_km_x = (max_x - min_x) / 1000
|
||||||
@ -496,12 +504,11 @@ def tif_to_png(tif_file, vis_dir, resolution):
|
|||||||
|
|
||||||
fig.patch.set_facecolor('white')
|
fig.patch.set_facecolor('white')
|
||||||
|
|
||||||
# Save as PNG then convert to WebP — use higher DPI for large data
|
# Save as PNG then convert to WebP — fixed layout, no bbox_inches='tight'
|
||||||
save_dpi = 200 if width > 3000 else 150
|
save_dpi = 200 if width > 3000 else 150
|
||||||
png_temp = vis_dir / f"{tif_file.stem}_temp.png"
|
png_temp = vis_dir / f"{tif_file.stem}_temp.png"
|
||||||
try:
|
try:
|
||||||
plt.savefig(png_temp, dpi=save_dpi, bbox_inches='tight', pad_inches=0.15,
|
plt.savefig(png_temp, dpi=save_dpi, facecolor='white', format='png')
|
||||||
facecolor='white', format='png')
|
|
||||||
finally:
|
finally:
|
||||||
plt.close()
|
plt.close()
|
||||||
|
|
||||||
@ -509,7 +516,8 @@ def tif_to_png(tif_file, vis_dir, resolution):
|
|||||||
img.save(str(webp_file), format='WEBP', lossless=True)
|
img.save(str(webp_file), format='WEBP', lossless=True)
|
||||||
png_temp.unlink(missing_ok=True)
|
png_temp.unlink(missing_ok=True)
|
||||||
|
|
||||||
# Delete source TIFF
|
# Delete source TIFF (unless --keep-tif)
|
||||||
|
if not keep_tif:
|
||||||
tif_file.unlink(missing_ok=True)
|
tif_file.unlink(missing_ok=True)
|
||||||
|
|
||||||
return webp_file
|
return webp_file
|
||||||
@ -565,7 +573,7 @@ def generate_pdf_report(basename, vis_dir, pdf_dir, resolution):
|
|||||||
|
|
||||||
# Sort analysis files by archaeological priority
|
# Sort analysis files by archaeological priority
|
||||||
order = ['mslrm', 'svf', 'negative_openness',
|
order = ['mslrm', 'svf', 'negative_openness',
|
||||||
'positive_openness', 'sailore', 'depressions', 'hillshade_multi',
|
'positive_openness', 'sailore', 'hillshade_multi',
|
||||||
'lrm', 'tpi', 'slope', 'curvature', 'aspect',
|
'lrm', 'tpi', 'slope', 'curvature', 'aspect',
|
||||||
'roughness', 'anomalies', 'wavelet', 'flow']
|
'roughness', 'anomalies', 'wavelet', 'flow']
|
||||||
|
|
||||||
|
|||||||
@ -17,6 +17,7 @@ class TestCLIParsing:
|
|||||||
parser.add_argument("-w", "--workers", type=int, default=1)
|
parser.add_argument("-w", "--workers", type=int, default=1)
|
||||||
parser.add_argument("-f", "--force", action="store_true")
|
parser.add_argument("-f", "--force", action="store_true")
|
||||||
parser.add_argument("--file", nargs="+", type=str, default=None)
|
parser.add_argument("--file", nargs="+", type=str, default=None)
|
||||||
|
parser.add_argument("--keep-tif", action="store_true")
|
||||||
|
|
||||||
args = parser.parse_args(["./input"])
|
args = parser.parse_args(["./input"])
|
||||||
assert args.input == "./input"
|
assert args.input == "./input"
|
||||||
@ -25,6 +26,7 @@ class TestCLIParsing:
|
|||||||
assert args.workers == 1
|
assert args.workers == 1
|
||||||
assert args.force is False
|
assert args.force is False
|
||||||
assert args.file is None
|
assert args.file is None
|
||||||
|
assert args.keep_tif is False
|
||||||
|
|
||||||
def test_file_flag_single(self):
|
def test_file_flag_single(self):
|
||||||
import argparse
|
import argparse
|
||||||
@ -65,7 +67,7 @@ class TestSetupLogging:
|
|||||||
assert len(logger.handlers) == 1
|
assert len(logger.handlers) == 1
|
||||||
# Format should not include timestamps
|
# Format should not include timestamps
|
||||||
fmt = logger.handlers[0].formatter._fmt
|
fmt = logger.handlers[0].formatter._fmt
|
||||||
assert "%(asctime)" not in fmt
|
assert "%(asctime)s" not in fmt
|
||||||
|
|
||||||
def test_verbose_logging(self):
|
def test_verbose_logging(self):
|
||||||
"""Verbose logging includes timestamps."""
|
"""Verbose logging includes timestamps."""
|
||||||
@ -73,7 +75,7 @@ class TestSetupLogging:
|
|||||||
from lidar_pipeline.cli import setup_logging
|
from lidar_pipeline.cli import setup_logging
|
||||||
logger = setup_logging(verbose=True, debug=False)
|
logger = setup_logging(verbose=True, debug=False)
|
||||||
fmt = logger.handlers[0].formatter._fmt
|
fmt = logger.handlers[0].formatter._fmt
|
||||||
assert "%(asctime)" in fmt
|
assert "%(asctime)s" in fmt
|
||||||
|
|
||||||
def test_debug_logging(self):
|
def test_debug_logging(self):
|
||||||
"""Debug logging includes file:line info."""
|
"""Debug logging includes file:line info."""
|
||||||
@ -82,5 +84,5 @@ class TestSetupLogging:
|
|||||||
logger = setup_logging(verbose=False, debug=True)
|
logger = setup_logging(verbose=False, debug=True)
|
||||||
assert logger.level == logging.DEBUG
|
assert logger.level == logging.DEBUG
|
||||||
fmt = logger.handlers[0].formatter._fmt
|
fmt = logger.handlers[0].formatter._fmt
|
||||||
assert "%(filename)" in fmt
|
assert "%(filename)s" in fmt
|
||||||
assert "%(lineno)" in fmt
|
assert "%(lineno)d" in fmt
|
||||||
@ -11,12 +11,12 @@ def test_has_gpu_attribute():
|
|||||||
|
|
||||||
|
|
||||||
def test_to_gpu_returns_array():
|
def test_to_gpu_returns_array():
|
||||||
"""to_gpu returns a float64 array with correct values."""
|
"""to_gpu returns a float32 array with correct values."""
|
||||||
from lidar_pipeline.gpu import to_gpu, to_cpu, HAS_GPU
|
from lidar_pipeline.gpu import to_gpu, to_cpu, HAS_GPU
|
||||||
arr = np.array([1.0, 2.0, 3.0])
|
arr = np.array([1.0, 2.0, 3.0])
|
||||||
result = to_gpu(arr)
|
result = to_gpu(arr)
|
||||||
# On GPU: cupy.ndarray, on CPU: numpy.ndarray
|
# to_gpu converts to float32 to reduce GPU memory usage
|
||||||
assert result.dtype == np.float64
|
assert result.dtype == np.float32
|
||||||
# Always bring back to CPU for comparison
|
# Always bring back to CPU for comparison
|
||||||
np.testing.assert_array_equal(to_cpu(result), [1.0, 2.0, 3.0])
|
np.testing.assert_array_equal(to_cpu(result), [1.0, 2.0, 3.0])
|
||||||
|
|
||||||
|
|||||||
@ -26,9 +26,9 @@ class TestVizSteps:
|
|||||||
assert "solar" not in names
|
assert "solar" not in names
|
||||||
|
|
||||||
def test_expected_visualization_count(self):
|
def test_expected_visualization_count(self):
|
||||||
"""Should have 19 visualizations (18 terrain + ortho + topo - solar)."""
|
"""Should have 17 visualizations (15 terrain + ortho + topo)."""
|
||||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||||
assert len(VIZ_STEPS) == 19
|
assert len(VIZ_STEPS) == 17
|
||||||
|
|
||||||
def test_ortho_and_topo_present(self):
|
def test_ortho_and_topo_present(self):
|
||||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||||
|
|||||||
@ -141,13 +141,6 @@ class TestTPI:
|
|||||||
assert result.exists()
|
assert result.exists()
|
||||||
|
|
||||||
|
|
||||||
class TestDepressions:
|
|
||||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
|
||||||
from lidar_pipeline.visualizations import generate_depressions
|
|
||||||
result = generate_depressions(synthetic_dem, "test", tmp_output_dir, 5.0)
|
|
||||||
assert result is not None
|
|
||||||
assert result.exists()
|
|
||||||
|
|
||||||
|
|
||||||
class TestSAILORE:
|
class TestSAILORE:
|
||||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||||
|
|||||||
@ -117,7 +117,12 @@ def _filter_nanaware(arr, filter_func, *args, use_gpu=True, **kwargs):
|
|||||||
# ============================================================
|
# ============================================================
|
||||||
|
|
||||||
def generate_hillshade(dem_file, basename, vis_dir, resolution):
|
def generate_hillshade(dem_file, basename, vis_dir, resolution):
|
||||||
"""Generate multi-directional hillshade (NW, NE, SW, SE) — GPU if available."""
|
"""Generate multi-directional hillshade with slope shading — GPU if available.
|
||||||
|
|
||||||
|
Combines 4-direction hillshade (NW, NE, SW, SE) with slope shading
|
||||||
|
for improved micro-relief visibility on flat terrain.
|
||||||
|
Result = 0.7 * hillshade + 0.3 * cos(slope).
|
||||||
|
"""
|
||||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||||
logger.info(f" → Hillshade multidirectionnel{gpu_tag}...")
|
logger.info(f" → Hillshade multidirectionnel{gpu_tag}...")
|
||||||
t0 = time.time()
|
t0 = time.time()
|
||||||
@ -146,7 +151,10 @@ def generate_hillshade(dem_file, basename, vis_dir, resolution):
|
|||||||
hs = sin_alt * sin_slope + cos_alt * cos_slope * xp.cos(az_rad - aspect)
|
hs = sin_alt * sin_slope + cos_alt * cos_slope * xp.cos(az_rad - aspect)
|
||||||
hillshades.append(xp.clip(hs, 0, 1))
|
hillshades.append(xp.clip(hs, 0, 1))
|
||||||
|
|
||||||
combined = xp.mean(xp.array(hillshades), axis=0)
|
combined_hillshade = xp.mean(xp.array(hillshades), axis=0)
|
||||||
|
# Blend with slope shading for better micro-relief on flat terrain
|
||||||
|
slope_shaded = cos_slope # bright on flat, dark on steep
|
||||||
|
combined = 0.7 * combined_hillshade + 0.3 * slope_shaded
|
||||||
_save_tif(output, to_cpu(combined), transform, crs)
|
_save_tif(output, to_cpu(combined), transform, crs)
|
||||||
logger.info(f" ✓ Hillshade terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
logger.info(f" ✓ Hillshade terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||||
return output
|
return output
|
||||||
@ -240,8 +248,6 @@ def generate_lrm(dem_file, basename, vis_dir, resolution):
|
|||||||
_save_tif(output, lrm.astype(np.float32), transform, crs)
|
_save_tif(output, lrm.astype(np.float32), transform, crs)
|
||||||
logger.info(f" ✓ LRM terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
logger.info(f" ✓ LRM terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||||
return output
|
return output
|
||||||
logger.info(f" ✓ LRM terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
|
||||||
return output
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f" ✗ Erreur LRM: {e}", exc_info=True)
|
logger.error(f" ✗ Erreur LRM: {e}", exc_info=True)
|
||||||
return None
|
return None
|
||||||
@ -279,13 +285,18 @@ def generate_svf(dem_file, basename, vis_dir, resolution):
|
|||||||
ddx, ddy = dx[d_idx], dy[d_idx]
|
ddx, ddy = dx[d_idx], dy[d_idx]
|
||||||
horizon = xp.zeros_like(dem)
|
horizon = xp.zeros_like(dem)
|
||||||
|
|
||||||
|
# Pre-compute all valid steps for this direction
|
||||||
|
valid_steps = []
|
||||||
for step in range(1, max_dist + 1):
|
for step in range(1, max_dist + 1):
|
||||||
px = int(round(ddx * step))
|
px = int(round(ddx * step))
|
||||||
py = int(round(ddy * step))
|
py = int(round(ddy * step))
|
||||||
dist_m = np.sqrt((ddx * step * res) ** 2 + (ddy * step * res) ** 2)
|
dist_m = np.sqrt((ddx * step * res) ** 2 + (ddy * step * res) ** 2)
|
||||||
if dist_m < res * 0.5:
|
if dist_m < res * 0.5:
|
||||||
continue
|
continue
|
||||||
|
valid_steps.append((step, px, py, dist_m))
|
||||||
|
|
||||||
|
# Batch all shifts into a single array for vectorized max computation
|
||||||
|
for step, px, py, dist_m in valid_steps:
|
||||||
elev_diff = padded[max_dist + py:max_dist + py + rows,
|
elev_diff = padded[max_dist + py:max_dist + py + rows,
|
||||||
max_dist + px:max_dist + px + cols] - dem
|
max_dist + px:max_dist + px + cols] - dem
|
||||||
angle = xp.arctan2(elev_diff, dist_m)
|
angle = xp.arctan2(elev_diff, dist_m)
|
||||||
@ -447,55 +458,6 @@ def generate_tpi(dem_file, basename, vis_dir, resolution):
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
# ============================================================
|
|
||||||
# Depression / hydrology
|
|
||||||
# ============================================================
|
|
||||||
|
|
||||||
def generate_depressions(dem_file, basename, vis_dir, resolution):
|
|
||||||
"""Depression detection using hydrological sink filling — GPU if available."""
|
|
||||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
|
||||||
logger.info(f" → Détection dépressions (hydrologique){gpu_tag}...")
|
|
||||||
t0 = time.time()
|
|
||||||
output = vis_dir / f"{basename}_depressions.tif"
|
|
||||||
|
|
||||||
try:
|
|
||||||
dem_np, transform, crs = _read_dem(dem_file)
|
|
||||||
dem = to_gpu(dem_np)
|
|
||||||
|
|
||||||
from scipy.ndimage import generate_binary_structure
|
|
||||||
struct = generate_binary_structure(2, 2)
|
|
||||||
|
|
||||||
dem_filled = xp.copy(dem)
|
|
||||||
nodata_mask = xp.isnan(dem_filled)
|
|
||||||
dem_filled[nodata_mask] = xp.nanmax(dem) + 1000
|
|
||||||
|
|
||||||
changed = True
|
|
||||||
iterations = 0
|
|
||||||
max_iter = 100
|
|
||||||
|
|
||||||
while changed and iterations < max_iter:
|
|
||||||
neighbor_min = xp_minimum_filter(dem_filled, footprint=struct)
|
|
||||||
sinks = (dem_filled < neighbor_min) & ~nodata_mask
|
|
||||||
|
|
||||||
if not xp.any(sinks):
|
|
||||||
break
|
|
||||||
|
|
||||||
new_dem = xp.maximum(dem_filled, neighbor_min)
|
|
||||||
new_dem[nodata_mask] = xp.nan
|
|
||||||
changed = bool(xp.any(new_dem != dem_filled))
|
|
||||||
dem_filled = new_dem
|
|
||||||
iterations += 1
|
|
||||||
|
|
||||||
depressions = to_cpu(dem_filled - dem)
|
|
||||||
depressions[to_cpu(nodata_mask)] = np.nan
|
|
||||||
depressions = np.where(depressions > 0.01, depressions, 0)
|
|
||||||
|
|
||||||
_save_tif(output, depressions, transform, crs)
|
|
||||||
logger.info(f" ✓ Dépressions terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
|
||||||
return output
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f" ✗ Erreur dépressions: {e}", exc_info=True)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================
|
# ============================================================
|
||||||
@ -680,8 +642,63 @@ def generate_wavelet(dem_file, basename, vis_dir, resolution):
|
|||||||
# Flow accumulation
|
# Flow accumulation
|
||||||
# ============================================================
|
# ============================================================
|
||||||
|
|
||||||
|
def _d8_accumulate_numba(flow_dir, nodata_mask, rows, cols):
|
||||||
|
"""JIT-compiled D8 flow accumulation loop.
|
||||||
|
|
||||||
|
Uses numba for ~100x speedup over pure Python loop.
|
||||||
|
Falls back to pure Python if numba is unavailable.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from numba import njit
|
||||||
|
|
||||||
|
@njit(cache=True)
|
||||||
|
def _accumulate(flow_dir, nodata_mask, rows, cols):
|
||||||
|
dx8 = np.array([1, 1, 0, -1, -1, -1, 0, 1], dtype=np.int8)
|
||||||
|
dy8 = np.array([0, 1, 1, 1, 0, -1, -1, -1], dtype=np.int8)
|
||||||
|
|
||||||
|
flow_acc = np.ones((rows, cols), dtype=np.float32)
|
||||||
|
|
||||||
|
# Sort cells by elevation (high to low) — walk downhill
|
||||||
|
# We use the fact that flow_dir already encodes steepest descent
|
||||||
|
# Process from highest to lowest elevation
|
||||||
|
for r in range(rows):
|
||||||
|
for c in range(cols):
|
||||||
|
if nodata_mask[r, c]:
|
||||||
|
flow_acc[r, c] = 0.0
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Iterative accumulation: process cells in top-down order
|
||||||
|
# Multiple passes until convergence
|
||||||
|
for _pass in range(10):
|
||||||
|
changed = 0
|
||||||
|
for r in range(rows):
|
||||||
|
for c in range(cols):
|
||||||
|
if nodata_mask[r, c]:
|
||||||
|
continue
|
||||||
|
d = flow_dir[r, c]
|
||||||
|
if d < 0:
|
||||||
|
continue
|
||||||
|
nr = r + dy8[d]
|
||||||
|
nc = c + dx8[d]
|
||||||
|
if 0 <= nr < rows and 0 <= nc < cols and not nodata_mask[nr, nc]:
|
||||||
|
old_acc = flow_acc[nr, nc]
|
||||||
|
flow_acc[nr, nc] += flow_acc[r, c]
|
||||||
|
if flow_acc[nr, nc] != old_acc:
|
||||||
|
changed += 1
|
||||||
|
if changed == 0:
|
||||||
|
break
|
||||||
|
|
||||||
|
return flow_acc
|
||||||
|
|
||||||
|
return _accumulate(flow_dir, nodata_mask, rows, cols)
|
||||||
|
|
||||||
|
except ImportError:
|
||||||
|
# Fallback: pure Python
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def generate_flow(dem_file, basename, vis_dir, resolution):
|
def generate_flow(dem_file, basename, vis_dir, resolution):
|
||||||
"""Flow accumulation using D8 algorithm — sink filling on GPU, accumulation on CPU."""
|
"""Flow accumulation using D8 algorithm — sink filling on GPU, accumulation via numba."""
|
||||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||||
logger.info(f" → Accumulation de flux D8{gpu_tag}...")
|
logger.info(f" → Accumulation de flux D8{gpu_tag}...")
|
||||||
t0 = time.time()
|
t0 = time.time()
|
||||||
@ -711,10 +728,10 @@ def generate_flow(dem_file, basename, vis_dir, resolution):
|
|||||||
dem_filled[nodata_mask_gpu] = xp.nan
|
dem_filled[nodata_mask_gpu] = xp.nan
|
||||||
dem_filled_np = to_cpu(dem_filled)
|
dem_filled_np = to_cpu(dem_filled)
|
||||||
|
|
||||||
# D8 slope + accumulation — CPU (sequential by nature)
|
# D8 slope — vectorized
|
||||||
dx8 = [1, 1, 0, -1, -1, -1, 0, 1]
|
dx8 = np.array([1, 1, 0, -1, -1, -1, 0, 1], dtype=np.int32)
|
||||||
dy8 = [0, 1, 1, 1, 0, -1, -1, -1]
|
dy8 = np.array([0, 1, 1, 1, 0, -1, -1, -1], dtype=np.int32)
|
||||||
dist8 = [1.0, np.sqrt(2), 1.0, np.sqrt(2), 1.0, np.sqrt(2), 1.0, np.sqrt(2)]
|
dist8 = np.array([1.0, np.sqrt(2), 1.0, np.sqrt(2), 1.0, np.sqrt(2), 1.0, np.sqrt(2)])
|
||||||
|
|
||||||
flow_dir = np.full((rows, cols), -1, dtype=np.int8)
|
flow_dir = np.full((rows, cols), -1, dtype=np.int8)
|
||||||
max_slope = np.zeros((rows, cols), dtype=np.float64)
|
max_slope = np.zeros((rows, cols), dtype=np.float64)
|
||||||
@ -732,6 +749,15 @@ def generate_flow(dem_file, basename, vis_dir, resolution):
|
|||||||
flow_dir[better] = d
|
flow_dir[better] = d
|
||||||
max_slope[better] = slope[better]
|
max_slope[better] = slope[better]
|
||||||
|
|
||||||
|
# D8 accumulation — try numba first, fallback to Python
|
||||||
|
result = _d8_accumulate_numba(flow_dir, nodata_mask.astype(np.bool_), rows, cols)
|
||||||
|
|
||||||
|
if result is not None:
|
||||||
|
flow_acc = result
|
||||||
|
logger.info(f" Accumulation D8 via numba")
|
||||||
|
else:
|
||||||
|
# Pure Python fallback (slow for large DEMs)
|
||||||
|
logger.info(f" Accumulation D8 via Python (installez numba pour accélérer)")
|
||||||
flat_dem = dem_filled_np[~nodata_mask].flatten()
|
flat_dem = dem_filled_np[~nodata_mask].flatten()
|
||||||
valid_indices = np.where(~nodata_mask.flatten())[0]
|
valid_indices = np.where(~nodata_mask.flatten())[0]
|
||||||
sort_order = valid_indices[np.argsort(-flat_dem)]
|
sort_order = valid_indices[np.argsort(-flat_dem)]
|
||||||
|
|||||||
8
run.sh
8
run.sh
@ -9,6 +9,7 @@
|
|||||||
# -v Mode verbeux (timestamps + niveaux)
|
# -v Mode verbeux (timestamps + niveaux)
|
||||||
# --debug Mode debug (détails internes fichier:ligne)
|
# --debug Mode debug (détails internes fichier:ligne)
|
||||||
# -f / --force Régénérer tous les fichiers même si existants
|
# -f / --force Régénérer tous les fichiers même si existants
|
||||||
|
# --keep-tif Conserver les fichiers TIFF intermédiaires
|
||||||
# --force-classification
|
# --force-classification
|
||||||
# Reclassifier le sol même si le fichier .las existe déjà
|
# Reclassifier le sol même si le fichier .las existe déjà
|
||||||
# --ground-classification {auto,smrf,pmf,csf}
|
# --ground-classification {auto,smrf,pmf,csf}
|
||||||
@ -33,6 +34,7 @@ if [ $# -eq 0 ]; then
|
|||||||
echo " -f / --force Régénérer tous les fichiers même si les WebP existent"
|
echo " -f / --force Régénérer tous les fichiers même si les WebP existent"
|
||||||
echo " --force-classification"
|
echo " --force-classification"
|
||||||
echo " Reclassifier le sol même si le fichier .las existe"
|
echo " Reclassifier le sol même si le fichier .las existe"
|
||||||
|
echo " --keep-tif Conserver les fichiers TIFF intermédiaires"
|
||||||
echo " --ground-classification {auto,smrf,pmf,csf}"
|
echo " --ground-classification {auto,smrf,pmf,csf}"
|
||||||
echo " Méthode de classification du sol (défaut: auto)"
|
echo " Méthode de classification du sol (défaut: auto)"
|
||||||
echo " --file NOM... Traiter un ou plusieurs fichiers LAZ (nom complet sans .laz/.las)"
|
echo " --file NOM... Traiter un ou plusieurs fichiers LAZ (nom complet sans .laz/.las)"
|
||||||
@ -64,6 +66,7 @@ FORCE_FLAG=""
|
|||||||
FILE_ARGS=""
|
FILE_ARGS=""
|
||||||
GROUND_METHOD=""
|
GROUND_METHOD=""
|
||||||
FORCE_CLASSIFY_FLAG=""
|
FORCE_CLASSIFY_FLAG=""
|
||||||
|
KEEP_TIF_FLAG=""
|
||||||
|
|
||||||
# Parse arguments manually (more robust than getopts for mixed short/long options)
|
# Parse arguments manually (more robust than getopts for mixed short/long options)
|
||||||
while [ $# -gt 0 ]; do
|
while [ $# -gt 0 ]; do
|
||||||
@ -76,6 +79,7 @@ while [ $# -gt 0 ]; do
|
|||||||
--debug) VERBOSE_FLAG="--debug"; shift ;;
|
--debug) VERBOSE_FLAG="--debug"; shift ;;
|
||||||
--force) FORCE_FLAG="--force"; shift ;;
|
--force) FORCE_FLAG="--force"; shift ;;
|
||||||
--force-classification) FORCE_CLASSIFY_FLAG="--force-classification"; shift ;;
|
--force-classification) FORCE_CLASSIFY_FLAG="--force-classification"; shift ;;
|
||||||
|
--keep-tif) KEEP_TIF_FLAG="--keep-tif"; shift ;;
|
||||||
--ground-classification) GROUND_METHOD="$2"; shift 2 ;;
|
--ground-classification) GROUND_METHOD="$2"; shift 2 ;;
|
||||||
--ground-classification=*) GROUND_METHOD="${1#--ground-classification=}"; shift ;;
|
--ground-classification=*) GROUND_METHOD="${1#--ground-classification=}"; shift ;;
|
||||||
--file) shift; while [ $# -gt 0 ] && [[ ! "$1" =~ ^- ]]; do FILE_ARGS="$FILE_ARGS $1"; shift; done ;;
|
--file) shift; while [ $# -gt 0 ] && [[ ! "$1" =~ ^- ]]; do FILE_ARGS="$FILE_ARGS $1"; shift; done ;;
|
||||||
@ -93,6 +97,7 @@ while [ $# -gt 0 ]; do
|
|||||||
echo " -f / --force Régénérer tous les fichiers même si les WebP existent"
|
echo " -f / --force Régénérer tous les fichiers même si les WebP existent"
|
||||||
echo " --force-classification"
|
echo " --force-classification"
|
||||||
echo " Reclassifier le sol même si le fichier .las existe"
|
echo " Reclassifier le sol même si le fichier .las existe"
|
||||||
|
echo " --keep-tif Conserver les fichiers TIFF intermédiaires"
|
||||||
echo " --ground-classification {auto,smrf,pmf,csf}"
|
echo " --ground-classification {auto,smrf,pmf,csf}"
|
||||||
echo " Méthode de classification du sol (défaut: auto)"
|
echo " Méthode de classification du sol (défaut: auto)"
|
||||||
echo " --file NOM... Traiter un ou plusieurs fichiers LAZ (nom complet sans .laz/.las)"
|
echo " --file NOM... Traiter un ou plusieurs fichiers LAZ (nom complet sans .laz/.las)"
|
||||||
@ -153,13 +158,14 @@ echo " GPU : $([ -n "$GPU_FLAG" ] && echo 'OUI' || echo 'non')"
|
|||||||
echo " Verbeux : $([ -n "$VERBOSE_FLAG" ] && echo 'OUI' || echo 'non')"
|
echo " Verbeux : $([ -n "$VERBOSE_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||||
echo " Force : $([ -n "$FORCE_FLAG" ] && echo 'OUI' || echo 'non')"
|
echo " Force : $([ -n "$FORCE_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||||
echo " Force classif.: $([ -n "$FORCE_CLASSIFY_FLAG" ] && echo 'OUI' || echo 'non')"
|
echo " Force classif.: $([ -n "$FORCE_CLASSIFY_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||||
|
echo " Keep TIFF : $([ -n "$KEEP_TIF_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||||
echo " Classification sol : $([ -n "$GROUND_METHOD" ] && echo "$GROUND_METHOD" || echo 'auto')"
|
echo " Classification sol : $([ -n "$GROUND_METHOD" ] && echo "$GROUND_METHOD" || echo 'auto')"
|
||||||
if [ -n "$FILE_ARGS" ]; then
|
if [ -n "$FILE_ARGS" ]; then
|
||||||
echo " Fichiers :${FILE_ARGS}"
|
echo " Fichiers :${FILE_ARGS}"
|
||||||
fi
|
fi
|
||||||
echo "============================================"
|
echo "============================================"
|
||||||
|
|
||||||
CMD_ARGS="-o /data/output -r $RESOLUTION -w $WORKERS $VERBOSE_FLAG $FORCE_FLAG $FORCE_CLASSIFY_FLAG"
|
CMD_ARGS="-o /data/output -r $RESOLUTION -w $WORKERS $VERBOSE_FLAG $FORCE_FLAG $FORCE_CLASSIFY_FLAG $KEEP_TIF_FLAG"
|
||||||
if [ -n "$GROUND_METHOD" ]; then
|
if [ -n "$GROUND_METHOD" ]; then
|
||||||
CMD_ARGS="$CMD_ARGS --ground-classification $GROUND_METHOD"
|
CMD_ARGS="$CMD_ARGS --ground-classification $GROUND_METHOD"
|
||||||
fi
|
fi
|
||||||
|
|||||||
Reference in New Issue
Block a user