Suppression éclairage solaire, GPU accéléré, --file multi, tests unitaires
- Suppression de generate_solar (éclairage solaire) des visualisations - Accélération GPU de hillshade, slope, aspect, curvature, depressions, anomalies, roughness, texture GLCM, flow (sink filling) - Nettoyage mémoire GPU entre visualisations (gpu_cleanup) - Correction OOM texture GLCM: calcul entropie bin par bin au lieu d'un tableau 3D massif sur GPU - Correction bug: xp_minimum_filter manquant dans imports visualizations - Option --file accepte plusieurs noms complets sans extension - run.sh affiche l'aide si appelé sans arguments - Option --test pour exécuter les tests unitaires dans Docker - Filtre ReturnNumber>=1 intégré dans le pipeline PDAL (plus d'erreur SMRF) - 60 tests unitaires: GPU, visualisations, rendering, DTM, pipeline, CLI - Ajout pytest au Dockerfile Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Jacquin Antoine
@ -28,7 +28,8 @@ RUN pip3 install --no-cache-dir \
|
||||
scikit-learn \
|
||||
scipy \
|
||||
tqdm \
|
||||
Pillow
|
||||
Pillow \
|
||||
pytest
|
||||
|
||||
# Install CuPy for GPU acceleration (optional - will fallback to numpy if not available)
|
||||
RUN pip3 install --no-cache-dir cupy-cuda12x || echo "CuPy not available - GPU acceleration disabled"
|
||||
|
||||
54
README.md
54
README.md
@ -2,7 +2,7 @@
|
||||
|
||||
Workflow automatisé pour générer des visualisations exploitables à partir de données LiDAR pour la détection de structures archéologiques.
|
||||
|
||||
## Visualisations (20 par fichier)
|
||||
## Visualisations (19 par fichier)
|
||||
|
||||
### Visualisations principales
|
||||
| # | Visualisation | Utilité archéologique |
|
||||
@ -11,30 +11,29 @@ Workflow automatisé pour générer des visualisations exploitables à partir de
|
||||
| 2 | **Pente (Slope)** | Murs de soutènement, talus, changements brusques |
|
||||
| 3 | **Aspect (Orientation)** | Direction des pentes, exposition |
|
||||
| 4 | **Courbure (Curvature)** | Fossés, terrasses, talus, concavité/convexité |
|
||||
| 5 | **Éclairage solaire** | Simulation de l'éclairage matinal |
|
||||
| 6 | **Sky-View Factor** | Structures, tumulus, fondations (ray-tracing 16 azimuts) |
|
||||
| 7 | **Local Relief Model** | Micro-reliefs, fossés, levées de terrain |
|
||||
| 8 | **Positive Openness** | Élévations, tumulus, bâtiments (ray-tracing 8 directions) |
|
||||
| 9 | **Negative Openness** | Cavités, fossés, souterrains (ray-tracing 8 directions) |
|
||||
| 5 | **Sky-View Factor** | Structures, tumulus, fondations (ray-tracing 16 azimuts) |
|
||||
| 6 | **Local Relief Model** | Micro-reliefs, fossés, levées de terrain |
|
||||
| 7 | **Positive Openness** | Élévations, tumulus, bâtiments (ray-tracing 8 directions) |
|
||||
| 8 | **Negative Openness** | Cavités, fossés, souterrains (ray-tracing 8 directions) |
|
||||
|
||||
### Visualisations avancées
|
||||
| # | Visualisation | Description | Détection |
|
||||
|---|--------------|-------------|-----------|
|
||||
| 10 | **MSRM** | Multi-Scale Relief Model (sigma 5/10/25/50/100m) | Tumulus, fossés, murs à toutes les échelles |
|
||||
| 11 | **TPI multi-échelle** | Topographic Position Index (5m + 100m) | Crêtes, vallées, plateformes |
|
||||
| 12 | **Dépressions** | Remplissage cuvettes + différence | Dolines, sinkholes, zones inondables |
|
||||
| 13 | **SAILORE** | LRM adaptatif (noyau = f(pente)) | Terrain hétérogène, tout relief |
|
||||
| 14 | **Rugosité** | Écart-type de l'élévation | Surfaces anthropiques vs naturelles |
|
||||
| 15 | **Anomalies statistiques** | Z-score + Local Moran's I | Anomalies topographiques significatives |
|
||||
| 16 | **Ondelette Mexican Hat** | CWT 2D multi-échelle | Tumulus, fossés circulaires |
|
||||
| 17 | **Texture GLCM** | Contraste, entropie, homogénéité | Labour, surfaces archéologiques |
|
||||
| 18 | **Accumulation de flux** | Algorithme D8 hydrologique | Fossés d'enceinte, routes antiques |
|
||||
| 9 | **MSRM** | Multi-Scale Relief Model (sigma 5/10/25/50/100m) | Tumulus, fossés, murs à toutes les échelles |
|
||||
| 10 | **TPI multi-échelle** | Topographic Position Index (5m + 100m) | Crêtes, vallées, plateformes |
|
||||
| 11 | **Dépressions** | Remplissage cuvettes + différence | Dolines, sinkholes, zones inondables |
|
||||
| 12 | **SAILORE** | LRM adaptatif (noyau = f(pente)) | Terrain hétérogène, tout relief |
|
||||
| 13 | **Rugosité** | Écart-type de l'élévation | Surfaces anthropiques vs naturelles |
|
||||
| 14 | **Anomalies statistiques** | Z-score + Local Moran's I | Anomalies topographiques significatives |
|
||||
| 15 | **Ondelette Mexican Hat** | CWT 2D multi-échelle | Tumulus, fossés circulaires |
|
||||
| 16 | **Texture GLCM** | Contraste, entropie, homogénéité | Labour, surfaces archéologiques |
|
||||
| 17 | **Accumulation de flux** | Algorithme D8 hydrologique | Fossés d'enceinte, routes antiques |
|
||||
|
||||
### Cartes de référence IGN
|
||||
| # | Visualisation | Source |
|
||||
|---|--------------|--------|
|
||||
| 19 | **Photographie aérienne IGN** | Orthophotographie WMTS |
|
||||
| 20 | **Carte topographique IGN** | Plan IGN V2 WMTS |
|
||||
| 18 | **Photographie aérienne IGN** | Orthophotographie WMTS |
|
||||
| 19 | **Carte topographique IGN** | Plan IGN V2 WMTS |
|
||||
|
||||
## Architecture modulaire
|
||||
|
||||
@ -45,13 +44,13 @@ lidar_pipeline/
|
||||
├── cli.py # argparse + logging + main()
|
||||
├── gpu.py # CuPy/numpy abstraction (HAS_GPU, to_gpu, to_cpu, xp_*)
|
||||
├── dtm.py # Classification PDAL + génération DTM
|
||||
├── visualizations.py # Fonctions generate_* (20 visualisations)
|
||||
├── visualizations.py # Fonctions generate_* (19 visualisations)
|
||||
├── ign.py # Téléchargement tuiles IGN + overlay
|
||||
├── rendering.py # Colormaps, tif_to_png, rapport PDF
|
||||
└── pipeline.py # LidarArchaeoPipeline (orchestration + registry)
|
||||
```
|
||||
|
||||
Ajouter une visualisation = 1 fonction + 1 entrée dans `VIZ_STEPS` + 1 entrée dans `COLORMAPS`.
|
||||
Ajouter une visualisation = 1 fonction + 1 entrée dans `VIZ_STEPS` + 1 entrée dans `COLORMAPS`. Supprimer = retirer les 3 entrées correspondantes.
|
||||
|
||||
## Installation Docker
|
||||
|
||||
@ -87,7 +86,7 @@ docker build -t lidar-lidar .
|
||||
-v Mode verbeux (timestamps + niveaux)
|
||||
--debug Mode debug (détails internes fichier:ligne)
|
||||
-f / --force Régénérer tous les fichiers même si les WebP existent
|
||||
--file NOM Traiter un seul fichier LAZ (pour tests)
|
||||
--file NOM... Traiter un ou plusieurs fichiers LAZ spécifiques (nom partiel)
|
||||
-h Afficher l'aide
|
||||
```
|
||||
|
||||
@ -111,8 +110,11 @@ docker build -t lidar-lidar .
|
||||
# Forcer la régénération de tous les fichiers
|
||||
./run.sh -g --force
|
||||
|
||||
# Traiter un seul fichier pour test
|
||||
./run.sh -g --file 6881
|
||||
# Traiter un fichier spécifique (test rapide)
|
||||
./run.sh -g --file LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc
|
||||
|
||||
# Traiter deux fichiers spécifiques
|
||||
./run.sh -g --file LHD_FXX_1000_6881_PTS_LAMB93_IGN69.copc LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc
|
||||
```
|
||||
|
||||
### Utilisation directe Docker
|
||||
@ -127,9 +129,9 @@ docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data
|
||||
docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data/output \
|
||||
lidar-lidar python3 -m lidar_pipeline /data/input -o /data/output -v
|
||||
|
||||
# Un seul fichier (test rapide)
|
||||
# Un ou plusieurs fichiers spécifiques
|
||||
docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data/output \
|
||||
lidar-lidar python3 -m lidar_pipeline /data/input -o /data/output --file 6881
|
||||
lidar-lidar python3 -m lidar_pipeline /data/input -o /data/output --file LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc
|
||||
|
||||
# Forcer la régénération
|
||||
docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data/output \
|
||||
@ -148,7 +150,7 @@ docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data
|
||||
│ │ │ ├── ..._hillshade_multi.webp
|
||||
│ │ │ ├── ..._svf.webp
|
||||
│ │ │ ├── ..._mslrm.webp
|
||||
│ │ │ └── ... (20 visualisations)
|
||||
│ │ │ └── ... (19 visualisations)
|
||||
│ │ └── fichier_6882/
|
||||
│ │ └── ...
|
||||
│ └── rapports/ # Rapports PDF A3 par fichier
|
||||
@ -177,7 +179,7 @@ docker run --rm --gpus all -v $(pwd)/input:/data/input:ro -v $(pwd)/output:/data
|
||||
| Workers | `-w` | 1 | Nombre de CPU pour traitement parallèle |
|
||||
| Output | `-o` | /data/output | Dossier de sortie |
|
||||
| Force | `-f/--force` | off | Régénérer même si les WebP existent |
|
||||
| File | `--file` | tous | Traiter un seul fichier (pour tests) |
|
||||
| File | `--file` | tous | Traiter un ou plusieurs fichiers LAZ (nom complet sans extension) |
|
||||
| Verbose | `-v` | off | Mode verbeux (timestamps + niveaux) |
|
||||
| Debug | `--debug` | off | Mode debug (détails internes) |
|
||||
|
||||
|
||||
@ -61,7 +61,7 @@ Exemples:
|
||||
python -m lidar_pipeline /data/input -o /data/output --force
|
||||
|
||||
Traiter un seul fichier (pour tests):
|
||||
python -m lidar_pipeline /data/input -o /data/output --file LHD_FXX_1000_6881_PTS_LAMB93_IGN69.copc.laz
|
||||
python -m lidar_pipeline /data/input -o /data/output --file LHD_FXX_1000_6881_PTS_LAMB93_IGN69.copc
|
||||
|
||||
Traitement parallèle (4 workers):
|
||||
python -m lidar_pipeline /data/input -o /data/output -w 4
|
||||
@ -95,9 +95,10 @@ Exemples:
|
||||
)
|
||||
parser.add_argument(
|
||||
"--file",
|
||||
nargs="+",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Traiter un seul fichier LAZ/LAS (pour tests, par nom partiel ou complet)"
|
||||
help="Traiter un ou plusieurs fichiers LAZ/LAS (nom complet sans extension, ex: LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose",
|
||||
@ -130,24 +131,35 @@ Exemples:
|
||||
force=args.force
|
||||
)
|
||||
|
||||
# If --file is specified, process only that single file
|
||||
# If --file is specified, process only matching files
|
||||
if args.file:
|
||||
from pathlib import Path
|
||||
input_dir = Path(args.input)
|
||||
# Find matching file
|
||||
matches = list(input_dir.glob(f"*{args.file}*")) + list(input_dir.glob(f"*{args.file}*.laz")) + list(input_dir.glob(f"*{args.file}*.las"))
|
||||
# Remove duplicates
|
||||
matches = list(dict.fromkeys(matches))
|
||||
if not matches:
|
||||
logger.error(f"Aucun fichier trouvé pour: {args.file}")
|
||||
# Each pattern is the full filename without extension (e.g. LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc)
|
||||
selected_files = []
|
||||
for pattern in args.file:
|
||||
matches = list(input_dir.glob(f"{pattern}.laz")) + list(input_dir.glob(f"{pattern}.las"))
|
||||
# Remove duplicates
|
||||
matches = list(dict.fromkeys(matches))
|
||||
if not matches:
|
||||
logger.warning(f"Aucun fichier trouvé pour: {pattern}")
|
||||
continue
|
||||
selected_files.extend(matches)
|
||||
# Remove duplicates across patterns
|
||||
seen = set()
|
||||
unique_files = []
|
||||
for f in selected_files:
|
||||
if f not in seen:
|
||||
seen.add(f)
|
||||
unique_files.append(f)
|
||||
if not unique_files:
|
||||
logger.error("Aucun fichier trouvé pour les motifs spécifiés")
|
||||
sys.exit(1)
|
||||
if len(matches) > 1:
|
||||
logger.info(f"Plusieurs fichiers correspondent, utilisation du premier:")
|
||||
for m in matches:
|
||||
logger.info(f" {m.name}")
|
||||
laz_file = matches[0]
|
||||
logger.info(f"Traitement du fichier: {laz_file.name}")
|
||||
pipeline.process_file(laz_file)
|
||||
logger.info(f"Traitement de {len(unique_files)} fichier(s) sélectionné(s)")
|
||||
for laz_file in unique_files:
|
||||
logger.info(f" → {laz_file.name}")
|
||||
for laz_file in unique_files:
|
||||
pipeline.process_file(laz_file)
|
||||
else:
|
||||
pipeline.process_all()
|
||||
except Exception as e:
|
||||
|
||||
@ -23,6 +23,9 @@ logger = logging.getLogger("lidar")
|
||||
def create_smrf_pipeline(input_laz, output_las):
|
||||
"""Create a PDAL pipeline JSON for SMRF ground classification.
|
||||
|
||||
Includes a filter for ReturnNumber/NumberOfReturns >= 1 to handle
|
||||
LiDAR HD files that may contain points with invalid return numbers.
|
||||
|
||||
Args:
|
||||
input_laz: Path to input LAZ/LAS file.
|
||||
output_las: Path to output classified LAS file.
|
||||
@ -33,6 +36,10 @@ def create_smrf_pipeline(input_laz, output_las):
|
||||
pipeline = {
|
||||
"pipeline": [
|
||||
str(input_laz),
|
||||
{
|
||||
"type": "filters.range",
|
||||
"limits": "ReturnNumber[1:],NumberOfReturns[1:]"
|
||||
},
|
||||
{
|
||||
"type": "filters.smrf",
|
||||
"ignore": "Classification[7:7]",
|
||||
@ -87,36 +94,7 @@ def classify_ground(laz_file, temp_dir):
|
||||
logger.info(" ✓ Classification sol terminée")
|
||||
return output_las
|
||||
except subprocess.CalledProcessError as e:
|
||||
error_msg = e.stderr.decode()
|
||||
logger.error(f" ✗ Erreur PDAL: {error_msg}")
|
||||
|
||||
# If error is about ReturnNumber=0, try filtering those points
|
||||
if "ReturnNumber" in error_msg and "NumberOfReturns" in error_msg:
|
||||
logger.info(" → Tentative de filtrage des points ReturnNumber=0...")
|
||||
|
||||
filtered_pipeline = [
|
||||
{"type": "readers.las", "filename": str(laz_file)},
|
||||
{"type": "filters.range", "limits": "ReturnNumber[1:],NumberOfReturns[1:]"},
|
||||
{"type": "filters.smrf", "scalar": 1.25},
|
||||
{"type": "filters.range", "limits": "Classification[2:2]"},
|
||||
{"type": "writers.las", "filename": str(output_las), "extra_dims": "all"}
|
||||
]
|
||||
|
||||
filtered_json = json.dumps(filtered_pipeline)
|
||||
with open(pipeline_file, 'w') as f:
|
||||
f.write(filtered_json)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
["pdal", "pipeline", str(pipeline_file)],
|
||||
capture_output=True, check=True
|
||||
)
|
||||
logger.info(" ✓ Classification sol terminée (points filtrés)")
|
||||
return output_las
|
||||
except subprocess.CalledProcessError as e2:
|
||||
logger.error(f" ✗ Erreur même avec filtrage: {e2.stderr.decode()}")
|
||||
return None
|
||||
|
||||
logger.error(f" ✗ Erreur classification PDAL: {e.stderr.decode()}")
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@ -70,4 +70,10 @@ def xp_minimum_filter(arr, footprint=None, size=None):
|
||||
"""Minimum filter — uses GPU if array is on GPU, CPU otherwise."""
|
||||
if HAS_GPU and isinstance(arr, cp.ndarray):
|
||||
return cp_ndimage.minimum_filter(arr, footprint=footprint, size=size)
|
||||
return ndimage.minimum_filter(arr, footprint=footprint, size=size)
|
||||
return ndimage.minimum_filter(arr, footprint=footprint, size=size)
|
||||
|
||||
|
||||
def gpu_cleanup():
|
||||
"""Free GPU memory. Call between visualizations to prevent OOM."""
|
||||
if HAS_GPU:
|
||||
cp.get_default_memory_pool().free_all_blocks()
|
||||
@ -17,11 +17,12 @@ import subprocess
|
||||
from .dtm import classify_ground, create_dtm_fast
|
||||
from .visualizations import (
|
||||
generate_hillshade, generate_slope, generate_aspect, generate_curvature,
|
||||
generate_solar, generate_lrm, generate_svf, generate_openness,
|
||||
generate_lrm, generate_svf, generate_openness,
|
||||
generate_mslrm, generate_tpi, generate_depressions, generate_sailore,
|
||||
generate_roughness, generate_anomalies, generate_wavelet, generate_texture,
|
||||
generate_flow,
|
||||
)
|
||||
from .gpu import gpu_cleanup
|
||||
from .ign import generate_ign_overlay
|
||||
from .rendering import tif_to_png, generate_pdf_report
|
||||
|
||||
@ -36,7 +37,6 @@ VIZ_STEPS = [
|
||||
('slope', generate_slope),
|
||||
('aspect', generate_aspect),
|
||||
('curvature', generate_curvature),
|
||||
('solar', generate_solar),
|
||||
('svf', generate_svf),
|
||||
('lrm', generate_lrm),
|
||||
('pos_open', lambda d, b, v, r: generate_openness(d, b, v, r, positive=True)),
|
||||
@ -167,6 +167,9 @@ class LidarArchaeoPipeline:
|
||||
vis_results[name] = None
|
||||
logger.error(f" [{idx}/{total}] ✗ {name}: {e}", exc_info=True)
|
||||
|
||||
# Free GPU memory between visualizations to prevent OOM
|
||||
gpu_cleanup()
|
||||
|
||||
# Convert to WebP (only newly generated TIFs, not skipped ones)
|
||||
logger.info(" Conversion images WebP:")
|
||||
for name, tif_file in vis_results.items():
|
||||
|
||||
@ -66,14 +66,6 @@ COLORMAPS = {
|
||||
'vmin_mode': 'fixed', 'vmin_val': 0,
|
||||
'vmax_mode': 'fixed', 'vmax_val': 360,
|
||||
},
|
||||
'solar': {
|
||||
'cmap': 'gray',
|
||||
'title': 'Éclairage Solaire',
|
||||
'legend': 'Illumination solaire (azimut 90°, altitude 30°)\nClair = Face éclairée | Sombre = Zone d\'ombre',
|
||||
'description': 'Simulation de l\'éclairage solaire matinal',
|
||||
'vmin_mode': 'fixed', 'vmin_val': 0,
|
||||
'vmax_mode': 'fixed', 'vmax_val': 1,
|
||||
},
|
||||
'curvature': {
|
||||
'cmap': 'RdYlBu_r',
|
||||
'title': 'Courbure (Convexité/Concavité du terrain)',
|
||||
|
||||
0
lidar_pipeline/tests/__init__.py
Normal file
0
lidar_pipeline/tests/__init__.py
Normal file
71
lidar_pipeline/tests/conftest.py
Normal file
71
lidar_pipeline/tests/conftest.py
Normal file
@ -0,0 +1,71 @@
|
||||
"""Shared fixtures for LiDAR pipeline tests."""
|
||||
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from rasterio.transform import from_bounds
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_output_dir(tmp_path):
|
||||
"""Create a temporary output directory."""
|
||||
out = tmp_path / "output"
|
||||
out.mkdir()
|
||||
return out
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def synthetic_dem(tmp_path):
|
||||
"""Create a small synthetic DEM GeoTIFF with realistic terrain features.
|
||||
|
||||
Includes:
|
||||
- A gaussian hill (tumulus)
|
||||
- A linear ridge (wall)
|
||||
- A depression (ditch)
|
||||
- Background noise
|
||||
"""
|
||||
size = 200
|
||||
x = np.linspace(0, 1000, size)
|
||||
y = np.linspace(0, 1000, size)
|
||||
X, Y = np.meshgrid(x, y)
|
||||
|
||||
# Base terrain with gentle slope
|
||||
dem = 100.0 + 0.01 * X + 0.005 * Y
|
||||
|
||||
# Gaussian hill (tumulus) at center
|
||||
dem += 5.0 * np.exp(-((X - 500)**2 + (Y - 500)**2) / (2 * 80**2))
|
||||
|
||||
# Linear ridge (wall) running SW-NE
|
||||
dist_to_wall = np.abs((X - 200) * 0.707 + (Y - 300) * 0.707) / np.sqrt(2)
|
||||
dem += 1.5 * np.exp(-dist_to_wall**2 / (2 * 10**2))
|
||||
|
||||
# Depression (ditch)
|
||||
dist_to_ditch = np.abs(X - 800)
|
||||
dem -= 2.0 * np.exp(-dist_to_ditch**2 / (2 * 15**2)) * (Y > 300) * (Y < 700)
|
||||
|
||||
# Small random noise
|
||||
rng = np.random.default_rng(42)
|
||||
dem += rng.normal(0, 0.05, dem.shape)
|
||||
|
||||
# Save as GeoTIFF (Lambert 93)
|
||||
dem_file = tmp_path / "test_dem.tif"
|
||||
transform = from_bounds(660000, 6700000, 661000, 6701000, size, size)
|
||||
|
||||
with rasterio.open(
|
||||
dem_file, 'w',
|
||||
driver='GTiff', height=size, width=size,
|
||||
count=1, dtype='float32',
|
||||
crs='EPSG:2154', transform=transform,
|
||||
compress='lzw'
|
||||
) as dst:
|
||||
dst.write(dem.astype('float32'), 1)
|
||||
|
||||
return dem_file
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def synthetic_dem_data(synthetic_dem):
|
||||
"""Return (array, transform, crs) from the synthetic DEM."""
|
||||
with rasterio.open(synthetic_dem) as src:
|
||||
return src.read(1), src.transform, src.crs
|
||||
86
lidar_pipeline/tests/test_cli.py
Normal file
86
lidar_pipeline/tests/test_cli.py
Normal file
@ -0,0 +1,86 @@
|
||||
"""Tests for CLI argument parsing."""
|
||||
|
||||
import pytest
|
||||
import sys
|
||||
|
||||
|
||||
class TestCLIParsing:
|
||||
def test_default_args(self):
|
||||
"""Default arguments are set correctly."""
|
||||
from lidar_pipeline.cli import main
|
||||
# We can't easily call main() without files, so test argparse directly
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("input", help="Input directory")
|
||||
parser.add_argument("-o", "--output", default="/data/output")
|
||||
parser.add_argument("-r", "--resolution", type=float, default=0.5)
|
||||
parser.add_argument("-w", "--workers", type=int, default=1)
|
||||
parser.add_argument("-f", "--force", action="store_true")
|
||||
parser.add_argument("--file", nargs="+", type=str, default=None)
|
||||
|
||||
args = parser.parse_args(["./input"])
|
||||
assert args.input == "./input"
|
||||
assert args.output == "/data/output"
|
||||
assert args.resolution == 0.5
|
||||
assert args.workers == 1
|
||||
assert args.force is False
|
||||
assert args.file is None
|
||||
|
||||
def test_file_flag_single(self):
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--file", nargs="+", type=str, default=None)
|
||||
args = parser.parse_args(["--file", "test_file"])
|
||||
assert args.file == ["test_file"]
|
||||
|
||||
def test_file_flag_multiple(self):
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--file", nargs="+", type=str, default=None)
|
||||
args = parser.parse_args(["--file", "file1", "file2", "file3"])
|
||||
assert args.file == ["file1", "file2", "file3"]
|
||||
|
||||
def test_force_flag(self):
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("-f", "--force", action="store_true")
|
||||
args = parser.parse_args(["-f"])
|
||||
assert args.force is True
|
||||
|
||||
def test_resolution_custom(self):
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("-r", "--resolution", type=float, default=0.5)
|
||||
args = parser.parse_args(["-r", "0.2"])
|
||||
assert args.resolution == 0.2
|
||||
|
||||
|
||||
class TestSetupLogging:
|
||||
def test_default_logging(self):
|
||||
"""Default logging shows messages without timestamps."""
|
||||
import logging
|
||||
from lidar_pipeline.cli import setup_logging
|
||||
logger = setup_logging(verbose=False, debug=False)
|
||||
assert logger.level == logging.INFO
|
||||
assert len(logger.handlers) == 1
|
||||
# Format should not include timestamps
|
||||
fmt = logger.handlers[0].formatter._fmt
|
||||
assert "%(asctime)" not in fmt
|
||||
|
||||
def test_verbose_logging(self):
|
||||
"""Verbose logging includes timestamps."""
|
||||
import logging
|
||||
from lidar_pipeline.cli import setup_logging
|
||||
logger = setup_logging(verbose=True, debug=False)
|
||||
fmt = logger.handlers[0].formatter._fmt
|
||||
assert "%(asctime)" in fmt
|
||||
|
||||
def test_debug_logging(self):
|
||||
"""Debug logging includes file:line info."""
|
||||
import logging
|
||||
from lidar_pipeline.cli import setup_logging
|
||||
logger = setup_logging(verbose=False, debug=True)
|
||||
assert logger.level == logging.DEBUG
|
||||
fmt = logger.handlers[0].formatter._fmt
|
||||
assert "%(filename)" in fmt
|
||||
assert "%(lineno)" in fmt
|
||||
41
lidar_pipeline/tests/test_dtm.py
Normal file
41
lidar_pipeline/tests/test_dtm.py
Normal file
@ -0,0 +1,41 @@
|
||||
"""Tests for DTM module."""
|
||||
|
||||
import json
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TestSMRFPipeline:
|
||||
def test_pipeline_json_valid(self):
|
||||
"""create_smrf_pipeline produces valid JSON with expected stages."""
|
||||
from lidar_pipeline.dtm import create_smrf_pipeline
|
||||
result = create_smrf_pipeline("/data/input/test.laz", "/data/output/test_ground.las")
|
||||
pipeline = json.loads(result)
|
||||
|
||||
assert "pipeline" in pipeline
|
||||
stages = pipeline["pipeline"]
|
||||
|
||||
# Should have: reader, range filter (ReturnNumber), SMRF, range filter (Classification), writer
|
||||
stage_types = [s.get("type") if isinstance(s, dict) else None for s in stages]
|
||||
|
||||
# First stage is the filename string (reader)
|
||||
assert isinstance(stages[0], str)
|
||||
assert "test.laz" in stages[0]
|
||||
|
||||
# Must contain SMRF filter
|
||||
assert "filters.smrf" in stage_types
|
||||
|
||||
# Must contain ReturnNumber filter
|
||||
range_stages = [s for s in stages if isinstance(s, dict) and s.get("type") == "filters.range"]
|
||||
assert len(range_stages) >= 1
|
||||
# At least one should filter ReturnNumber
|
||||
assert any("ReturnNumber" in str(s.get("limits", "")) for s in range_stages)
|
||||
|
||||
def test_pipeline_output_path(self):
|
||||
"""Pipeline output path is set correctly."""
|
||||
from lidar_pipeline.dtm import create_smrf_pipeline
|
||||
result = create_smrf_pipeline("/input/a.laz", "/output/a_ground.las")
|
||||
pipeline = json.loads(result)
|
||||
# Last stage should be writer with correct output path
|
||||
writer = [s for s in pipeline["pipeline"] if isinstance(s, dict) and s.get("type") == "writers.las"][0]
|
||||
assert writer["filename"] == "/output/a_ground.las"
|
||||
93
lidar_pipeline/tests/test_gpu.py
Normal file
93
lidar_pipeline/tests/test_gpu.py
Normal file
@ -0,0 +1,93 @@
|
||||
"""Tests for GPU helper module."""
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
|
||||
def test_has_gpu_attribute():
|
||||
"""HAS_GPU must be a boolean."""
|
||||
from lidar_pipeline.gpu import HAS_GPU
|
||||
assert isinstance(HAS_GPU, bool)
|
||||
|
||||
|
||||
def test_to_gpu_returns_array():
|
||||
"""to_gpu returns a float64 array with correct values."""
|
||||
from lidar_pipeline.gpu import to_gpu, to_cpu, HAS_GPU
|
||||
arr = np.array([1.0, 2.0, 3.0])
|
||||
result = to_gpu(arr)
|
||||
# On GPU: cupy.ndarray, on CPU: numpy.ndarray
|
||||
assert result.dtype == np.float64
|
||||
# Always bring back to CPU for comparison
|
||||
np.testing.assert_array_equal(to_cpu(result), [1.0, 2.0, 3.0])
|
||||
|
||||
|
||||
def test_to_cpu_noop_numpy():
|
||||
"""to_cpu on a numpy array is a no-op."""
|
||||
from lidar_pipeline.gpu import to_cpu
|
||||
arr = np.array([1.0, 2.0])
|
||||
result = to_cpu(arr)
|
||||
assert result is arr
|
||||
|
||||
|
||||
def test_xp_gaussian_filter():
|
||||
"""xp_gaussian_filter blurs a point source correctly."""
|
||||
from lidar_pipeline.gpu import xp_gaussian_filter
|
||||
arr = np.zeros((50, 50), dtype=np.float64)
|
||||
arr[25, 25] = 1.0
|
||||
result = xp_gaussian_filter(arr, sigma=3)
|
||||
assert result.shape == (50, 50)
|
||||
# Center should still be the highest value
|
||||
center_val = float(np.asarray(result)[25, 25])
|
||||
corner_val = float(np.asarray(result)[0, 0])
|
||||
assert center_val > corner_val
|
||||
assert center_val > 0.01 # Not all energy is lost
|
||||
|
||||
|
||||
def test_xp_uniform_filter_cpu():
|
||||
"""xp_uniform_filter works on CPU arrays."""
|
||||
from lidar_pipeline.gpu import xp_uniform_filter
|
||||
arr = np.ones((50, 50), dtype=np.float64)
|
||||
arr[25, 25] = 100.0
|
||||
result = xp_uniform_filter(arr, size=5)
|
||||
# Mean should be close to 1 everywhere except near center
|
||||
assert result.shape == (50, 50)
|
||||
assert result[0, 0] == pytest.approx(1.0, abs=0.01)
|
||||
|
||||
|
||||
def test_xp_minimum_filter_cpu():
|
||||
"""xp_minimum_filter works on CPU arrays."""
|
||||
from lidar_pipeline.gpu import xp_minimum_filter
|
||||
arr = np.ones((50, 50), dtype=np.float64)
|
||||
arr[25, 25] = 0.0
|
||||
result = xp_minimum_filter(arr, size=3)
|
||||
assert result.shape == (50, 50)
|
||||
# Around the minimum, values should be 0
|
||||
assert result[25, 25] == 0.0
|
||||
assert result[24, 25] == 0.0
|
||||
|
||||
|
||||
def test_log_gpu_status(caplog):
|
||||
"""log_gpu_status emits a log message."""
|
||||
import logging
|
||||
from lidar_pipeline.gpu import log_gpu_status
|
||||
with caplog.at_level(logging.INFO, logger="lidar"):
|
||||
log_gpu_status()
|
||||
assert any("GPU" in r.message or "CPU" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not pytest.importorskip("cupy", reason="CuPy not available"),
|
||||
reason="Requires GPU + CuPy"
|
||||
)
|
||||
def test_to_gpu_roundtrip():
|
||||
"""to_gpu -> to_cpu preserves data when GPU is available."""
|
||||
import cupy as cp
|
||||
from lidar_pipeline.gpu import to_gpu, to_cpu, HAS_GPU
|
||||
if not HAS_GPU:
|
||||
pytest.skip("No GPU available")
|
||||
arr = np.array([1.0, 2.0, 3.0], dtype=np.float32)
|
||||
gpu_arr = to_gpu(arr)
|
||||
assert isinstance(gpu_arr, cp.ndarray)
|
||||
result = to_cpu(gpu_arr)
|
||||
assert isinstance(result, np.ndarray)
|
||||
np.testing.assert_array_almost_equal(result, [1.0, 2.0, 3.0])
|
||||
79
lidar_pipeline/tests/test_pipeline.py
Normal file
79
lidar_pipeline/tests/test_pipeline.py
Normal file
@ -0,0 +1,79 @@
|
||||
"""Tests for pipeline orchestration."""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TestVizSteps:
|
||||
def test_viz_steps_not_empty(self):
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
assert len(VIZ_STEPS) > 0
|
||||
|
||||
def test_viz_steps_have_callable_functions(self):
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
for name, func in VIZ_STEPS:
|
||||
assert callable(func), f"VIZ_STEPS entry '{name}' is not callable"
|
||||
|
||||
def test_viz_steps_names_unique(self):
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
names = [name for name, _ in VIZ_STEPS]
|
||||
assert len(names) == len(set(names)), "VIZ_STEPS has duplicate names"
|
||||
|
||||
def test_no_solar_in_viz_steps(self):
|
||||
"""Solar visualization was removed."""
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
names = [name for name, _ in VIZ_STEPS]
|
||||
assert "solar" not in names
|
||||
|
||||
def test_expected_visualization_count(self):
|
||||
"""Should have 19 visualizations (18 terrain + ortho + topo - solar)."""
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
assert len(VIZ_STEPS) == 19
|
||||
|
||||
def test_ortho_and_topo_present(self):
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
names = [name for name, _ in VIZ_STEPS]
|
||||
assert "ortho" in names
|
||||
assert "topo" in names
|
||||
|
||||
|
||||
class TestLidarArchaeoPipeline:
|
||||
def test_init_creates_dirs(self, tmp_path):
|
||||
from lidar_pipeline.pipeline import LidarArchaeoPipeline
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
output_dir = tmp_path / "output"
|
||||
|
||||
pipeline = LidarArchaeoPipeline(str(input_dir), str(output_dir))
|
||||
assert (tmp_path / "output").exists()
|
||||
assert (tmp_path / "output" / "DTM").exists()
|
||||
assert (tmp_path / "output" / "visualisations").exists()
|
||||
assert (tmp_path / "output" / "rapports").exists()
|
||||
|
||||
def test_init_raises_on_missing_input(self, tmp_path):
|
||||
from lidar_pipeline.pipeline import LidarArchaeoPipeline
|
||||
with pytest.raises(ValueError, match="introuvable"):
|
||||
LidarArchaeoPipeline("/nonexistent/path", str(tmp_path / "output"))
|
||||
|
||||
def test_find_laz_files_empty(self, tmp_path):
|
||||
from lidar_pipeline.pipeline import LidarArchaeoPipeline
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
pipeline = LidarArchaeoPipeline(str(input_dir), str(tmp_path / "output"))
|
||||
files = pipeline.find_laz_files()
|
||||
assert files == []
|
||||
|
||||
def test_find_laz_files(self, tmp_path):
|
||||
from lidar_pipeline.pipeline import LidarArchaeoPipeline
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
(input_dir / "test.laz").touch()
|
||||
(input_dir / "other.las").touch()
|
||||
(input_dir / "readme.txt").touch()
|
||||
|
||||
pipeline = LidarArchaeoPipeline(str(input_dir), str(tmp_path / "output"))
|
||||
files = pipeline.find_laz_files()
|
||||
names = [f.name for f in files]
|
||||
assert "test.laz" in names
|
||||
assert "other.las" in names
|
||||
assert "readme.txt" not in names
|
||||
98
lidar_pipeline/tests/test_rendering.py
Normal file
98
lidar_pipeline/tests/test_rendering.py
Normal file
@ -0,0 +1,98 @@
|
||||
"""Tests for rendering module (colormaps, tif_to_png)."""
|
||||
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from rasterio.transform import from_bounds
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _make_test_tif(tmp_path, data=None, size=50):
|
||||
"""Create a small test GeoTIFF and return its path."""
|
||||
if data is None:
|
||||
rng = np.random.default_rng(42)
|
||||
data = rng.normal(0, 1, (size, size)).astype(np.float32)
|
||||
|
||||
transform = from_bounds(660000, 6700000, 661000, 6701000, size, size)
|
||||
tif_file = tmp_path / "test_vis.tif"
|
||||
with rasterio.open(
|
||||
tif_file, 'w', driver='GTiff', height=size, width=size,
|
||||
count=1, dtype='float32', crs='EPSG:2154', transform=transform,
|
||||
compress='lzw'
|
||||
) as dst:
|
||||
dst.write(data, 1)
|
||||
return tif_file
|
||||
|
||||
|
||||
class TestColormaps:
|
||||
def test_colormaps_dict_exists(self):
|
||||
from lidar_pipeline.rendering import COLORMAPS
|
||||
assert isinstance(COLORMAPS, dict)
|
||||
|
||||
def test_all_viz_steps_have_colormaps(self):
|
||||
"""Every VIZ_STEPS entry should have a corresponding COLORMAPS entry or render correctly."""
|
||||
from lidar_pipeline.pipeline import VIZ_STEPS
|
||||
from lidar_pipeline.rendering import COLORMAPS
|
||||
# Some viz names differ from colormap keys
|
||||
name_map = {
|
||||
'pos_open': 'positive_openness',
|
||||
'neg_open': 'negative_openness',
|
||||
}
|
||||
# IGN overlays (ortho, topo) are RGB images — no colormap needed
|
||||
skip = {'ortho', 'topo'}
|
||||
for name, _ in VIZ_STEPS:
|
||||
if name in skip:
|
||||
continue
|
||||
cmap_key = name_map.get(name, name)
|
||||
assert cmap_key in COLORMAPS, f"Missing colormap for: {name} (looked as {cmap_key})"
|
||||
|
||||
def test_colormap_has_required_keys(self):
|
||||
"""Each colormap entry must have cmap, title, legend, description."""
|
||||
from lidar_pipeline.rendering import COLORMAPS
|
||||
required = {'cmap', 'title', 'legend', 'description'}
|
||||
for name, entry in COLORMAPS.items():
|
||||
missing = required - set(entry.keys())
|
||||
assert not missing, f"Colormap '{name}' missing keys: {missing}"
|
||||
|
||||
|
||||
class TestTifToPng:
|
||||
def test_converts_tif_to_webp(self, tmp_path):
|
||||
from lidar_pipeline.rendering import tif_to_png
|
||||
tif_file = _make_test_tif(tmp_path)
|
||||
result = tif_to_png(tif_file, tmp_path, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
assert result.suffix == '.webp'
|
||||
|
||||
def test_removes_source_tif(self, tmp_path):
|
||||
from lidar_pipeline.rendering import tif_to_png
|
||||
tif_file = _make_test_tif(tmp_path)
|
||||
assert tif_file.exists()
|
||||
tif_to_png(tif_file, tmp_path, 5.0)
|
||||
assert not tif_file.exists(), "Source TIF should be deleted after conversion"
|
||||
|
||||
def test_webp_has_content(self, tmp_path):
|
||||
from lidar_pipeline.rendering import tif_to_png
|
||||
tif_file = _make_test_tif(tmp_path)
|
||||
result = tif_to_png(tif_file, tmp_path, 5.0)
|
||||
assert result.stat().st_size > 1000 # Must be a real image
|
||||
|
||||
|
||||
class TestApplyColormap:
|
||||
def test_symmetric_mode(self, tmp_path):
|
||||
from lidar_pipeline.rendering import COLORMAPS, tif_to_png
|
||||
# LRM uses symmetric mode
|
||||
data = np.random.default_rng(42).normal(0, 0.5, (50, 50)).astype(np.float32)
|
||||
tif_file = _make_test_tif(tmp_path, data)
|
||||
result = tif_to_png(tif_file, tmp_path, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_percentile_mode(self, tmp_path):
|
||||
from lidar_pipeline.rendering import tif_to_png
|
||||
# Most visualizations use percentile mode
|
||||
data = np.random.default_rng(42).normal(50, 10, (50, 50)).astype(np.float32)
|
||||
tif_file = _make_test_tif(tmp_path, data)
|
||||
result = tif_to_png(tif_file, tmp_path, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
216
lidar_pipeline/tests/test_visualizations.py
Normal file
216
lidar_pipeline/tests/test_visualizations.py
Normal file
@ -0,0 +1,216 @@
|
||||
"""Tests for visualization functions.
|
||||
|
||||
Each test creates a small synthetic DEM and runs a visualization function,
|
||||
checking that it produces a valid output file.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# --- Core terrain visualizations (no GPU required) ---
|
||||
|
||||
class TestHillshade:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_hillshade
|
||||
result = generate_hillshade(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
assert result.suffix == ".tif"
|
||||
|
||||
def test_output_values_valid(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_hillshade
|
||||
result = generate_hillshade(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
assert data.shape[0] > 0
|
||||
assert np.nanmin(data) >= 0
|
||||
assert np.nanmax(data) <= 1
|
||||
|
||||
|
||||
class TestSlope:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_slope
|
||||
result = generate_slope(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_slope_values_degrees(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_slope
|
||||
result = generate_slope(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
assert np.nanmin(data) >= 0
|
||||
assert np.nanmax(data) <= 90
|
||||
|
||||
|
||||
class TestAspect:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_aspect
|
||||
result = generate_aspect(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_aspect_values_0_360(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_aspect
|
||||
result = generate_aspect(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
valid = data[~np.isnan(data)]
|
||||
assert np.nanmin(valid) >= 0
|
||||
assert np.nanmax(valid) <= 360
|
||||
|
||||
|
||||
class TestCurvature:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_curvature
|
||||
result = generate_curvature(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
# --- GPU-accelerated visualizations ---
|
||||
|
||||
class TestLRM:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_lrm
|
||||
result = generate_lrm(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_lrm_has_positive_negative(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_lrm
|
||||
result = generate_lrm(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
# LRM should have both positive and negative values
|
||||
assert np.nanmax(data) > 0
|
||||
assert np.nanmin(data) < 0
|
||||
|
||||
|
||||
class TestSVF:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_svf
|
||||
result = generate_svf(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_svf_values_0_1(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_svf
|
||||
result = generate_svf(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
valid = data[~np.isnan(data)]
|
||||
assert np.nanmin(valid) >= 0
|
||||
assert np.nanmax(valid) <= 1
|
||||
|
||||
|
||||
class TestOpenness:
|
||||
def test_positive_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_openness
|
||||
result = generate_openness(synthetic_dem, "test", tmp_output_dir, 5.0, positive=True)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_negative_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_openness
|
||||
result = generate_openness(synthetic_dem, "test", tmp_output_dir, 5.0, positive=False)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestMSLRM:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_mslrm
|
||||
result = generate_mslrm(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestTPI:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_tpi
|
||||
result = generate_tpi(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestDepressions:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_depressions
|
||||
result = generate_depressions(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestSAILORE:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_sailore
|
||||
result = generate_sailore(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestRoughness:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_roughness
|
||||
result = generate_roughness(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_roughness_non_negative(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_roughness
|
||||
result = generate_roughness(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
# Standard deviation is always >= 0
|
||||
assert np.nanmin(data) >= 0
|
||||
|
||||
|
||||
class TestAnomalies:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_anomalies
|
||||
result = generate_anomalies(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestWavelet:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_wavelet
|
||||
result = generate_wavelet(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestTexture:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_texture
|
||||
result = generate_texture(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
|
||||
class TestFlow:
|
||||
def test_generates_tif(self, synthetic_dem, tmp_output_dir):
|
||||
from lidar_pipeline.visualizations import generate_flow
|
||||
result = generate_flow(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
assert result is not None
|
||||
assert result.exists()
|
||||
|
||||
def test_flow_log_values(self, synthetic_dem, tmp_output_dir):
|
||||
import rasterio
|
||||
from lidar_pipeline.visualizations import generate_flow
|
||||
result = generate_flow(synthetic_dem, "test", tmp_output_dir, 5.0)
|
||||
with rasterio.open(result) as src:
|
||||
data = src.read(1)
|
||||
# log1p(x) >= 0 for x >= 0
|
||||
valid = data[~np.isnan(data)]
|
||||
assert np.nanmin(valid) >= 0
|
||||
@ -10,11 +10,10 @@ from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import rasterio
|
||||
from scipy import ndimage
|
||||
from scipy.ndimage import generic_filter, gaussian_filter, uniform_filter, minimum_filter
|
||||
from scipy.ndimage import generic_filter
|
||||
from scipy.stats import binned_statistic_2d
|
||||
|
||||
from .gpu import HAS_GPU, to_gpu, to_cpu, xp_gaussian_filter, xp_uniform_filter
|
||||
from .gpu import HAS_GPU, to_gpu, to_cpu, xp_gaussian_filter, xp_uniform_filter, xp_minimum_filter
|
||||
|
||||
logger = logging.getLogger("lidar")
|
||||
|
||||
@ -58,31 +57,38 @@ def _read_dem(dem_file):
|
||||
# ============================================================
|
||||
|
||||
def generate_hillshade(dem_file, basename, vis_dir, resolution):
|
||||
"""Generate multi-directional hillshade (NW, NE, SW, SE)."""
|
||||
logger.info(" → Hillshade multidirectionnel...")
|
||||
"""Generate multi-directional hillshade (NW, NE, SW, SE) — GPU if available."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Hillshade multidirectionnel{gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_hillshade_multi.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dy, dx = np.gradient(dem)
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np)
|
||||
dy, dx = xp.gradient(dem)
|
||||
|
||||
azimuts = [315, 45, 225, 135]
|
||||
altitude = 30
|
||||
hillshades = []
|
||||
|
||||
for az in azimuts:
|
||||
az_rad = np.radians(az)
|
||||
alt_rad = np.radians(altitude)
|
||||
slope = np.arctan(np.sqrt(dx**2 + dy**2))
|
||||
aspect = np.arctan2(dy, dx)
|
||||
hs = np.sin(alt_rad) * np.sin(slope) + \
|
||||
np.cos(alt_rad) * np.cos(slope) * np.cos(az_rad - aspect)
|
||||
hillshades.append(np.clip(hs, 0, 1))
|
||||
slope = xp.arctan(xp.sqrt(dx**2 + dy**2))
|
||||
aspect = xp.arctan2(dy, dx)
|
||||
sin_slope = xp.sin(slope)
|
||||
cos_slope = xp.cos(slope)
|
||||
|
||||
combined = np.mean(hillshades, axis=0)
|
||||
_save_tif(output, combined, transform, crs)
|
||||
logger.info(f" ✓ Hillshade terminé ({time.time()-t0:.1f}s)")
|
||||
alt_rad = xp.radians(xp.array(altitude))
|
||||
sin_alt = xp.sin(alt_rad)
|
||||
cos_alt = xp.cos(alt_rad)
|
||||
|
||||
for az in azimuts:
|
||||
az_rad = xp.radians(xp.array(az))
|
||||
hs = sin_alt * sin_slope + cos_alt * cos_slope * xp.cos(az_rad - aspect)
|
||||
hillshades.append(xp.clip(hs, 0, 1))
|
||||
|
||||
combined = xp.mean(xp.array(hillshades), axis=0)
|
||||
_save_tif(output, to_cpu(combined), transform, crs)
|
||||
logger.info(f" ✓ Hillshade terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur hillshade: {e}", exc_info=True)
|
||||
@ -90,17 +96,19 @@ def generate_hillshade(dem_file, basename, vis_dir, resolution):
|
||||
|
||||
|
||||
def generate_slope(dem_file, basename, vis_dir, resolution):
|
||||
"""Generate slope map (degrees)."""
|
||||
logger.info(" → Pente (Slope)...")
|
||||
"""Generate slope map (degrees) — GPU if available."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Pente (Slope){gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_slope.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dy, dx = np.gradient(dem)
|
||||
slope = np.arctan(np.sqrt(dx**2 + dy**2)) * 180 / np.pi
|
||||
_save_tif(output, slope, transform, crs)
|
||||
logger.info(f" ✓ Pente terminée ({time.time()-t0:.1f}s)")
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np)
|
||||
dy, dx = xp.gradient(dem)
|
||||
slope = xp.arctan(xp.sqrt(dx**2 + dy**2)) * 180 / xp.pi
|
||||
_save_tif(output, to_cpu(slope), transform, crs)
|
||||
logger.info(f" ✓ Pente terminée ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur slope: {e}", exc_info=True)
|
||||
@ -108,18 +116,20 @@ def generate_slope(dem_file, basename, vis_dir, resolution):
|
||||
|
||||
|
||||
def generate_aspect(dem_file, basename, vis_dir, resolution):
|
||||
"""Generate aspect (slope orientation) map."""
|
||||
logger.info(" → Aspect (Orientation)...")
|
||||
"""Generate aspect (slope orientation) map — GPU if available."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Aspect (Orientation){gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_aspect.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dy, dx = np.gradient(dem)
|
||||
aspect = np.arctan2(dy, dx) * 180 / np.pi
|
||||
aspect = np.mod(aspect, 360)
|
||||
_save_tif(output, aspect, transform, crs)
|
||||
logger.info(f" ✓ Aspect terminé ({time.time()-t0:.1f}s)")
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np)
|
||||
dy, dx = xp.gradient(dem)
|
||||
aspect = xp.arctan2(dy, dx) * 180 / xp.pi
|
||||
aspect = xp.mod(aspect, 360)
|
||||
_save_tif(output, to_cpu(aspect), transform, crs)
|
||||
logger.info(f" ✓ Aspect terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur aspect: {e}", exc_info=True)
|
||||
@ -127,49 +137,28 @@ def generate_aspect(dem_file, basename, vis_dir, resolution):
|
||||
|
||||
|
||||
def generate_curvature(dem_file, basename, vis_dir, resolution):
|
||||
"""Generate curvature (terrain concavity/convexity) map."""
|
||||
logger.info(" → Courbure (Curvature)...")
|
||||
"""Generate curvature (terrain concavity/convexity) map — GPU if available."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Courbure (Curvature){gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_curvature.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dz_dx = np.gradient(dem, axis=1)
|
||||
dz_dy = np.gradient(dem, axis=0)
|
||||
d2z_dx2 = np.gradient(dz_dx, axis=1)
|
||||
d2z_dy2 = np.gradient(dz_dy, axis=0)
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np)
|
||||
dz_dx = xp.gradient(dem, axis=1)
|
||||
dz_dy = xp.gradient(dem, axis=0)
|
||||
d2z_dx2 = xp.gradient(dz_dx, axis=1)
|
||||
d2z_dy2 = xp.gradient(dz_dy, axis=0)
|
||||
curvature = (d2z_dx2 + d2z_dy2) / 2
|
||||
_save_tif(output, curvature, transform, crs)
|
||||
logger.info(f" ✓ Courbure terminée ({time.time()-t0:.1f}s)")
|
||||
_save_tif(output, to_cpu(curvature), transform, crs)
|
||||
logger.info(f" ✓ Courbure terminée ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur curvature: {e}", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
def generate_solar(dem_file, basename, vis_dir, resolution):
|
||||
"""Generate solar irradiance simulation."""
|
||||
logger.info(" → Éclairage Solaire...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_solar.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dy, dx = np.gradient(dem)
|
||||
slope = np.arctan(np.sqrt(dx**2 + dy**2))
|
||||
aspect = np.arctan2(dy, dx)
|
||||
az_rad = np.radians(90)
|
||||
alt_rad = np.radians(30)
|
||||
solar = np.sin(alt_rad) * np.sin(slope) + \
|
||||
np.cos(alt_rad) * np.cos(slope) * np.cos(az_rad - aspect)
|
||||
solar = np.clip(solar, 0, 1)
|
||||
_save_tif(output, solar, transform, crs)
|
||||
logger.info(f" ✓ Solaire terminé ({time.time()-t0:.1f}s)")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur solar: {e}", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
# ============================================================
|
||||
# GPU-accelerated visualizations
|
||||
@ -390,45 +379,46 @@ def generate_tpi(dem_file, basename, vis_dir, resolution):
|
||||
# ============================================================
|
||||
|
||||
def generate_depressions(dem_file, basename, vis_dir, resolution):
|
||||
"""Depression detection using hydrological sink filling."""
|
||||
logger.info(" → Détection dépressions (hydrologique)...")
|
||||
"""Depression detection using hydrological sink filling — GPU if available."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Détection dépressions (hydrologique){gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_depressions.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
|
||||
from scipy.ndimage import binary_dilation, generate_binary_structure
|
||||
|
||||
dem_filled = dem.copy()
|
||||
nodata_mask = np.isnan(dem_filled)
|
||||
dem_filled[nodata_mask] = np.nanmax(dem) + 1000
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np)
|
||||
|
||||
from scipy.ndimage import generate_binary_structure
|
||||
struct = generate_binary_structure(2, 2)
|
||||
|
||||
dem_filled = xp.copy(dem)
|
||||
nodata_mask = xp.isnan(dem_filled)
|
||||
dem_filled[nodata_mask] = xp.nanmax(dem) + 1000
|
||||
|
||||
changed = True
|
||||
iterations = 0
|
||||
max_iter = 100
|
||||
|
||||
while changed and iterations < max_iter:
|
||||
from scipy.ndimage import minimum_filter as scipy_min_filter
|
||||
neighbor_min = scipy_min_filter(dem_filled, footprint=struct)
|
||||
neighbor_min = xp_minimum_filter(dem_filled, footprint=struct)
|
||||
sinks = (dem_filled < neighbor_min) & ~nodata_mask
|
||||
|
||||
if not np.any(sinks):
|
||||
if not xp.any(sinks):
|
||||
break
|
||||
|
||||
new_dem = np.maximum(dem_filled, neighbor_min)
|
||||
new_dem[nodata_mask] = np.nan
|
||||
changed = np.any(new_dem != dem_filled)
|
||||
new_dem = xp.maximum(dem_filled, neighbor_min)
|
||||
new_dem[nodata_mask] = xp.nan
|
||||
changed = bool(xp.any(new_dem != dem_filled))
|
||||
dem_filled = new_dem
|
||||
iterations += 1
|
||||
|
||||
depressions = dem_filled - dem
|
||||
depressions[nodata_mask] = np.nan
|
||||
depressions = to_cpu(dem_filled - dem)
|
||||
depressions[to_cpu(nodata_mask)] = np.nan
|
||||
depressions = np.where(depressions > 0.01, depressions, 0)
|
||||
|
||||
_save_tif(output, depressions, transform, crs)
|
||||
logger.info(f" ✓ Dépressions terminé ({time.time()-t0:.1f}s)")
|
||||
logger.info(f" ✓ Dépressions terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur dépressions: {e}", exc_info=True)
|
||||
@ -489,24 +479,27 @@ def generate_sailore(dem_file, basename, vis_dir, resolution):
|
||||
# ============================================================
|
||||
|
||||
def generate_roughness(dem_file, basename, vis_dir, resolution):
|
||||
"""Surface roughness - standard deviation of elevation in a window."""
|
||||
logger.info(" → Rugosité de surface...")
|
||||
"""Surface roughness - standard deviation of elevation in a window (GPU-accelerated)."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Rugosité de surface{gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_roughness.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np.astype(np.float64))
|
||||
window_size = int(5 / resolution)
|
||||
if window_size % 2 == 0:
|
||||
window_size += 1
|
||||
|
||||
def std_filter(arr):
|
||||
return np.nanstd(arr)
|
||||
# Vectorized std: sqrt(E[X²] - (E[X])²) via uniform_filter (GPU-accelerated)
|
||||
local_mean = xp_uniform_filter(dem, size=window_size)
|
||||
local_mean_sq = xp_uniform_filter(dem * dem, size=window_size)
|
||||
roughness = xp.sqrt(local_mean_sq - local_mean * local_mean)
|
||||
|
||||
roughness = generic_filter(dem.astype(np.float64), std_filter,
|
||||
size=window_size, mode='nearest')
|
||||
roughness = to_cpu(roughness)
|
||||
_save_tif(output, roughness, transform, crs)
|
||||
logger.info(f" ✓ Rugosité terminée ({time.time()-t0:.1f}s)")
|
||||
logger.info(f" ✓ Rugosité terminée ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur rugosité: {e}", exc_info=True)
|
||||
@ -518,29 +511,33 @@ def generate_roughness(dem_file, basename, vis_dir, resolution):
|
||||
# ============================================================
|
||||
|
||||
def generate_anomalies(dem_file, basename, vis_dir, resolution):
|
||||
"""Statistical anomaly detection - z-score of local relief + Local Moran's I."""
|
||||
logger.info(" → Détection anomalies statistiques...")
|
||||
"""Statistical anomaly detection - z-score of local relief + Local Moran's I — GPU if available."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Détection anomalies statistiques{gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_anomalies.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
dem = to_gpu(dem_np)
|
||||
|
||||
lrm = dem - gaussian_filter(dem, sigma=15 / resolution)
|
||||
lrm_mean = np.nanmean(lrm)
|
||||
lrm_std = max(np.nanstd(lrm), 0.01)
|
||||
lrm = dem - xp_gaussian_filter(dem, sigma=15 / resolution)
|
||||
lrm_mean = xp.nanmean(lrm)
|
||||
lrm_std = max(float(xp.nanstd(lrm)), 0.01)
|
||||
z_score = (lrm - lrm_mean) / lrm_std
|
||||
|
||||
window = int(10 / resolution)
|
||||
if window % 2 == 0:
|
||||
window += 1
|
||||
|
||||
local_mean = uniform_filter(z_score, size=window)
|
||||
morans_i = z_score * (local_mean - np.nanmean(z_score)) / np.nanstd(z_score)
|
||||
anomaly_score = np.abs(z_score) * np.sign(morans_i)
|
||||
local_mean = xp_uniform_filter(z_score, size=window)
|
||||
z_mean = xp.nanmean(z_score)
|
||||
z_std = max(float(xp.nanstd(z_score)), 0.01)
|
||||
morans_i = z_score * (local_mean - z_mean) / z_std
|
||||
anomaly_score = xp.abs(z_score) * xp.sign(morans_i)
|
||||
|
||||
_save_tif(output, anomaly_score, transform, crs)
|
||||
logger.info(f" ✓ Anomalies terminé ({time.time()-t0:.1f}s)")
|
||||
_save_tif(output, to_cpu(anomaly_score), transform, crs)
|
||||
logger.info(f" ✓ Anomalies terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur anomalies: {e}", exc_info=True)
|
||||
@ -595,21 +592,24 @@ def generate_wavelet(dem_file, basename, vis_dir, resolution):
|
||||
# ============================================================
|
||||
|
||||
def generate_texture(dem_file, basename, vis_dir, resolution):
|
||||
"""GLCM texture analysis on hillshade - contrast, entropy, homogeneity."""
|
||||
logger.info(" → Texture GLCM...")
|
||||
"""GLCM-inspired texture analysis — contrast, entropy, homogeneity (GPU-accelerated)."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Texture GLCM{gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_texture.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
|
||||
gy, gx = np.gradient(dem, resolution)
|
||||
# Hillshade — compute on CPU to avoid holding DEM on GPU during texture
|
||||
gy, gx = np.gradient(dem_np, resolution)
|
||||
slope = np.arctan(np.sqrt(gx**2 + gy**2))
|
||||
alt_rad = np.radians(45)
|
||||
az_rad = np.radians(315)
|
||||
aspect = np.arctan2(gy, gx)
|
||||
shading = (np.sin(alt_rad) * np.cos(slope) +
|
||||
np.cos(alt_rad) * np.sin(slope) *
|
||||
np.cos(az_rad - np.arctan2(gy, gx)))
|
||||
np.cos(az_rad - aspect))
|
||||
hillshade = np.clip(shading, 0, 1)
|
||||
|
||||
valid = hillshade[~np.isnan(hillshade)]
|
||||
@ -617,31 +617,39 @@ def generate_texture(dem_file, basename, vis_dir, resolution):
|
||||
raise ValueError("No valid data for texture analysis")
|
||||
lo, hi = np.percentile(valid, (1, 99))
|
||||
img = np.clip((hillshade - lo) / max(hi - lo, 0.001), 0, 1)
|
||||
img_uint8 = (img * 255).astype(np.uint8)
|
||||
del hillshade, shading, slope, aspect, gy, gx # free memory
|
||||
|
||||
window = int(5 / resolution)
|
||||
if window % 2 == 0:
|
||||
window += 1
|
||||
|
||||
def local_variance(arr):
|
||||
return np.var(arr.astype(np.float64))
|
||||
# Contrast (variance) — GPU-accelerated
|
||||
img_gpu = to_gpu(img.astype(np.float32))
|
||||
local_mean = xp_uniform_filter(img_gpu, size=window)
|
||||
local_mean_sq = xp_uniform_filter(img_gpu * img_gpu, size=window)
|
||||
contrast = to_cpu(local_mean_sq - local_mean * local_mean).astype(np.float64)
|
||||
del img_gpu, local_mean, local_mean_sq # free GPU memory
|
||||
|
||||
def local_entropy(arr):
|
||||
hist, _ = np.histogram(arr.astype(np.float64), bins=16, range=(0, 256))
|
||||
hist = hist / max(hist.sum(), 1)
|
||||
hist = hist[hist > 0]
|
||||
return -np.sum(hist * np.log2(hist))
|
||||
# Entropy — compute bin-by-bin to avoid large 3D allocation
|
||||
n_bins = 16
|
||||
img_uint8 = np.clip(img * 255, 0, 255).astype(np.uint8)
|
||||
quantized = (img_uint8 // (256 // n_bins)).astype(np.int32)
|
||||
entropy = np.zeros_like(img, dtype=np.float64)
|
||||
win_area = max(window * window, 1)
|
||||
|
||||
def local_homogeneity(arr):
|
||||
arr_f = arr.astype(np.float64)
|
||||
return np.mean(1.0 / (1.0 + (arr_f - np.mean(arr_f)) ** 2))
|
||||
for b in range(n_bins):
|
||||
plane = (quantized == b).astype(np.float32)
|
||||
plane_gpu = to_gpu(plane)
|
||||
prob_plane = to_cpu(xp_uniform_filter(plane_gpu, size=window))
|
||||
prob_val = prob_plane / win_area
|
||||
prob_val = np.clip(prob_val, 1e-10, None)
|
||||
entropy -= prob_val * np.log2(prob_val)
|
||||
del plane_gpu # free GPU memory per bin
|
||||
|
||||
contrast = generic_filter(img_uint8.astype(np.float64), local_variance,
|
||||
size=window, mode='nearest')
|
||||
entropy = generic_filter(img_uint8.astype(np.float64), local_entropy,
|
||||
size=window, mode='nearest')
|
||||
homogeneity = generic_filter(img_uint8.astype(np.float64), local_homogeneity,
|
||||
size=window, mode='nearest')
|
||||
del quantized, img_uint8 # free CPU memory
|
||||
|
||||
# Homogeneity — 1 / (1 + variance)
|
||||
homogeneity = 1.0 / (1.0 + contrast)
|
||||
|
||||
def norm(arr):
|
||||
valid_arr = arr[~np.isnan(arr)]
|
||||
@ -650,12 +658,10 @@ def generate_texture(dem_file, basename, vis_dir, resolution):
|
||||
std_val = max(np.std(valid_arr), 0.01)
|
||||
return (arr - np.mean(valid_arr)) / std_val
|
||||
|
||||
texture_combined = (0.4 * norm(contrast) +
|
||||
0.4 * norm(entropy) -
|
||||
0.2 * norm(homogeneity))
|
||||
texture_combined = 0.4 * norm(contrast) + 0.4 * norm(entropy) - 0.2 * norm(homogeneity)
|
||||
|
||||
_save_tif(output, texture_combined, transform, crs)
|
||||
logger.info(f" ✓ Texture terminée ({time.time()-t0:.1f}s)")
|
||||
logger.info(f" ✓ Texture terminée ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur texture GLCM: {e}", exc_info=True)
|
||||
@ -667,55 +673,58 @@ def generate_texture(dem_file, basename, vis_dir, resolution):
|
||||
# ============================================================
|
||||
|
||||
def generate_flow(dem_file, basename, vis_dir, resolution):
|
||||
"""Flow accumulation using D8 algorithm.
|
||||
|
||||
Identifies drainage patterns, ditches, and enclosure boundaries.
|
||||
"""
|
||||
logger.info(" → Accumulation de flux D8...")
|
||||
"""Flow accumulation using D8 algorithm — sink filling on GPU, accumulation on CPU."""
|
||||
gpu_tag = " [GPU]" if HAS_GPU else ""
|
||||
logger.info(f" → Accumulation de flux D8{gpu_tag}...")
|
||||
t0 = time.time()
|
||||
output = vis_dir / f"{basename}_flow.tif"
|
||||
|
||||
try:
|
||||
dem, transform, crs = _read_dem(dem_file)
|
||||
rows, cols = dem.shape
|
||||
nodata_mask = np.isnan(dem)
|
||||
dem_np, transform, crs = _read_dem(dem_file)
|
||||
rows, cols = dem_np.shape
|
||||
nodata_mask = np.isnan(dem_np)
|
||||
|
||||
from scipy.ndimage import minimum_filter as scipy_min_filter, generate_binary_structure
|
||||
|
||||
dem_filled = dem.copy()
|
||||
dem_filled[nodata_mask] = np.nanmax(dem) + 1000
|
||||
# Sink filling — GPU-accelerated
|
||||
dem_gpu = to_gpu(dem_np)
|
||||
nodata_mask_gpu = xp.isnan(dem_gpu)
|
||||
dem_filled = xp.copy(dem_gpu)
|
||||
dem_filled[nodata_mask_gpu] = xp.nanmax(dem_gpu) + 1000
|
||||
|
||||
from scipy.ndimage import generate_binary_structure
|
||||
struct = generate_binary_structure(2, 2)
|
||||
|
||||
for _ in range(50):
|
||||
neighbor_min = scipy_min_filter(dem_filled, footprint=struct)
|
||||
sinks = (dem_filled < neighbor_min) & ~nodata_mask
|
||||
if not np.any(sinks):
|
||||
neighbor_min = xp_minimum_filter(dem_filled, footprint=struct)
|
||||
sinks = (dem_filled < neighbor_min) & ~nodata_mask_gpu
|
||||
if not xp.any(sinks):
|
||||
break
|
||||
dem_filled = np.where(sinks, neighbor_min, dem_filled)
|
||||
dem_filled = xp.where(sinks, neighbor_min, dem_filled)
|
||||
|
||||
dem_filled[nodata_mask] = np.nan
|
||||
dem_filled[nodata_mask_gpu] = xp.nan
|
||||
dem_filled_np = to_cpu(dem_filled)
|
||||
|
||||
# D8 slope + accumulation — CPU (sequential by nature)
|
||||
dx8 = [1, 1, 0, -1, -1, -1, 0, 1]
|
||||
dy8 = [0, 1, 1, 1, 0, -1, -1, -1]
|
||||
dist8 = [1.0, np.sqrt(2), 1.0, np.sqrt(2), 1.0, np.sqrt(2), 1.0, np.sqrt(2)]
|
||||
|
||||
flow_dir = np.full((rows, cols), -1, dtype=np.int8)
|
||||
max_slope = np.full((rows, cols), 0.0)
|
||||
max_slope = np.zeros((rows, cols), dtype=np.float64)
|
||||
|
||||
padded = np.pad(dem_filled, 1, mode='constant',
|
||||
constant_values=np.nanmax(dem_filled) + 10000)
|
||||
padded = np.pad(dem_filled_np, 1, mode='constant',
|
||||
constant_values=np.nanmax(dem_filled_np[~np.isnan(dem_filled_np)]) + 10000)
|
||||
|
||||
for d in range(8):
|
||||
nx = 1 + dx8[d]
|
||||
ny = 1 + dy8[d]
|
||||
neighbor_elev = padded[ny:ny + rows, nx:nx + cols]
|
||||
slope = (dem_filled - neighbor_elev) / (dist8[d] * resolution)
|
||||
slope = (dem_filled_np - neighbor_elev) / (dist8[d] * resolution)
|
||||
slope[nodata_mask] = -1
|
||||
better = slope > max_slope
|
||||
flow_dir[better] = d
|
||||
max_slope[better] = slope[better]
|
||||
|
||||
flat_dem = dem_filled[~nodata_mask].flatten()
|
||||
flat_dem = dem_filled_np[~nodata_mask].flatten()
|
||||
valid_indices = np.where(~nodata_mask.flatten())[0]
|
||||
sort_order = valid_indices[np.argsort(-flat_dem)]
|
||||
|
||||
@ -733,7 +742,7 @@ def generate_flow(dem_file, basename, vis_dir, resolution):
|
||||
|
||||
flow_log = np.log1p(flow_acc)
|
||||
_save_tif(output, flow_log, transform, crs)
|
||||
logger.info(f" ✓ Flux terminé ({time.time()-t0:.1f}s)")
|
||||
logger.info(f" ✓ Flux terminé ({time.time()-t0:.1f}s){gpu_tag}")
|
||||
return output
|
||||
except Exception as e:
|
||||
logger.error(f" ✗ Erreur flux: {e}", exc_info=True)
|
||||
|
||||
93
run.sh
93
run.sh
@ -9,11 +9,39 @@
|
||||
# -v Mode verbeux (timestamps + niveaux)
|
||||
# --debug Mode debug (détails internes fichier:ligne)
|
||||
# -f / --force Régénérer tous les fichiers même si existants
|
||||
# --file NOM Traiter un seul fichier LAZ (pour tests)
|
||||
# --file NOM... Traiter un ou plusieurs fichiers LAZ spécifiques
|
||||
# --test Exécuter les tests unitaires
|
||||
# -h Afficher l'aide complète
|
||||
|
||||
set -e
|
||||
|
||||
# Afficher l'aide si aucun argument
|
||||
if [ $# -eq 0 ]; then
|
||||
echo "Pipeline LiDAR Archéologique"
|
||||
echo ""
|
||||
echo "Usage: $0 [options]"
|
||||
echo ""
|
||||
echo " -r RESOLUTION Résolution en m/px (défaut: 0.5)"
|
||||
echo " -w WORKERS Nombre de workers CPU parallèles (défaut: 1)"
|
||||
echo " -g Activer l'accélération GPU NVIDIA"
|
||||
echo " -v Mode verbeux (timestamps + niveaux)"
|
||||
echo " --debug Mode debug (détails internes fichier:ligne)"
|
||||
echo " -f / --force Régénérer tous les fichiers même si les WebP existent"
|
||||
echo " --file NOM... Traiter un ou plusieurs fichiers LAZ (nom complet sans .laz/.las)"
|
||||
echo " --test Exécuter les tests unitaires"
|
||||
echo " -h Afficher cette aide"
|
||||
echo ""
|
||||
echo "Exemples:"
|
||||
echo " $0 -g # Avec accélération GPU"
|
||||
echo " $0 -g -w 4 # GPU + 4 workers parallèles"
|
||||
echo " $0 -g -v # GPU + mode verbeux"
|
||||
echo " $0 -r 0.2 -g --debug # Haute résolution + GPU + debug"
|
||||
echo " $0 -f # Forcer la régénération de tous les fichiers"
|
||||
echo " $0 --file LHD_FXX_1000_6882_PTS_LAMB93_IGN69.copc # Un fichier"
|
||||
echo " $0 --file LHD_...6881.copc LHD_...6882.copc # Plusieurs fichiers"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
INPUT_DIR="${SCRIPT_DIR}/input"
|
||||
OUTPUT_DIR="${SCRIPT_DIR}/output"
|
||||
@ -23,7 +51,7 @@ WORKERS=1
|
||||
GPU_FLAG=""
|
||||
VERBOSE_FLAG=""
|
||||
FORCE_FLAG=""
|
||||
FILE_FILTER=""
|
||||
FILE_ARGS=""
|
||||
|
||||
while getopts "r:w:gvf-:" opt; do
|
||||
case $opt in
|
||||
@ -51,27 +79,58 @@ while getopts "r:w:gvf-:" opt; do
|
||||
echo " -v Mode verbeux (timestamps + niveaux)"
|
||||
echo " --debug Mode debug (détails internes fichier:ligne)"
|
||||
echo " -f / --force Régénérer tous les fichiers même si les WebP existent"
|
||||
echo " --file NOM Traiter un seul fichier LAZ (pour tests)"
|
||||
echo " --file NOM... Traiter un ou plusieurs fichiers LAZ (nom complet sans .laz/.las)"
|
||||
echo " -h Afficher cette aide"
|
||||
echo ""
|
||||
echo "Exemples:"
|
||||
echo " $0 # Traitement standard"
|
||||
echo " $0 -g # Avec accélération GPU"
|
||||
echo " $0 -g -w 4 # GPU + 4 workers parallèles"
|
||||
echo " $0 -g -v # GPU + mode verbeux"
|
||||
echo " $0 -r 0.2 -g --debug # Haute résolution + GPU + debug"
|
||||
echo " $0 -f # Forcer la régénération de tous les fichiers"
|
||||
echo " $0 --file 6881 # Traiter un seul fichier (pour tests)"
|
||||
echo " $0 -g -w 4 # GPU + 4 workers parallèles"
|
||||
echo " $0 -g -v # GPU + mode verbeux"
|
||||
echo " $0 -r 0.2 -g --debug # Haute résolution + GPU + debug"
|
||||
echo " $0 -f # Forcer la régénération de tous les fichiers"
|
||||
echo " $0 --file 6881 # Traiter un seul fichier"
|
||||
echo " $0 --file 6881 6882 # Traiter deux fichiers spécifiques"
|
||||
exit 0
|
||||
;;
|
||||
*) echo "Option invalide. Utilisez -h pour l'aide." >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Handle --file option separately (not easily done in getopts)
|
||||
if [[ " $@ " == *" --file "* ]]; then
|
||||
# Extract the argument after --file
|
||||
FILE_FILTER=$(echo "$@" | sed -n 's/.*--file *\([^ ]*\).*/\1/p')
|
||||
# Check for --test flag first
|
||||
if [[ " $* " == *" --test "* ]]; then
|
||||
# Build l'image si elle n'existe pas
|
||||
if ! docker image inspect "$IMAGE_NAME" >/dev/null 2>&1; then
|
||||
echo "Build de l'image Docker..."
|
||||
docker build -t "$IMAGE_NAME" "$SCRIPT_DIR"
|
||||
fi
|
||||
echo "============================================"
|
||||
echo " Tests unitaires LiDAR Pipeline"
|
||||
echo "============================================"
|
||||
docker run --rm $GPU_FLAG \
|
||||
"$IMAGE_NAME" \
|
||||
python3 -m pytest -v --pyargs lidar_pipeline.tests
|
||||
exit $?
|
||||
fi
|
||||
|
||||
# Collect --file arguments (everything after --file until next option)
|
||||
if [[ "$*" == *" --file "* ]] || [[ "$*" == *" --file"* && "$*" == *"--file "* ]]; then
|
||||
# Extract all arguments after --file
|
||||
PAST_FILE=false
|
||||
for arg in "$@"; do
|
||||
if [ "$arg" = "--file" ]; then
|
||||
PAST_FILE=true
|
||||
continue
|
||||
fi
|
||||
if $PAST_FILE; then
|
||||
# Stop if we hit another option
|
||||
if [[ "$arg" == -* ]]; then
|
||||
PAST_FILE=false
|
||||
else
|
||||
FILE_ARGS="$FILE_ARGS $arg"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
FILE_ARGS=$(echo "$FILE_ARGS" | xargs)
|
||||
fi
|
||||
|
||||
# Build l'image si elle n'existe pas
|
||||
@ -92,14 +151,14 @@ echo " Workers : ${WORKERS}"
|
||||
echo " GPU : $([ -n "$GPU_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||
echo " Verbeux : $([ -n "$VERBOSE_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||
echo " Force : $([ -n "$FORCE_FLAG" ] && echo 'OUI' || echo 'non')"
|
||||
if [ -n "$FILE_FILTER" ]; then
|
||||
echo " Fichier : ${FILE_FILTER}"
|
||||
if [ -n "$FILE_ARGS" ]; then
|
||||
echo " Fichiers :${FILE_ARGS}"
|
||||
fi
|
||||
echo "============================================"
|
||||
|
||||
CMD_ARGS="-o /data/output -r $RESOLUTION -w $WORKERS $VERBOSE_FLAG $FORCE_FLAG"
|
||||
if [ -n "$FILE_FILTER" ]; then
|
||||
CMD_ARGS="$CMD_ARGS --file $FILE_FILTER"
|
||||
if [ -n "$FILE_ARGS" ]; then
|
||||
CMD_ARGS="$CMD_ARGS --file $FILE_ARGS"
|
||||
fi
|
||||
|
||||
docker run --rm $GPU_FLAG \
|
||||
|
||||
Reference in New Issue
Block a user