257 lines
14 KiB
Markdown
257 lines
14 KiB
Markdown
<div align="center">
|
||
|
||
# lidar_rendu
|
||
|
||
**See through the forest.** Turn France's open LiDAR HD point clouds into seamless,
|
||
archaeology-grade relief maps — served as a fast slippy map, reusable XYZ tiles
|
||
and print-ready PDF field sheets.
|
||
|
||
[](LICENSE)
|
||
|
||
|
||
-76B900?logo=nvidia&logoColor=white)
|
||
|
||
|
||
|
||
<img src="docs/img/neuf-brisach.png" alt="Neuf-Brisach fortress in oriented relief" width="900">
|
||
|
||
<sub>Neuf-Brisach (Haut-Rhin), Vauban's star fortress (UNESCO World Heritage), rendered from one 1 km LiDAR HD tile.
|
||
Every bastion, moat and ravelin reads at a glance; buildings are black (no ground points).</sub>
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
## Why
|
||
|
||
The French mapping agency (IGN) publishes **LiDAR HD**: a nationwide airborne laser
|
||
scan, ~10 points/m², free under the *Licence Ouverte 2.0*. Hidden under forest
|
||
canopy and fields are hollow ways, trenches, burial mounds, field systems and
|
||
forgotten walls — but raw point clouds are hard to read, and naïve DEM renderings
|
||
show seams at every tile edge, stripes from the flight lines and colour blotches
|
||
that look like relief but aren't.
|
||
|
||
`lidar_rendu` is an end-to-end pipeline that solves exactly that:
|
||
|
||
- **One command** from an IGN tile ID to a browsable map.
|
||
- **One carefully designed layer** — *oriented relief* — instead of fifteen
|
||
colormaps to flip through.
|
||
- **Seamless at any scale**: fixed scales, no per-tile statistics, 100 m
|
||
overlap borrowed from the eight neighbouring tiles.
|
||
- **Measurement-aware**: a companion *precision* layer shows where the terrain is
|
||
measured and where it is interpolated, so you never mistake a gap for a feature.
|
||
|
||
## Gallery
|
||
|
||
<table>
|
||
<tr>
|
||
<td width="50%"><img src="docs/img/hartmannswillerkopf.png" alt="Hartmannswillerkopf battlefield"></td>
|
||
<td width="50%"><img src="docs/img/neuf-brisach-a4.png" alt="A4 PDF field sheet of Neuf-Brisach"></td>
|
||
</tr>
|
||
<tr>
|
||
<td><b>Hartmannswillerkopf</b> (tile 1011_6760) — the WWI battlefield of 1915 under dense
|
||
forest: roads, trench lines and shell craters appear on the slope.</td>
|
||
<td><b>PDF field sheet</b> exported from the map (A4 landscape, 1:5,000): Lambert 93 grid,
|
||
true-north arrow, scale bar, legend and data-quality inset.</td>
|
||
</tr>
|
||
</table>
|
||
|
||
## Highlights
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Oriented relief** | A single RGB image: CIELAB *lightness* carries local openness (bumps light, hollows dark), *hue* carries slope orientation at constant lightness — so colour never fakes relief. |
|
||
| **Flight-strip calibration** | Each tile mixes several passes, sometimes offset by a few cm. The pipeline estimates per-strip offsets, per-scan-line shift and roll, and a per-beam cross-track profile, then removes them before rasterising. No more stripes. |
|
||
| **Honest gap filling** | At 0.2 m, ~80 % of pixels contain no point. Gaps are closed morphologically within the point envelope, with a radius that follows local point spacing — nothing is extrapolated outward. |
|
||
| **Seamless tiles** | DTMs are built on the nominal 1 km tile plus a 100 m buffer taken from the neighbours (downloaded automatically), then cropped back exactly. |
|
||
| **Fast** | GPU (CuPy) when available, numba otherwise. Ground extraction straight from IGN classes with laspy (~5 s per tile), relief kernel ~5 s on a 12-core CPU, AVIF encoding in 0.6 s. |
|
||
| **Slippy map** | Built-in Leaflet UI: tile selection with IGN metadata, relief / precision / side-by-side compare, adjustable intensity, light & dark themes, mobile bottom sheet, shareable links. |
|
||
| **Standard tiles** | `/tiles/{layer}/{z}/{x}/{y}.png` in the OpenStreetMap scheme, plus TileJSON, WMTS and a JOSM imagery file — drop it into QGIS, JOSM, iD, uMap or MapLibre. |
|
||
| **Generate from the map** | Draw a rectangle: the tiles are downloaded from IGN, processed, and appear on the map row by row as they finish. |
|
||
| **Print** | Vector PDF sheets (A4/A3, 1:1,000 – 1:10,000) with grid, WGS84 corners, meridian convergence, legend and point-density inset. |
|
||
| **Runs on a Raspberry Pi** | A lightweight image (no PDAL, no GPU) serves the map and delegates generation to a worker machine; it keeps working offline. |
|
||
|
||
## Quick start
|
||
|
||
Requirements: Docker with Compose ≥ 2.24. An NVIDIA GPU is optional.
|
||
|
||
```bash
|
||
git clone https://github.com/<you>/lidar_rendu.git && cd lidar_rendu
|
||
|
||
./start.sh # build the image, start the map on http://localhost:8973/
|
||
./start.sh process --fetch-tiles 1037,6779 # download one IGN tile (Neuf-Brisach) and process it
|
||
./start.sh logs # follow the server log
|
||
./start.sh stop # stop everything
|
||
```
|
||
|
||
`start.sh` creates `input/` and `output/` owned by you and checks whether Docker
|
||
can reach a GPU; if not, it adds `docker-compose.cpu.yml` and everything runs on
|
||
the CPU (slower, same output).
|
||
|
||
Tile IDs are the IGN grid coordinates in km (Lambert 93): `1037,6779` is the tile
|
||
whose top-left corner is at X = 1,037 km, Y = 6,779 km. You can also skip the
|
||
command line entirely: open the map, go to the **Génération** tab and draw a
|
||
zone.
|
||
|
||
> The user interface and log messages are in French; code identifiers are in English.
|
||
|
||
## How it works
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
A[IGN LiDAR HD<br/>COPC .laz tile] --> B[Ground points<br/>IGN classes, laspy]
|
||
N[8 neighbour tiles<br/>100 m buffer] --> B
|
||
B --> C[Strip & scan-line<br/>calibration]
|
||
C --> D[DTM 0.2 m<br/>+ bounded gap fill]
|
||
D --> E[Oriented relief<br/>+ precision layer]
|
||
E --> F[AVIF tiles<br/>cropped to 1 km]
|
||
F --> G[Inventory +<br/>XYZ pyramid]
|
||
G --> H[Map · XYZ/WMTS · PDF]
|
||
```
|
||
|
||
1. **Download** — the COPC tile and, if missing, its eight neighbours from the IGN
|
||
Géoplateforme. Tiles are processed north to south, so the map fills top down.
|
||
2. **Ground** — IGN's ground class (2) extracted directly with laspy; PDAL (SMRF /
|
||
CSF) remains available. See [ground classification](docs/GROUND_CLASSIFICATION.md).
|
||
3. **Calibration** — robust per-strip vertical offsets, then a joint least-squares
|
||
adjustment of every scan line (offset + roll) against the other strips, plus a
|
||
per-beam angular profile. Results are written to a JSON sidecar next to the DTM.
|
||
4. **DTM** — rasterised at 0.2 m on the tile + buffer; small gaps closed within the
|
||
point envelope; ground density saved alongside.
|
||
5. **Render** — oriented relief (openness at 5 / 10 / 20 m on a detrended DTM,
|
||
16 directions, plus 35 % hillshade) and precision (16 log-scale density levels).
|
||
Fixed scales everywhere: identical terrain gives identical colour on every tile.
|
||
6. **Publish** — AVIF quadrants encoded once from the source raster, inventory
|
||
refreshed after every tile, XYZ pyramid pre-generated in the background.
|
||
|
||
## The map
|
||
|
||
- **Click a tile** to select it: extent, IGN acquisition date, sensors, point count,
|
||
download link, and the calibration applied to each flight strip.
|
||
- **Three view modes** — relief, precision, or *compare* with a draggable split bar.
|
||
- **Relief intensity** slider (0.5×–2×), remembered per browser and carried in the link.
|
||
- **Share** — the URL encodes position, mode, comparison and intensity.
|
||
- **Use in JOSM / QGIS** button — copies the tile URL for the current layer.
|
||
- Keyboard: `1`–`5` tabs, `P` next view mode, `Esc` deselect / collapse.
|
||
- Works on phones: the panel becomes a three-height bottom sheet; GPS centring over HTTPS.
|
||
|
||
Full reference: [docs/MAPS.md](docs/MAPS.md).
|
||
|
||
## Use the tiles anywhere
|
||
|
||
| Client | URL |
|
||
|---|---|
|
||
| QGIS, iD, uMap, Leaflet, MapLibre | `http://<host>:8973/tiles/relief_oriente/{z}/{x}/{y}.png` |
|
||
| JOSM (TMS) | `http://<host>:8973/tiles/relief_oriente/{zoom}/{x}/{y}.png` |
|
||
| JOSM (all layers at once) | `http://<host>:8973/tiles/josm.imagery.xml` |
|
||
| TileJSON 3.0 | `http://<host>:8973/tiles/relief_oriente.json` |
|
||
| WMTS 1.0 | `http://<host>:8973/tiles/wmts.xml` |
|
||
|
||
256 px PNG, EPSG:3857, transparent outside coverage, CORS enabled, zoom 5–19
|
||
(19 ≈ 0.2 m/px, one screen pixel per LiDAR pixel).
|
||
|
||
## Command line
|
||
|
||
The processing pipeline is `python3 -m lidar_pipeline`; with Docker Compose,
|
||
`./start.sh process [options]` runs it on `input/` → `output/`.
|
||
|
||
| Option | Default | Purpose |
|
||
|---|---|---|
|
||
| `--fetch-tiles COL,ROW …` | — | Download these LiDAR HD tiles from IGN first |
|
||
| `--file NAME …` | all of `input/` | Process only these tiles (name without `.laz`) |
|
||
| `-r RES` | `0.2` | Resolution in m/px (comma-separated for several) |
|
||
| `-g [GPU]` | CPU | GPU(s): `-g`, `-g 0`, `-g 0,2`, `-g all` |
|
||
| `-w N` | `auto` | Parallel workers (bounded by free VRAM per GPU) |
|
||
| `--only VIZ …` / `--skip VIZ …` | map layers | Choose visualisations |
|
||
| `--edge-buffer M` | `100` | Buffer borrowed from neighbour tiles (0 = off) |
|
||
| `--ground-classification` | `ign` | `ign`, `auto`, `smrf`, `csf` |
|
||
| `--no-strip-align` | on | Disable flight-strip calibration |
|
||
| `--format`, `--quality` | `avif`, `60` | Output encoding |
|
||
| `-f`, `--force` | off | Regenerate even if outputs exist |
|
||
| `--rebuild-index` | — | Rebuild the tile inventory only |
|
||
| `-v`, `--debug` | — | Verbose / debug logging |
|
||
|
||
By default only the map layers are produced (`relief_oriente`, `densite_sol`).
|
||
Classic visualisations remain one flag away, e.g.
|
||
`./start.sh process --only hillshade slope svf pos_open`:
|
||
|
||
| Key | Visualisation | | Key | Visualisation |
|
||
|---|---|---|---|---|
|
||
| `hillshade` | Multi-directional hillshade | | `svf` | Sky-View Factor |
|
||
| `slope` | Slope | | `roughness` | Roughness |
|
||
| `aspect` | Aspect | | `wavelet` | Mexican-hat wavelet (multi-scale) |
|
||
| `mslrm` | Multi-scale relief model | | `flow_acc` | D8 flow accumulation |
|
||
| `sailore` | Slope-adaptive local relief | | `solar` | Solar illumination |
|
||
| `pos_open` / `neg_open` | Positive / negative openness | | `anomaly` | Statistical anomalies |
|
||
| `ortho` / `topo` | IGN orthophoto / topographic map | | | |
|
||
|
||
## Deployment
|
||
|
||
| Setup | Command | Port |
|
||
|---|---|---|
|
||
| All-in-one (map + generation) | `docker compose up -d --build serve` (or `./start.sh`) | 8973 |
|
||
| Processing worker for remote maps | `docker compose -f docker-compose.worker.yml up -d --build` | 8973 |
|
||
| Lightweight map (Raspberry Pi, no PDAL/GPU) | `docker compose -f docker-compose.maps.yml up -d --build` | 8975 |
|
||
|
||
The lightweight map mirrors the worker's tiles, delegates generation to it, and
|
||
keeps serving from disk when the worker is off. Tokens (`LIDAR_API_TOKEN`) and a
|
||
network allow-list (`LIDAR_REGEN_CIDR`, private ranges by default) protect the
|
||
generation API. Step-by-step guide: [docs/DEPLOY_WEBAPP.md](docs/DEPLOY_WEBAPP.md).
|
||
|
||
Always pass `--build`: the code is baked into the image, and the build is
|
||
near-instant thanks to the layer cache.
|
||
|
||
## Project layout
|
||
|
||
```
|
||
lidar_pipeline/
|
||
├── cli.py command line
|
||
├── pipeline.py orchestration, worker pool, VRAM-aware GPU slots
|
||
├── fetch_ign.py IGN catalogue and downloads
|
||
├── dtm.py ground extraction, strip calibration, DTM, gap filling
|
||
├── visualizations.py generate_* functions (relief, openness, SVF, …)
|
||
├── rendering.py colormaps, GeoTIFF → AVIF, 1 km crop
|
||
├── index.py catalogue, thumbnails, sub-tiles, inventory, layer registry
|
||
├── quality.py per-tile quality sidecar (density, acquisition dates)
|
||
├── tiles.py XYZ pyramid: reprojection, cache, background maintenance
|
||
├── mapserve.py FastAPI server: map, tiles, TileJSON/WMTS, generation API
|
||
├── export_pdf.py PDF field sheets (Pillow + pyproj + reportlab)
|
||
├── gpu.py CuPy / NumPy abstraction with CPU fallback
|
||
├── web/ map UI (HTML, CSS, JS)
|
||
└── tests/ ~400 pytest tests
|
||
```
|
||
|
||
## Development
|
||
|
||
```bash
|
||
./run.sh --test # rebuild the image and run the full test suite
|
||
docker run --rm lidar-lidar python3 -m pytest -v --pyargs lidar_pipeline.tests.test_tiles
|
||
```
|
||
|
||
Adding a visualisation takes four edits: a `generate_X()` in `visualizations.py`,
|
||
an entry in `VIZ_STEPS` (`pipeline.py`), a colormap in `rendering.py`, and a legend
|
||
in `VIZ_LEGENDS` (`index.py`). Design decisions and conventions are documented in
|
||
[AGENTS.md](AGENTS.md) (French).
|
||
|
||
Contributions are welcome: open an issue or a pull request.
|
||
|
||
## Documentation
|
||
|
||
- [docs/MAPS.md](docs/MAPS.md) — map UI, tile contract, cache, PDF export, environment variables
|
||
- [docs/DEPLOY_WEBAPP.md](docs/DEPLOY_WEBAPP.md) — two-machine deployment (Raspberry Pi + worker)
|
||
- [docs/GROUND_CLASSIFICATION.md](docs/GROUND_CLASSIFICATION.md) — ground filters benchmark and literature
|
||
|
||
## Data and licences
|
||
|
||
- **Code**: [MIT](LICENSE).
|
||
- **LiDAR HD, orthophotos, maps**: © IGN, [Licence Ouverte 2.0](https://www.etalab.gouv.fr/licence-ouverte-open-licence/).
|
||
Attribution is required and is shown on every map, tile endpoint and PDF.
|
||
- **Base map**: © [OpenStreetMap](https://www.openstreetmap.org/copyright) contributors.
|
||
Before using these renderings as a tracing source in OpenStreetMap, check the
|
||
community's position on the source.
|
||
|
||
Built on [PDAL](https://pdal.io), [laspy](https://laspy.readthedocs.io),
|
||
[rasterio](https://rasterio.readthedocs.io), [CuPy](https://cupy.dev),
|
||
[numba](https://numba.pydata.org), [FastAPI](https://fastapi.tiangolo.com),
|
||
[Leaflet](https://leafletjs.com) and [reportlab](https://www.reportlab.com).
|