<div align="center">

# lidar_rendu

**See through the forest.** Turn France's open LiDAR HD point clouds into seamless,
archaeology-grade relief maps — served as a fast slippy map, reusable XYZ tiles
and print-ready PDF field sheets.

[![License: MIT](https://img.shields.io/badge/license-MIT-2ea44f.svg)](LICENSE)
![Docker](https://img.shields.io/badge/run%20with-Docker%20Compose-2496ED?logo=docker&logoColor=white)
![Python](https://img.shields.io/badge/python-3.10+-3776AB?logo=python&logoColor=white)
![GPU](https://img.shields.io/badge/GPU-optional%20(CUDA%20%2F%20CuPy)-76B900?logo=nvidia&logoColor=white)
![Data](https://img.shields.io/badge/data-IGN%20LiDAR%20HD-0b6fb7)
![Tiles](https://img.shields.io/badge/tiles-XYZ%20%C2%B7%20TileJSON%20%C2%B7%20WMTS-7952b3)

<img src="docs/img/neuf-brisach.png" alt="Neuf-Brisach fortress in oriented relief" width="900">

<sub>Neuf-Brisach (Haut-Rhin), Vauban's star fortress (UNESCO World Heritage), rendered from one 1 km LiDAR HD tile.
Every bastion, moat and ravelin reads at a glance; buildings are black (no ground points).</sub>

</div>

---

## Why

The French mapping agency (IGN) publishes **LiDAR HD**: a nationwide airborne laser
scan, ~10 points/m², free under the *Licence Ouverte 2.0*. Hidden under forest
canopy and fields are hollow ways, trenches, burial mounds, field systems and
forgotten walls — but raw point clouds are hard to read, and naïve DEM renderings
show seams at every tile edge, stripes from the flight lines and colour blotches
that look like relief but aren't.

`lidar_rendu` is an end-to-end pipeline that solves exactly that:

- **One command** from an IGN tile ID to a browsable map.
- **One carefully designed layer** — *oriented relief* — instead of fifteen
  colormaps to flip through.
- **Seamless at any scale**: fixed scales, no per-tile statistics, 100 m
  overlap borrowed from the eight neighbouring tiles.
- **Measurement-aware**: a companion *precision* layer shows where the terrain is
  measured and where it is interpolated, so you never mistake a gap for a feature.

## Gallery

<table>
<tr>
<td width="50%"><img src="docs/img/hartmannswillerkopf.png" alt="Hartmannswillerkopf battlefield"></td>
<td width="50%"><img src="docs/img/neuf-brisach-a4.png" alt="A4 PDF field sheet of Neuf-Brisach"></td>
</tr>
<tr>
<td><b>Hartmannswillerkopf</b> (tile 1011_6760) — the WWI battlefield of 1915 under dense
forest: roads, trench lines and shell craters appear on the slope.</td>
<td><b>PDF field sheet</b> exported from the map (A4 landscape, 1:5,000): Lambert 93 grid,
true-north arrow, scale bar, legend and data-quality inset.</td>
</tr>
</table>

## Highlights

| | |
|---|---|
| **Oriented relief** | A single RGB image: CIELAB *lightness* carries local openness (bumps light, hollows dark), *hue* carries slope orientation at constant lightness — so colour never fakes relief. |
| **Flight-strip calibration** | Each tile mixes several passes, sometimes offset by a few cm. The pipeline estimates per-strip offsets, per-scan-line shift and roll, and a per-beam cross-track profile, then removes them before rasterising. No more stripes. |
| **Honest gap filling** | At 0.2 m, ~80 % of pixels contain no point. Gaps are closed morphologically within the point envelope, with a radius that follows local point spacing — nothing is extrapolated outward. |
| **Seamless tiles** | DTMs are built on the nominal 1 km tile plus a 100 m buffer taken from the neighbours (downloaded automatically), then cropped back exactly. |
| **Fast** | GPU (CuPy) when available, numba otherwise. Ground extraction straight from IGN classes with laspy (~5 s per tile), relief kernel ~5 s on a 12-core CPU, AVIF encoding in 0.6 s. |
| **Slippy map** | Built-in Leaflet UI: tile selection with IGN metadata, relief / precision / side-by-side compare, adjustable intensity, light & dark themes, mobile bottom sheet, shareable links. |
| **Standard tiles** | `/tiles/{layer}/{z}/{x}/{y}.png` in the OpenStreetMap scheme, plus TileJSON, WMTS and a JOSM imagery file — drop it into QGIS, JOSM, iD, uMap or MapLibre. |
| **Generate from the map** | Draw a rectangle: the tiles are downloaded from IGN, processed, and appear on the map row by row as they finish. |
| **Print** | Vector PDF sheets (A4/A3, 1:1,000 – 1:10,000) with grid, WGS84 corners, meridian convergence, legend and point-density inset. |
| **Runs on a Raspberry Pi** | A lightweight image (no PDAL, no GPU) serves the map and delegates generation to a worker machine; it keeps working offline. |

## Quick start

Requirements: Docker with Compose ≥ 2.24. An NVIDIA GPU is optional.

```bash
git clone https://github.com/<you>/lidar_rendu.git && cd lidar_rendu

./start.sh                                 # build the image, start the map on http://localhost:8973/
./start.sh process --fetch-tiles 1037,6779 # download one IGN tile (Neuf-Brisach) and process it
./start.sh logs                            # follow the server log
./start.sh stop                            # stop everything
```

`start.sh` creates `input/` and `output/` owned by you and checks whether Docker
can reach a GPU; if not, it adds `docker-compose.cpu.yml` and everything runs on
the CPU (slower, same output).

Tile IDs are the IGN grid coordinates in km (Lambert 93): `1037,6779` is the tile
whose top-left corner is at X = 1,037 km, Y = 6,779 km. You can also skip the
command line entirely: open the map, go to the **Génération** tab and draw a
zone.

> The user interface and log messages are in French; code identifiers are in English.

## How it works

```mermaid
flowchart LR
    A[IGN LiDAR HD<br/>COPC .laz tile] --> B[Ground points<br/>IGN classes, laspy]
    N[8 neighbour tiles<br/>100 m buffer] --> B
    B --> C[Strip & scan-line<br/>calibration]
    C --> D[DTM 0.2 m<br/>+ bounded gap fill]
    D --> E[Oriented relief<br/>+ precision layer]
    E --> F[AVIF tiles<br/>cropped to 1 km]
    F --> G[Inventory +<br/>XYZ pyramid]
    G --> H[Map · XYZ/WMTS · PDF]
```

1. **Download** — the COPC tile and, if missing, its eight neighbours from the IGN
   Géoplateforme. Tiles are processed north to south, so the map fills top down.
2. **Ground** — IGN's ground class (2) extracted directly with laspy; PDAL (SMRF /
   CSF) remains available. See [ground classification](docs/GROUND_CLASSIFICATION.md).
3. **Calibration** — robust per-strip vertical offsets, then a joint least-squares
   adjustment of every scan line (offset + roll) against the other strips, plus a
   per-beam angular profile. Results are written to a JSON sidecar next to the DTM.
4. **DTM** — rasterised at 0.2 m on the tile + buffer; small gaps closed within the
   point envelope; ground density saved alongside.
5. **Render** — oriented relief (openness at 5 / 10 / 20 m on a detrended DTM,
   16 directions, plus 35 % hillshade) and precision (16 log-scale density levels).
   Fixed scales everywhere: identical terrain gives identical colour on every tile.
6. **Publish** — AVIF quadrants encoded once from the source raster, inventory
   refreshed after every tile, XYZ pyramid pre-generated in the background.

## The map

- **Click a tile** to select it: extent, IGN acquisition date, sensors, point count,
  download link, and the calibration applied to each flight strip.
- **Three view modes** — relief, precision, or *compare* with a draggable split bar.
- **Relief intensity** slider (0.5×–2×), remembered per browser and carried in the link.
- **Share** — the URL encodes position, mode, comparison and intensity.
- **Use in JOSM / QGIS** button — copies the tile URL for the current layer.
- Keyboard: `1`–`5` tabs, `P` next view mode, `Esc` deselect / collapse.
- Works on phones: the panel becomes a three-height bottom sheet; GPS centring over HTTPS.

Full reference: [docs/MAPS.md](docs/MAPS.md).

## Use the tiles anywhere

| Client | URL |
|---|---|
| QGIS, iD, uMap, Leaflet, MapLibre | `http://<host>:8973/tiles/relief_oriente/{z}/{x}/{y}.png` |
| JOSM (TMS) | `http://<host>:8973/tiles/relief_oriente/{zoom}/{x}/{y}.png` |
| JOSM (all layers at once) | `http://<host>:8973/tiles/josm.imagery.xml` |
| TileJSON 3.0 | `http://<host>:8973/tiles/relief_oriente.json` |
| WMTS 1.0 | `http://<host>:8973/tiles/wmts.xml` |

256 px PNG, EPSG:3857, transparent outside coverage, CORS enabled, zoom 5–19
(19 ≈ 0.2 m/px, one screen pixel per LiDAR pixel).

## Command line

The processing pipeline is `python3 -m lidar_pipeline`; with Docker Compose,
`./start.sh process [options]` runs it on `input/` → `output/`.

| Option | Default | Purpose |
|---|---|---|
| `--fetch-tiles COL,ROW …` | — | Download these LiDAR HD tiles from IGN first |
| `--file NAME …` | all of `input/` | Process only these tiles (name without `.laz`) |
| `-r RES` | `0.2` | Resolution in m/px (comma-separated for several) |
| `-g [GPU]` | CPU | GPU(s): `-g`, `-g 0`, `-g 0,2`, `-g all` |
| `-w N` | `auto` | Parallel workers (bounded by free VRAM per GPU) |
| `--only VIZ …` / `--skip VIZ …` | map layers | Choose visualisations |
| `--edge-buffer M` | `100` | Buffer borrowed from neighbour tiles (0 = off) |
| `--ground-classification` | `ign` | `ign`, `auto`, `smrf`, `csf` |
| `--no-strip-align` | on | Disable flight-strip calibration |
| `--format`, `--quality` | `avif`, `60` | Output encoding |
| `-f`, `--force` | off | Regenerate even if outputs exist |
| `--rebuild-index` | — | Rebuild the tile inventory only |
| `-v`, `--debug` | — | Verbose / debug logging |

By default only the map layers are produced (`relief_oriente`, `densite_sol`).
Classic visualisations remain one flag away, e.g.
`./start.sh process --only hillshade slope svf pos_open`:

| Key | Visualisation | | Key | Visualisation |
|---|---|---|---|---|
| `hillshade` | Multi-directional hillshade | | `svf` | Sky-View Factor |
| `slope` | Slope | | `roughness` | Roughness |
| `aspect` | Aspect | | `wavelet` | Mexican-hat wavelet (multi-scale) |
| `mslrm` | Multi-scale relief model | | `flow_acc` | D8 flow accumulation |
| `sailore` | Slope-adaptive local relief | | `solar` | Solar illumination |
| `pos_open` / `neg_open` | Positive / negative openness | | `anomaly` | Statistical anomalies |
| `ortho` / `topo` | IGN orthophoto / topographic map | | | |

## Deployment

| Setup | Command | Port |
|---|---|---|
| All-in-one (map + generation) | `docker compose up -d --build serve` (or `./start.sh`) | 8973 |
| Processing worker for remote maps | `docker compose -f docker-compose.worker.yml up -d --build` | 8973 |
| Lightweight map (Raspberry Pi, no PDAL/GPU) | `docker compose -f docker-compose.maps.yml up -d --build` | 8975 |

The lightweight map mirrors the worker's tiles, delegates generation to it, and
keeps serving from disk when the worker is off. Tokens (`LIDAR_API_TOKEN`) and a
network allow-list (`LIDAR_REGEN_CIDR`, private ranges by default) protect the
generation API. Step-by-step guide: [docs/DEPLOY_WEBAPP.md](docs/DEPLOY_WEBAPP.md).

Always pass `--build`: the code is baked into the image, and the build is
near-instant thanks to the layer cache.

## Project layout

```
lidar_pipeline/
├── cli.py            command line
├── pipeline.py       orchestration, worker pool, VRAM-aware GPU slots
├── fetch_ign.py      IGN catalogue and downloads
├── dtm.py            ground extraction, strip calibration, DTM, gap filling
├── visualizations.py generate_* functions (relief, openness, SVF, …)
├── rendering.py      colormaps, GeoTIFF → AVIF, 1 km crop
├── index.py          catalogue, thumbnails, sub-tiles, inventory, layer registry
├── quality.py        per-tile quality sidecar (density, acquisition dates)
├── tiles.py          XYZ pyramid: reprojection, cache, background maintenance
├── mapserve.py       FastAPI server: map, tiles, TileJSON/WMTS, generation API
├── export_pdf.py     PDF field sheets (Pillow + pyproj + reportlab)
├── gpu.py            CuPy / NumPy abstraction with CPU fallback
├── web/              map UI (HTML, CSS, JS)
└── tests/            ~400 pytest tests
```

## Development

```bash
./run.sh --test                     # rebuild the image and run the full test suite
docker run --rm lidar-lidar python3 -m pytest -v --pyargs lidar_pipeline.tests.test_tiles
```

Adding a visualisation takes four edits: a `generate_X()` in `visualizations.py`,
an entry in `VIZ_STEPS` (`pipeline.py`), a colormap in `rendering.py`, and a legend
in `VIZ_LEGENDS` (`index.py`). Design decisions and conventions are documented in
[AGENTS.md](AGENTS.md) (French).

Contributions are welcome: open an issue or a pull request.

## Documentation

- [docs/MAPS.md](docs/MAPS.md) — map UI, tile contract, cache, PDF export, environment variables
- [docs/DEPLOY_WEBAPP.md](docs/DEPLOY_WEBAPP.md) — two-machine deployment (Raspberry Pi + worker)
- [docs/GROUND_CLASSIFICATION.md](docs/GROUND_CLASSIFICATION.md) — ground filters benchmark and literature

## Data and licences

- **Code**: [MIT](LICENSE).
- **LiDAR HD, orthophotos, maps**: © IGN, [Licence Ouverte 2.0](https://www.etalab.gouv.fr/licence-ouverte-open-licence/).
  Attribution is required and is shown on every map, tile endpoint and PDF.
- **Base map**: © [OpenStreetMap](https://www.openstreetmap.org/copyright) contributors.
  Before using these renderings as a tracing source in OpenStreetMap, check the
  community's position on the source.

Built on [PDAL](https://pdal.io), [laspy](https://laspy.readthedocs.io),
[rasterio](https://rasterio.readthedocs.io), [CuPy](https://cupy.dev),
[numba](https://numba.pydata.org), [FastAPI](https://fastapi.tiangolo.com),
[Leaflet](https://leafletjs.com) and [reportlab](https://www.reportlab.com).

Hartmannswillerkopf (tile 1011_6760) — the WWI battlefield of 1915 under dense forest: roads, trench lines and shell craters appear on the slope.	PDF field sheet exported from the map (A4 landscape, 1:5,000): Lambert 93 grid, true-north arrow, scale bar, legend and data-quality inset.