lidar_rendu/docs/MAPS.md

# XYZ tile map (`lidar-maps` image)

A classic "slippy" map — the same tile scheme as Google Maps and
**OpenStreetMap** — served by a dedicated lightweight image. Pipeline
renders thus become a **reusable imagery basemap** in JOSM, iD, QGIS, uMap,
MapLibre or OsmAnd, on top of the browsing interface it also provides.

This is the ONLY web interface in the project (the old historical webapp has
been removed): it also embeds **tile generation** — local on the full image
(the full pipeline server, port 8973, `./run.sh --serve`), delegated via
`LIDAR_GENERATION_URL` on the lightweight machine (see
`docs/DEPLOY_WEBAPP.md`).

```bash
docker compose -f docker-compose.maps.yml up -d --build   # http://localhost:8975/
docker compose -f docker-compose.maps.yml logs -f maps
./run.sh --serve-maps            # equivalent, foreground container
```

## Interface

A **single tabbed panel** (`web/map.{html,css,js}`) groups all settings,
replacing the old stack of stacked blocks: **View** (main layer,
relief/precision mode, basemap), **Tile** (the selected tile's card),
**PDF** (PDF export), **Generate** (hidden if the generator is unavailable
or not authorized for that browser) and **Share**. On desktop, the panel occupies a
fixed column, collapsible into an icon strip (**‹**, always usable: clicking
a tab redeploys the panel). Below 720 px wide, it becomes a **bottom sheet**
with three heights — closed, half, full — changed by dragging the handle, by
simply tapping it (one notch), or by tapping the already-active tab again.

Clicking the map **selects** the LiDAR HD tile under the cursor (dashed
outline) and fills the **Tile** tab (footprint, IGN info, pass
alignment) without changing the displayed tab; clicking the same tile again
deselects it, clicking another moves the selection. Keyboard shortcuts:
**1–5** (panel tabs, no effect if the tab is hidden), **P** (next display
mode), **Escape** (deselects the tile, then collapses the panel). Two
light/dark themes (◐/☀/☾ button, `auto` by default = follows the system);
settings and panel state are kept in `localStorage` (`lidarMapView_v2`,
`lidar-print`, `lidar-panel`, `lidar-theme` — silently ignored in private
browsing).

## Generating tiles from the map

- **＋ Area** (Generate tab) — draw a rectangle: the 1 km LHD tiles it
  intersects are downloaded from the IGN geoplatform and then processed
  (0.2 m, options below); successive zones and clicks are additive;
- **⤒ Complete** — all tiles already present in `input/` that are
  missing at least one of the requested layers;
- **↻ Generate/Regenerate this tile** (button on the tile card, Tile tab) —
  a specific tile, even with no existing data.

Run options: target layers (default: the panel layers) and forced
regeneration. Ground classification (IGN, ground only) and edge stitching (a
100 m band taken from neighboring tiles) are enforced: no setting available.
A request submitted while a run is already in progress goes into a
**persisted queue** (never interrupting work in progress); progress is shown
tile by tile (orange frames = rendering in progress, blue = waiting, red =
failed) with a log and a **Stop** button (SIGTERM then SIGKILL). During a
run, each completed tile appears on the map as it finishes: the inventory is
updated after each tile and the pyramid maintenance job prioritizes its tiles
(polled every 10 s).

The buttons are hidden if the browser's IP falls outside
`LIDAR_REGEN_CIDR` (default: localhost + private RFC1918 ranges) or if no
generation backend exists (lightweight image without `LIDAR_GENERATION_URL`).
On the lightweight machine, everything is forwarded to the worker
(`LIDAR_GENERATION_URL`), which runs the full pipeline; its tiles are then
pulled back on demand via the `/api/tiles` inventory plus versioned static
assets (`LIDAR_SOURCE_URL`).

## Tiling contract

| Point | Value |
|---|---|
| Projection | EPSG:3857, OSM XYZ scheme (north-west origin, `y` toward the south) |
| Canonical URL | `/tiles/{layer}/{z}/{x}/{y}.png` — 256 px, PNG RGBA |
| High-density variant | `/tiles/{layer}/{z}/{x}/{y}@2x.avif` — 512 px AVIF (internal interface; png/webp also accepted) |
| Zoom levels | 5 → 19 native (0.2 m/px ≈ z19 in France); beyond that, client-side over-zoom |
| Outside coverage | fully transparent tile (overlayable), header `X-Tile-Empty: 1` |
| Pending (cache-only mode) | transparent tile, headers `X-Tile-Empty: 1` + `X-Tile-Pending: 1`, never cached by the browser |
| CORS | `Access-Control-Allow-Origin: *` on `/tiles/*` |
| Attribution | `LIDAR_ATTRIBUTION`, default "LiDAR HD © IGN — Licence Ouverte 2.0" |

Discovery: `/tiles/{layer}.json` (TileJSON 3.0.0), `/tiles/wmts.xml`
(WMTS 1.0.0, `GoogleMapsCompatible` grid), `/tiles/josm.imagery.xml`
(all layers at once in JOSM).

## Using the tiles elsewhere

- **JOSM** — *Imagery → Imagery preferences → + TMS*, paste
  `http://<host>:8975/tiles/slope/{zoom}/{x}/{y}.png`. To add everything at
  once: *Imagery preferences → Offline/Custom → Add imagery source XML* with
  `http://<host>:8975/tiles/josm.imagery.xml`.
- **iD** — *Background → Custom*, paste
  `http://<host>:8975/tiles/slope/{z}/{x}/{y}.png`.
- **QGIS** — *XYZ Tiles → New Connection* (same URL, max zoom 19), or
  *WMS/WMTS → New* with `http://<host>:8975/tiles/wmts.xml`.
- **uMap / MapLibre / Leaflet** — same XYZ template, or the TileJSON.
- **OsmAnd** — online tile source, XYZ template, max zoom 19.

The **XYZ imagery background** section of the map's **Share** tab shows
these URLs for the selected layer, with a **Copy** button for the XYZ template.

## Full pyramid pre-generated

By default, **all** levels up to native (19 in standard OSM numbering =
0.2 m/px, served as `18@2x` by the interface) are written to disk in AVIF
and pre-generated by the background maintenance job (on a map fed by an
upstream source: downloaded from `LIDAR_MAPS_URL`). No level is rendered
on the fly: on-demand rendering on a Raspberry Pi (~170 ms per tile, 3 at a
time) took 1.5 to 6 s per screen at zoom levels 17–19. The interface is
capped at zoom 19 (1 screen pixel = 1 LiDAR pixel): never an upscaled tile.
Measured order of magnitude: ~110,000 @2x tiles for 3,240 tiles, of which
81,000 at native level, ~5 GB.

Reduced storage (measured on disk): a lower `LIDAR_TILE_CACHE_MAX_Z` and/or
`LIDAR_TILE_EVEN_LEVELS=1` (even levels only; the interface then downsamples
the level above's tiles at odd zooms, other levels are rendered on the fly
with an in-memory cache, `LIDAR_TILE_MEMORY_CACHE_MB`).

A map fed by an upstream tile-source server (`LIDAR_SOURCE_URL`, the Pi's
case) pulls in and **keeps locally** the sources for each tile as soon as it
appears (thumbnails and 500 m quadrants; the full tile, a duplicate of its
quadrants, is not pulled in), without waiting for a visit. All levels remain
available while the rendering container is off; a source missed while it was
off is picked up on the next scan.

## Tile sheet and compass rose

Clicking the map selects the LiDAR HD tile under the cursor (dashed yellow
frame) and fills the **Tile** tab — name, Lambert 93 footprint, then,
if the tile has been rendered, resolution, generation date and vertical pass
alignment (beams, offsets, line correction) — whether it has been generated
or not, without changing the displayed tab. IGN information arrives
separately (`GET /api/map/ign?col&row`, STAC catalog cached in
`output/ign_meta/`): LiDAR scan date and time, sensors, mission, operator,
edit date, classification process, point count and a download link for the
`.copc.laz` point cloud on the geoplatform. A slow or unreachable catalog
never blocks selection. Clicking the same tile again (or Escape) deselects
it.

When the oriented relief layer is displayed, a compass rose shows the color
of each slope orientation (same CIELAB formula as the render); lightness
carries the local relief (light = bump, dark = hollow).

## Display: relief and precision

The map serves only the layers in `PANEL_VIZ` (`index.py`): the **oriented
relief** (`relief_oriente`, the main display layer, which merges local
openness and slope orientation) and the **precision** layer
(`densite_sol`: density of the ground points retained for the DTM). Other
visualizations present on disk are neither listed nor served as tiles.

There is no more layer stack. The panel offers three modes, one click each,
or the **P** key to cycle to the next:

- **Relief** — the oriented relief alone;
- **Precision** — density alone, in 16 shades of gray (fixed log
  scale: level k starting at 0.25 × 2^(k/2) pts/m², black ≤ 0.35 or no
  points, white ≥ 45); reads as a geometric-reliability map;
- **Compare** — a slider (mouse or touch) separates two layers
  chosen from two menus (relief, precision, or bare OSM background) on
  either side of the cursor; the same layer on both sides adds no split.

The View tab also carries a **relief intensity** slider
(0.5×–2×, 1× by default): a comfort contrast applied to the displayed
layer's container (CSS `contrast()`), remembered and shared in the link
(`&I=`, written only if ≠ 1×) but never fixed as a server default nor
applied to the exported PDF (which keeps the standard render). A
collapsible **How to read the map** block reuses,
for the displayed layer(s), the reading text from `VIZ_LEGENDS` (also served
by `/api/map/meta` and the TileJSON).

The precision legend (16 levels, tooltip in pts/m² on each level) is shown
as soon as precision is visible, either alone or on one side of the Compare
slider. The slider clips only the LiDAR layers (a CSS `clip-path` on each
layer's container), never the basemap.

The share link carries the main layer, the mode, the comparison and the
intensity:
`#z/lat/lng&M=relief_oriente&P=compare&C=relief:precision:30&I=1.4&B=1:85:1`
(`&C=left:right:position%` only in Compare mode). Old links from the layer
stack (`&L=…`) and the old "both" mode (`&P=both:opacity`, opacity then
ignored) open without error, in relief mode.

### Freezing the configuration

The **★ Set as default** button saves the current
display — main layer, mode, basemap — to `output/.map-defaults.json` (never
the intensity, a per-browser comfort setting). Any browser with no local
setting then starts from this configuration; **↺ Reset**
forgets the local state and reverts to it.

```bash
curl http://localhost:8975/api/map/defaults              # served configuration
curl -X DELETE http://localhost:8975/api/map/defaults    # revert to the registry
```

With no saved file, defaults come from the pipeline registry (`DEFAULT_VIZ`,
`PRECISION_VIZ`, `DEFAULT_VIEW_MODE` in `index.py`). Received values are
filtered: an unknown layer (or precision itself) is rejected as the main
layer, the mode is validated, basemap opacity is clamped to 0–1. A file from
the old stack (`order`/`on`/`blend`) is ignored, except for the basemap.

## PDF export (field print sheet)

The **PDF** (PDF export) tab shows the settings (A4/A3 format, landscape/portrait,
scale 1:1,000 to 1:10,000, optional title) and, while it stays open, a
**dashed yellow frame** showing the area that will be printed (it disappears
when switching tabs). The frame is **anchored to the terrain**: it is
placed at the center of the view when opened (or keeps its previous
position if it's still visible), the map zooms to show it in full above the
panel, and one can then navigate freely without it moving. It is moved by
dragging its **✥ handle** (mouse or touch); on release, the exact Lambert 93
geometry is recomputed (`GET /api/export/frame`) — a release immediately
followed by a click does not select a tile (300 ms guard). **⌖ Centre here**
brings it back to the center of the view, **⤢ Show frame** zooms onto it; changing format, orientation or scale
recenters the view. Settings and frame position are kept in the browser's
`localStorage`. **Export PDF** downloads the sheet (`GET
/api/export/pdf`), named `relief_{x_km}_{y_km}_1-{scale}.pdf` (Lambert 93
center in km, to three decimal places).

The sheet (module `lidar_pipeline/export_pdf.py`) is composed directly in
Lambert 93 from the already-rendered sources (no pass through the XYZ
tiles), then drawn vectorially (text, grid, legend) with `reportlab` —
Pillow + pyproj + reportlab only, **no numpy**: it runs equally well on the
full image and on the Pi's lightweight image alone. Contents:

- the **oriented relief** map, cropped to the requested scale (300 dpi at
  A4, 250 dpi at A3 — bounds Pi memory usage, ~36 MB at A3); outside the
  available tiles' coverage, the area stays white and hatched;
- a **Lambert 93 grid** (100 m spacing at scales 1:1,000/1:2,000, 500 m at
  1:5,000, 1,000 m at 1:10,000) graduated in the margin, and the printed
  area's **WGS84 corners** at all four angles;
- a **geographic north arrow** accounting for meridian convergence (the map
  is oriented to the L93 grid north, not geographic north — the difference
  is shown in degrees);
- a **graphic scale bar** (alternating bar) and the **numeric scale**;
- an **8-point orientation rose** (N/NE/E/SE/S/SW/W/NW), same CIELAB formula
  as the tile sheet's rose;
- the oriented relief's legend text, reused from `VIZ_LEGENDS` (single
  source shared with the interface and the TileJSON);
- a **quality panel**: a thumbnail of ground-point density (50 m cells,
  color classes), key figures (average density, weakest cell, share of
  interpolated area, acquisition period), areas **hatched in white** where
  quality data is not available, and a "**Missing data (no relief)**" line listing the tiles in the area that
  haven't been generated yet;
- a title block: title (by default, the list of covered tiles), scale,
  format, dpi, L93 center, area size, export date and IGN source mention.

Only one PDF export at a time: a second call while an export is running gets
`429` (retry). Export is **never delegated** to `LIDAR_GENERATION_URL`:
unlike tile generation, the lightweight map alone (a Pi with no worker) can
export by itself, from the sources already pulled to disk.

> **License** — LiDAR HD is distributed under the **Licence Ouverte 2.0**
> ("Open License 2.0"): IGN attribution is mandatory and must remain visible
> to the end user. Before using these renders as an OpenStreetMap **survey**
> layer, check the position of the community (OSM-FR) on the source in
> question.

## How a tile is built

1. The tile's footprint is converted to Lambert 93 (5×5 sampling of the
   outline: edges aren't straight in L93).
2. The 1 km tiles it intersects are found by their name
   (`LHD_FXX_{col}_{row}` → X ∈ [col, col+1] km, Y ∈ [row−1, row] km).
3. For each one, the coarsest **source tier** that is still sufficient is
   chosen among those the pipeline already produces: `index_thumbs`
   thumbnail (≈3.9 m/px), intermediate `_mid` thumbnail (1.56 m/px),
   `index_subtiles` quadrants (2500 px, 4× less to decode than the full
   tile), then the full tile. The three folders are scanned
   **independently**: a partial cache — a lightweight machine that only
   pulls in quadrants and thumbnails, never full tiles — remains fully
   usable.
4. The useful window is cropped, then reprojected via a **perspective
   transform** (`Image.transform(..., PERSPECTIVE)`), tile by tile: the
   measured alignment is sub-pixel.
5. The result is encoded (PNG or WebP) and written to
   `output/index_xyz/{layer}/{z}/{x}/{y}[@2x].{ext}`.

No GDAL/PDAL dependency: **Pillow + pyproj** only.

### Cache and expiry

- A cached tile is re-served as long as **no contributing tile is more
  recent than it**: regenerating a tile only invalidates its own tiles.
- A tile with no data is memoized with an `.empty` marker — never
  recomputed.
- Decoded source images are kept in a small LRU cache: AVIF decoding
  dominates the cost, and neighboring tiles reuse it.
- Client-side, the interface appends `?v=<stamp>` (most recent mtime) and
  then gets an immutable cache; OSM clients use the bare URL, served with
  short revalidation.

### Measurements (real 0.2 m tiles, 3×3 km block, GPU-less container)

| | first render | from cache |
|---|---|---|
| z10–z14 | 50–200 ms | 3–9 ms |
| z16–z18 (native resolution) | 46–130 ms | 3–9 ms |

Weight of a slope tile at z17, measured on the same tile:

| encoding | size | fidelity |
|---|---|---|
| PNG RGBA (default) | 210 KB | lossless |
| Palettized PNG (`LIDAR_TILE_PNG_PALETTE=1`) | 55 KB | average deviation 5.8 levels |
| WebP q78 (`…/{y}.webp`) | 38 KB | lossy, visually clean |

The canonical PNG stays **lossless** by default: these renders are meant for
interpretation, not illustration. For use as an imagery basemap where
bandwidth matters, two levers: request the `.webp` URL (supported by QGIS,
iD, MapLibre, uMap) or enable `LIDAR_TILE_PNG_PALETTE=1`.

Alignment checks performed on real data: an OSM path overlays exactly onto
the trace visible in the slope layer, and the seam gap between neighboring
tiles stays **below the terrain's natural noise** (measured gap 35–53
levels against a median of 49–53 between two random neighboring columns
within the same tile).

## Load and small machines

The service is bounded at every stage, nothing is unlimited:

| Stage | Default | Setting |
|---|---|---|
| Concurrent renders | 2 | `LIDAR_TILE_WORKERS` |
| Concurrent tile downloads | 2 | `LIDAR_TILE_FETCH_WORKERS` |
| Same tile requested in parallel | 1 render, others wait for its result | — |
| Decoded source memory | 192 MB | `LIDAR_TILE_SOURCE_CACHE_MB` |

Excess requests wait on the semaphore **before** any decoding: they consume
neither CPU nor memory. A browser over HTTP/1.1 only opens 6 connections per
origin anyway, across all layers combined; behind an HTTP/2 proxy this cap
disappears and only these settings hold the load.

Pre-warming (`/api/map/warm`) is sequential: it cannot saturate the machine,
only take time.

On a 2 GB Raspberry Pi, `LIDAR_TILE_WORKERS=1` and
`LIDAR_TILE_SOURCE_CACHE_MB=64` remain comfortable.

## Cache-only + background maintenance (small Raspberry Pi)

On a machine that must never compute while someone is browsing (a Pi that
also hosts other services), two variables invert the load:

```yaml
environment:
  - LIDAR_TILE_CACHE_ONLY=1     # browsing no longer renders anything
  - LIDAR_TILE_BACKGROUND=1     # a background task maintains the pyramid
```

- **`LIDAR_TILE_CACHE_ONLY=1`** — a missing or stale tile is served
  transparent with `X-Tile-Pending: 1` and `Cache-Control: no-store` (the
  browser re-requests it: as soon as maintenance has rendered it, it
  appears). No more local rendering on the request path; with
  `LIDAR_MAPS_URL`, a missing tile (not yet handled by maintenance) is
  pulled in from there — a download of a few dozen KB, cached — then
  served: browsing covers all zoom levels without ever computing on the
  small machine.
- **`LIDAR_TILE_BACKGROUND=1`** — a poller rescans tiles at a regular
  interval (`LIDAR_TILE_BACKGROUND_INTERVAL`, 120 s); each new or
  regenerated tile (the worker just produced it, or the on-demand cache just
  pulled it in) queues its pyramid. Low-priority renderers (`os.nice`) drain
  it at a rate of one tile per `LIDAR_TILE_BACKGROUND_PAUSE` second (1 s).
  First start-up: ALL tiles are unknown, the whole pyramid is rebuilt —
  already-fresh tiles only cost a `stat`. Levels are queued from the
  smallest zoom to the largest: the map fills in coarsely first.

| Variable | Default | Role |
|---|---|---|
| `LIDAR_TILE_BACKGROUND_MAX_Z` | native (`18` at @2x, `19` at 256 px) | maximum level maintained in the background, URL numbering |
| `LIDAR_TILE_BACKGROUND_SCALE` | `2` | tiles maintained: 2 = 512 px (the interface's) |
| `LIDAR_TILE_BACKGROUND_FMT` | `avif` | format of maintained tiles (the interface's) |
| `LIDAR_TILE_BACKGROUND_PAUSE` | `1.0` | pause (s) between two renders — the discretion lever |
| `LIDAR_TILE_BACKGROUND_INTERVAL` | `120` | seconds between two tile scans |
| `LIDAR_TILE_BACKGROUND_QUEUE_MAX` | `65536` | bounded queue — each tile remembers the first pyramid tile it was refused for lack of room and resumes from there on the next scans, until its whole pyramid has gone through |

Control: `GET /api/map/background` (state, counters, queue), `POST
/api/map/background` (immediate scan), `POST /api/map/warm` (manual
exhaustive pre-computation, all zooms/formats, see below). Finally, capping
the container itself (`mem_limit` + `memswap_limit` in a compose override)
guarantees a misbehaving render can no longer bring down the machine: the
OOM killer would only ever hit `lidar-maps`.

## Pre-warming

```bash
curl -X POST http://localhost:8975/api/map/warm \
  -H 'Content-Type: application/json' \
  -d '{"layers": ["slope"], "z_min": 10, "z_max": 16}'
curl http://localhost:8975/api/map/warm     # progress / report
```

Without `bounds`, the coverage of available tiles is used. Useful after a
large run so the first visit is instant.

## Two machines

Two complementary upstreams, depending on what the local machine has.

`LIDAR_SOURCE_URL` designates **the full pipeline server** (port 8973): the
tile inventory comes from its `/api/tiles`, and each source image is pulled
in on the first render that needs it. The map container then starts with no
local data at all and serves the entire remote catalog:

```bash
docker run -d --name lidar-maps --network host \
  -e LIDAR_SOURCE_URL=http://127.0.0.1:8973 \
  -e LIDAR_OUTPUT_DIR=/data/output -e LIDAR_PORT=8975 \
  -v "$PWD/output-maps:/data/output" --user "$(id -u):$(id -g)" lidar-maps
```

Measured on 849 real tiles served over an SSH tunnel: first tile in an area
0.9–3.2 s (including downloading a 1.8 MB quadrant), then **~1 ms** from the
local cache, which only keeps what has been consulted.

`LIDAR_MAPS_URL` designates an upstream tile server (the processing
machine): a tile missing locally is pulled in from there (a few dozen KB)
then cached — instead of pulling in whole tiles. A circuit breaker suspends
attempts for 2 minutes after 3 consecutive failures: the map stays usable
even with the worker off.

## Environment variables

| Variable | Default | Role |
|---|---|---|
| `LIDAR_OUTPUT_DIR` | `/data/output` | tiles read from, `index_xyz/` cache written to |
| `LIDAR_PORT` | `8975` | listening port |
| `LIDAR_SOURCE_URL` | — | full pipeline server: inventory + tiles pulled in on demand |
| `LIDAR_SOURCE_TOKEN` | — | token if the upstream server requires `LIDAR_API_TOKEN` |
| `LIDAR_MAPS_URL` | — | upstream tile server (remote map) |
| `LIDAR_ATTRIBUTION` | LiDAR HD © IGN | attribution served (TileJSON, WMTS, JOSM, map) |
| `LIDAR_TILE_PNG_PALETTE` | — | `1`: palettized PNG (~4× lighter, ~6-level deviation) |
| `LIDAR_TILE_SOURCE_CACHE_MB` | `192` | memory budget for the decoded source-image cache |
| `LIDAR_TILE_WORKERS` | `2` | concurrent tile renders |
| `LIDAR_TILE_FETCH_WORKERS` | `2` | concurrent tile downloads (`LIDAR_SOURCE_URL` mode) |
| `LIDAR_SSL_CERTFILE` / `LIDAR_SSL_KEYFILE` | — | direct HTTPS (GPS on a phone) |

## Troubleshooting

- **Empty map, `layers: 0`** (`curl /healthz`): no render in
  `output/visualisations/`, or the folder is mounted at the wrong path.
- **Tiles transparent everywhere**: zoom out of range (5–19) or an area with
  no tiles; the `X-Tile-Empty` header confirms this.
- **First visit is slow**: normal, each tile is rendered once — pre-warm
  (see above).
- **JOSM rejects the URL**: use the `{zoom}/{x}/{y}` template (JOSM), not
  `{z}/{x}/{y}` (Leaflet/QGIS).
- **Layer missing from the menu**: it doesn't exist on disk for these tiles —
  `/api/map/meta` lists what is actually available.

## References

- `lidar_pipeline/tiles.py` — grid, reprojection, cache, pre-warming.
- `lidar_pipeline/mapserve.py` — tile/TileJSON/WMTS/JOSM routes and the map API.
- `lidar_pipeline/web/map.{html,css,js}` — interface (tabbed panel, Leaflet `L.tileLayer`, native LOD), re-read by `lidar_pipeline/mapui.py`.
- `Dockerfile.maps`, `docker-compose.maps.yml`, `./run.sh --serve-maps`.