lidar_rendu/docs/MAPS.md

XYZ tile map (lidar-maps image)

A classic "slippy" map — the same tile scheme as Google Maps and OpenStreetMap — served by a dedicated lightweight image. Pipeline renders thus become a reusable imagery basemap in JOSM, iD, QGIS, uMap, MapLibre or OsmAnd, on top of the browsing interface it also provides.

This is the ONLY web interface in the project (the old historical webapp has been removed): it also embeds tile generation — local on the full image (the full pipeline server, port 8973, ./run.sh --serve), delegated via LIDAR_GENERATION_URL on the lightweight machine (see docs/DEPLOY_WEBAPP.md).

docker compose -f docker-compose.maps.yml up -d --build   # http://localhost:8975/
docker compose -f docker-compose.maps.yml logs -f maps
./run.sh --serve-maps            # equivalent, foreground container

Interface

A single tabbed panel (web/map.{html,css,js}) groups all settings, replacing the old stack of stacked blocks: View (main layer, relief/precision mode, basemap), Tile (the selected tile's card), PDF (PDF export), Generate (hidden if the generator is unavailable or not authorized for that browser) and Share. On desktop, the panel occupies a fixed column, collapsible into an icon strip (‹, always usable: clicking a tab redeploys the panel). Below 720 px wide, it becomes a bottom sheet with three heights — closed, half, full — changed by dragging the handle, by simply tapping it (one notch), or by tapping the already-active tab again.

Clicking the map selects the LiDAR HD tile under the cursor (dashed outline) and fills the Tile tab (footprint, IGN info, pass alignment) without changing the displayed tab; clicking the same tile again deselects it, clicking another moves the selection. Keyboard shortcuts: 1–5 (panel tabs, no effect if the tab is hidden), P (next display mode), Escape (deselects the tile, then collapses the panel). Two light/dark themes (◐/☀/☾ button, auto by default = follows the system); settings and panel state are kept in localStorage (lidarMapView_v2, lidar-print, lidar-panel, lidar-theme — silently ignored in private browsing).

Generating tiles from the map

• ＋ Area (Generate tab) — draw a rectangle: the 1 km LHD tiles it intersects are downloaded from the IGN geoplatform and then processed (0.2 m, options below); successive zones and clicks are additive;
• ⤒ Complete — all tiles already present in input/ that are missing at least one of the requested layers;
• ↻ Generate/Regenerate this tile (button on the tile card, Tile tab) — a specific tile, even with no existing data.

Run options: target layers (default: the panel layers) and forced regeneration. Ground classification (IGN, ground only) and edge stitching (a 100 m band taken from neighboring tiles) are enforced: no setting available. A request submitted while a run is already in progress goes into a persisted queue (never interrupting work in progress); progress is shown tile by tile (orange frames = rendering in progress, blue = waiting, red = failed) with a log and a Stop button (SIGTERM then SIGKILL). During a run, each completed tile appears on the map as it finishes: the inventory is updated after each tile and the pyramid maintenance job prioritizes its tiles (polled every 10 s).

The buttons are hidden if the browser's IP falls outside LIDAR_REGEN_CIDR (default: localhost + private RFC1918 ranges) or if no generation backend exists (lightweight image without LIDAR_GENERATION_URL). On the lightweight machine, everything is forwarded to the worker (LIDAR_GENERATION_URL), which runs the full pipeline; its tiles are then pulled back on demand via the /api/tiles inventory plus versioned static assets (LIDAR_SOURCE_URL).

Tiling contract

Point	Value
Projection	EPSG:3857, OSM XYZ scheme (north-west origin, y toward the south)
Canonical URL	/tiles/{layer}/{z}/{x}/{y}.png — 256 px, PNG RGBA
High-density variant	/tiles/{layer}/{z}/{x}/{y}@2x.avif — 512 px AVIF (internal interface; png/webp also accepted)
Zoom levels	5 → 19 native (0.2 m/px ≈ z19 in France); beyond that, client-side over-zoom
Outside coverage	fully transparent tile (overlayable), header X-Tile-Empty: 1
Pending (cache-only mode)	transparent tile, headers X-Tile-Empty: 1 + X-Tile-Pending: 1, never cached by the browser
CORS	Access-Control-Allow-Origin: * on /tiles/*
Attribution	LIDAR_ATTRIBUTION, default "LiDAR HD © IGN — Licence Ouverte 2.0"

Discovery: /tiles/{layer}.json (TileJSON 3.0.0), /tiles/wmts.xml (WMTS 1.0.0, GoogleMapsCompatible grid), /tiles/josm.imagery.xml (all layers at once in JOSM).

Using the tiles elsewhere

• JOSM — Imagery → Imagery preferences → + TMS, paste http://<host>:8975/tiles/slope/{zoom}/{x}/{y}.png. To add everything at once: Imagery preferences → Offline/Custom → Add imagery source XML with http://<host>:8975/tiles/josm.imagery.xml.
• iD — Background → Custom, paste http://<host>:8975/tiles/slope/{z}/{x}/{y}.png.
• QGIS — XYZ Tiles → New Connection (same URL, max zoom 19), or WMS/WMTS → New with http://<host>:8975/tiles/wmts.xml.
• uMap / MapLibre / Leaflet — same XYZ template, or the TileJSON.
• OsmAnd — online tile source, XYZ template, max zoom 19.

The XYZ imagery background section of the map's Share tab shows these URLs for the selected layer, with a Copy button for the XYZ template.

Full pyramid pre-generated

By default, all levels up to native (19 in standard OSM numbering = 0.2 m/px, served as 18@2x by the interface) are written to disk in AVIF and pre-generated by the background maintenance job (on a map fed by an upstream source: downloaded from LIDAR_MAPS_URL). No level is rendered on the fly: on-demand rendering on a Raspberry Pi (~170 ms per tile, 3 at a time) took 1.5 to 6 s per screen at zoom levels 17–19. The interface is capped at zoom 19 (1 screen pixel = 1 LiDAR pixel): never an upscaled tile. Measured order of magnitude: ~110,000 @2x tiles for 3,240 tiles, of which 81,000 at native level, ~5 GB.

Reduced storage (measured on disk): a lower LIDAR_TILE_CACHE_MAX_Z and/or LIDAR_TILE_EVEN_LEVELS=1 (even levels only; the interface then downsamples the level above's tiles at odd zooms, other levels are rendered on the fly with an in-memory cache, LIDAR_TILE_MEMORY_CACHE_MB).

A map fed by an upstream tile-source server (LIDAR_SOURCE_URL, the Pi's case) pulls in and keeps locally the sources for each tile as soon as it appears (thumbnails and 500 m quadrants; the full tile, a duplicate of its quadrants, is not pulled in), without waiting for a visit. All levels remain available while the rendering container is off; a source missed while it was off is picked up on the next scan.

Tile sheet and compass rose

Clicking the map selects the LiDAR HD tile under the cursor (dashed yellow frame) and fills the Tile tab — name, Lambert 93 footprint, then, if the tile has been rendered, resolution, generation date and vertical pass alignment (beams, offsets, line correction) — whether it has been generated or not, without changing the displayed tab. IGN information arrives separately (GET /api/map/ign?col&row, STAC catalog cached in output/ign_meta/): LiDAR scan date and time, sensors, mission, operator, edit date, classification process, point count and a download link for the .copc.laz point cloud on the geoplatform. A slow or unreachable catalog never blocks selection. Clicking the same tile again (or Escape) deselects it.

When the oriented relief layer is displayed, a compass rose shows the color of each slope orientation (same CIELAB formula as the render); lightness carries the local relief (light = bump, dark = hollow).

Display: relief and precision

The map serves only the layers in PANEL_VIZ (index.py): the oriented relief (relief_oriente, the main display layer, which merges local openness and slope orientation) and the precision layer (densite_sol: density of the ground points retained for the DTM). Other visualizations present on disk are neither listed nor served as tiles.

There is no more layer stack. The panel offers three modes, one click each, or the P key to cycle to the next:

• Relief — the oriented relief alone;
• Precision — density alone, in 16 shades of gray (fixed log scale: level k starting at 0.25 × 2^(k/2) pts/m², black ≤ 0.35 or no points, white ≥ 45); reads as a geometric-reliability map;
• Compare — a slider (mouse or touch) separates two layers chosen from two menus (relief, precision, or bare OSM background) on either side of the cursor; the same layer on both sides adds no split.

The View tab also carries a relief intensity slider (0.5×–2×, 1× by default): a comfort contrast applied to the displayed layer's container (CSS contrast()), remembered and shared in the link (&I=, written only if ≠ 1×) but never fixed as a server default nor applied to the exported PDF (which keeps the standard render). A collapsible How to read the map block reuses, for the displayed layer(s), the reading text from VIZ_LEGENDS (also served by /api/map/meta and the TileJSON).

The precision legend (16 levels, tooltip in pts/m² on each level) is shown as soon as precision is visible, either alone or on one side of the Compare slider. The slider clips only the LiDAR layers (a CSS clip-path on each layer's container), never the basemap.

The share link carries the main layer, the mode, the comparison and the intensity: #z/lat/lng&M=relief_oriente&P=compare&C=relief:precision:30&I=1.4&B=1:85:1 (&C=left:right:position% only in Compare mode). Old links from the layer stack (&L=…) and the old "both" mode (&P=both:opacity, opacity then ignored) open without error, in relief mode.

Freezing the configuration

The ★ Set as default button saves the current display — main layer, mode, basemap — to output/.map-defaults.json (never the intensity, a per-browser comfort setting). Any browser with no local setting then starts from this configuration; ↺ Reset forgets the local state and reverts to it.

curl http://localhost:8975/api/map/defaults              # served configuration
curl -X DELETE http://localhost:8975/api/map/defaults    # revert to the registry

With no saved file, defaults come from the pipeline registry (DEFAULT_VIZ, PRECISION_VIZ, DEFAULT_VIEW_MODE in index.py). Received values are filtered: an unknown layer (or precision itself) is rejected as the main layer, the mode is validated, basemap opacity is clamped to 0–1. A file from the old stack (order/on/blend) is ignored, except for the basemap.

PDF export (field print sheet)

The PDF (PDF export) tab shows the settings (A4/A3 format, landscape/portrait, scale 1:1,000 to 1:10,000, optional title) and, while it stays open, a dashed yellow frame showing the area that will be printed (it disappears when switching tabs). The frame is anchored to the terrain: it is placed at the center of the view when opened (or keeps its previous position if it's still visible), the map zooms to show it in full above the panel, and one can then navigate freely without it moving. It is moved by dragging its ✥ handle (mouse or touch); on release, the exact Lambert 93 geometry is recomputed (GET /api/export/frame) — a release immediately followed by a click does not select a tile (300 ms guard). ⌖ Centre here brings it back to the center of the view, ⤢ Show frame zooms onto it; changing format, orientation or scale recenters the view. Settings and frame position are kept in the browser's localStorage. Export PDF downloads the sheet (GET /api/export/pdf), named relief_{x_km}_{y_km}_1-{scale}.pdf (Lambert 93 center in km, to three decimal places).

The sheet (module lidar_pipeline/export_pdf.py) is composed directly in Lambert 93 from the already-rendered sources (no pass through the XYZ tiles), then drawn vectorially (text, grid, legend) with reportlab — Pillow + pyproj + reportlab only, no numpy: it runs equally well on the full image and on the Pi's lightweight image alone. Contents:

• the oriented relief map, cropped to the requested scale (300 dpi at A4, 250 dpi at A3 — bounds Pi memory usage, ~36 MB at A3); outside the available tiles' coverage, the area stays white and hatched;
• a Lambert 93 grid (100 m spacing at scales 1:1,000/1:2,000, 500 m at 1:5,000, 1,000 m at 1:10,000) graduated in the margin, and the printed area's WGS84 corners at all four angles;
• a geographic north arrow accounting for meridian convergence (the map is oriented to the L93 grid north, not geographic north — the difference is shown in degrees);
• a graphic scale bar (alternating bar) and the numeric scale;
• an 8-point orientation rose (N/NE/E/SE/S/SW/W/NW), same CIELAB formula as the tile sheet's rose;
• the oriented relief's legend text, reused from VIZ_LEGENDS (single source shared with the interface and the TileJSON);
• a quality panel: a thumbnail of ground-point density (50 m cells, color classes), key figures (average density, weakest cell, share of interpolated area, acquisition period), areas hatched in white where quality data is not available, and a "Missing data (no relief)" line listing the tiles in the area that haven't been generated yet;
• a title block: title (by default, the list of covered tiles), scale, format, dpi, L93 center, area size, export date and IGN source mention.

Only one PDF export at a time: a second call while an export is running gets 429 (retry). Export is never delegated to LIDAR_GENERATION_URL: unlike tile generation, the lightweight map alone (a Pi with no worker) can export by itself, from the sources already pulled to disk.

License — LiDAR HD is distributed under the Licence Ouverte 2.0 ("Open License 2.0"): IGN attribution is mandatory and must remain visible to the end user. Before using these renders as an OpenStreetMap survey layer, check the position of the community (OSM-FR) on the source in question.

How a tile is built

1. The tile's footprint is converted to Lambert 93 (5×5 sampling of the outline: edges aren't straight in L93).
2. The 1 km tiles it intersects are found by their name (LHD_FXX_{col}_{row} → X ∈ [col, col+1] km, Y ∈ [row−1, row] km).
3. For each one, the coarsest source tier that is still sufficient is chosen among those the pipeline already produces: index_thumbs thumbnail (≈3.9 m/px), intermediate _mid thumbnail (1.56 m/px), index_subtiles quadrants (2500 px, 4× less to decode than the full tile), then the full tile. The three folders are scanned independently: a partial cache — a lightweight machine that only pulls in quadrants and thumbnails, never full tiles — remains fully usable.
4. The useful window is cropped, then reprojected via a perspective transform (Image.transform(..., PERSPECTIVE)), tile by tile: the measured alignment is sub-pixel.
5. The result is encoded (PNG or WebP) and written to output/index_xyz/{layer}/{z}/{x}/{y}[@2x].{ext}.

No GDAL/PDAL dependency: Pillow + pyproj only.

Cache and expiry

• A cached tile is re-served as long as no contributing tile is more recent than it: regenerating a tile only invalidates its own tiles.
• A tile with no data is memoized with an .empty marker — never recomputed.
• Decoded source images are kept in a small LRU cache: AVIF decoding dominates the cost, and neighboring tiles reuse it.
• Client-side, the interface appends ?v=<stamp> (most recent mtime) and then gets an immutable cache; OSM clients use the bare URL, served with short revalidation.

Measurements (real 0.2 m tiles, 3×3 km block, GPU-less container)

	first render	from cache
z10–z14	50–200 ms	3–9 ms
z16–z18 (native resolution)	46–130 ms	3–9 ms

Weight of a slope tile at z17, measured on the same tile:

encoding	size	fidelity
PNG RGBA (default)	210 KB	lossless
Palettized PNG (LIDAR_TILE_PNG_PALETTE=1)	55 KB	average deviation 5.8 levels
WebP q78 (…/{y}.webp)	38 KB	lossy, visually clean

The canonical PNG stays lossless by default: these renders are meant for interpretation, not illustration. For use as an imagery basemap where bandwidth matters, two levers: request the .webp URL (supported by QGIS, iD, MapLibre, uMap) or enable LIDAR_TILE_PNG_PALETTE=1.

Alignment checks performed on real data: an OSM path overlays exactly onto the trace visible in the slope layer, and the seam gap between neighboring tiles stays below the terrain's natural noise (measured gap 35–53 levels against a median of 49–53 between two random neighboring columns within the same tile).

Load and small machines

The service is bounded at every stage, nothing is unlimited:

Stage	Default	Setting
Concurrent renders	2	LIDAR_TILE_WORKERS
Concurrent tile downloads	2	LIDAR_TILE_FETCH_WORKERS
Same tile requested in parallel	1 render, others wait for its result	—
Decoded source memory	192 MB	LIDAR_TILE_SOURCE_CACHE_MB

Excess requests wait on the semaphore before any decoding: they consume neither CPU nor memory. A browser over HTTP/1.1 only opens 6 connections per origin anyway, across all layers combined; behind an HTTP/2 proxy this cap disappears and only these settings hold the load.

Pre-warming (/api/map/warm) is sequential: it cannot saturate the machine, only take time.

On a 2 GB Raspberry Pi, LIDAR_TILE_WORKERS=1 and LIDAR_TILE_SOURCE_CACHE_MB=64 remain comfortable.

Cache-only + background maintenance (small Raspberry Pi)

On a machine that must never compute while someone is browsing (a Pi that also hosts other services), two variables invert the load:

environment:
  - LIDAR_TILE_CACHE_ONLY=1     # browsing no longer renders anything
  - LIDAR_TILE_BACKGROUND=1     # a background task maintains the pyramid

• LIDAR_TILE_CACHE_ONLY=1 — a missing or stale tile is served transparent with X-Tile-Pending: 1 and Cache-Control: no-store (the browser re-requests it: as soon as maintenance has rendered it, it appears). No more local rendering on the request path; with LIDAR_MAPS_URL, a missing tile (not yet handled by maintenance) is pulled in from there — a download of a few dozen KB, cached — then served: browsing covers all zoom levels without ever computing on the small machine.
• LIDAR_TILE_BACKGROUND=1 — a poller rescans tiles at a regular interval (LIDAR_TILE_BACKGROUND_INTERVAL, 120 s); each new or regenerated tile (the worker just produced it, or the on-demand cache just pulled it in) queues its pyramid. Low-priority renderers (os.nice) drain it at a rate of one tile per LIDAR_TILE_BACKGROUND_PAUSE second (1 s). First start-up: ALL tiles are unknown, the whole pyramid is rebuilt — already-fresh tiles only cost a stat. Levels are queued from the smallest zoom to the largest: the map fills in coarsely first.

Variable	Default	Role
LIDAR_TILE_BACKGROUND_MAX_Z	native (18 at @2x, 19 at 256 px)	maximum level maintained in the background, URL numbering
LIDAR_TILE_BACKGROUND_SCALE	2	tiles maintained: 2 = 512 px (the interface's)
LIDAR_TILE_BACKGROUND_FMT	avif	format of maintained tiles (the interface's)
LIDAR_TILE_BACKGROUND_PAUSE	1.0	pause (s) between two renders — the discretion lever
LIDAR_TILE_BACKGROUND_INTERVAL	120	seconds between two tile scans
LIDAR_TILE_BACKGROUND_QUEUE_MAX	65536	bounded queue — each tile remembers the first pyramid tile it was refused for lack of room and resumes from there on the next scans, until its whole pyramid has gone through

Control: GET /api/map/background (state, counters, queue), POST /api/map/background (immediate scan), POST /api/map/warm (manual exhaustive pre-computation, all zooms/formats, see below). Finally, capping the container itself (mem_limit + memswap_limit in a compose override) guarantees a misbehaving render can no longer bring down the machine: the OOM killer would only ever hit lidar-maps.

Pre-warming

curl -X POST http://localhost:8975/api/map/warm \
  -H 'Content-Type: application/json' \
  -d '{"layers": ["slope"], "z_min": 10, "z_max": 16}'
curl http://localhost:8975/api/map/warm     # progress / report

Without bounds, the coverage of available tiles is used. Useful after a large run so the first visit is instant.

Two machines

Two complementary upstreams, depending on what the local machine has.

LIDAR_SOURCE_URL designates the full pipeline server (port 8973): the tile inventory comes from its /api/tiles, and each source image is pulled in on the first render that needs it. The map container then starts with no local data at all and serves the entire remote catalog:

docker run -d --name lidar-maps --network host \
  -e LIDAR_SOURCE_URL=http://127.0.0.1:8973 \
  -e LIDAR_OUTPUT_DIR=/data/output -e LIDAR_PORT=8975 \
  -v "$PWD/output-maps:/data/output" --user "$(id -u):$(id -g)" lidar-maps

Measured on 849 real tiles served over an SSH tunnel: first tile in an area 0.9–3.2 s (including downloading a 1.8 MB quadrant), then ~1 ms from the local cache, which only keeps what has been consulted.

LIDAR_MAPS_URL designates an upstream tile server (the processing machine): a tile missing locally is pulled in from there (a few dozen KB) then cached — instead of pulling in whole tiles. A circuit breaker suspends attempts for 2 minutes after 3 consecutive failures: the map stays usable even with the worker off.

Environment variables

Variable	Default	Role
LIDAR_OUTPUT_DIR	/data/output	tiles read from, index_xyz/ cache written to
LIDAR_PORT	8975	listening port
LIDAR_SOURCE_URL	—	full pipeline server: inventory + tiles pulled in on demand
LIDAR_SOURCE_TOKEN	—	token if the upstream server requires LIDAR_API_TOKEN
LIDAR_MAPS_URL	—	upstream tile server (remote map)
LIDAR_ATTRIBUTION	LiDAR HD © IGN	attribution served (TileJSON, WMTS, JOSM, map)
LIDAR_TILE_PNG_PALETTE	—	1: palettized PNG (~4× lighter, ~6-level deviation)
LIDAR_TILE_SOURCE_CACHE_MB	192	memory budget for the decoded source-image cache
LIDAR_TILE_WORKERS	2	concurrent tile renders
LIDAR_TILE_FETCH_WORKERS	2	concurrent tile downloads (LIDAR_SOURCE_URL mode)
LIDAR_SSL_CERTFILE / LIDAR_SSL_KEYFILE	—	direct HTTPS (GPS on a phone)

Troubleshooting

• Empty map, layers: 0 (curl /healthz): no render in output/visualisations/, or the folder is mounted at the wrong path.
• Tiles transparent everywhere: zoom out of range (5–19) or an area with no tiles; the X-Tile-Empty header confirms this.
• First visit is slow: normal, each tile is rendered once — pre-warm (see above).
• JOSM rejects the URL: use the {zoom}/{x}/{y} template (JOSM), not {z}/{x}/{y} (Leaflet/QGIS).
• Layer missing from the menu: it doesn't exist on disk for these tiles — /api/map/meta lists what is actually available.

References

• lidar_pipeline/tiles.py — grid, reprojection, cache, pre-warming.
• lidar_pipeline/mapserve.py — tile/TileJSON/WMTS/JOSM routes and the map API.
• lidar_pipeline/web/map.{html,css,js} — interface (tabbed panel, Leaflet L.tileLayer, native LOD), re-read by lidar_pipeline/mapui.py.
• Dockerfile.maps, docker-compose.maps.yml, ./run.sh --serve-maps.