Skip to content

Commit

Permalink
Update documentation and spec (#258)
Browse files Browse the repository at this point in the history
  • Loading branch information
mactrem authored Jul 23, 2024
1 parent 6159818 commit f5048c1
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 8 deletions.
20 changes: 15 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,23 @@
# MapLibre Tiles (MLT)

The MLT format is mainly inspired by the [MVT format](https://github.com/mapbox/vector-tile-spec) specification, but has been redesigned from the ground up to improve the following areas:

- **Improved compression ratio** - up to 6x on large tiles, based on a column oriented layout with (custom) lightweight encodings
- **Better decoding performance** - fast lightweight encodings which can be used in combination with SIMD/vectorization instructions
The MapLibre Tile specification is mainly inspired by the [Mapbox Vector Tile (MVT)](https://github.com/mapbox/vector-tile-spec) specification,
but has been redesigned from the ground up to address the challenges of rapidly growing geospatial data volumes
and complex next-generation geospatial source formats as well as to leverage the capabilities of modern hardware and APIs.
MLT is specifically designed for modern and next generation graphics APIs to enable high-performance processing and rendering of
large (planet-scale) 2D and 2.5 basemaps. In particular, MLT offers the following features:
- **Improved compression ratio**: up to 6x on large tiles, based on a column oriented layout with recursively applied (custom)
lightweight encodings. This leads to reduced latency, storage, and egress costs and, in particular, improved cache utilization
- **Better decoding performance**: fast lightweight encodings which can be used in combination with SIMD/vectorization instructions
- **Support for linear referencing and m-values** to efficiently support the upcoming next generation source formats such as Overture Maps (GeoParquet)
- **Support 3D coordinates**, i.e. elevation
- **Support complex types**, including nested properties, lists and maps
- **Improved processing performance**: Based on an in-memory format that can be processed efficiently on the CPU and GPU and loaded directly into GPU buffers partially (like polygons in WebGL) or completely (in case of WebGPU compute shader usage) without additional processing
- **Improved processing performance**, based on storage and in-memory formats that are specifically designed for modern GL APIs,
allowing for efficient processing on both CPU and GPU. The formats are designed to be loaded into
GPU buffers with little or no additional processing

📝 For a more in-depth exploration of MLT have a look at the [following slides](https://github.com/mactrem/presentations/blob/main/FOSS4G_2024_Europe/FOSS4G_2024_Europe.pdf) or watch
[this talk](https://foss4ge.television.ee/stream/2024-07-03-LASTOOLS).


> [!CAUTION]
> This format is still in the active research and development phase and is not ready for production use.
Expand Down
11 changes: 8 additions & 3 deletions spec/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,10 +330,11 @@ and the DuckDB execution format and is tailored for the visualization use case.
usage within a GPU compute shader.
The main design goals of the MLT in-memory format are:

- Define a platform-agnostic representation to avoid the expensive materialization costs in particular for strings
- Enable advanced CPU throughput by optimizing the memory layout for cache locality and SIMD instructions
- Allow random (mainly constant time) access to all data, so it can also be parallel processed on the GPU (WebGPU compute shader)
- Allow random (mainly constant time) access to all data, so it can also be parallel processed on the GPU (compute shader)
- Provide compressed data structure which can be directly processed without decoding
- Parts of the geometries of a tile should be already in a renderable form and can be therefore directly loaded into GPU buffers with no additional processing
- Provide the geometries of a tile in a representation that can be (directly) loaded into GPU buffers with little or no additional processing

The data are stored in continuous memory buffers, referred to as vectors, with additional metadata and an optional nullability bitmap
for representing null values. The storage format contains a `VectorType` field in the metadata that tells the decoder
Expand All @@ -353,9 +354,13 @@ The MLT in-memory format supports the following vectors:
> **_Note:_** Further evaluation is needed to determine if the [latest research findings](https://arxiv.org/pdf/2306.15374.pdf)
> can be used to enable random access on delta encoded values as well
In case a compressed vector can be used, this has the additional advantage that the conversion from the storage
In case a compressed vector can be used, this has the additional advantage that the conversion from the storage
into in-memory format is basically a zero-copy operation.

Following Apache Arrow's approach based on the [Intel performance guide](https://www.intel.com/content/www/us/en/developer/topic-technology/data-center/overview.html),
we recommend that decoders should allocate memory on aligned addresses with a multiple of 64-bytes (if possible).


### Technical Terminology

in the following a small glossary to help disambiguate
Expand Down

0 comments on commit f5048c1

Please # to comment.