RFC: How should MLT support nested `List` / `Map` types (format decision)?

[DoFabien]

Context / Motivation

I’d like to contribute support for nested property types—starting with List<T> and later Map<K,V>—in the official TypeScript decoder (this repo, ts/).

In parallel, I’m developing a Rust encoder that converts GeoBuf, FlatGeoBuf, and GeoJSON into MLT. I need List support (and likely Map later) to represent real-world attributes—for example:

• A parcel with multiple transaction years: [2015, 2018, 2021]
• A feature with repeated price history: [{price: 100000, year: 2015}, {price: 150000, year: 2021}]

That’s why I’m motivated to contribute this work upstream.

Before implementing anything, I need guidance on the wire-format choice, because the current Tag 0x01 format appears intentionally “MVT-equivalent”.

What I found in the current repo

Tag 0x01 type codes (current state)

Across the existing implementations (C++ / Java / TS / Rust parser), Tag 0x01 supports:

Code range	Meaning
0–3	ID (32/64 + nullable variants)
4	GEOMETRY
10–29	scalar types (nullable via bit0)
30	STRUCT (only complex type with children)

There is no distinct LIST or MAP type code today.

Relevant references:

• C++ type map: cpp/include/mlt/metadata/type_map.hpp
• TS type map: ts/src/metadata/tileset/typeMap.ts
• Java type map: java/mlt-core/src/main/java/org/maplibre/mlt/converter/encodings/MltTypeMap.java
• Rust (v01 parser): rust/mlt-nom/src/v01/column.rs
• Proto schema shows ComplexType = { GEOMETRY, STRUCT }: spec/schema/mlt_tileset_metadata.proto

Implication

To support List / Map in the TS decoder, we likely need to extend the format (or introduce a new tag), otherwise there is nothing encoded to decode.

Decision needed

What is the preferred approach for adding nested types (List/Map) to MLT while keeping compatibility expectations clear?

Option A — Extend Tag 0x01

Add new type codes for LIST and MAP (including nullable variants) and define their stream layouts.

Pros:

• Minimal framing change (still tag == 1)
• Easiest to iterate on in all decoders/encoders

Cons:

• Might weaken the “MVT-equivalent” framing goal of Tag 0x01
• Requires coordinating updates across languages and fixtures

Option B — Introduce a new tag (e.g. 0x02) for nested types

Keep Tag 0x01 strictly “MVT-equivalent”, and add a new block/tag for tiles that use nested types.

Pros:

• Preserves the meaning of Tag 0x01
• Clear separation between “baseline” and “extended” tiles

Cons:

• More format surface area: new framing + more tooling changes (CLI, tests, fixtures)

Specific questions

1. Which option is preferred for MLT going forward (A: extend 0x01 vs B: new tag)?
2. If Option A:
   • Do we want to preserve the current “nullable via bit0” pattern for new type codes?
   • Are there reserved code ranges we should use for new complex types?
   • Should LIST<T> be represented as a complex type with exactly one child (the element type)?
3. If Option B:
   • What should the new tag framing look like (block structure, metadata placement, versioning)?
4. For both options:
   • What is the intended JS output shape? (e.g., null vs [], maps as Record<string, T> vs Array<[K,V]>)
   • Any constraints on supported nesting depth in v1 (e.g., List<List<T>>)?
   • How should this be tested/validated (new fixtures, or small synthetic tiles)?
5. For MAP<K,V> specifically: I'd prefer to defer detailed design until LIST is validated, unless maintainers prefer to define both layouts at once.

Strawman proposal (only if Option A is accepted)

Reserve new type codes while keeping bit0 == nullable consistent with existing encodings:

• LIST = 32, OPT_LIST = 33
• MAP = 34, OPT_MAP = 35

Potential stream layout for LIST<T>:

1. Optional PRESENT stream (if nullable)
2. A per-feature length stream (lengths → offsets)
3. Child column streams for T, decoded over sum(lengths) concatenated values

Goal of this issue

Get a maintainer decision on the format direction, so I can:

• implement the TS decoder changes,
• add tests and fixtures accordingly,
• and submit a PR that matches the project’s intended evolution path.

[HarelM]

This is probably a question to @mactrem & @nyurik.
Generally speaking, I think this is a long time requirement from MVT which I believe MLT should address/support.

[mactrem]

I've been exploring nested type support as part of MLT's core design, both conceptually and through a proof-of-concept implementation. Here are my current thoughts on the approach:

Structural Encoding Approach
MLT uses Present/Length pair-based encoding (as an alternative to Dremel encoding) for nested data types (structs, lists, maps):

• Each optional nesting level has a present stream
• Each variable-length type (e.g., lists) has a length stream
• Types are flattened via pre-order traversal

Current Status
I've implemented a proof-of-concept using Overture Maps data to evaluate efficiency and feasibility.

Limitations
The POC currently relies on an external schema definition based on Protocol Buffers (which will be supported as an optimization later, but was not included in the first version). However, the schema definition for complex types needs to be embedded within the file itself in the metadta section rather than referenced externally. The current metadata definition in the format only supports scalar data types.

Related Documentation
For more details on the proposed type system, see: MLT Type System Specification

@nyurik What's your opinion on how we should handle the schema embedding approach?

@DoFabien Happy to discuss this further on Slack if you're interested.

[DoFabien]

Thank you for the detailed update.

It is great to hear that the Present/Length stream approach is the chosen direction.

Regarding your question on schema embedding: I agree that for a robust implementation, the schema should be fully self-contained within the tile metadata, rather than relying on external proto files.

To address the limitation you mentioned ("the current metadata definition only supports scalar data types"), I think we could extend the existing FieldMetadata definition to support recursion, while remaining fully backward-compatible.

Conceptually, instead of a strictly flat list, it could look like this:

message FieldMetadata {
    uint32 id = 1;
    string name = 2;
    Type type = 3;

    // ... existing fields ...

    // Optional recursive definition for nested types:
    // - For LIST: contains 1 item (the element type)
    // - For STRUCT / MAP: contains N items (the child fields)
    repeated FieldMetadata children = 15;
}

Existing implementations could safely ignore the children field, while more advanced encoders/decoders could use it to support LIST / STRUCT semantics without flattening.

Context on my implementation:
I am working on a backend service (currently in advanced POC stage) that processes MLT tiles dynamically. It reads cached MLTs, performs filtering using Lists internally, and generates a new MLT tile on the fly. Currently, I flatten the output for the client to stay compatible with existing decoders, so the lack of nested type support is not an immediate blocker.
However, my roadmap involves exposing these nested structures directly to the client without flattening. At that point, I will rely on the TypeScript decoder to handle them natively.
Aligning on the schema now ensures readiness for that future transition. (I really enjoy working with the MLT format, by the way!)

I can update my Rust encoder to emit the physical Present/Length streams as described in the Type System spec.

If this metadata direction makes sense, I’m also happy to help implement or test the recursive metadata support in the TS decoder.

Happy to discuss or iterate on this if useful.

[mactrem]

Yes, that’s an option.

In the POC implementation with the external Protobuf based schema definition, i used a more advanced type system approach see https://github.com/mactrem/mlt-evaluation/blob/main/spec/schema/mlt_tileset_metadata.proto

However, for the embedded case, we could start with a simplified approach in the first iteration