Expose tiles and textures to interact with terrain in custom layers

[su-noh]

The render_to_texture.ts file, render the tiles by multiple layers and use the finished tile texture(were to pulled from RenderPool) to render the terrain.

However, in the if statement below, LAYER object does not have custom property, so the custom type layer is not entered in the if statement and is not pushed to this._stacks.
see the code here.

For the above reasons, the tile texture is not directly accessible on the custom type layer.

Also, in RenderPool, the tile texture is returned to the pool and reused, so even if access is possible, the custom layer does not know if it is the right timing tile texture.

That is why the custom layer does not have access to the textures used in the final terrain and therefore cannot interact with the terrain provided by maplibre.

To correct this issue, we need to add custom property to the LAYER object and add Tile and Framebuffer to the render parameter to make it accessible to the custom layer. Alternatively, you can add an optional new function such as .renderToTile to CustomLayerInterface.

// render_to_texture.ts

/**
 * lookup table which layers should rendered to texture
 */
const LAYERS: { [keyof in StyleLayer['type']]?: boolean } = {
+   custom: true,
    background: true,
    fill: true,
    line: true,
    raster: true,
    hillshade: true
};

The function of painter.renderLayer(painter, painter.style.sourceCaches[layer.source], layer, coords, options); does not seem to have a big problem because the function that handles the custom layer is working properly.

However, the coords parameter is not undefined. so must be passed to the custom layer.

// painter.ts

renderLayer(painter: Painter, sourceCache: SourceCache, layer: StyleLayer, coords: Array<OverscaledTileID>, renderOptions: RenderOptions) {

    ...

    } else if (isCustomStyleLayer(layer)) {
-         drawCustom(painter, sourceCache, layer, renderOptions);
+         drawCustom(painter, sourceCache, layer, coords, renderOptions); 
    }
}

// draw_custom.ts

-export function drawCustom(painter: Painter, sourceCache: SourceCache, layer: CustomStyleLayer, renderOptions: RenderOptions) {
+export function drawCustom(painter: Painter, sourceCache: SourceCache, layer: CustomStyleLayer, coords: Array<OverscaledTileID>,renderOptions: RenderOptions) {

[HarelM]

Not sure I fully follow, but if you have a good idea how to improve this without introducing any breaking changes feel free to open a PR.

[amdevelopment-world]

i have the same problem, i created a pretty simple custom layer that applies a texture with N pixels per side per tile, with some irrelevant bicubic interpolation, i'm using the vertexShaderPrelude and define strings to setup my vertex shader, then i call the projectTile function, in render i create my mesh i use createTileMesh, so, in an ideal world, calling projectTile would take terrain into account, and the mesh for a tile would be the more complex 3D one when terrain is on, BUT, and this is a big one, i used this example https://maplibre.org/maplibre-gl-js/docs/examples/globe-custom-tiles/ as a starting point, so, i only recreate a shader when variantName changes, so, even (in my dreams ?) both the vertexShaderPrelude and define AND the createTileMesh would seamlessly work, i would not know, as im not including "terrain" on, in the key for my shader cache, anyway, im a noob, but, either i'm an idiot, that is very likely, or, createTileMesh and the vertex shader injected source code could be improved, or, in the very best case, the sample code is outdated, i know this is probably not the most typicall use of maplibre, but it's one of the ways someone can push boundaries, so i think it's worth helping poor devs like me and anyone who tries something similare

if someone can tell me how to do this i will create a simple example to help anyone else who wants to do something similar, custom layers, when shared, make this platform even greater

sorry if i missed some obvious API or doc that helps to do this, there is so much to read !

[HarelM]

Custom layer is a complicated solution, so I'm not sure I'll be able to help here.
If you find a solution feel free to share it.