Extension points for mesh- and material-related customizations

[GhisBntly]

Feature

This is a proposal for API additions needed to implement mesh- and material-related customizations beyond the current cesium-unreal capabilities.

Our use case

We need to process the rendered cesium tiles with advanced materials that cannot be done with the current capabilities, and also because of a limitation of Unreal Engine shaders:

• Our material shader needs to test an ID on each vertex to determine the shading: the ID is an unsigned 64bits integer, which Unreal shaders do not support, and we cannot simply cast to 32 bits because high-order bits contain relevant information.
• Other external data not present in the tileset's metadata also need to be correlated with the ID to determine what shading characteristics to apply at any given time, which adds specific requirements:
  • The custom shading may need to change the material Blend Mode to Translucent
  • The custom shading may need the "total" bounding box in the scene (as far as the currently loaded tiles allow to determine) for each ID (which identifies a consistent sub-mesh). The rough estimate given by the first loaded and thus coarser tiles is usually sufficient.
    • (@me/Bentley) Is it really important? Including those as extra metadata in the generated tileset is probably preferrable, ultimately.

Given these requirements, this is the solution we developed, inspired from metadata-based material customization in cesium-unreal:

• When tiles are loaded and Unreal meshes are built, we inspect each vertex to map its 64bits ID (found in the glTF tiles' metadata) to its 32bits FeatureID
• We may need to force the use of BaseTranslucent instead of BaseOpaque as parent of the UDynamicMaterialInstance created by cesium-unreal
• For each tile, we create textures which will be indexed by FeatureIDs (baked into the mesh UVs) in our custom material to apply the desired shading according to the external (non-tileset) data mentioned.
• Managing the per-tile information for meshes, materials, textures and ID vs. FeatureIDs mappings means we need some basic notifications on tiles lifecycle: when a tile has finished loading, when its visibility changes, and when it is about to be unloaded.

Implementation in our cesium-unreal's fork

Translating to the cesium-unreal APIs, those are extension points we added (temp names from our forked implementation):

1. In loadPrimitive, BakeFeatureIDsInVertexUVs = hook allowing API user to enforce the baking of FeatureIDs into a UV layer, without requiring a CesiumFeaturesMetadataComponent
   • The rationale is unclear, I guess we wanted to avoid creating a component because we didn't need any actual encoding of metadata, we only need to access the FeatureIDs in the UVs.
   • Also, we initially wanted to bake the FeatureID's on-demand, to avoid doing it for IDs that do not need it. But we failed to reliably edit the static mesh in-game (might not be possible) so we had to backtrack on that and now just bake for all meshes. The probability of having tiles with no need at all for FeatureID's in the UVs is probably low anyway.
   • Should we revert to using a CesiumFeaturesMetadataComponent or is there an advantage in performance and memory to be had by not creating this component? In that case, we could add a tileset flag to ensure this baking.
   • (@me/Bentley:WIP) we need the default FeatureID to be 0xFFFFFFFF (our shader relies on that) and not 0 as set in CesiumGltfComponent.cpp => moved to Metadata: Add a way to propagate the NullFeatureId to the GPU #1690
2. In loadPrimitive, bNeedsCPUAccess is passed to LODResources.VertexBuffers.PositionVertexBuffer.Init (idem index buffer) to allow access to vertex/index data after mesh creation:
   • (@me/Bentley) Check impact on memory footprint?
   • (@me/Bentley) Check if we can do without these buffer accesses (eg. compute Elements boundings boxes some other way? + check other usages). => we can, so the CPU access feature was dropped.
3. In loadPrimitiveGameThreadPart, OnMeshConstructed = event for API users to subscribe to when a game object (UCesiumGltfPrimitiveComponent) is created
   • UCesiumGltfPrimitiveComponent is a Private interface: thus, we need either to make it Public, or pass some more parameters that we have found to be needed: native tile, material instance, CesiumGltf::MeshPrimitive, FCesiumPrimitiveFeatures, FCesiumPrimitiveMetadata, GltfToUnrealTexCoordMap
   • Our implementation needs that to: parse metadata, count FeatureID's per tile (for texture dimensions), compute per-Element BBoxes, and maintain its own structures of tiles/meshcomponents/materials (esp. for mapping to/from Elements)
4. In loadPrimitiveGameThreadPart, CreateMaterial_GameThreadPart = hook to allow the API user to override the choice of BaseMaterial when creating a UCesiumGltfPrimitiveComponent's material instance: this is needed typically when the external data mandates that the material should be translucent, whereas cesium-unreal's criterions only would lead to an opaque material instance. Also needed to support other material customizations like our Glass material type, etc.
5. OnTileConstructed, at the end of CreateOnGameThread = event allowing custom processing after all meshes of a tile have been created (in the game thread)
6. OnVisibilityChanged, at the end of CreateOnGameThread = event allowing custom processing when a tile's visibility is changed by cesium's render selection algorithm
7. BeforeTileDestruction = event allowing custom processing just before a tile's render content is unloaded

I guess I could have split into sub-issues, but I was unsure how many inter-dependencies the above items would end up having, and whether or not it would clarify or confuse discussions...

Looking forward to your input!

[j9liu]

Thanks @GhisBntly for the detailed write-up! I can't say I have answers for everything at the moment, but I'll try to address some of these things...

Should we revert to using a CesiumFeaturesMetadataComponent or is there an advantage in performance and memory to be had by not creating this component? In that case, we could add a tileset flag to ensure this baking.

CesiumFeaturesMetadataComponent is intended for making feature IDs / metadata accessible in an Unreal material, which seems aligned to your use case. Despite the name, you can configure it to only use feature IDs and ignore any metadata.

For context, this component is designed to be configured in the Editor; users can manually query what properties exist on a tileset, then restrict which properties to actually load on the GPU at runtime. This does involve prior knowledge about what feature IDs are in the tileset -- and maybe that has worked for you so far -- but I acknowledge that it could be awkward to use if you don't have that knowledge in advance.

In general, I don't see any particular performance drawbacks to having the component. It only deals with the feature IDs or metadata properties that you specify on it, ignoring everything else. It doesn't do anything on the game Tick(), it just influences the load code in CesiumGltfComponent. But definitely let us know if you run into anything unexpected!

UCesiumGltfPrimitiveComponent is a Private interface: thus, we need either to make it Public, or pass some more parameters that we have found to be needed: native tile, material instance, CesiumGltf::MeshPrimitive, FCesiumPrimitiveFeatures, FCesiumPrimitiveMetadata, GltfToUnrealTexCoordMap

I'm not personally against this, but it'd help to know more about how GltfToUnrealTexCoordMap, CesiumGltf::MeshPrimitive and the native Tile would be used. I think that the other things -- the Unreal material instance, FCesiumPrimitiveFeatures, and FCesiumPrimitiveMetadata -- should already be accessible on a given UPrimitiveComponent through some other methods, so I'm curious about the other elements involved!

CreateMaterial_GameThreadPart = hook to allow the API user to override the choice of BaseMaterial when creating a UCesiumGltfPrimitiveComponent's material instance: this is needed typically when the external data mandates that the material should be translucent, whereas cesium-unreal's criterions only would lead to an opaque material instance.

Yeah I think we should allow the option to use translucent materials on opaque glTF models. Our material interface is a bit fragmented (due to handling opaque vs. translucent vs. water mask) so I wonder if it's a good opportunity to revisit this UI/UX. I think it is technically possible to put a translucent material in the opaque Material property on the tileset, but that might not be intuitive or convenient.

[GhisBntly]

Thanks for reading through the whole thing, @j9liu - here's some more ;^^

UCesiumGltfPrimitiveComponent is a Private interface: thus, we need either to make it Public, or pass some more parameters that we have found to be needed: native tile, material instance, CesiumGltf::MeshPrimitive, FCesiumPrimitiveFeatures, FCesiumPrimitiveMetadata, GltfToUnrealTexCoordMap

I'm not personally against this, but it'd help to know more about how GltfToUnrealTexCoordMap, CesiumGltf::MeshPrimitive and the native Tile would be used.

• GltfToUnrealTexCoordMap is used to configure the material parameter telling which UV layer to use to access FeatureIDs. I need to check whether cesium's code can do it for us, or why it wouldn't (maybe because we don't actually encode any property into our material, and only use the FeatureIDs baked in UVs)
  • It's probably simpler than that once all dead code is removed, I think only the UV layer index is needed.
• CesiumGltf::MeshPrimitive:
  • it is used to make a call to getExtension for one of our custom extension, but we realized that we didn't have to create a new one and could reuse an existing one
  • also used to get the features accessor for BakeFeatureIDsInVertexUVs's support code, but this should also disappear if we use CesiumFeaturesMetadataComponent (I tried it and still have a regression to investigate, but I have hope to make it work)
• The native tile is passed because we use its TileID as internal hashing key for our internal data (see items 5., 6. and 7.), but also because we have extended cesium-native with 2 features that I want to discuss in separate threads. Direct access to the native tile was an implementation shortcut, since to make them generic extensions, we'd probably have to expose them in cesium-unreal (and other engines...):
  • glTF tuning, a tile loading post-process step that depends on a set of evolving rules, thus we need to detect whether the current incarnation of a tile is up-to-date with the latest rules set
  • the tile render-readiness flag, to defer showing a tile until some of our asynchronous processing has finished

I think that the other things -- the Unreal material instance, FCesiumPrimitiveFeatures, and FCesiumPrimitiveMetadata -- should already be accessible on a given UPrimitiveComponent through some other methods, so I'm curious about the other elements involved!

FCesiumPrimitiveFeatures, and FCesiumPrimitiveMetadata are Public but need to be accessed from a UCesiumGltfPrimitiveComponent (or its interface ICesiumPrimitive), both of which being Module-private in the Unreal meaning. So I don't think we can have access to them outside the CesiumRuntime module :-/

CreateMaterial_GameThreadPart = hook to allow the API user to override the choice of BaseMaterial when creating a UCesiumGltfPrimitiveComponent's material instance: this is needed typically when the external data mandates that the material should be translucent, whereas cesium-unreal's criterions only would lead to an opaque material instance.

(...) I think it is technically possible to put a translucent material in the opaque Material property on the tileset, but that might not be intuitive or convenient.

Let me rephrase: we don't want BaseMaterial to be translucent, we want to be able to customize, for each UCesiumGltfPrimitiveComponent, whether its material should be instantiated from BaseMaterial or from BaseMaterialTranslucent. This choice depends on the primitive's metadata.

[GhisBntly]

I refactored our code and opened the above (draft) PR with it:

• I added a set of Public UINTERFACE's (see Public/CesiumLoadedTile.h: ICesiumLoadedTileBase, ICesiumLoadedTile, ICesiumLoadedTilePrimitive) to have cleaner signatures for CesiumMeshBuildCallbacks methods: they give access to the data we need.
• I added bAllowMeshBuffersCPUAccess as a tileset member
• I included the "tile render-readiness" flag (default=true) which I had not mentioned initially: as explained in another discussion, it can be toggled off to delay the frame/tick at which the tile can be shown, while waiting for some asynchronous post-processing to finish, for example. The flag and setter is in Cesium3DTilesSelection::Tile, and accessible through ICesiumLoadedTile so that it can typically be called from CesiumMeshBuildCallbacks::OnTileConstructed

[j9liu]

Sorry for the delayed response @GhisBntly! I see that you opened #1669 for this already, so I'll take a look in a bit 😄

Let me rephrase: we don't want BaseMaterial to be translucent, we want to be able to customize, for each UCesiumGltfPrimitiveComponent, whether its material should be instantiated from BaseMaterial or from BaseMaterialTranslucent. This choice depends on the primitive's metadata.

Ahhh gotcha, thanks for the correction! Yeah, that kind of granularity for assigning materials currently isn't possible, so I see the need for your custom callback.

FCesiumPrimitiveFeatures, and FCesiumPrimitiveMetadata are Public but need to be accessed from a UCesiumGltfPrimitiveComponent (or its interface ICesiumPrimitive), both of which being Module-private in the Unreal meaning. So I don't think we can have access to them outside the CesiumRuntime module :-/

This has actually come up for other users too -- we had someone open #715 for a similar problem, but we were wary of exposing UCesiumGltfPrimitiveComponent because its implementation details are subject to change. I'd say we still have that same attitude today, but we might need to revisit this if there's no good way to enable the same functionality with public APIs... 🤔