Advice on ECS and instancing GLTFs

[viridia]

As previously mentioned, I'm working on porting my game/engine from three.js to Bevy. I'd like to get some advice on how to handle instancing.

Let me explain how the existing engine works: The game supports very large worlds (many kilometers) which are broken up into chunks called "precincts", each precinct is 64 x 64 meters. These are dynamically cached in memory as the player moves around the world. Within a precinct there are lots of instanced objects:

Each 1-meter section of wall in the screenshot is an instance of a GLTF model. Because the GLTF models often share the same textures, they are typically packed into a single .glb file - so for example, the "dungeon-walls.glb" file has about three dozen separate models which are referenced by name.

When a gltf file is loaded, the first thing that happens is that all of the materials are replaced. The GLTF material.userData field specifies the kinds of special effects (cartoon outlines, flame animation, glow effect, sprites, unlit, etc.) so each material is replaced with a custom material that implements that special effect, but otherwise keeps the same properties (color, texture, etc.)

The precinct map consists of a list of instance placements - that is, the model id, position, rotation and scale of each instance. Models ids are integers which reference a table of asset keys, also stored in the precinct map.

Each precinct has an InstancedModels manager which maintains and update the list of instances. Instances are organized by model type, so for example all of the "straight-wall-section" instances are stored in a single map entry which has an array of transforms, one transform per instance.

Walls which have animated components (doors and gates) are not instanced, but instead are treated as normal scene graph elements. The instance manager handles this automatically by checking to see if the wall section has any properties which would make instancing impossible.

The actual rendering of instances is relatively straightforward using three.js's InstancedMesh class. The built-in three.js shaders are written such that they can be used for either instanced or non-instanced rendering, but only for models in which the instance data is a transform. For something like particle systems, where each instance has multiple parameters, a custom shader must be used.

So here's a list of problems I need to solve:

• Referencing models - models are typically referenced via the asset name and fragment, e.g. "walls-dungeon.glb#StraightWallSection". Bevy currently doesn't support this (asset paths can only reference meshes by index, which is not going to work here). Alternatively, I can do a traversal through the GLTF world structure, but the data structures that Bevy provides are complicated and not easy to deal with. I'd need to build an index in order to avoid a linear search, and that index will need to stay in memory for as long as the GLTF asset is loaded.
• Replacing materials - as mentioned, I need to traverse the entire GLTF tree and replace all the materials with custom ones that support the special effects. This also means I need to get access to the GLTF custom attributes on the materials, which is also going to be a challenge.
• Instanced rendering - Bevy doesn't support instancing out of the box, so I'll need to create custom pipelines and shaders, and I'll need to do this for every material shader that I've written. Also, since some objects are not instanced, I'll have to write both an instanced and non-instanced version of each shader. I've been reading over the instancing examples, and honestly I'm having a really hard time understanding the code.
• Frustrum Culling - the Bevy instancing example disables frustum culling, but that's a deal-breaker for me. Because precincts in the cache may be off-screen, there may be a lot of instances in the scene. In fact, for models which have a high instance count, the instance manager sometimes splits the instance list into smaller pieces which have smaller bounding boxes. Computing the bounding box for the whole set is relatively easy, but I need to convince Bevy to use it.
• Instance Lifecycle and ECS - There's a question about how to organize all this in an ECS world. Should the instance manager be a component, or should each model type (and its list of transforms) be a component? The answer I'm thinking is "both", since you need a quick way to iterate through all the children of the instance manager when editing levels in the game editor. Also, how do you signal when the instance list needs to be rebuilt? Also, GLTF models can have more than one primitive, and each primitive needs to be instanced separately. How should I organize that information in terms of ECS components? How should I organize it in terms of pipelines? Also, you can't traverse the GLTF model until the load state of the handle is ready, so you need a way to represent the "deferred" traversal (e.g.. a data structure that says "here's 3 transforms, we want all the primitives of mesh X in asset Y, but only when the handle is ready). Is that data structure a component, or is it an element in a vector in a component?
• Detecting non-instanced models - the code for detecting whether a model should be instanced or not needs to be ported.
• Visbility changes - the game engine supports cutaway views, so when you walk in a building your view isn't obscured by the upper floors - this is not a clipping plane (which would look bad), and it's not a render layer bit (since layer bits are already being used by the portal system), so we need to iterate through all the instances and set their visibility based on the set of active cutaway volumes.

As you can see that's a rather daunting set of tasks, and I've left a lot out to keep this post from being any longer. But I'm interested in hearing any advice or guidance from folks who have more experience in Bevy/ECS/Rust than I do.

And if you're wondering, "why are you even doing this at all?", well there's a lot of reasons, but if you look at the second screenshot you'll notice there are no point-light shadows, because point-light shadow buffers are very expensive in three.js and you can't have very many of them. There are tons of other similar limitations in three.js and after three years I'm just tired of fighting with them.

[dmyyy]

With regards to instancing I think @superdump is taking a look at general mesh batching/instancing this cycle. Not sure how one can tell if a mesh has been instanced or not - maybe a label component