Skip to content

Commit e7b64b6

Browse files
Add support for arbitrary/third party glTF Extension processing via GltfExtensionHandler (#22106)
# Objective Currently Bevy doesn't support arbitrary glTF extensions. The ones it does support are hardcoded. We should support glTF extensions, as this is a primary mechanism for sharing behavior via data exported from applications like Blender. I personally have found usecases in exporting component data, lightmap textures/information, and processing other kinds of data (AnimationGraph, 3d meshes into 2d, etc). ## Solution This PR introduces a new `GltfExtensionHandler` trait that users can implement and add to the glTF loader processing via inserting into a Resource. There are two example processors currently added, with a third that I'd like to add after this PR. - `examples/gltf/gltf_extension_animation_graph.rs` duplicates the functionality of `animation_mesh`, constructing AnimationGraphs via extension processing and applying them to be played on the relevant nodes. - `examples/gltf/gltf_extension_mesh_2d.rs` duplicates the functionality of the `custom_gltf_vertex_attribute` example, showing how the extension processing could be used to convert 3d meshes to 2d meshes alongside custom materials. Both of these examples re-use existing assets and thus don't *actually use* extension data, but show how one could access the relevant data to say, only convert specifically labelled Mesh3ds to 2d, or process many animations into multiple graphs based on extension-data based labelling introduced in Blender. A third example I want to introduce after this PR is the same core functionality Skein requires: an example that uses reflected component data stored in glTF extensions and inserts that data onto the relevant entities, resulting in scenes that are "ready to go". ## Comparison to Extras In comparison to extensions: data placed in glTF extras is well supported through the `GltfExtras` category of components. Extras only support adding an additional `extras` field to any object. Data stored in extras is application-specific. It should be usable by Bevy developers to implement their own, application-specific, data transfer. This is supported by applications like Blender through the application of Custom Properties. Once data is used by more than one application, it belongs in a glTF extension. ## What is a glTF Extension? Extensions are named with a prefix like `KHR` or `EXT`. Bevy has already reserved the `BEVY` namespace for this, which is listed in the official [prefix list](https://github.com/KhronosGroup/glTF/blob/7bbd90978cad06389eee3a36882c5ef2f2039faf/extensions/Prefixes.md). For a glTF file, an extension must be listed in `extensionsUsed` and optionally `extensionsRequired`. ``` { "extensionsRequired": [ "KHR_texture_transform" ], "extensionsUsed": [ "KHR_texture_transform" ] } ``` Extension data is allowed in any place extras are also allowed, but also allow much more flexibility. Extensions are also allowed to define global data, add additional binary chunks, and more. For meshes, extensions can add additional attribute names, accessor types, and/or component types `KHR_lights_punctual` is a contained and understandable example of an extension: https://github.com/KhronosGroup/glTF/blob/7bbd90978cad06389eee3a36882c5ef2f2039faf/extensions/2.0/Khronos/KHR_lights_punctual/README.md . This one happens to be already hardcoded into Bevy's handling, so it doesn't benefit from arbitrary extension processing, but there are additional [ratified](https://github.com/KhronosGroup/glTF/tree/7bbd90978cad06389eee3a36882c5ef2f2039faf/extensions#ratified-khronos-extensions) and [in-progress](https://github.com/KhronosGroup/glTF/tree/7bbd90978cad06389eee3a36882c5ef2f2039faf/extensions#in-progress-khronos-and-multi-vendor-extensions-and-projects) extensions, as well as [vendor](https://github.com/KhronosGroup/glTF/tree/7bbd90978cad06389eee3a36882c5ef2f2039faf/extensions#vendor-extensions) and other arbitrary extensions that would benefit from userland support. ## Implementation This initial implementation is reasonably minimal: enabling extension processing for objects/etc as they're loaded which may also define extension data, including the scene world. This may leave out useful functionality; as detailed in the next section: "What's not implemented". Extension handlers are defined by implementing a trait which can optionally define hooks and data. Extension handler data is cloned to start with a fresh slate for each glTF load, which limits scope to "one glTF load". So while state can be maintained across hooks during a single load, users who want to combine or handle multiple glTF assets should do so in the main app, not in an extension handler. Following this, because the extensions are stored as `dyn GltfExtension` *and* we want to clone them to isolate state to a single load, `dyn_clone` must be included as a workaround to enable this cloning. An extension handler has to be added to the list of handler by accessing a `Resource` and pushing an instantiated handler into it. This Resource keeps the list of extension handlers so that a new glTF loader can bootstrap them. The design of the hooks is such that: - If no extensions handlers are registered, none are called for processing - If an extension handler is defined, it receives all "events" - handlers are defined by a trait, and default implementations are called if an override is not specified. - default implementations are no-ops It is important that extensions receive all events because certain information is not embedded in extension data. For example, processing animation data into an animation graph could require both processing animations with extension data, tracking the animation roots through hooks like `on_node`, *and* applying those graphs in the `on_scene_completed` hook. - Extension data is passed to hooks as `Option<&serde_json::Value>` which is only passing references around as the data has already been converted to `Value` by the `gltf` crate. - `LoadContext` is required for creating any new additional assets, like `AnimationGraph`s. - *scene* World access is provided in hooks like `on_scene_completed`, which allows calculating data over the course of a glTF load and applying it to a Scene. ### What's not implemented This PR chooses to *not* implement some features that it could. Instead the approach in this PR is to offer up the data that Bevy has already processed to extensions to do more with that data. - Overriding `load_image`/`process_loaded_texture` - This could allow projects like bevy_web_codecs, [which currently forks the entire gltf loader](https://github.com/jf908/bevy_web_codecs/tree/373bbf29be6555c7603fd6867a01159ab0f20fed/bevy_web_codecs_gltf). Associated [issue](#21185). However I believe this needs some design work dedicated to what exactly happens here to support that use case. - This PR doesn't include any refactoring of the glTF loader, which I feel is important for a first merge. - ~~There is some benefit to passing in the relevant `gltf::*` object to every hook. For example, I believe this is the only way to access extension data for `KHR_lights_punctual`, and [`KHR_materials_variants`](https://docs.rs/gltf/1.4.1/gltf/struct.Document.html#method.variants) or other extensions with "built-in" support. I haven't done this in all places.~~ (edit: after external implementation I decided this was a good idea and added it to more places) ## Testing ``` cargo run --example gltf_extension_animation_graph cargo run --example gltf_extension_mesh_2d ``` --- ## Showcase Both examples running: https://github.com/user-attachments/assets/f9e7c3c9-cdad-4d33-ace7-7c2ca5469d5e https://github.com/user-attachments/assets/baa9bc92-ca3b-46ad-a3f0-2f74bbc29b68 <details> <summary>An example that showcases converting Mesh3d to Mesh2d</summary> ```rust #[derive(Default, Clone)] struct GltfExtensionProcessorToMesh2d; impl GltfExtensionProcessor for GltfExtensionProcessorToMesh2d { fn extension_ids(&self) -> &'static [&'static str] { &[""] } fn dyn_clone(&self) -> Box<dyn GltfExtensionHandler> { Box::new((*self).clone()) } fn on_spawn_mesh_and_material( &mut self, load_context: &mut LoadContext<'_>, _gltf_node: &gltf::Node, entity: &mut EntityWorldMut, ) { if let Some(mesh3d) = entity.get::<Mesh3d>() && let Some(_) = entity.get::<MeshMaterial3d<StandardMaterial>>() { let material_handle = load_context.add_loaded_labeled_asset("AColorMaterial", (CustomMaterial {}).into()); let mesh_handle = mesh3d.0.clone(); entity .remove::<(Mesh3d, MeshMaterial3d<StandardMaterial>)>() .insert((Mesh2d(mesh_handle), MeshMaterial2d(material_handle.clone()))); } } } ``` </details> --------- Co-authored-by: Kristoffer Søholm <k.soeholm@gmail.com>
1 parent d042f8d commit e7b64b6

File tree

9 files changed

+821
-9
lines changed

9 files changed

+821
-9
lines changed

Cargo.toml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -732,6 +732,7 @@ event-listener = "5.3.0"
732732
anyhow = "1"
733733
accesskit = "0.21"
734734
nonmax = "0.5"
735+
gltf = "1.4"
735736

736737
[target.'cfg(not(target_family = "wasm"))'.dev-dependencies]
737738
ureq = { version = "3.0.8", features = ["json"] }
@@ -4201,6 +4202,28 @@ description = "Loads and renders a glTF file as a scene, including the gltf extr
42014202
category = "glTF"
42024203
wasm = true
42034204

4205+
[[example]]
4206+
name = "gltf_extension_animation_graph"
4207+
path = "examples/gltf/gltf_extension_animation_graph.rs"
4208+
doc-scrape-examples = true
4209+
4210+
[package.metadata.example.gltf_extension_animation_graph]
4211+
name = "glTF extension AnimationGraph"
4212+
description = "Uses glTF data to build an AnimationGraph via extension processing"
4213+
category = "glTF"
4214+
wasm = true
4215+
4216+
[[example]]
4217+
name = "gltf_extension_mesh_2d"
4218+
path = "examples/gltf/gltf_extension_mesh_2d.rs"
4219+
doc-scrape-examples = true
4220+
4221+
[package.metadata.example.gltf_extension_mesh_2d]
4222+
name = "glTF extension processing to build Mesh2ds from glTF data"
4223+
description = "Uses glTF extension data to convert incoming Mesh3d/MeshMaterial3d assets to 2d"
4224+
category = "glTF"
4225+
wasm = true
4226+
42044227
[[example]]
42054228
name = "query_gltf_primitives"
42064229
path = "examples/gltf/query_gltf_primitives.rs"

assets/models/barycentric/barycentric.gltf

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,18 @@
11
{
2+
"scene": 0,
3+
"scenes": [
4+
{
5+
"nodes": [
6+
0
7+
]
8+
}
9+
],
10+
"nodes": [
11+
{
12+
"name": "box",
13+
"mesh": 0
14+
}
15+
],
216
"accessors": [
317
{
418
"bufferView": 0,

crates/bevy_gltf/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ gltf = { version = "1.4.0", default-features = false, features = [
5555
"names",
5656
"utils",
5757
] }
58+
async-lock = { version = "3.0", default-features = false }
5859
thiserror = { version = "2", default-features = false }
5960
base64 = "0.22.0"
6061
fixedbitset = "0.5"

crates/bevy_gltf/src/lib.rs

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,8 @@ pub mod prelude {
155155
pub use crate::{assets::Gltf, assets::GltfExtras, label::GltfAssetLabel};
156156
}
157157

158+
use crate::extensions::GltfExtensionHandlers;
159+
158160
pub use {assets::*, label::GltfAssetLabel, loader::*};
159161

160162
// Has to store an Arc<Mutex<...>> as there is no other way to mutate fields of asset loaders.
@@ -249,7 +251,8 @@ impl Plugin for GltfPlugin {
249251
.init_asset::<GltfPrimitive>()
250252
.init_asset::<GltfMesh>()
251253
.init_asset::<GltfSkin>()
252-
.preregister_asset_loader::<GltfLoader>(&["gltf", "glb"]);
254+
.preregister_asset_loader::<GltfLoader>(&["gltf", "glb"])
255+
.init_resource::<GltfExtensionHandlers>();
253256
}
254257

255258
fn finish(&self, app: &mut App) {
@@ -267,11 +270,14 @@ impl Plugin for GltfPlugin {
267270
let default_sampler = default_sampler_resource.get_internal();
268271
app.insert_resource(default_sampler_resource);
269272

273+
let extensions = app.world().resource::<GltfExtensionHandlers>();
274+
270275
app.register_asset_loader(GltfLoader {
271276
supported_compressed_formats,
272277
custom_vertex_attributes: self.custom_vertex_attributes.clone(),
273278
default_sampler,
274279
default_use_model_forward_direction: self.use_model_forward_direction,
280+
extensions: extensions.0.clone(),
275281
});
276282
}
277283
}

crates/bevy_gltf/src/loader/extensions/mod.rs

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,270 @@ mod khr_materials_anisotropy;
44
mod khr_materials_clearcoat;
55
mod khr_materials_specular;
66

7+
use alloc::sync::Arc;
8+
use async_lock::RwLock;
9+
10+
use bevy_animation::AnimationClip;
11+
use bevy_asset::{Handle, LoadContext};
12+
use bevy_ecs::{
13+
entity::Entity,
14+
resource::Resource,
15+
world::{EntityWorldMut, World},
16+
};
17+
use bevy_pbr::StandardMaterial;
18+
use bevy_platform::collections::{HashMap, HashSet};
19+
use gltf::Node;
20+
21+
use crate::GltfMesh;
22+
723
pub(crate) use self::{
824
khr_materials_anisotropy::AnisotropyExtension, khr_materials_clearcoat::ClearcoatExtension,
925
khr_materials_specular::SpecularExtension,
1026
};
27+
28+
/// Stores the `GltfExtensionHandler` implementations so that they
29+
/// can be added by users and also passed to the glTF loader
30+
#[derive(Resource, Default)]
31+
pub struct GltfExtensionHandlers(pub Arc<RwLock<Vec<Box<dyn GltfExtensionHandler>>>>);
32+
33+
/// glTF Extensions can attach data to any objects in a glTF file.
34+
/// This is done by inserting data in the `extensions` sub-object, and
35+
/// data in the extensions sub-object is keyed by the id of the extension.
36+
/// For example: `KHR_materials_variants`, `EXT_meshopt_compression`, or `BEVY_my_tool`
37+
///
38+
/// A list of publicly known extensions and their ids can be found
39+
/// in the [KhronosGroup/glTF](https://github.com/KhronosGroup/glTF/blob/main/extensions/README.md)
40+
/// git repo. Vendors reserve prefixes, such as the `BEVY` prefix,
41+
/// which is also listed in the [KhronosGroup repo](https://github.com/KhronosGroup/glTF/blob/main/extensions/Prefixes.md).
42+
///
43+
/// The `GltfExtensionHandler` trait should be implemented to participate in
44+
/// processing glTF files as they load, and exposes glTF extension data via
45+
/// a series of hook callbacks.
46+
///
47+
/// The type a `GltfExtensionHandler` is implemented for can define data
48+
/// which will be cloned for each new glTF load. This enables stateful
49+
/// handling of glTF extension data during a single load.
50+
pub trait GltfExtensionHandler: Send + Sync {
51+
/// Required for dyn cloning
52+
fn dyn_clone(&self) -> Box<dyn GltfExtensionHandler>;
53+
54+
/// When loading a glTF file, a glTF object that could contain extension
55+
/// data will cause the relevant hook to execute once for each id in this list.
56+
/// Each invocation will receive the extension data for one of the extension ids,
57+
/// along with the `extension_id` itself so implementors can differentiate
58+
/// between different calls and parse data correctly.
59+
///
60+
/// The hooks are always called, even if there is no extension data
61+
/// for a specified id. This is useful for scenarios where additional
62+
/// extension data isn't required, but processing should still happen.
63+
///
64+
/// Most implementors will pick one extension for this list, causing the
65+
/// relevant hooks to fire once per object. An implementor that does not
66+
/// wish to receive any data but still wants hooks to be called can use
67+
/// an empty string `""` as the extension id, which is also the default
68+
/// value if the function is not implemented by an implementor. If the
69+
/// empty string is used, all extension data in hooks will be `None`.
70+
///
71+
/// Some implementors will choose to list multiple extensions here.
72+
/// This is an advanced use case and the alternative of having multiple
73+
/// independent handlers should be considered as an option first.
74+
/// If multiple extension ids are listed here, the hooks will fire once
75+
/// for each extension id, and each successive call will receive the data for
76+
/// a separate extension. The extension id is also included in hook arguments
77+
/// for this reason, so multiple extension id implementors can differentiate
78+
/// between the data received.
79+
fn extension_ids(&self) -> &'static [&'static str] {
80+
&[""]
81+
}
82+
83+
/// Called when the "global" data for an extension
84+
/// at the root of a glTF file is encountered.
85+
#[expect(
86+
unused,
87+
reason = "default trait implementations do not use the arguments because they are no-ops"
88+
)]
89+
fn on_root_data(&mut self, extension_id: &str, value: Option<&serde_json::Value>) {}
90+
91+
#[cfg(feature = "bevy_animation")]
92+
#[expect(
93+
unused,
94+
reason = "default trait implementations do not use the arguments because they are no-ops"
95+
)]
96+
/// Called when an individual animation is processed
97+
fn on_animation(
98+
&mut self,
99+
extension_id: &str,
100+
extension_data: Option<&serde_json::Value>,
101+
gltf_animation: &gltf::Animation,
102+
name: Option<&str>,
103+
handle: Handle<AnimationClip>,
104+
) {
105+
}
106+
107+
#[cfg(feature = "bevy_animation")]
108+
#[expect(
109+
unused,
110+
reason = "default trait implementations do not use the arguments because they are no-ops"
111+
)]
112+
/// Called when all animations have been collected.
113+
/// `animations` is the glTF ordered list of `Handle<AnimationClip>`s
114+
/// `named_animations` is a `HashMap` from animation name to `Handle<AnimationClip>`
115+
/// `animation_roots` is the glTF index of the animation root object
116+
fn on_animations_collected(
117+
&mut self,
118+
load_context: &mut LoadContext<'_>,
119+
animations: &[Handle<AnimationClip>],
120+
named_animations: &HashMap<Box<str>, Handle<AnimationClip>>,
121+
animation_roots: &HashSet<usize>,
122+
) {
123+
}
124+
125+
/// Called when an individual texture is processed
126+
#[expect(
127+
unused,
128+
reason = "default trait implementations do not use the arguments because they are no-ops"
129+
)]
130+
fn on_texture(
131+
&mut self,
132+
extension_id: &str,
133+
extension_data: Option<&serde_json::Value>,
134+
texture: Handle<bevy_image::Image>,
135+
) {
136+
}
137+
138+
/// Called when an individual material is processed
139+
#[expect(
140+
unused,
141+
reason = "default trait implementations do not use the arguments because they are no-ops"
142+
)]
143+
fn on_material(
144+
&mut self,
145+
extension_id: &str,
146+
extension_data: Option<&serde_json::Value>,
147+
load_context: &mut LoadContext<'_>,
148+
gltf_material: &gltf::Material,
149+
name: Option<&str>,
150+
material: Handle<StandardMaterial>,
151+
) {
152+
}
153+
154+
/// Called when an individual glTF Mesh is processed
155+
#[expect(
156+
unused,
157+
reason = "default trait implementations do not use the arguments because they are no-ops"
158+
)]
159+
fn on_gltf_mesh(
160+
&mut self,
161+
extension_id: &str,
162+
extension_data: Option<&serde_json::Value>,
163+
load_context: &mut LoadContext<'_>,
164+
gltf_mesh: &gltf::Mesh,
165+
name: Option<&str>,
166+
mesh: Handle<GltfMesh>,
167+
) {
168+
}
169+
170+
/// mesh and material are spawned as a single Entity,
171+
/// which means an extension would have to decide for
172+
/// itself how to merge the extension data.
173+
#[expect(
174+
unused,
175+
reason = "default trait implementations do not use the arguments because they are no-ops"
176+
)]
177+
fn on_spawn_mesh_and_material(
178+
&mut self,
179+
load_context: &mut LoadContext<'_>,
180+
primitive: &gltf::Primitive,
181+
mesh: &gltf::Mesh,
182+
material: &gltf::Material,
183+
entity: &mut EntityWorldMut,
184+
) {
185+
}
186+
187+
/// Called when an individual Scene is done processing
188+
#[expect(
189+
unused,
190+
reason = "default trait implementations do not use the arguments because they are no-ops"
191+
)]
192+
fn on_scene_completed(
193+
&mut self,
194+
extension_id: &str,
195+
extension_data: Option<&serde_json::Value>,
196+
scene: &gltf::Scene,
197+
name: Option<&str>,
198+
world_root_id: Entity,
199+
scene_world: &mut World,
200+
load_context: &mut LoadContext<'_>,
201+
) {
202+
}
203+
204+
/// Called when a node is processed
205+
#[expect(
206+
unused,
207+
reason = "default trait implementations do not use the arguments because they are no-ops"
208+
)]
209+
fn on_gltf_node(
210+
&mut self,
211+
extension_id: &str,
212+
extension_data: Option<&serde_json::Value>,
213+
load_context: &mut LoadContext<'_>,
214+
gltf_node: &Node,
215+
entity: &mut EntityWorldMut,
216+
) {
217+
}
218+
219+
/// Called with a `DirectionalLight` node is spawned
220+
/// which is typically created as a result of
221+
/// `KHR_lights_punctual`
222+
#[expect(
223+
unused,
224+
reason = "default trait implementations do not use the arguments because they are no-ops"
225+
)]
226+
fn on_spawn_light_directional(
227+
&mut self,
228+
extension_id: &str,
229+
extension_data: Option<&serde_json::Value>,
230+
load_context: &mut LoadContext<'_>,
231+
gltf_node: &Node,
232+
entity: &mut EntityWorldMut,
233+
) {
234+
}
235+
/// Called with a `PointLight` node is spawned
236+
/// which is typically created as a result of
237+
/// `KHR_lights_punctual`
238+
#[expect(
239+
unused,
240+
reason = "default trait implementations do not use the arguments because they are no-ops"
241+
)]
242+
fn on_spawn_light_point(
243+
&mut self,
244+
extension_id: &str,
245+
extension_data: Option<&serde_json::Value>,
246+
load_context: &mut LoadContext<'_>,
247+
gltf_node: &Node,
248+
entity: &mut EntityWorldMut,
249+
) {
250+
}
251+
/// Called with a `SpotLight` node is spawned
252+
/// which is typically created as a result of
253+
/// `KHR_lights_punctual`
254+
#[expect(
255+
unused,
256+
reason = "default trait implementations do not use the arguments because they are no-ops"
257+
)]
258+
fn on_spawn_light_spot(
259+
&mut self,
260+
extension_id: &str,
261+
extension_data: Option<&serde_json::Value>,
262+
load_context: &mut LoadContext<'_>,
263+
gltf_node: &Node,
264+
entity: &mut EntityWorldMut,
265+
) {
266+
}
267+
}
268+
269+
impl Clone for Box<dyn GltfExtensionHandler> {
270+
fn clone(&self) -> Self {
271+
self.dyn_clone()
272+
}
273+
}

0 commit comments

Comments
 (0)