glTF 2.1 and complex scenes

[lilleyse]

There's a growing need for complex scenes in glTF while staying true to its core purpose as an efficient transmission format. Some examples include:

• Referencing external assets to build complex scenes, e.g. game environments
• Referencing individual mesh/material/animation resources from other files
• Supporting level of detail for 3D commerce, geospatial, and AEC/BIM/CAD (3d-tiles#553)

Previously this discussion was centered on glX. After much debate about whether this should be a separate file format or a set of extensions an interesting idea was proposed: what about a version bump?

What the glTF 2.1 proposal includes:

• Maintain backward and forward compatibility with glTF 2.0.
• All new features including graceful deprecation would be part of the new version of the glTF spec
• New features would include not only external references, but long desired improvement for the core spec that might include options like: 4GB file limit, binary json, etc

Importantly, while glTF 2.1 would gain additional features to support complex scenes, the intent is not to replace USD or replicate its scene composition approach. Their respective strengths of transmission vs. authoring still stand.

Some of the features being considered for glTF 2.1:

Complex scenes

• External references JSON refer other JSON? #37
• Bounding volume hierarchies
• Level of detail (including refinement criteria)

Package format

• .glx or .glz (similar to USDZ)

Other ideas

• Updates to PBR model
• Opportunity to clean up and modularize areas of the core spec

Definitely interested in hearing everyone's thoughts about this approach and other ideas for glTF 2.1.

[npolys]

There is a solution in ISO X3D!
The specification and implementations are here now.

The subset of concepts REQUIRED for assets and experiences is being developed in the 3D Web Interoperability group at Metaverse Standards Forum. The Web of Worlds concept and API are described here:
https://metaverse-standards.org/news/blog/building-the-world-wide-webiverse/
https://metaverse-standards.org/news/blog/linked-spatial-experiences-the-web-of-worlds/

We like glTF as a ' JPEG of 3D'; keep it simple!

X3Dv4 lets users build scenes, light, and define interactivity including formats like X3D and glTF.
Some engines load X3D, glTF, PLY, OBJ, STL, IFC, DICOM, ... but reasonably, client server negotiation could support any IANA <model-*> type .

So consider the principle of Parsimony: simplicity.

Simple Linking Standard that uses a basic scenegraph and URI/URL mechanism (with fallback semantics) to compose scenes and to deliver experiences.
Indeed, a subset syntax (let's call it a LinkedDataProfile) could be easily agreed upon.
eg Considering a new X3D WOW Profile:

X3D Core (Metadata): https://www.web3d.org/documents/specifications/19775-1/V4.0/Part01/components/core.html
Networking: https://www.web3d.org/documents/specifications/19775-1/V4.0/Part01/components/networking.html
Grouping: https://www.web3d.org/documents/specifications/19775-1/V4.0/Part01/components/grouping.html

A SMALL set of nodes that meet the use cases (format-agnostic linked experiences); for example:

MetadataSet (and MetadataNodes)

Inline
Anchor

Transform
Group
StaticGroup
Switch
LOD

The event -based behavior graph of X3D is certainly another topic/ thread!

[aaronfranke]

I would like to see external references in glTF itself. This is a good feature for the format, and it makes sense to put it in 2.1. There is no harm to implementations which can't support it, since such data can be ignored.

In the glTF External Reference (glTFX) repository, I opened an issue linking to my repository where I implemented glTFX in Godot Engine. This was pretty simple to do, taking only about a day or so, and the concepts directly map to Godot scenes, allowing a hierarchy of glTFX files to be imported as a hierarchy of Godot scenes, and vice versa.

Having this be in the base spec would be even simpler, because then we would not need to worry about distinguishing between files which contain complex objects (meshes, lights, physics, etc) from those that don't, allows animating these nodes, and it would open the door for future extensions like KHR_interactivity to work with these nodes.

In my own format, I added this feature, allowing model files to reference other model files.

Details

This is done with a "models" array in the top-level JSON, following the IANA name "model". Each item contains a MIME type, URI (or buffer view), and an optional name. These are then referenced by index on nodes, like "model": 0, similarly to how glTF references meshes, cameras, etc. Here is an example. The red lines show a glTF being instanced, using the MIME type model/gltf+json, and the orange lines show a G4MF being instanced, using the MIME type model/g4mf+json (pending registration).

I happily recommend that glTF follows the same format. I also recommend that the MIME type be explicitly defined; there is value in including non-glTF formats in glTF, such as including OCIF files to represent GUIs. The glTF 2.1 specification could explicitly state that only "model/gltf+json" and "model/gltf-binary" are supported in the base specification, but other formats can be added by extensions.

[crlf0710]

Re: External references: No opposition to the idea of allowing referencing external resources and assets, i found the current status is already not at all optimal. From the perspective of a glTF "bundler", the current status quo is that one has to actually fully parse and inspect into the fields to build an actual list of referenced external resources. What makes it even worse, those links can occur within extensions, so one can never tell if all referenced resources are already correctly bundled unless it actually supports every single glTF extension in the world.

I'll be very grateful if a future version of glTF spec can actually move these references to a central place within the format, so the glTF bundlers can grab a list of all referenced external things without actually wholly parsing all the glTF data, especially, the extensions.

[javagl]

That ^ is a very valid point.

... the current status quo is that one has to actually fully parse and inspect into the fields to build an actual list of referenced external resources

This is ... true-ish 🙂 parsing the glTF JSON is some sort of baseline - you can hardly do anything without that anyhow. And one could argue that you can get away with just iterating over the buffers and images (At least, I did that).

But...

What makes it even worse, those links can occur within extensions, so one can never tell if all referenced resources are already correctly bundled unless it actually supports every single glTF extension in the world.

This is the important part here. One could still argue that this is "rare", and whatever extensions are storing, it could/should/will usually be a buffer (or image) anyhow. But a dedicated place to securely grab all references from would increase the flexibility and reduce the number of guesses and assumptions that have to be made.

[aaronfranke]

As an example of what @crlf0710 is discussing, in KHR_audio_emitter, it references audio files (MP3 etc). These are often external (so not in a buffer) and are not images, so if a trivial implementation were to make an attempt to just read a .gltf file and gather all the files it references, such as a tool to upload a glTF online, it would miss the audio files. So the trivial uploader would need to have one or more things done to make it work:

1. Increase the complexity of the tool to support KHR_audio_emitter's audio data references and every other future extension that references external files.
2. Require that users select all files used by the glTF file manually.
3. As a hack, just blindly read every property named "uri" and check for files at those paths.

There is a similar problem with node indices, mesh indices, etc. A tool that processes glTF files needs to be careful to not remove or reorder any existing nodes, mesh, etc, because it could break indices, or if it needs to do that, then unknown extensions need to be discarded to avoid producing invalid files. I chatted with @donmccurdy about adding an option to glTF-Transform to preserve data from unknown extensions, even if just opting in to each extension with a command line argument, but he strongly felt that the file and ID referencing problems were so significant that glTF-Transform has no choice but to discard all unknown extensions always.

[lexaknyazev]

There's always an option to exclusively use buffer views when referring to resources from extension objects. However that additional level of indirection may be quite annoying when such resources are stored in separate files.

[Igor-dvr]

Apart from external references and already mentioned stuff:

• rigid bodies, implicit shapes
• more animation related stuff
• scene lighting, dome light...

this would really help a lot of users.

[hybridherbst]

Some things that I'd like to see from 2.1:

• somewhat stricter spec language in a few places, e.g. replace "SHOULD" with "MUST" for URI normalization. See for example Relative URI Issues atteneder/glTFast#738 (comment)
• allowing multiple internal buffers (current spec allows at most 1 internal buffer and arbitrary number of external buffers) because it makes diffable modifications easier
• +1 for thinking about "resources" as a data type so that extensions can be more opaque (don't need to be fully known for some operations on the glTF)