glTF object custom user data/extras storage

[AndreCostaaa]

Hi @spnda,

I would like to store my own data tied to the Node class
Usually, in C, this is done by setting a void* attribute that can then point to anything, I guess with C++ this would be done with generic instead

template <typename UserDataT>
FASTGLTF_EXPORT struct Node {
    UserDataT userData;
}

Would this change be something you'd like? If not do you have another possible approach?

[spnda]

Something along the lines of your proposal has been discussed a couple of times, but implementing this using a templated type requires a large rewrite of the Parser class, moving a lot if not all of the parsing logic into headers since the type is not constant.

std::any is a fair contender, however it needs to store the type that is currently actually stored which requires RTTI which some users may not want or have disabled. Therefore, this was not a good solution in my books. Especially on embedded RTTI is not something you really want since it massively bloats the executable, and uses storage one might not have. Which is why I currently am not considering merging #125.

Since no good solution was ever found during the discussions I had on Discord I never implemented anything inline with your needs. The current best solution is to have your own array of node_userdata that lives along side the nodes array where you keep your user data. This isn't as clean as I'd like it but it's definitely functional. Perhaps some other system can be implemented that works better using custom types similar to std::any or just a plain void*.

Also, ideally, this would not just exist just for nodes, but for every type used in an asset.

[AndreCostaaa]

For me, you can never go wrong with void* but this is coming from a C developer

My current concern is that in LVGL, we need to find our node data when iterating fastgltf's nodes

See here:

https://github.com/AndreCostaaa/lvgl/blob/06a42e6aad0f4b53a05cd3370f80de84831a699f/src/libs/gltf/gltf_data/lv_gltf_model_node.cpp#L118-L130

Using it here during the rendering:

https://github.com/AndreCostaaa/lvgl/blob/06a42e6aad0f4b53a05cd3370f80de84831a699f/src/libs/gltf/gltf_view/lv_gltf_view_render.cpp#L1223-L1233

Right now I need to make the decision to either store those node datas inside a tree data structure > which could result in very bad cache coherency and big memory overhead or inside an array > better cache coherency but for very big models it will become slow

Do you have an idea? Are you against adding a void * to every structure?

[spnda]

Right now I need to make the decision to either store those node datas inside a tree data structure > which could result in very bad cache coherency and big memory overhead or inside an array > better cache coherency but for very big models it will become slow

Your other option is to keep your own copies of each node. Essentially copy or claim the memory from the std::vector and put it into your own array where you can create a hybrid type of your own stuff and the fastgltf type. One of the great things with glTFs is that it's all just indices, and not pointers. So the data structure itself does not care one bit where your nodes are.

Are you against adding a void * to every structure?

Not really, I guess it is the best option in the end without templating the entire Asset type (cause that was incredibly fucked, I tried it locally a while back). But if I do it it wouldn't just be on the Node type, it would be everywhere. I also wonder if it can be disabled on a per-type basis using macros as enums? And, well, I would still prefer if there was something like std::any or std::shared_ptr but without RTI and the other drawbacks. std::shared_ptr<void*> would be the better to use in this case since we're using C++ and some people would be more accustomed to it, though that, again, would create problems in your mostly C environment....

[FishOfTheNorthStar]

This sounds great. I've often wished I could attach custom data to nodes, materials, and lights. Seems like it could come in real handy with all kinds of stuff, so I like the sound of 'it would be everywhere'.

We're adjusting model transforms at runtime, and without something like this, there's an unfortunate amount of item lookups, this will help eliminate those entirely. Since the node traversal was already happening anyways, it'll be much more efficient.

This could be a great foundation for other stuff later, too. With this in place, the idea of adding a user specified encoder/decoder becomes a lot more realistic. Or several, even. Different nodes could use different codecs, and that would make them a lot like classes. Could be a great glTF extension some day.

[AndreCostaaa]

I will investigate what it would take to use my own version of a Node structure and iterate over that instead

std::shared_ptr<void*> would be the better to use in this case since we're using C++ and some people would be more accustomed to it, though that, again, would create problems in your mostly C environment....

Isn't this limiting the usage of the attribute? Ie, with a plain void* you could store a size_t value, using shared_ptr<void*> means you need to allocate a pointer to a size_t so that it can be freed when the Node is deleted

[spnda]

Isn't this limiting the usage of the attribute? Ie, with a plain void* you could store a size_t value, using shared_ptr<void*> means you need to allocate a pointer to a size_t so that it can be freed when the Node is deleted

Well, kinda, yes. I just feel like a shared_ptr/unique_ptr is more fitting for this use case, I don't think it would be a common use case but it is something to think about. I think this also kinda shows why this is a bigger problem, and why void* isn't perfect, either.