Transform annotation/hotpot positions alongside model-viewer.orientation changes

[rmhall]

Loving this library - amazing work. I've been using it extensively for a couple projects and thanks to these discussions I've been able to pull off pretty much everything I have needed to date.

I was curious to see if anyone had suggestions on the most optimal way to accomplish a specific task related to annotation positioning when altering the orientation of a model.

I am displaying a .glb model of a hardware device (a tablet with a display) that has a 16:9 orientation. The app allows users to apply their own textures to the screen through various means. Since it's a screen users may want to see that screen in landscape or portrait orientation depending on their content. I currently allow users to do just that by rotating the model in the view by animating the "orientation" property after they have double clicked or invoked/requested the app to to toggle between orientations of 0deg and 90deg. That all works great and its straightforward to animate that property.

The issue arises if they also have annotations/hotspots turned on as visible, as those are initially defined based on the models default state of landscape orientation. If I animate that orientation value like:

modelViewer.orientation = angle_animating_between_0_and_90+'deg 0deg 0deg';

The hotspots are still tied to the original orientation and thus are in the wrong place once the model orientation is rotating and completed rotating - they do not travel/sync with the models orientation.

I've attached a video illustrating the concept and core issue using our astronaut friend:

moonman_rotate.mp4

And here is a simple reduced test case using jQuery and GreenSock to do exactly what I've described above and in the video to illustrate how changing the orientation of the model moves the model, but the hotspot positions do not sync along with that orientation change:

const mv = document.querySelector('model-viewer');
	var anim;
	function rotate() {
		anim = gsap.to(mv, {orientation: "90deg 0deg 0deg", duration: 1, repeat: 100, yoyo: true, onUpdate: function() {
			mv.updateFraming();
		}});
	}
	var rotateOn = false;
	var singleton = true;
	$(document).on("click", function(event) {
		rotateOn = !rotateOn;
		if(rotateOn) {
			if(singleton) {
				rotate();
				singleton=false;
			} else {
				anim.play();
			}			
		} else {
			anim.pause();
		}
	});

I could define another set of the same hotspots with the coordinates tied to either the 0degree or 90degree orientation (portrait vs landscape) and then turn them off while things are animating and simply turn on the second set of hotspots that map to the new orientation angle, after the rotation occurs, but that requires the creation/maintenance of two variations of that data and simply wont look as nice as it would if the hotspots/annotations were also animating their positions over time with the rest of the rotation of the models orientation. It also would be limited to those two discrete orientations and not dynamic orientation changes, so not really a worthwhile path to pursue in my mind.

Ideally I am looking for a way to leverage the updateHotspot method to tailor the hotspots position and normal values (and possibly even the orbit and target values for the hotspots too) dynamically in realtime (through some sort of transformation) at the same time I am changing the main model orientation values, so that they can transform and animate right along with the degrees of the orientation and resulting animation of the model.

I was hoping someone may have some guidance on a good approach to what kind of transformation I could apply to tackle this, or perhaps something I've overlooked that is already baked in that would do this automatically when changing the models orientation to keep the hotspot positions in sync.

[elalish]

I would say this is working as intended, but you're referring to a feature request that's been on the back burner for some time: animated hotspots. You're referring specifically to rigid-body animation, but really the same problem applies to skinned meshes, where it would be nice to define a hotspot such that it could follow e.g. the hoof of a galloping horse. The only trick is we need a different way to define the coordinates, since they're no longer simply in the world coordinate system.

If you're interested in trying to solve this, a PR would be most welcome.

[rmhall]

Hi @elalish Thanks, and yes, absolutely the example is working as intended. My apologies didn't realize there was already a feature request for animated hotspots. I didn't actually see any issues or tickets on that topic, but of course actually just found this one after I posted: #1053

And that is definitely in the same vein, wether it's an animated model or just animating properties of the model, the end goal is the same, anchoring the hotspots. Reading that ticket, I guess my main question, since that was about two years ago now, has support for barycentric coordinates been added in or exposed yet? Would love to see an example on how @fivelive accomplished his barycentric coordinate calculations. I may need to go see if there is anything from the underlying three.js that I could tap into or figure out a good way to transform all those external coordinates to keep them in sync with changes to the model. I suppose I'll be looking at this a bit more deeply. I do already see that you can retrieve barycentric coordinates from three.js: https://threejs.org/docs/#api/en/math/Triangle.getBarycoord - so will read a bit more there...

In case anyone else is following along on this, I did also run across this, which gets to the heart of it with threejs and may be valuable as a starting point: https://stackoverflow.com/questions/63391138/how-to-create-hotspots-in-3d-enviroment-using-threejs and ultimately leads to this article too: https://manu.ninja/webgl-three-js-annotations/

If I end up solving this with anything novel or useful, even if it's just an example on how to go about it, I'd be happy to share and if it involves the library itself, happy to do a PR. Thanks @elalish, keep up the great work!

[elalish]

Yes, you're on exactly the right track. This shouldn't really be too hard to implement with three.js - we can also add the triangle/barycentric coordinates as extra output of positionAndNormalFromPoint like we added uv, to make these easier to select in our editor. The real question is what's a good way to define the point in a string attribute? Maybe something like "5 2734 0.5 0.3 0.2" meaning glTF mesh index 5, triangle index 2,734, barycentric coords: [0.5, 0.3, 0.2]? What if they don't sum to 1? Normalize? For hiding, use the triangle normal for direction? Or specify it somehow?

You're welcome to open a draft PR just to talk through the API before you start an implementation if you want.

[rmhall]

Great - thanks for the confirmation on the direction, appreciate it!

This is not necessarily my area of expertise, so I don't feel I can weigh in intelligently on your responses and questions on the best way to define those items, until I've done a bit more reviewing myself. However, off-hand that sounds like a great path to extend the results of calls to positionAndNormalFromPoint and a great jumping off point for me to explore.

Thanks for that and for the suggestion for a draft PR to discuss further. I feel I first need to spend some additional time looking more deeply at the current implementation controlling the hotspots to understand the current mechanics better, and then I can start digging into the above and maybe open a draft PR to take further. Thanks again @elalish - much appreciated!