Placement of objects in AR mode

[elbobo]

Hi,

This is more of a query than a bug and I understand it relates to both quick look and scene-viewer rather than model-viewer itself but am posting here in the hope that someone can enlighten me.

I'm trying to understand how models are placed in space when in AR mode. It seems to me from testing that once the plane is detected, the model is placed directly on the plane at a point as though a straight line is drawn directly from the camera view to the plane.

Assuming that's correct, that makes sense. What is a little bit more baffling is that IOS and android seem to handle this slightly differently. IOS seems to place the model so that the origin point of the model matches the anchor point of the viewer. Android on the other hand, seems to match the front edge of the model's bounding box to the anchor point. Is that what others experience?

To clarify what I think is going on to myself (and to help get some advice) I've made a diagram.

The difference is not ideal but as long as its consistent that's fine. My final query is one that I can't find any logic to. Sometimes when models contain animation, they are elevated above the ground level even though they exist in the original file, at same height as a static version. This seems to happen more in IOS than android but as I say there is no consistency to it. You can see an example of what I mean by that here https://ninth-worried-mice.glitch.me/ you can see the cube is elevated into the air even though it is on the groun when in model-viewer (and in the original file).

If anyone could clarify any of the above for me it would be much appreciated.

Great work on the project

Testing on
Safari, iPhone XR, IOS 13.3
Chrome, firefox, Pixel 3, Android 9

Versions

Latest version of model viewer

[cdata]

Hi @elbobo thanks for the thoughtful investigation and the inquiry.

This project is not the authoritative source for Quicklook or Scene Viewer information. That said, I can try to offer some commentary about your investigation and its findings:

Scene Viewer and Quicklook are different native "apps" made by different teams at different companies. To the extent that they share anything in common at all, it is a coincidence due to the fact that their designers arrived at the same common sense conclusion about what makes a good user experience when placing objects in AR via a handheld device.

There is a ton of room for subjectivities here. One app may decide that the best way to place an object is based on the object's bounding box. Another app may decide to place objects based on their relative position in the model's root scene. Ultimately there is no canonical "spec" for how AR placement should work (perhaps we need one? 😅), so every app should be assumed to have come to its own conclusions about what makes for good UX.

As far as animations are concerned, I would drop this into the same bucket as above. There are probably articulable reasons why any given app does something peculiar when animations are playing, but those reasons will be specific to that app. I'll CC @tpsiaki who may be able to comment on any peculiarities related to Scene Viewer.

If you happen to have specific questions about <model-viewer>, I'm happy to answer those as well. We are about to embark on UX refinement for browser-based AR via WebXR, so this topic is top-of-mind for us right now. CC @elalish who will be spending time in this area soon.

[tpsiaki]

Hi @elbobo - you are correct regarding how the positioning of the models works in Scene Viewer. Most of the initial use cases for Scene Viewer involved placing an object in front of the user, starting with the animals in Google Search (some of which are quite large).

This behavior was what we decided on to avoid the experience where the user starts with the model placed in a way that will intersect with their phone.

I understand that for more immersive experiences this behavior might be undesirable, so if there are particular use cases where a different behavior would be helpful we'd love to hear about it.
We're happy to consider exposing alternative options if needed.

[elbobo]

Hi @tpsiaki thanks for the reply and insight, really helpful.

The project I was working on at the time was put on hold but is now alive again so it got me thinking.. In response to your point about considering different use cases I would like to suggest that if possible different model placement options be made available (I like the idea that the default is 'as is' but via model viewer you could perhaps pass an argument to scene-viewer for placement options) A couple of possibilities below which I think would cover a lot of cases (titles just for ref).

1. 'Immersive' To allow placement like quick look (the IOS drawing on my original diagram) so the origin is the centre of the model rather than the bounding box. As you say, the benefit of this would be more 'immersive' AR experiences where the user is dropped into the centre of a scene and has objects surrounding them, rather than initially being stood outside looking in until they walk inside.

2. 'Subterranean' As I understand it, models will always be rendered from the ground plane up, regardless of whether their origin has been set below zero in the creation tool. It would be great if it were possible to allow objects to sit beneath the plane. 2 reasons for this that AR is great for
   a) "making the invisible, visible": showing things beneath the surface is quite a common idea I have come across when people are thinking of a unique value of AR
   b) AR is great for showing scale, and allowing people to stand for example, on the top of something very high, or at the top of something very deep is another one of its unique benefits over other media.

I would be interested to hear if you had considered either of these as possibilities. I appreciate that webXR may be the solution to some of this but as it stands I still find the 'scene-viewer' and 'quick-look' route to be more reliable in terms of quality (graphics, persistence etc.)

Thanks

[elalish]

This is also a bit different in WebXR mode. Here the model can be turned before it is placed (if you move your phone a bit), so instead of anchoring on the middle of the forward edge of the bounding box, it uses the point where the view vector to the bounding box center crosses over the floor projection of the bounding box, but it amounts to the same thing if the object is facing the camera. In addition, the hit test ray I use to place the object is not the one from the center of the screen, but one pointed 20 degrees downward, which helps both with hitting a detected plane quicker and with placing the object so that it is better centered onscreen.

[hybridherbst]

I'd like to bump this issue up again @elalish . Especially with larger models the placement on Android WebXR is completely uncontrollable and leads to the desired scene just being placed very far away.

Basically everyone has to (when they know how to) drag the model closer again after initially placing it; we're seeing a large amount of users on Android drop out at this point in contrast to iOS where the placement is where expected.

Incidentally, I was just reminded by that from the 2.0 announcement video on Twitter which shows exactly that problem: https://twitter.com/modelviewer/status/1570064958909456385?t=vveUk9YjEcua2IKiw7RgZQ&s=19

Placement is much too far away and you're dragging the model closer again to the point where the pivot matches with the expected placement point.

[elalish]

Agreed, I noticed this too while filming the video, but decided not to hold the release for it. I'd love some UX feedback on how close is right. The way it works right now is that the model is placed at the same camera radius as was used in 3D when the button was pressed (note when you zoom in first it places closer in AR). However, it doesn't look right because the FoV of the 3D camera doesn't match the FoV of the physical camera.

I'm considering just re-framing it based on the physical camera's FoV and ignoring any orbit changes that have happened in 3D. I think my particular case of a tall model in a small room is going to place beyond the wall regardless, since it's just too cramped (seeing the inside of the model is so disconcerting I think it's a priority to avoid that). But it'll be better at least. Does that sound reasonable?

I also noticed another issue, which is that tap-recenter is somehow still happening in AR mode, so I need to disable that.

[hybridherbst]

If you want to check out a real-world example where this is an issue, you can look at a project we just launched: https://prefrontalcortex.de/stadtklimahalle/baumoase/ (there's 4 models you can click through)

These models are intended to be placed 1:1 on an open space. On iOS, that works pretty well - when users place them, the "center of attention" / "middle of the model" is placed where they click. On Android, it's placed super far away, due to what I think are bounding box calculations pushing it there.

UX-wise, I don't think the orientation a user has put on the model in the browser should actually affect the AR placement, at least not always - these are different media.

And to be absolutely honest here, I had always felt the AR placement of modelviewer to be pretty random and kind of chugged that off as the plane tracking on Android being usually worse - only with your explanation now I was able to test and understand that the user rotation and user scale has this direct influence on placement!
I assume that in return means that if we want control over the behaviour, we might be able to reset model rotation and scale to something desired right before launching AR mode... interesting :)

[hybridherbst]

Giving this a bit more thought: I understand why you chose this approach for relatively small e-commerce models that are supposed to be placed in 1:1. I don't think it works well for larger models or even arbitrary models that are allowed to get scaled.

There are a number of different solutions for model placing that I can think off:

1. keeping rotation and view and scale but changing position (the current behaviour)
2. keeping rotation and view and pivot but scaling the model down instead of pushing it far away (I believe 8th wall does that by default)
3. keeping rotation and pivot as user-chosen but resetting the view (what would be desired for our use case here).

Would it help if we make some small examples for each of those cases to show why this is a choice per model / per use case to make?
Things I can come up with:

• a 20m tall dinosaur (couple of different scenarios here) that I want to see shrinked down in AR
• a small insect that I want to see enlarged in AR
• a 1:1 model of a river that I want to stand next to in AR
• a house where I want to stand in front of the door in AR
• a bike
• and probably some more.

[elalish]

Thanks, this is super-helpful feedback! Regarding different scales of assets, I think about this a little differently - let me know if this resonates with you or not:

In 3D mode, the scale of the object doesn't matter (if you scale it, it looks identical, because we scale our default framing the same way). In AR, the scale is meaningful (and I would say generally should be fixed, to respect author intent). Part of the reason I added the scale attribute was to allow devs both an easy way to fix units (without having to re-author the glTF), but also for these "scale model" cases. I would say that a mosquito should be modeled not at its physical size of 3mm, but at its "display scale" size of, say 1m. Likewise with the dinosaur - maybe full size is fun, but for education, probably a 2m scale size is better. I'd prefer the author to control this via scale or the glTF itself, since it has no effect on 3D mode anyway.

The really different cases are the landscape-type things. Here you want to place yourself inside the model, so our framing goals of having the whole model in frame are totally pointless. I think our current framing method might be better for these, since by basing it on the current camera, if you've set it to be inside-out, I think it'll respect it. And yes, a demo proving/disproving this would be highly beneficial.

I'm hopeful we can thread the needle here and find something that works pretty well generally. Your input is much appreciated!