3D pose renders behind the camera (CV vs GL convention); matrixGL_RH throws

[kalwalt]

Summary

When a marker is successfully tracked, the getMarker event fires with a valid pose (real rotation + translation), but three.js examples still show no 3D object. The object is placed behind the camera and gets frustum-clipped.

Surfaced while implementing #30 (static-image Teblid example): the marker tracks correctly (Marker tracked ! Num. matches : 1000) and a found pose arrives, yet no sphere is visible.

Evidence (runtime)

Sample pose delivered to the scene (three.js column-major elements):

translation = (0.953, -0.384, +5.824)

Projection matrix from the same run (column-major):

[-1.785, 0, 0, 0,  0, 2.380, 0, 0,  0, 0, -1.0002, -1,  0, 0, -0.20002, 0]

This is a standard OpenGL projection (proj[11] = -1, camera looks down −Z). Plugging the object origin (0.953, -0.384, 5.824, 1) through it:

• clip.w = -view.z = -5.824 (negative → behind camera)
• ndc.z ≈ 1.03 (> 1 → beyond far plane)

So the object is clipped and never rasterized, despite correct tracking.

Root cause

The pose field (from getPoseMatrix() → transMatToGLMat()) is in OpenCV camera convention (object in front ⇒ +Z), which is incompatible with the OpenGL projection used for rendering (object in front ⇒ −Z). The right-handed GL matrix matrixGL_RH (arglCameraViewRHf) — the one that should be fed to three.js — currently throws an exception (see recent dev commits experimenting with the pose matrix).

Suggested direction

• Provide a correct GL modelview/pose matrix (apply the full CV→GL handedness flip: negate Y and Z appropriately on both rotation and translation), or fix arglCameraViewRHf / matrixGL_RH so it no longer throws and can be used directly.
• Quick verification: negating the translation Z on the pose before setMatrix(root.matrix, ...) should bring the object into view — useful as a confirmation while the proper conversion is sorted out.

Notes

• The fix likely lives in the emscripten/WebARKitLib submodule (pose conversion: invertPose/cvToGl/arglCameraViewRHf) and would require rebuilding dist/WebARKit.js.
• Affects all three.js worker examples, not just the static-image one from (feat): static image example with Teblid tracker #30.
• Tracking itself is working; this is purely a pose-convention / rendering issue.

Related latent bug (already worked around in #30)

WebARKitTracker::processFrame calls GetTrackedFeaturesWarped() when a marker is detected on the very first frame (_frameCount == 0), before GetInitialFeatures() has populated the tracked-feature set. perspectiveTransform then runs on an empty point set and throws (CV_Assert(scn + 1 == m.cols)). The #30 static example works around this by feeding one blank warmup frame, but the library should guard the pose block against an empty feature set.

[kalwalt]

Filed upstream against the library where the fix belongs: webarkit/WebARKitLib#34 (code is on the dev branch).

It covers both the pose-convention problem (tracked object renders behind the camera; matrixGL_RH throws) and the related first-frame perspectiveTransform crash, with the runtime evidence from here.