Crash: perspectiveTransform on empty point set when marker detected on the first frame

[kalwalt]

Summary

WebARKitTracker::WebARKitTrackerImpl::processFrame throws an uncaught C++/OpenCV exception when a marker is detected on the very first processed frame (_frameCount == 0). The pose block calls TrackingPointSelector::GetTrackedFeaturesWarped() before the tracked-feature selection has been populated, so cv::perspectiveTransform runs on an empty point set and asserts.

Root cause

In WebARKit/WebARKitTrackers/WebARKitOpticalTracking/WebARKitTracker.cpp::processFrame:

• The optical-flow branch is guarded by if (_frameCount > 0 && _prevPyramid.size() > 0) (~line 371). The only call to GetInitialFeatures() — which fills _selectedPts — lives inside that branch (~line 374).
• On the first frame _frameCount == 0, so that branch is skipped and _selectedPts stays empty.
• But MatchFeatures can still set _isDetected = true on that same first frame.
• The pose block if (_isDetected || _isTracking) (~line 389) then calls GetTrackedFeaturesWarped() (~line 391):

std::vector<cv::Point2f> GetTrackedFeaturesWarped() {
    std::vector<cv::Point2f> selectedPoints = GetTrackedFeatures(); // empty
    std::vector<cv::Point2f> warpedPoints;
    perspectiveTransform(selectedPoints, warpedPoints, _homography); // throws
    return warpedPoints;
}

With an empty input, OpenCV's perspectiveTransform hits CV_Assert(scn + 1 == m.cols) — the empty vector yields a default 1-channel Mat (scn = 1) against a 3×3 homography (m.cols = 3), i.e. 2 == 3 → exception. Stack (WASM):

cv::error -> cv::perspectiveTransform -> TrackingPointSelector::GetTrackedFeaturesWarped()
          -> WebARKit::processFrame(...)

Why it's normally masked

With a live camera, detection essentially never succeeds on the literal first processed frame, so _frameCount has already advanced past 0 by the time _isDetected becomes true. A static image detects immediately on frame 0 and triggers this deterministically (surfaced in the webarkit-testing static Teblid example).

Suggested fix (library)

Guard the pose block against an empty tracked-feature set, e.g.:

if (_isDetected || _isTracking) {
    std::vector<cv::Point2f> imgPoints = _trackSelection.GetTrackedFeaturesWarped();
    std::vector<cv::Point3f> objPoints = _trackSelection.GetTrackedFeatures3d();
    if (imgPoints.size() >= 4 && objPoints.size() == imgPoints.size()) {
        // ... cameraPoseFromPoints / pose update ...
    }
}

or make GetTrackedFeaturesWarped() (and GetTrackedFeatures3d()) return early when _selectedPts is empty, and/or populate the selection on first detection so frame 0 has a valid set. Either avoids calling perspectiveTransform on empty input.

Current workaround (downstream)

The webarkit-testing static-image example feeds one blank warmup frame before the real image, so detection cannot succeed on _frameCount == 0. That sidesteps the crash but the proper guard belongs in the library.

Repro

webarkit-testing threejs_teblid_static_image_ES6_example.html without the warmup frame: the tracker detects the marker on the first processed frame and processFrame throws.

Refs: webarkit/webarkit-testing#30, #31