Integrate ARToolKit WASM detector with ImageBitmap→OffscreenCanvas→detector pipeline

[Copilot]

Implements real-time marker detection pipeline in Web Worker using ARToolKit WASM. Transfers ImageBitmap from main thread, processes via OffscreenCanvas, runs detection, emits structured results.

Worker Pipeline

New: src/worker/artoolkit/loader.js

• ARToolKit WASM detector interface with defensive fallback
• Gracefully degrades to stub mode when WASM unavailable
• Returns { id, confidence, poseMatrix, corners } per detection

Modified: src/worker/worker.js

// Initialize detector on worker startup
detector = await createDetector();

// Process ImageBitmap → ImageData via OffscreenCanvas
const imageData = processImageBitmap(imageBitmap);
const detections = detector.detect(imageData);

// Always cleanup transferred ImageBitmap
imageBitmap.close();

Key behaviors:

• OffscreenCanvas created once, reused per frame
• ImageBitmap properly closed after processing
• Console logs detection events in worker context
• Node.js worker_threads compatibility maintained

Plugin Updates

Modified: src/plugin.js

• Added console logging for detection results in main thread
• Logs marker found events with ID and confidence

Live Example

New: examples/simple-marker/

• Webcam capture with createImageBitmap() from video
• Frame submission via engine eventBus with ImageBitmap transfer
• Status monitoring (camera, plugin, worker, frame/marker counts)
• Live console output display

Documentation

Modified: README.md

• WASM setup options: npm package, manual binary, or CDN
• Event handling patterns and API reference
• Example usage with code snippets
• Detection pipeline architecture diagram

WASM Binary Required

⚠️ WASM binary not included. To enable real detection:

npm install artoolkit5-js
# or place artoolkit.wasm in src/worker/artoolkit/

Without WASM: worker initializes successfully (with warnings), returns empty detections, all APIs functional.

Detection Flow

Video → createImageBitmap() → [transfer] → Worker: OffscreenCanvas → ImageData → 
ARToolKit WASM → postMessage → Plugin → emit(ar:markerFound/Updated/Lost)

Features:

• Zero-copy ImageBitmap transfer (neutered after postMessage)
• OffscreenCanvas reused across frames (not recreated)
• Detection scope: simple square markers only (no NFT)
• Cross-platform: browser Worker + Node worker_threads

Original prompt

Integrate a real ARToolKit detector (artoolkit5-js WASM) into the worker and wire ImageBitmap → OffscreenCanvas → detector pipeline for simple square markers (no NFT). Create a new feature branch and open a draft PR.

Requirements (high level):

• Add artoolkit5-js (or minimal WASM loader) as a dependency or vendor the runtime inside the worker. Use a lightweight approach: include a small loader that fetches a wasm binary from an npm package or a CDN. If artoolkit5-js is not available as a direct npm package, vendor a minimal wasm + JS init stub under src/worker/artoolkit/ with a placeholder loader that expects wasm file at ./artoolkit.wasm. Document in README that wasm file must be provided or install steps.
• Update src/worker/worker.js to initialize the WASM detector at worker startup (on 'init'). Use top-level async or module imports as needed (worker is loaded as module). The worker must:
  • create an OffscreenCanvas with width/height set when first frame arrives (or during init if options provided).
  • On receiving processFrame with payload.imageBitmap (browser path): draw the ImageBitmap into OffscreenCanvas, get ImageData (or pass canvas directly to the detector API), run detection on the image.
  • For Node mode (worker_threads) keep previous stub behavior (still work with frameId).
  • Only support simple markers: use marker detection API of artoolkit to detect square markers and return marker id and pose matrix if found.
  • Emit detectionResult messages with detections array: { id, confidence, poseMatrix (array), corners }.
  • Log to worker console the detection events (console.log) so they appear in browser/devtools or Node worker stdout.
• Ensure worker cleans up ImageBitmap.close() and reuses OffscreenCanvas to avoid recreation per frame.
• Update src/plugin.js if needed to forward imageBitmap, and ensure plugin logs detection events to main console upon receipt (already emits events; add console.log in _onWorkerMessage for detectionResult to make it visible in main thread logs).
• Add a small example page in examples/simple-marker/ that demonstrates using the plugin in a browser with a webcam: captures frames with createImageBitmap and posts to the plugin via engine.eventBus. The page should load the plugin from src (no bundler), initialize plugin, and log detections to console.

Implementation notes and constraints:

• Do not implement NFT markers — only simple square markers (pattern markers). Use existing artoolkit detection APIs for marker detection.
• Keep the integration defensive: if wasm fails to load, worker should fall back to the stub and emit an error message to main thread.
• Keep changes minimally invasive: add worker/artoolkit loader files under src/worker/, update worker.js to use it, add example files under examples/simple-marker/ (index.html and script module). Update README.md with instructions to provide the wasm file and run the example.
• Add unit/integration tests are optional — focus on runnable example and console output.

Deliverables (what the PR should contain):

• New branch copilot/integrate-artoolkit-wasm
• Worker changes implementing wasm initialization and frame processing pipeline
• artotoolkit loader stub under src/worker/artoolkit/ (loader.js + README note or placeholder wasm)
• examples/simple-marker/ example page demonstrating webcam → ImageBitmap → plugin → worker → console logs
• README updates describing how to obtain/place the wasm file and run the example locally (using a simple static server)
• Open a draft pull request against main with a clear description of changes and any manual setup steps.

Make sure the PR is draft and titled as the problem_title. Include links to changed files in the PR body and note that the wasm binary must be added to run real detection.

Run any repo-file checks needed to ensure paths are correct. Use the existing branch copilot/scaffold-artoolkit as the base for the new branch.

Do not include NFT marker support. Make detection console output visible both in worker console and main thread console when detections occur.

This pull request was created as a result of the following prompt from Copilot chat.

Integrate a real ARToolKit detector (artoolkit5-js WASM) into the worker and wire ImageBitmap → OffscreenCanvas → detector pipeline for simple square markers (no NFT). Create a new feature branch and open a draft PR.

Requirements (high level):

• Add artoolkit5-js (or minimal WASM loader) as a dependency or vendor the runtime inside the worker. Use a lightweight approach: include a small loader that fetches a wasm binary from an npm package or a CDN. If artoolkit5-js is not available as a direct npm package, vendor a minimal wasm + JS init stub under src/worker/artoolkit/ with a placeholder loader that expects wasm file at ./artoolkit.wasm. Document in README that wasm file must be provided or install steps.
• Update src/worker/worker.js to initialize the WASM detector at worker startup (on 'init'). Use top-level async or module imports as needed (worker is loaded as module). The worker must:
  • create an OffscreenCanvas with width/height set when first frame arrives (or during init if options provided).
  • On receiving processFrame with payload.imageBitmap (browser path): draw the ImageBitmap into OffscreenCanvas, get ImageData (or pass canvas directly to the detector API), run detection on the image.
  • For Node mode (worker_threads) keep previous stub behavior (still work with frameId).
  • Only support simple markers: use marker detection API of artoolkit to detect square markers and return marker id and pose matrix if found.
  • Emit detectionResult messages with detections array: { id, confidence, poseMatrix (array), corners }.
  • Log to worker console the detection events (console.log) so they appear in browser/devtools or Node worker stdout.
• Ensure worker cleans up ImageBitmap.close() and reuses OffscreenCanvas to avoid recreation per frame.
• Update src/plugin.js if needed to forward imageBitmap, and ensure plugin logs detection events to main console upon receipt (already emits events; add console.log in _onWorkerMessage for detectionResult to make it visible in main thread logs).
• Add a small example page in examples/simple-marker/ that demonstrates using the plugin in a browser with a webcam: captures frames with createImageBitmap and posts to the plugin via engine.eventBus. The page should load the plugin from src (no bundler), initialize plugin, and log detections to console.

Implementation notes and constraints:

• Do not implement NFT markers — only simple square markers (pattern markers). Use existing artoolkit detection APIs for marker detection.
• Keep the integration defensive: if wasm fails to load, worker should fall back to the stub and emit an error message to main thread.
• Keep changes minimally invasive: add worker/artoolkit loader files under src/worker/, update worker.js to use it, add example files under examples/simple-marker/ (index.html and script module). Update README.md with instructions to provide the wasm file and run the example.
• Add unit/integration tests are optional — focus on runnable example and console output.

Deliverables (what the PR should contain):

• New branch copilot/integrate-artoolkit-wasm
• Worker changes implementing wasm initialization and frame processing pipeline
• artotoolkit loader stub under src/worker/artoolkit/ (loader.js + README note or placeholder wasm)
• examples/simple-marker/ example page demonstrating webcam → ImageBitmap → plugin → worker → console logs
• README updates describing how to obtain/place the wasm file and run the example locally (using a simple static server)
• Open a draft pull request against main with a clear description of changes and any manual setup steps.

Make sure the PR is draft and titled as the problem_title. Include links to changed files in the PR body and note that the wasm binary must be added to run real detection.

Run any repo-file checks needed to ensure paths are correct. Use the existing branch copilot/scaffold-artoolkit as the base for the new branch.

Do not include NFT marker support. Make detection console output visible both in worker console and main thread console when detections occur.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[Copilot]

Pull Request Overview

This PR transforms the ARToolKit plugin from a simple stub into a fully functional AR marker detection system. It integrates ARToolKit WASM in a Web Worker, implements pattern marker loading, and adds comprehensive documentation and examples.

Key changes:

• Integrates ARToolKit5-js WASM library for real marker detection in the worker
• Implements pattern marker loading API with promise-based request/response handling
• Adds comprehensive documentation with setup instructions and examples
• Creates a complete working example demonstrating webcam-based marker detection

Reviewed Changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/worker/worker.js	Transforms worker stub into full ARToolKit integration with marker loading, filtering, and frame processing
src/worker/artoolkit/loader.js	Adds ARToolKit WASM loader stub for future binary integration
src/worker/artoolkit/README.md	Documents ARToolKit WASM setup options and fallback behavior
src/plugin.js	Adds loadMarker API with promise-based worker communication and getMarker event forwarding
package.json	Adds @ar-js-org/artoolkit5-js dependency
package-lock.json	Locks dependency versions including artoolkit5-js and its transitive dependencies
examples/simple-marker/script.js	Creates comprehensive webcam-based marker detection example
examples/simple-marker/index.html	Provides interactive demo UI for marker loading
examples/simple-marker/data/patt.hiro	Adds Hiro pattern file for marker detection
examples/simple-marker/data/camera_para.dat	Adds ARToolKit camera calibration data
examples/simple-marker/README.md	Documents example setup and usage instructions
README.md	Expands documentation with features, API reference, architecture, and troubleshooting

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.