Integrate ARToolKit WASM detector with ImageBitmap→OffscreenCanvas→detector pipeline (#4)

* scaffold ARToolKit plugin with initial setup and basic functionality (#1)

* scaffold ARToolKit plugin with initial setup and basic functionality

* feat(worker): add detection worker stub and wire to plugin

* fix(plugin): restore ArtoolkitPlugin class and integrate worker lifecycle

* scaffold ARToolKit plugin with initial setup and basic functionality
- missed plugin.js!

* feat(worker): enhance cross-platform support for worker lifecycle and messaging

* feat(plugin): enhance ImageBitmap handling and add browser smoke test

* feat(plugin): enhance marker tracking configuration and sweep logic

* Initial plan

* Implement ARToolKit WASM integration with worker pipeline

Co-authored-by: kalwalt <[email protected]>

* feat(worker): integrate @ar-js-org/artoolkit5-js for marker detection in browser and Node

* Initial plan

* feat(worker): add pattern marker loading support with worker integration

- Add loadMarker message handling in worker with ARController simulation
- Add plugin.loadMarker() method with Promise-based API
- Add logging for detectionResult and loadMarkerResult messages
- Create examples/simple-marker with index.html, patt.hiro, and README
- Support both browser and Node worker environments

Co-authored-by: kalwalt <[email protected]>

* fix(worker): prevent CORS error by checking environment before importing node:worker_threads

Only attempt to import node:worker_threads when in Node.js environment (typeof self === 'undefined').
In browser environments, skip the import entirely to avoid CORS policy violations.

Co-authored-by: kalwalt <[email protected]>

* feat: update index.html for ARToolKit plugin integration with improved UI and event handling

* feat(plugin): forward worker getMarker events over eventBus

* feat(worker): forward ARToolKit getMarker events from worker to main thread

* feat: integrate ARToolKit with worker for enhanced marker detection and processing

* feat: enhance ARToolKit integration with module URL support and improved initialization handling

* feat: add camera parameters URL support and enhance marker filtering in ARToolKit integration

* Update src/worker/worker.js

- The context hint should be willReadFrequently: true since getImageData is called on line 291 as a fallback. The 'false' value may hurt performance when ImageData is actually needed.

Co-authored-by: Copilot <[email protected]>

* remove not needed line

Co-authored-by: Copilot <[email protected]>

* [nitpick] Corrected spelling of 'doctype' to 'DOCTYPE'. While lowercase is valid HTML5, uppercase is the conventional standard.

Co-authored-by: Copilot <[email protected]>

* Return statement in finally block will override any thrown exceptions from the catch block, preventing proper error propagation. Remove the return statement from the finally block

Co-authored-by: Copilot <[email protected]>

* docs: update README for ARToolKit integration, clarify serving requirements and usage instructions

---------

Co-authored-by: Walter Perdan <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: kalwalt <[email protected]>
Co-authored-by: Copilot <[email protected]>

Author: 3 people

README.md

@@ -1,24 +1,169 @@
-# arjs-plugin-artoolkit
-```markdown
 # @ar-js-org/arjs-plugin-artoolkit
 
-Minimal ARToolKit detection plugin scaffold for AR.js core.
+ARToolKit marker detection plugin for AR.js core with WebAssembly support.
+
+## Features
+
+- Web Worker-based detection — marker detection runs off the main thread
+- ImageBitmap support — zero-copy frame transfer (browser)
+- Cross-platform — browser Workers and Node.js worker_threads (stub path)
+- ARToolKit integration — square pattern markers
+- Event-driven API — marker found/updated/lost + raw getMarker forwarding
+- Filtering — only forwards PATTERN_MARKER events above a minimum confidence
+
+## Installation
+
+```bash
+npm install @ar-js-org/arjs-plugin-artoolkit
+```
+
+## Configuration (module + assets)
+
+To ensure the Worker can import ARToolKit in the browser, pass an explicit ESM URL (recommended):
+
+```js
+const plugin = new ArtoolkitPlugin({
+  worker: true,
+  artoolkitModuleUrl: '/node_modules/@ar-js-org/artoolkit5-js/dist/ARToolkit.js', // explicit ESM
+  cameraParametersUrl: '/path/to/camera_para.dat',                                // optional override
+  wasmBaseUrl: '/node_modules/@ar-js-org/artoolkit5-js/dist/',                    // optional; if your build needs it
+  minConfidence: 0.6                                                              // forward only confident detections
+});
+```
+
+CDN fallback (if you don’t serve node_modules):
+- Set `artoolkitModuleUrl` to a CDN ESM endpoint, e.g. a jsDelivr/UNPKG URL for the package’s ESM bundle.
+
+Notes:
+- The previous “loader.js” and manual WASM placement flow is no longer used.
+- If your ARToolKit build fetches auxiliary assets (WASM, data), set `wasmBaseUrl` accordingly.
 
 ## Usage
 
-Register with the Engine plugin manager:
+### Register and enable
 
 ```js
 import { ArtoolkitPlugin } from '@ar-js-org/arjs-plugin-artoolkit';
 
-engine.pluginManager.register('artoolkit', new ArtoolkitPlugin({ /* options */ }));
+const plugin = new ArtoolkitPlugin({
+  worker: true,
+  lostThreshold: 5,       // frames before a marker is considered lost
+  frameDurationMs: 100,   // expected ms per frame (affects lost timing)
+  artoolkitModuleUrl: '/node_modules/@ar-js-org/artoolkit5-js/dist/ARToolkit.js',
+  cameraParametersUrl: '/data/camera_para.dat',
+  minConfidence: 0.6
+});
+
+engine.pluginManager.register('artoolkit', plugin);
 await engine.pluginManager.enable('artoolkit');
 ```
 
-The plugin emits events on the engine event bus:
-- `ar:markerFound`
-- `ar:markerUpdated`
-- `ar:markerLost`
+### Events
+
+The plugin emits the following events on your engine’s event bus:
+
+```js
+// Marker first detected
+engine.eventBus.on('ar:markerFound', ({ id, poseMatrix, confidence, corners }) => {
+  // poseMatrix is Float32Array(16)
+});
+
+// Marker updated (tracking)
+engine.eventBus.on('ar:markerUpdated', (data) => {
+  // same shape as markerFound
+});
+
+// Marker lost
+engine.eventBus.on('ar:markerLost', ({ id }) => {});
+
+// Worker lifecycle
+engine.eventBus.on('ar:workerReady', () => {});
+engine.eventBus.on('ar:workerError', (error) => {});
+
+// Raw ARToolKit getMarker (filtered: PATTERN_MARKER only, above minConfidence)
+engine.eventBus.on('ar:getMarker', (payload) => {
+  // payload = { type, matrix: number[16], marker: { idPatt, cfPatt, idMatrix?, cfMatrix?, vertex? } }
+});
+```
+
+### Sending frames
+
+```js
+// Create ImageBitmap from a <video> or <canvas>
+const imageBitmap = await createImageBitmap(video);
+
+// Emit an engine update; the plugin transfers the ImageBitmap to the worker
+engine.eventBus.emit('engine:update', {
+  id: frameId,
+  timestamp: Date.now(),
+  imageBitmap,
+  width: imageBitmap.width,
+  height: imageBitmap.height
+});
+
+// The ImageBitmap is transferred and cannot be reused; the worker will close it.
+```
+
+### Loading a pattern marker
+
+```js
+const { markerId, size } = await plugin.loadMarker('/examples/simple-marker/data/patt.hiro', 1);
+```
+
+## Examples
+
+A complete webcam-based example is available under `examples/simple-marker/`.
+
+Serve from the repository root so that ES modules and node_modules paths resolve:
+
+```bash
+# From repository root
+npx http-server -p 8080
+# or
+python3 -m http.server 8080
+```
+
+Open:
+- http://localhost:8080/examples/simple-marker/index.html
+
+The example demonstrates:
+- Webcam capture with getUserMedia
+- ImageBitmap creation and frame submission
+- Event handling and console output
+- Raw `ar:getMarker` payloads for debugging
+
+## API Reference
+
+### ArtoolkitPlugin options
+
+```ts
+{
+  worker?: boolean;            // Enable worker (default: true)
+  lostThreshold?: number;      // Frames before 'lost' (default: 5)
+  frameDurationMs?: number;    // ms per frame (default: 200)
+  sweepIntervalMs?: number;    // Lost-sweep interval (default: 100)
+  artoolkitModuleUrl?: string; // ESM URL for ARToolKit (recommended)
+  cameraParametersUrl?: string;// Camera params file URL
+  wasmBaseUrl?: string;        // Base URL for ARToolKit assets (if required)
+  minConfidence?: number;      // Minimum confidence to forward getMarker (default: 0.6)
+}
+```
+
+### Methods
+
+- `async init(core)` — initialize with engine core
+- `async enable()` — start worker and subscribe to frames
+- `async disable()` — stop worker and timers
+- `dispose()` — alias for disable
+- `getMarkerState(markerId)` — current tracked state
+- `async loadMarker(patternUrl: string, size = 1)` — load and track a pattern
+
+## Troubleshooting
 
-This repo contains a skeleton. Detection and worker/WASM integration will be implemented in follow-up work.
-```
+- “Failed to resolve module specifier” in the Worker:
+    - Provide `artoolkitModuleUrl` or serve `/node_modules` from your dev server
+- Worker not starting:
+    - Serve via HTTP/HTTPS; ensure ES modules and Workers are supported
+- No detections:
+    - Confirm camera started, correct marker pattern, sufficient lighting
+    - Adjust `minConfidence` to reduce/raise filtering

examples/simple-marker/README.md

@@ -0,0 +1,132 @@
+# Simple Marker Example
+
+This example demonstrates how to load and track a pattern marker using the ARToolKit plugin.
+
+## Setup Instructions
+
+### 1. Install Dependencies
+
+From the repository root, install the dependencies:
+
+```bash
+npm install
+```
+
+### 2. Serve the Example
+
+You must serve from the repository root so that:
+- ES modules resolve (../../src/plugin.js)
+- The worker module URL (../../node_modules/...) is reachable
+
+You can use any static file server. Examples:
+
+#### Option A: Using Python
+
+```bash
+# From repository root
+python3 -m http.server 8080
+```
+
+Then open: http://localhost:8080/examples/simple-marker/index.html
+
+#### Option B: Using Node.js http-server
+
+```bash
+# Install http-server globally if not already installed
+npm install -g http-server
+
+# From repository root
+http-server -p 8080
+```
+
+Then open: http://localhost:8080/examples/simple-marker/index.html
+
+#### Option C: Using VS Code Live Server
+
+If you're using VS Code with the Live Server extension:
+1. Right-click on `examples/simple-marker/index.html`
+2. Select "Open with Live Server"
+
+### 3. Using the Example
+
+1. Wait for the worker to be ready (status will change to “Worker ready”)
+2. Click “Start Camera” to begin sending frames
+3. Click “Load Marker” to load the Hiro pattern marker
+4. Show the marker to the camera and watch the event log and console
+
+## Module resolution
+
+The example config (in `index.html`) passes explicit URLs so the worker can import ARToolKit and camera params:
+
+```js
+const plugin = new ArtoolkitPlugin({
+  worker: true,
+  artoolkitModuleUrl: '/node_modules/@ar-js-org/artoolkit5-js/dist/ARToolkit.js',
+  cameraParametersUrl: '/examples/simple-marker/data/camera_para.dat'
+});
+```
+
+If your server can’t serve `/node_modules`, either:
+- Adjust `artoolkitModuleUrl` to a path your server exposes, or
+- Use a CDN ESM URL as a fallback (see project README for details)
+
+## What’s Happening
+
+This example demonstrates:
+
+1. Plugin Initialization: creating and enabling `ArtoolkitPlugin`
+2. Worker Communication: the plugin starts a Worker for detection
+3. Pattern Loading: `plugin.loadMarker('/examples/simple-marker/data/patt.hiro', 1)`
+4. Event Handling:
+    - `ar:workerReady` — Worker initialized
+    - `ar:markerFound` — First detection of a marker
+    - `ar:markerUpdated` — Subsequent tracking updates
+    - `ar:markerLost` — Marker no longer visible
+    - `ar:getMarker` — Raw ARToolKit getMarker payload (type, matrix, marker fields)
+
+## Pattern File
+
+The `data/patt.hiro` file is a standard ARToolKit pattern. You can replace it with your own pattern and update the URL accordingly.
+
+## Code Overview
+
+Key parts of the example:
+
+```javascript
+// Create plugin instance with worker enabled and explicit module/params URLs
+const plugin = new ArtoolkitPlugin({
+  worker: true,
+  artoolkitModuleUrl: '/node_modules/@ar-js-org/artoolkit5-js/dist/ARToolkit.js',
+  cameraParametersUrl: '/examples/simple-marker/data/camera_para.dat'
+});
+
+// Initialize and enable
+await plugin.init(core);
+await plugin.enable();
+
+// Load a pattern marker
+const result = await plugin.loadMarker('/examples/simple-marker/data/patt.hiro', 1);
+console.log(`Marker loaded with ID: ${result.markerId}`);
+```
+
+## Troubleshooting
+
+- Worker not loading?
+    - Ensure you’re serving via HTTP/HTTPS from the repository root (not `file://`)
+    - Check console for module resolution/CORS errors
+- Module import errors?
+    - Make sure `/node_modules/@ar-js-org/artoolkit5-js/dist/ARToolkit.js` is reachable, or use a CDN URL
+- Marker not loading?
+    - Verify the pattern file path is correct and accessible
+    - Ensure the worker is ready before calling `loadMarker()`
+- No detections?
+    - Click “Start Camera” before “Load Marker”
+    - Ensure good lighting and the correct marker
+    - Increase confidence tolerance if needed (see README options)
+
+## Browser Support
+
+This example requires:
+- ES modules
+- Web Workers
+- Modern browser (Chrome 80+, Firefox 75+, Safari 13.1+, Edge 80+)