feat: Improve QR scanning reliability for dark borderless codes#759
feat: Improve QR scanning reliability for dark borderless codes#759AntonioVentilii wants to merge 12 commits intomainfrom
Conversation
…k-borderless-codes
…k-borderless-codes
Use the BarcodeDetector API polyfill (backed by the same ZXing C++ WASM engine) instead of zxing-wasm directly. This simplifies the component by removing the hidden canvas — BarcodeDetector.detect() accepts the video element directly. Made-with: Cursor
…des (#12162) # Note This PR is a fork of dfinity/gix-components#759. Awaiting that review, we simply copy the components in our repo and add it to the next release, since it is a time-sensitive issue. # Motivation The QR code reader was using `html5-qrcode`, which has a known issue with inverted (dark-themed, borderless) QR codes on mobile — they simply cannot be scanned. The library is also unmaintained. Known upstream issues: - mebjas/html5-qrcode#766 - mebjas/html5-qrcode#468 - mebjas/html5-qrcode#94 Since the root cause is a limitation of `html5-qrcode` itself (no support for inverted QR codes), and the library is no longer maintained, this PR replaces it with [`barcode-detector`](https://www.npmjs.com/package/barcode-detector) — a [Barcode Detection API](https://developer.mozilla.org/en-US/docs/Web/API/BarcodeDetector) polyfill powered by ZXing C++ WebAssembly under the hood. The underlying engine natively supports inverted code detection (`tryInvert` is enabled by default). We tested it manually on a mobile device and dark borderless QR codes are now successfully scanned. Thanks to @sea-snake for the tip! # Changes (copied from gix-components's `QRCodeReader`) - Removed the `html5-qrcode` dependency entirely in component `QRCodeReader`. - In component `QRCodeReader`: - Camera access is now handled directly via the browser `navigator.mediaDevices.getUserMedia` API, requesting the rear-facing camera (`facingMode: "environment"`) at up to 1920×1080 resolution. - The video feed is rendered in a native `<video>` element. - QR decoding uses the standard `BarcodeDetector` API (via `barcode-detector/ponyfill`). A detector instance is created with `formats: ["qr_code"]` and `detector.detect(videoElement)` is called at ~10 fps via `setInterval(scan, 100)`. The polyfill accepts the `<video>` element directly — no intermediate `<canvas>` needed. - A frame-processing guard (`isProcessingFrame`) prevents overlapping decode calls. - Cleanup on destroy properly clears the scan interval and stops all media tracks. - The scan overlay is now implemented with a `.scan-overlay` container with a `.scan-region` box using `box-shadow` to mask the surrounding area and a border styled with the primary color. This replaces the old `:global(#qr-shaded-region)` overrides that targeted `html5-qrcode`'s internal DOM. - Updated the library reference in the QR code reader docs page from `jsQR` to `barcode-detector`. - Added `barcode-detector@^3.1.1` to dependencies and removed `html5-qrcode`. # Tests Manual testing on mobile: dark borderless QR codes are now successfully scanned (this was the main validation, as the core issue only reproduced on real mobile devices). <img width="1071" height="1428" alt="IMG_5315" src="https://github.com/user-attachments/assets/f25cbe60-3d92-472c-917e-ffdfb5f1b343" /> <img width="1179" height="2556" alt="IMG_5314" src="https://github.com/user-attachments/assets/6fd3d30d-2080-4464-8263-80ce3d544698" />
yhabib
left a comment
There was a problem hiding this comment.
Left some small questions/nits.
- Can you add the CSP comment?
- Do you know how this new dep. impacts the bundle size?
| .scan-region { | ||
| width: min(90cqw, 90cqh); | ||
| aspect-ratio: 1; | ||
| box-shadow: 0 0 0 9999px white; |
There was a problem hiding this comment.
[question] have you tested this with a dark theme? I don't think it adapts well 🤔
| const [qrResult] = results; | ||
|
|
||
| if (nonNullish(qrResult)) { | ||
| dispatch("nnsQRCode", qrResult.rawValue); |
There was a problem hiding this comment.
[question] Is it intended to keep pushing events even after successful one? should it not stop? do you know how it worked with the previous library?
| // Make sure the camera is stopped if the component is destroyed before | ||
| // scanning is starte. | ||
| await html5QrCode?.stop(); | ||
| if (nonNullish(videoElement)) { |
There was a problem hiding this comment.
[nit] I guess this check is purely TS, no? Otherwise why not stop the stream in the else statement?
| } | ||
| }; | ||
|
|
||
| scanInterval = setInterval(scan, 100); |
There was a problem hiding this comment.
[nit] setInterval(scan, 100) fires every 100ms regardless of how long detect() takes. isProcessingFrame guard prevents overlap, but frames will be skipped if detection is slow. Eg
Time 0ms: setInterval fires → scan() starts → detect() begins (takes ~150ms)
Time 100ms: setInterval fires → scan() starts → isProcessingFrame is true → early return
Time 200ms: setInterval fires → scan() starts → detect() just finished at ~150ms, so isProcessingFrame is false → detect() begins again
Time 300ms: setInterval fires → scan() starts → isProcessingFrame is true → early return
What about sth like:
const scan = async () => {
if (isDestroyed || isNullish(videoElement) ||
videoElement.readyState < HTMLMediaElement.HAVE_CURRENT_DATA) {
return;
}
try {
const results = await detector.detect(videoElement);
const [qrResult] = results;
if (nonNullish(qrResult)) {
dispatch("nnsQRCode", qrResult.rawValue);
}
} catch {
// No QR visible
}
if (!isDestroyed) scanInterval = setTimeout(scan, 100);
};
scanInterval = setTimeout(scan, 100);
Motivation
The QR code reader was using
html5-qrcode, which has a known issue with inverted (dark-themed, borderless) QR codes on mobile — they simply cannot be scanned. The library is also unmaintained.Known upstream issues:
A first attempt (#757) tried to fix this by increasing the scan region tolerance within the same library, but that approach still failed on mobile (see test results in that PR).
Since the root cause is a limitation of
html5-qrcodeitself (no support for inverted QR codes), and the library is no longer maintained, this PR replaces it withbarcode-detector— a Barcode Detection API polyfill powered by ZXing C++ WebAssembly under the hood. The underlying engine natively supports inverted code detection (tryInvertis enabled by default). We tested it manually on a mobile device and dark borderless QR codes are now successfully scanned.Thanks to @sea-snake for the tip!
Changes
html5-qrcodedependency entirely in componentQRCodeReader.QRCodeReader:navigator.mediaDevices.getUserMediaAPI, requesting the rear-facing camera (facingMode: "environment") at up to 1920×1080 resolution.<video>element.BarcodeDetectorAPI (viabarcode-detector/ponyfill). A detector instance is created withformats: ["qr_code"]anddetector.detect(videoElement)is called at ~10 fps viasetInterval(scan, 100). The polyfill accepts the<video>element directly — no intermediate<canvas>needed.isProcessingFrame) prevents overlapping decode calls..scan-overlaycontainer with a.scan-regionbox usingbox-shadowto mask the surrounding area and a border styled with the primary color. This replaces the old:global(#qr-shaded-region)overrides that targetedhtml5-qrcode's internal DOM.jsQRtobarcode-detector.barcode-detector@^3.1.1to dependencies and removedhtml5-qrcode.Tests
Manual testing on mobile: dark borderless QR codes are now successfully scanned (this was the main validation, as the core issue only reproduced on real mobile devices).
Screenshots
Updated E2E snapshots.