QR code generator

[prusnak]

In Trezor, we have a QR code generator module based on https://www.nayuki.io/page/qr-code-generator-library

Would there be an interest in having a native module for QR code encoding - extmod/moduqr.c? Optionally built of course.

I think a lot of people need this functionality and https://github.com/JASchilz/uQR is just too slow :-/

Edit: I have no strong preference of having this in the upstream as the moduqr would look probably very different to what we do know, so I don't mind having it in my tree at all. However, I am just thinking alound whether having QR code encoder in the standard library would be helpful.

[mattytrentini]

I'm interested in generating QR Codes in MicroPython. I had intended to try porting the Python variant of Nayuki's QR code generator but I'm not yet sure if performance will be reasonable...

Have you had a chance to profile uQR? How slow is too slow? :)

So, yes, I'm keen to have a native module if the performance of a pure MicroPython module is particularly poor. Happy to help out with the implementation.

[peterhinch]

From a quick look at uQR there would seem to be scope for optimisation both for size and speed. For example

G15 = (
    (1 << 10) | (1 << 8) | (1 << 5) | (1 << 4) | (1 << 2) | (1 << 1) |
    (1 << 0))

is an expensive way to assign 0x537 to a variable. There is at least one global dict which could be replaced by a lambda. The alternative code emitters could be used. The final output is an extravagant (in RAM) way to store a bitmap. And that's just from a quick scan of the code.

A couple of questions for anyone thinking of porting or optimising this:

• Are there features which can be culled for a "micro" version?
• Should the output got into a framebuf instance?
• What platforms is it expected to support? ESP8266 could be hard.

All this boils down to "how slow is too slow". If it's >= an order of magnitude there's no choice than to write it in C (IMO). And before anyone suggests it, C programming is only something I do in extremis so the answer is no. :)

[prusnak]

@mattytrentini I haven't profiled Python version of Nayuki's QR code generator nor do I plan to do so. But the uQR from micropython-lib was indeed really slow, don't have the numbers though. Maybe if you do the benchmark of these two, we can decide whether it is worth having this code natively or not.

[peterhinch]

[EDIT]To fix an error measuring RAM usage.
I did a quick test of JASchilz/uQR on a Pyboard 1.0. Compilation failed with a memory error, so I ran with frozen bytecode. A precompiled .mpy file also worked.

import utime
import micropython, gc
gc.collect()
print(micropython.mem_info())
from uQR import QRCode
t = utime.ticks_ms()
qr = QRCode()
dt1 = utime.ticks_diff(utime.ticks_ms(), t)
t = utime.ticks_ms()
qr.add_data('uQR rocks!')
dt2 = utime.ticks_diff(utime.ticks_ms(), t)
t = utime.ticks_ms()
matrix = qr.get_matrix()
dt3 = utime.ticks_diff(utime.ticks_ms(), t)
print(dt1, dt2, dt3)
gc.collect()
print(micropython.mem_info())

The outcome was:

dt1 == 1ms
dt2 == 0ms
dt3 == 464ms
RAM use 34.8KiB

Most of the RAM was consumed by the line

from uQR import QRCode

I'm surprised that the author managed to get it to run on an ESP8266.

I then repeated the test, changing the string to 'uQR rocks! This is a longer string more like a typical URL' with the following outcome:

dt1 == 0ms
dt2 == 2ms
dt3 == 1200ms
RAM use 41.8KiB

RAM use is getting excessive, but this is largely due to the inefficient format of matrix which I estimate to be using 41*41*4 = 6.7KiB to store 41*6 = 246 bytes of information (6 bytes is more than sufficient to store 41 bits).

The code looks like a port of "big iron" Python to MicroPython with no attempt at optimisation. How much of an improvement can be made to speed would need proper profiling. But I have no idea what speed you guys need. If you need more than 10x my "finger in the air" guess is that you're best off going to C. Less and we might be in with a chance with optimisation and I may be able to help.

What are we expecting here?

[prusnak]

It would be interesting to see the same kind of benchmark @peterhinch did but with the Nayuki's code. Anyone?

[peterhinch]

@prusnak
If you mean this source file it would need porting to MicroPython first.

[prusnak]

@peterhinch yes and yes

[peterhinch]

Alas I'm concluding that this project is dead in the water. I'm losing interest.

Nobody has responded to my requests for a performance target or options for a 'micro' feature subset; still less expressed any interest in actually contributing effort.

I can port, profile and optimise code. Without agreed objectives and help from people with an understanding of the code design my efforts would probably be wasted. I don't plan to tackle this as a solo project.

[mattytrentini]

Apologies @peterhinch, I've been crazy-busy at work and was also away this weekend. Even so, I have a long list of MicroPython work I want to tackle and QR codes are unfortunately not the highest priority at the moment...I really appreciate you looking into this though!

My performance needs are modest; halving those times would be quite usable for me (0.5-1 sec for reasonably-sized QR codes would be manageable - though my gut says that's slow). I agree that there appears to be some significant improvements possible. I'd be most likely using the ESP32 or possibly an STM32F4. Not an ESP8266. Framebuf would be a logical render target but I'm also curious about how it could be integrated into LittlevGL as that shows promise as a GUI toolkit.

But I need to be careful about over-committing; I really can't commit to doing much with QR codes right now...perhaps @prusnak might have more time!

[peterhinch]

One potential optimisation is to use Native or Viper code for performance critical functions. This is unavailable on ESP32. My gut feeling (without having done any profiling) is that a 2:1 improvement will be readily available even without recourse to native code. On a Pyboard D with appropriate use of native code it might be considerably quicker...

[petrkr]

Did only call tests, not profiling code.

get_matrix took: 1884ms
set_frame_memory took: 84ms
display_frame took: 2ms
efb_show took: 92ms
display_qr took: 371ms

but already did already few optimizations like

-        return lambda i, j: (int(i / 2) + int(j / 3)) % 2 == 0
+        return lambda i, j: (i // 2 + j // 3) % 2 == 0

which give 1.8sec from 1.9sec

Board: ESP32

[mattytrentini]

I revisited this topic and ported the Nayuki Project's Python code to MicroPython. Also documented the porting effort in the hope it may be useful for others porting from Python to MicroPython.

I've not made any performance improvements, simply ported the code so the identical codebase works on both Python and MicroPython.

And performance, on my TinyPICO is slow: typically ~1.9 seconds to generate a QR code with a short string!

There are quite a few obvious performance improvements to be made. That said, I'm giving a talk at PyCon AU: Extending MicroPython: Using C for good! in a few weeks so my next steps are to build the C port into a MicroPython module, explain how that's done and then measure it...

I'll have to revisit performance improvements after PyCon AU! If someone else wants to see how fast they can make Nayuki then by all means give it a go; it's at least running on MicroPython now... :)

[nayuki]

@mattytrentini: Thanks for your MicroPython porting work! I apologize for the trouble you went through because my Python library is not optimized for this use case.

To be clear, my Python QR Code lib is designed to prioritize absolute correctness, feature-completeness, human-reading clarity and conciseness, and then performance last. As one example, reed_solomon_multiply() is implemented as a slow but understandable algorithm instead of as an opaque 256×256 lookup table.

Python, as an interpreted language, is already demonstrably slow on desktop computers. When doing pure numeric and array operations (as opposed to I/O or call C modules), Python is roughly 30× slower than a compiled language like Java which has fixed-size integers.

For the microcontrollers I worked with, they are about 100× slower than desktop computers. This exacerbates Python's slowness; also their small memory may not be appropriate for running a Python interpreter. So specifically for embedded systems, I advocate using the C version of my QR Code generator library (photo, video), which is very streamlined in terms of CPU time and memory usage. On a side note, the Moddable project packaged up my C lib into their JavaScript framework.

[doc-hex]

I've taken @nayuki 's C library and wrapped it into a module. I tried to keep the API simple and still powerful enough for the things I want to do, like control the version number precisely.

Needs just a little more to make it a dynamically-loadable native-code .mpy, but nearly there. Works great compiled into Micropython, both versions 1.12 and 1.9.4 (tested).

mpy-qr

[TerryIoannou]

Would it be better just create an API on your server that just generated a qrcode and then send it to your Microboard as a png and then displays it within the dimention of the screen you have connected with your microboard ?

and from the microboard send data back to the api after its scanned or am I missing something ?

[petrkr]

Would it be better just create an API on your server that just generated a qrcode and then send it to your Microboard as a png and then displays it within the dimention of the screen you have connected with your microboard ?

and from the microboard send data back to the api after its scanned or am I missing something ?

Very good idea, but tell me, how exactly this will work on offline device?

[TerryIoannou]

I have not thought of that actually which makes me wonder more about the current project I am working on thank you for the thought xD if you have the solution to it then that would be perfect!

[peterhinch]

See the QRMap widget in MicroGui.

[jonnor]

There seems to be several good QR code alternatives here. Perhaps this issue can be closed now? Maybe use a discussion if further discussion is wanted.

Just triaging a bit to maybe one day get below 1k open issues :D

[petrkr]

There seems to be several good QR code alternatives here. Perhaps this issue can be closed now? Maybe use a discussion if further discussion is wanted.

Just triaging a bit to maybe one day get below 1k open issues :D

That alternatives are very slow.
Tooks like 2 sec to generate and display QR code.

Prusnak posted C codec which do everything in just 300ms.

So perhaps you can close it whenever there will be solution how to use those optimized libraries.

There us QR code, which is compiled as C user module, but that is part of BIN, would be nice if it can be compiled as 'so' library and put on filesystem like mpy can

[petrkr]

Talking about this one

https://github.com/coinkite/mpy-qr

[mattytrentini]

I gave a (admittedly not my finest!) talk a few years ago about using a native module to generate QR codes: Extending MicroPython: Using C for good! (slides here). There is a gist that contains the guts of the example. Generating QR codes took the same as when using the underlying C library; 11ms.

Perhaps I should polish that up and make a proper library out of it...

[scruss]

Maybe try WendelHime/uQR: QR Code Generator for MicroPython: it's a fork of the original uQR that supports more input characters. While it's pure MicroPython, I can get frames to render as fast as I need them.

Its output format is a little weird. It would be nice for it to be able to render to a framebuf, say, but it's good enough for me

[petrkr]

And speed? Does it render qr within resonable time? Let's say 300ms?

[scruss]

Not 300, but 490 ms on ESP32 for this code:

from uQR import QRCode
import time

qr = QRCode()
begin = time.ticks_ms()
qr.add_data("uQR rocks!")
matrix = qr.get_matrix()
end = time.ticks_ms()
print("Elapsed / ms:", time.ticks_diff(end, begin))

For me, pure MicroPython almost always beats compiled-in C module because maintaining all the toolchains is very boring.

Maybe some users would find this library useful.

[peterhinch]

@jonnor I have converted this to a discussion. Maybe new readers will post other solutions.