micropython feature suggestions

[kjm1102]

Is there a mechanism for suggesting changes to upython for those of us lacking the wherewithal to do pull requests? For example I really liked the nvs_set & nvs_get functionality of pycom python, & that rtc memory preserved those values through a reset. Seemed to be a good alternative to rtc.memory() for multiple variables that need to be persevered through a deepsleep.

[jimmo]

This is the exact place to do that!

But in this case, NVS is actually supported. See https://docs.micropython.org/en/latest/library/esp32.html?highlight=nvs#esp32.NVS for docs and https://github.com/micropython/micropython/blob/master/tests/esp32/esp32_nvs.py for examples of how to use it.

[jimmo]

See the example I linked to:

from esp32 import NVS

NVS is the class, esp32 is the module that you import it from.

[kjm1102]

tnx Jim so is

nvs = NVS("mp-test")

the preferred way of initialising or should I substitute something else for "mp-test" (nvs=NVS() gives a TypeError wanting a positional argument)?

[jimmo]

It's the namespace name, you can choose whatever you want, just as long as you choose something. You can also have more than one namespace if you want to have different sets of key/values for different purposes. Think of it like a very simple one-layer directory structure:

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html

[kjm1102]

Understood.

How do I recover a string stored as a blob? The example shows how to recover the strings length but not the string itself? My best guess fails.

from esp32 import NVS
nvs=NVS('locals')
blob1 = "testing a string as a blob"
nvs.set_blob("blob1", blob1)
print(nvs.get_blob("blob1"))
Traceback (most recent call last):
  File "<stdin>", line 8, in <module>
TypeError: function takes 3 positional arguments but 2 were given

[karfas]

Is something wrong with https://docs.micropython.org/en/v1.15/library/esp32.html ?
This reads

NVS.get_blob(key, buffer)

There is no buffer argument in your get_blob() call.

[karfas]

@kjm1102: You are mixing things up.
The NVS class provides a key-value store in EEPROM in a separate partition. This is also used by some ESP-IDF functions.
RTC.memory() gives you access to a small portion of RTC memory (RAM).

Regarding RTC memory there are two of my PRs (and possibly others) open:

• (April '21) esp32/machine_rtc: ESP32 machine.RTC().usermem() returning a bytearray. #7133
• (Mai '22) esp32/machine_rtc: Preserve RTC user memory over most reset causes. #7298

Additional, you could try https://github.com/karfas/upy-esp32-lib - there is some python code for managing RTC memory content.

[kjm1102]

Nope don't get it. Nowhere in there can I see anything stored with nvs.set_blob recovered with nvs.get_blob. The _i32 example is fine integer stored, integer recovered, no problems.

[jimmo]

buf1 = bytearray(len(blob1))
len1 = nvs.get_blob("blob1", buf1)
print(buf1)

this gets the blob, storing it in buf1 (which is then printed). very similar to readinto (used in socket/file/i2c).

[kjm1102]

Ah buf1 doing double duty, and buf1.decode() has the string, finally! The pycom version seemed a bit more intuitive but then what would I know.

What's with the nvs.commit() window dressing? Seems to work OK without it?

[karfas]

Can you please read the documentation ?
The documentation (see link above) clearly states

Changes to NVS need to be committed to flash by calling the commit method. Failure to call commit results in changes being lost at the next reset.

[jimmo]

The pycom version seemed a bit more intuitive but then what would I know.

It's done this way for efficiency, saves allocating a buffer unnecessarily for the result, instead you reuse a preallocated buffer. Same as readinto for socket/file/i2c.

[rkompass]

Referring to the title micropython feature suggestions (in plural) and encouraged by @jimmo s This is the exact place to do that! I would like to express some thoughts regarding the development process. (I consider myself also as lacking the wherewithal to do pull requests atm, so I rather think I'm not in a position to do that, but it circulates in my mind anyway.)

1. My impression is and you (@jimmo) stated that also explicitly, that you have a very very long todo list.
2. About every two weeks or so a new architecture comes into play. This of course is a result of micropythons huge success.
3. Also at least from the appearance of esp8266 onwards the development has to deal more and more with topics like networking, bluetooth, threading, webserver etc, which are important for the internet of things but may be considered a bit external from the pure python, native, viper, assembler, interrupt and common api for peripherals stuff. Many ideas concerning the latter are found in the early discussions but apparently there was no time to work on them.
4. The number of core developers at the same time seems to keep rather constant. I explain this to myself by the complexity of uPy s internals, which are not documented (you have to explore the sources to get a grip of it).

So I'm wondering whether it would be a good idea to collect and sort out the topics on the todo list from time to time and then start a kickstarter-like process to have an incentive (and focus?) for core development. I would like to support such a thing. But perhaps the approaches to different issues have to 'get ripe' before any real coding should start and it would be contraproductive to try to push it?
Probably you have discussed this from time to time. I would be glad to get an impression what the position of core developers is related to this.

[jimmo]

Thanks @rkompass -- you raise some excellent points. You're definitely right that a huge amount of our effort actually has very little to do with the core "Python" part.

As far as making the todo list a bit more open and clearer, that's definitely something I'm keen to do (if nothing else than to force us to focus too!). We talked a little bit about some of this stuff on the CircuitPython Show interview recently. And you're also right that it's very difficult to figure out how something should work until you actually sit down to code it -- this is also one of the things that makes reviewing PRs so much work.

I'm not sure Damien would ever want to do another Kickstarter, but GitHub sponsorship has definitely been a huge help in this area.