btree isn't available

[gwynge]

I see btree is part of micropython libraries https://docs.micropython.org/en/latest/library/btree.html

But I cannot import btree on my Pico W.

Is the doc incorrect? Or am I missing the library somehow?

rp2-pico-w-20221011-unstable-v1.19.1-543-gab317a0d6

[dlech]

You would need to compile your own firmware with MICROPY_PY_BTREE enabled. It is currently only enabled in some ESP and Unix variants.

[pobbie]

No file "mpconfigport.mk", no file at all with extension .mk

[dlech]

OK, then you need to do the equivalent in the CMakeLists.txt. See the esp32 port for an example.

[pobbie]

Ok.
I added set(MICROPY_PY_BTREE ON) to CMakeLists.txt and that made something,
now I get a CMake error:
CMake Error at /home/pi/pico/micropython/extmod/extmod.cmake:60 (add_library):
No SOURCES given to target: micropy_extmod_btree
Call Stack (most recent call first):
CMakeLists.txt:64 (include)
CMake Generate step failed. Build files cannot be regenerated correctly.
make: *** [Makefile:32: all] Fel 1

[dlech]

Hmm... still needs some work to make RP2040 work like ESP32 CMake.

[dlech]

I made #9992 with a fix. Also, it looks like you might not have the lib/berkeley-db-1.xx git submodule checked out.

[gwynge]

Thanks for clarifying. Is this variance documented? If not - should it be?

[pobbie]

Where do I put this code? CMakeLists.txt? Från: David Lechner ***@***.***> Skickat: den 17 november 2022 00:25 Till: micropython/micropython ***@***.***> Kopia: Per-Olof Bergström ***@***.***>; Manual ***@***.***> Ämne: Re: [micropython/micropython] btree isn't available (Discussion #9626) Maybe try this? diff --git a/extmod/extmod.cmake b/extmod/extmod.cmake index 2207a951a..f0653a568 100644

…

--- a/extmod/extmod.cmake +++ b/extmod/extmod.cmake @@ -79,6 +79,7 @@ if(MICROPY_PY_BTREE) ) target_compile_definitions(micropy_extmod_btree PRIVATE + MICROPY_PY_BTREE=1

__DBINTERFACE_PRIVATE=1 mpool_error=printf abort=abort_ @@ -93,6 +94,7 @@ if(MICROPY_PY_BTREE) ) list(APPEND MICROPY_DEF_CORE + MICROPY_PY_BTREE=1

__DBINTERFACE_PRIVATE=1 "virt_fd_t=void*" ) - Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Forgs%2Fmicropython%2Fdiscussions%2F9626%23discussioncomment-4161998&data=05%7C01%7C%7C9bb6fccfae674347eff108dac829c7a8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638042379010918578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=31k0S%2BJAvCPbcfwQY7UFnUptxM3x9oKvU6qBkcOYtg8%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAC67ZPB3XIJISNNLHRIHJRDWIVUMXANCNFSM6AAAAAARFPTGVA&data=05%7C01%7C%7C9bb6fccfae674347eff108dac829c7a8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638042379010918578%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xiuykJRAO2KycXHU5E8%2F1Dby3e3kjRvTroSNFaFgXGw%3D&reserved=0>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[kapetan]

Hi. I'm trying to make a MicroPython build with btree included for the Pico W. But can't get it to work, has anybody had any success with it?

I added MICROPY_PY_BTREE to ports/rp2/CMakeLists.txt (see below).

diff --git a/ports/rp2/CMakeLists.txt b/ports/rp2/CMakeLists.txt
index 281b0c3bc..dae142b65 100644
--- a/ports/rp2/CMakeLists.txt
+++ b/ports/rp2/CMakeLists.txt
@@ -56,6 +56,7 @@ endif()
 
 # Enable extmod components that will be configured by extmod.cmake.
 # A board may also have enabled additional components.
+set(MICROPY_PY_BTREE ON)
 set(MICROPY_SSL_MBEDTLS ON)
 
 # Use the local cyw43_driver instead of the one in pico-sdk
@@ -434,6 +435,8 @@ target_link_libraries(${MICROPY_TARGET} micropy_lib_mbedtls)
 
 target_link_libraries(${MICROPY_TARGET} usermod)
 
+target_link_libraries(${MICROPY_TARGET} micropy_extmod_btree)
+
 target_include_directories(${MICROPY_TARGET} PRIVATE
     ${MICROPY_INC_CORE}
     ${MICROPY_INC_USERMOD}

It builds successfully and in the output of make -C ports/rp2 BOARD=RPI_PICO_W I can see btree being included (see below), but when I flash the Pico with the firmware.uf2 file and try to execute import btree I get ImportError: no module named 'btree'.

[  3%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_close.c.obj
[  3%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_conv.c.obj
[  3%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_debug.c.obj
[  4%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_delete.c.obj
[  4%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_get.c.obj
[  4%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_open.c.obj
[  4%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_overflow.c.obj
[  4%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_page.c.obj
[  5%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_put.c.obj
[  5%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_search.c.obj
[  5%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_seq.c.obj
[  5%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_split.c.obj
[  5%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/btree/bt_utils.c.obj
[  6%] Building C object CMakeFiles/micropy_extmod_btree.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/lib/berkeley-db-1.xx/mpool/mpool.c.obj
[  6%] Built target micropy_extmod_btree
...
[ 40%] Building C object CMakeFiles/firmware.dir/Users/mirza/workspace/line/fluid-spotter-firmware/runtime/micropython/extmod/modbtree.c.obj
...

[kapetan]

Additional changes were required to rp2/CMakeLists.txt and rp2/mpconfigport.h to make btree build for Pico W.

diff --git a/ports/rp2/CMakeLists.txt b/ports/rp2/CMakeLists.txt
index 281b0c3bc..45b7f48b1 100644
--- a/ports/rp2/CMakeLists.txt
+++ b/ports/rp2/CMakeLists.txt
@@ -56,6 +56,7 @@ endif()
 
 # Enable extmod components that will be configured by extmod.cmake.
 # A board may also have enabled additional components.
+set(MICROPY_PY_BTREE ON)
 set(MICROPY_SSL_MBEDTLS ON)
 
 # Use the local cyw43_driver instead of the one in pico-sdk
@@ -434,6 +435,8 @@ target_link_libraries(${MICROPY_TARGET} micropy_lib_mbedtls)
 
 target_link_libraries(${MICROPY_TARGET} usermod)
 
+target_link_libraries(${MICROPY_TARGET} micropy_extmod_btree)
+
 target_include_directories(${MICROPY_TARGET} PRIVATE
     ${MICROPY_INC_CORE}
     ${MICROPY_INC_USERMOD}
@@ -468,6 +471,7 @@ set_source_files_properties(
 )
 
 target_compile_definitions(${MICROPY_TARGET} PRIVATE
+    ${MICROPY_DEF_CORE}
     FFCONF_H=\"${MICROPY_OOFATFS_DIR}/ffconf.h\"
     LFS1_NO_MALLOC LFS1_NO_DEBUG LFS1_NO_WARN LFS1_NO_ERROR LFS1_NO_ASSERT
     LFS2_NO_MALLOC LFS2_NO_DEBUG LFS2_NO_WARN LFS2_NO_ERROR LFS2_NO_ASSERT
diff --git a/ports/rp2/mpconfigport.h b/ports/rp2/mpconfigport.h
index 1cc9b9616..1ac86cb8f 100644
--- a/ports/rp2/mpconfigport.h
+++ b/ports/rp2/mpconfigport.h
@@ -85,6 +85,7 @@
 #ifndef MICROPY_USE_INTERNAL_ERRNO
 #define MICROPY_USE_INTERNAL_ERRNO              (1)
 #endif
+#define MICROPY_STREAMS_POSIX_API               (1)
 
 // Fine control over Python builtins, classes, modules, etc
 #define MICROPY_PY_BUILTINS_HELP_TEXT           rp2_help_text

Now it's possible to import btree after flashing the the Pico W but btree.open just hangs forever and only way to recover is to power cycle the device.

import btree
f = open('data.db', 'w+b')
db = btree.open(f) # Hangs here

[AndreasTheCat]

@kapetan :
Any idea's why btree.open hangs on PicoW?
Btw: Am still looking for a "lightest-weight" KV-Store. Btree seems fine for my requirements concerning CRUD, but are there any other known solutions?!

[kapetan]

I don't know why it crashes, haven't had the time to look into it. Would be nice if it worked though.

[kapetan]

I did some debugging and it looks like the Pico W is running out of memory when trying to allocate the btree structure.

Attaching a terminal I get the following output.

*** PANIC ***

Out of memory

Which is caused by this line in bt_open.c: https://github.com/micropython/berkeley-db-1.xx/blob/85373b548f1fb0119a463582570b44189dfb09ae/btree/bt_open.c#L190

sizeof(BTREE) evaulates to 548.

[Gadgetoid]

I'm a day late and a dollar short here, but I had an interest in btree and ran into this same problem.

It took me an embarrassingly long time to realise it's the classic problem of MicroPython module calling malloc at all (this has been the bane of my life since I started working with the Pico).

malloc wont work unless MICROPY_C_HEAP_SIZE is set to at least 4096. Judging by how many calls there are to malloc in btree then I'd guess this needs to be larger still.

Other than setting MICROPY_C_HEAP_SIZE to a sufficiently generous value there's no easy workaround to this so TLDR if you want to get further then add the following to the above:

 if(NOT MICROPY_C_HEAP_SIZE)
-    set(MICROPY_C_HEAP_SIZE 0)
+    set(MICROPY_C_HEAP_SIZE 32768)
 endif()

Which will steal 32k from MicroPython's heap so regular C has some memory to work with.

This, at least, does not hang immediately!

If you want to reimplement btree's malloc/free/realloc usage in terms of m_tracked_calloc and m_tracked_free you could commit a crime like this: pimoroni@0cc56b6

I left memory debug output in modbtree_malloc.c so you can see what's happening.

[jmhunter]

Now that the Pico 2 is out, has anyone tried using btree with the Pico 2 / Pico 2 W?

Right now I only have a Pico W with me, and have run into this issue where I'm struggling to enable btree - but I'm wondering if a Pico 2 would have no such problems..

[Gadgetoid]

The issues are pretty fundamental to MicroPython and stem from btree trying to malloc. (see my response above)

It works on Pico if you reimplement memory allocation or reserve some heap for C. See: pimoroni@0cc56b6

[ctimmer]

Is there enough interest in bringing btree up to micropython standards? I've done a lot of c programming (although not for a number of years). Looks like the bulk of fixes would be swapping out malloc/free code for their micrpython equivalents.

Being retired I have lots of free time.

Just a thought,
Curt