Support installing the elf2uf2 and pioasm utilities and the documentation

[clausecker]

The build system should support building and installing the elf2uf2 and pioasm utilities, documentation as well as pre-compiled library files for users of build systems other than CMake. Right now, it is kinda hard to even coax the build system into producing these binaries separately. The goal here is to support making binary packages of the SDK utilities for easy installation through the distribution's package manager as well as making it easier to build RPi Pico projects that don't use CMake.

[kilograham]

The goal here is to support making binary packages of the SDK for easy installation through the distribution's package manager

Who's goal is this? Is this easier than checking out a single github project, or having the SDK as a submodule

making it easier to build RPi Pico projects without having to fight with CMake.

The SDK uses cmake for its build. It's really not that hard. It has a bunch of benefits as described in chapter 2.1 here https://datasheets.raspberrypi.org/pico/raspberry-pi-pico-c-sdk.pdf

making binary packages of the SDK ...

Unless you have very good reasons, you always want to build and link using the CMake. Not wanting to learn any cmake is not a good reason!

The build system should support building and installing the elf2uf2 and pioasm utilities,

You would still have to build them once; separate installation packages across all supported platforms is still a lot of extra work.

[clausecker]

Who's goal is this? Is this easier than checking out a single github project, or having the SDK as a submodule

My goal. I do not want to rebuild the entire standard library for every project or even use CMake for my own projects. I am not a fan of CMake; this is not a matter of refusing to learn it, but rather about things like integration into existing projects that already use a different build system or have special requirements. It is also about giving the user the choice to select a build system instead of forcing CMake onto everybody.

You would still have to build them once; separate installation packages across all supported platforms is still a lot of extra work.

That is correct. But once built and installed, the tools and libraries can be used with custom build systems, giving the programmer more flexibility. I also kinda don't understand the point of wasting tons of space rebuilding the library and tools for each and every project. Seems kinda wasteful.

I do not ask the project to provide binary packages. Merely to make it so that generating binary packages is possible, e.g. by adding targets to build all utilities and to precompile the standard library for a given board (let the distribution worry about building the library for each board separately). A distribution can then pick up the toolkit and make a package.

[kilograham]

On elf2uf2 etc, these can be built separately from within their own build directory

https://github.com/raspberrypi/pico-sdk/tree/master/tools/elf2uf2
https://github.com/raspberrypi/pico-sdk/tree/master/tools/pioasm

(i.e. cd whicever, mkdir build, cd build, cmake .., make)

It is also about giving the user the choice to select a build system instead of forcing CMake onto everybody.

You certainly aren't forced to use CMake, but it is the only supported way. This is not because we are fanatics, this is because CMake allows us to do some very powerful things very simply (for us and the end user) that would be a lot of work to do in another build system. If you choose not to use CMake, then you implicitly lose a lot of flexibility within how the SDK works, and things become more brittle.

but rather about things like integration into existing projects that already use a different build system or have special requirements

You can look at https://github.com/raspberrypi/pico-playground/blob/master/standalone/static_sdk/CMakeLists.txt. If possible you are much better off making a static library out of the other thing.

Note, running the tool above will show just how many things you weren't having to worry about in terms of command line defines etc.

[kilograham]

You can look at https://github.com/raspberrypi/pico-playground/blob/master/standalone/static_sdk/CMakeLists.txt.

or run make VERBOSE=1

[clausecker]

Thank you for the answer.

On elf2uf2 etc, these can be built separately from within their own build directory

Yes; though slightly inconvenient, this is workaround. Have these built by default when building the SDK on top level (as opposed to, as a dependency) would already be a nice improvement. Also, an install target to install at least these tools and the documentation could be helpful. The project already has logic to check for the tools in the user's PATH it seems and they don't depend on the project configuration at all, so compiling them once ahead of time seems useful enough.

You certainly aren't forced to use CMake, but it is the only supported way.

If it's the only supported way it seems a lot like I am pretty much forced to use CMake. I don't really get why the traditional static library approach doesn't work for you; perhaps there are some shenanigans with generated or conditional code that I haven't found yet (didn't dug into the src directory too deeply). And I don't get how recompiling each module for every project is supposed to be an advantage, but I understand if you don't want to argue this point.

Arch “packaged” the SDK by just shipping the source. But even that seems dubious as they don't initialise any submodules of tinyusb and it is unclear which are needed for which configurations. Shipping all of them would be too much wasted space, clocking in at almost 400 MB (there is stuff like a full rtos tree in there).

The pico-playground link with the static library you posted does seem like it is useful. I'll go ahead and give it a try.

Note, running the tool above will show just how many things you weren't having to worry about in terms of command line defines etc.

I do see you use a lot of -Wl,--wrap=... options; it honestly doesn't seem to be too bad.

As for some background: we've been porting mecrisp-stellaris, a Forth system, to the Pico board. The build process is a bit special with a custom linker script, custom boot loader, custom runtime and ABI, and an intermediate step where the image is loaded into an emulator to compile additional definitions in. Right now, we use a conversion script from Microsoft's UF2 tools and have copied all magic constants for the peripherals in by hand. It might be useful to make use of some parts of the SDK (such as pin definitions) to allow for reuse of the port to other boards, reduce the number of magic constants as well as to embed metadata into the image. But integrating CMake into the build process of a project that already builds images for dozens of other boards without requiring such dependencies seems to be a lot of work and added complexity for not really much of a gain.

[lurch]

And I don't get how recompiling each module for every project is supposed to be an advantage, but I understand if you don't want to argue this point.

Just for an example, have a look at how many compile-time settings https://github.com/raspberrypi/pico-project-generator allows you to set. A lot of the configuration steps (?) that you might have to do with a statically-built library, are instead done at compile time leading to more optimised executables.

Each time you run cmake it doesn't rebuild the entire standard library, it only compiles the parts of the code actually used by your project. It's all very clever 😉 (Magic @kilograham stuff that I don't fully understand myself yet!)

The build process is a bit special with a custom linker script, custom boot loader, custom runtime and ABI, and an intermediate step where the image is loaded into an emulator to compile additional definitions in.

LOL, that sounds a bit more complicated than just following the instructions in https://datasheets.raspberrypi.org/pico/getting-started-with-pico.pdf and running cmake? 😉

If it's the only supported way it seems a lot like I am pretty much forced to use CMake.

I think Graham might have been implying that whilst CMake is the only build system that we'll be supporting in the SDK, the entire SDK sourcecode is BSD-licensed, so you're free to wrap your own build-system around the SDK's source-files?

This issue reminds me a lot of #17

It might be useful to make use of some parts of the SDK (such as pin definitions) to allow for reuse of the port to other boards

See https://github.com/raspberrypi/pico-sdk/tree/master/src/boards/include/boards ?

[clausecker]

LOL, that sounds a bit more complicated than just following the instructions in https://datasheets.raspberrypi.org/pico/getting-started-with-pico.pdf and running cmake? wink

Our build system is literally two shell scripts per board (one that runs as, ld, objcopy to make an initial binary, one to load that binary into the emulator and compile in additional definitions). Configuring CMake would be a lot more complex, especially if it's just for one board. And documentation of your build system is rather poor; for example, it doesn't say what demands you place on a custom linker script or even how to set one up. And even what the supported linker scripts do (e.g. what the exact memory regions the binary is loaded into are and what sort of guarantees are made about this) and how they can be used is not really explained at all.

Coming back to Mecrisp Stellaris, it seems like a work flow that involves passing the binary through an emulator to load additional code won't be supported at all without some ugly hacks (e.g. because converting a binary image to an UF2 file seems to be something you don't support at all; we would have to somehow turn the binary image back into an ELF file), so how is using cmake going to be easy and straightforward here?

A lot of the configuration steps (?) that you might have to do with a statically-built library, are instead done at compile time leading to more optimised executables.

such as ...? Many such configuration steps (e.g. default configuration for peripherals) can also be executed at link time by adding an assembly file that sets up a bunch of symbols with the desired configuration.

I mean I get that some people like to have the fiction of zero overhead when chosing defaults, but does it really matter in practice?

Each time you run cmake it doesn't rebuild the entire standard library, it only compiles the parts of the code actually used by your project. It's all very clever wink (Magic @kilograham stuff that I don't fully understand myself yet!)

I honestly prefer simple and straightforward over “very clever.” Hundreds of tiny one-file libraries with many layers of abstraction spread all over the SDK that all add custom options to the build and may pull in additional random dependencies just make it an absolute nightmare to figure out what exactly is going on. Sure you lose 0.5% performance if you don't have hundreds of macros removing conditional code everywhere in case random features aren't used, but the amount of mind space you save by not having to think about all of this complexity is more than worth it.

(btw, I just noticed the SDK doesn't even build on FreeBSD because it tries to pull in system C++ headers that get very confused by the embedded environment and complain loudly; so much for “very clever”).

I mean sure, if all you do is program this one board you'll eventually get used to it and accept the craziness as given, but to our project the RP2040 is one chip among dozens to support and demands to integrate a different build system with strange rules and conventions just to get started with the SDK really don't make this any better. It may be plug&play for greenfield development, but if you for some reason can't bend the entire project around the quirky demands of the SDK, you are left out cold.

I think Graham might have been implying that whilst CMake is the only build system that we'll be supporting in the SDK, the entire SDK sourcecode is BSD-licensed, so you're free to wrap your own build-system around the SDK's source-files?

Right now the more economic option is to carry on not using the SDK because it seems like it's going to be quite unlikely that it can be used in any useful way.

[lurch]

Please accept my apologies for my slightly tongue-in-cheek comments.

Yes, for your usage the cmake-build-system in the SDK might not be what you're looking for (we had similar "complaints" from the PlatformIO people), but https://www.raspberrypi.org/documentation/rp2040/getting-started/#getting-started-with-c and https://datasheets.raspberrypi.org/pico/getting-started-with-pico.pdf kinda illustrate the user that the pico-sdk is targeted at? 🤷

I think #394 is a good example of how much "mind space" the system Graham has created saves the average Pico user - no need to muck about with include-paths or linker-settings, all the user needs to do is add a single hardware_adc keyword to their CMakeLists.txt and all of a sudden the user has access to RP2040's ADC functionality.

But I'm going to duck out of this conversation now because there's not really anything useful I can add.

[kilograham]

But I'm going to duck out of this conversation now because there's not really anything useful I can add.

yes... not really very helpful

[kilograham]

As for some background: we've been porting mecrisp-stellaris, a Forth system, to the Pico board

This would have been good context up front :-)

Also, an install target to install at least these tools and the documentation could be helpful. The project already has logic to check for the tools in the user's PATH it seems and they don't depend on the project configuration at all, so compiling them once ahead of time seems useful enough.

Yes, having the tools build as part of the build, rather than installation, was a) good for bootstrapping, but more importantly b) a lot simpler when you need to support Linux, Mac, and Windows (both with/without cygwin/wsl/wsl2) - always worth keeping that in the back of your mind when considering all these things.

Having an install option totally makes sense, and please open a separate issue.

Arch “packaged” the SDK by just shipping the source. But even that seems dubious as they don't initialise any submodules of tinyusb and it is unclear which are needed for which configurations. Shipping all of them would be too much wasted space, clocking in at almost 400 MB (there is stuff like a full rtos tree in there).

I am out of the loop of anyone packaging SDK. tinyusb has a horrible swamp of submodules (fortunately none are needed for pico_sdk), however they are removing all of them I think in a subsequent release.

... but I understand if you don't want to argue this point.

Yeah, i'm really not going to argue any of the decisions that have been made (all with good reasons); you have your queries many of which i could put to rest with a better understanding of how things work, however discussing all that is not germane, as really doesn't help you moving forward.

You are trying to build a custom language runtime environment, and target a pre-existing different build system (probably with support for multiple targets and indeed a "board config" notion too). This is not something we have tried to make easy. Most of the benefits of the way the SDK works don't really help you, and equally you don't actually care to be able to customize the SDK as you have a single use.

Other people have already done this (at least Rust, MicroPython, CircuitPython, Arduino Core and more i can't think of off the top of my head) the former is bare metal, the MicroPython uses CMake for this and a few other targets, CircuitPython uses the SDK but compiles and links itself. Also of course the RTOSes have to decide if they are built on top of/along side the SDK or not)

Note the Hundreds of tiny one-file libraries with many layers of abstraction is really a very shallow tree and then some aggregations and logical inter-dependencies - I'm not sure you've read the First chapter or two of the C++ book mentioned above). The bootrom actually uses the SDK and relies mostly on the hardware_ level of libraries

If you absolutely don't want to use CMake in your build, you either have to

a) call CMake from make to build a static lib... you could generate a CMake file on the fly even.
b) do the compilation yourself. This is what CircuitPython does, but they take on the onus of making sure the options/files are up to date with each SDK release.

The complicating factor with RP2040 is that you must prepend a "2nd stage" flash onto the binary, since the RP2040 has no knowledge of the type of flash in use - this boot stage 2 is dependent on the type of flash used (though you can use a slow generic one). pico_standard_link CMake library does this for you,

If you are doing custom linking stuff anyway, then yes you need to adapt our linker script. Again yes, the link scripts are not well documented, but such things are way off the happy path - that is on the list though (along with templateization)

Anyways, for your use case what does the SDK get you possibly? Most likely of use:

• startup /clock setting code
• float/double implementation which is both fast, and saves you space being in the boot rom
• transparent use of hardware divider etc.
• hardware_ level libraries to help you implement your stuff.

Looking at the pico-playground tool will give you a good idea what compile/link options you need if you want to build yourself.
Again of course these may be build platform specific.

P.S.

elf2uf2 is not really required; you can take the .BIN and pass it thru Microsoft's python tool

P.P.S.

pioasm should not be required if you don't call pico_generate_pio_header from your CMakeLists.txt

P.P.P.S.

(btw, I just noticed the SDK doesn't even build on FreeBSD because it tries to pull in system C++ headers that get very confused by the embedded environment and complain loudly; so much for “very clever”).

cleverness was not my word, nor the goal. We do not test on FreeBSD, if you open an issue with the error message that may help identify the cause.

[clausecker]

Thank you for your response.

This would have been good context up front :-)

Yes, having the tools build as part of the build, rather than installation, was a) good for bootstrapping, but more importantly b) a lot simpler when you need to support Linux, Mac, and Windows (both with/without cygwin/wsl/wsl2) - always worth keeping that in the back of your mind when considering all these things.

Having an install option totally makes sense, and please open a separate issue.

This issue actually already is about the install option (see the first comment). If you like, I can take “pre-compiled library files” out of the scope so it should already pretty much be what you suggest.

The Forth system was more of an example for how not every project can realistically be structured around the Pico SDK's demands on a build system.

I am out of the loop of anyone packaging SDK. tinyusb has a horrible swamp of submodules (fortunately none are needed for pico_sdk), however they are removing all of them I think in a subsequent release.

Ah, that sounds like good news at least.

The complicating factor with RP2040 is that you must prepend a "2nd stage" flash onto the binary, since the RP2040 has no knowledge of the type of flash in use - this boot stage 2 is dependent on the type of flash used (though you can use a slow generic one). pico_standard_link CMake library does this for you,

Correct. For Mecrisp Stellaris, we actually wrote a custom boot loader for this purpose.

If you are doing custom linking stuff anyway, then yes you need to adapt our linker script. Again yes, the link scripts are not well documented, but such things are way off the happy path - that is on the list though (along with templateization)

Sounds good!

Anyways, for your use case what does the SDK get you possibly? Most likely of use:

Specifically, I wanted to get macros for register definitions and the included binaries. Especially pioasm might be useful for some projects.

elf2uf2 is not really required; you can take the .BIN and pass it thru Microsoft's python tool

Yes, that's what we do right now.

cleverness was not my word, nor the goal. We do not test on FreeBSD, if you open an issue with the error message that may help identify the cause.

Yes, @lurch said that, not you. I'll see if I can open an issue about that.

[aallan]

We do not test on FreeBSD, if you open an issue with the error message that may help identify the cause.

I'll see if I can open an issue about that.

I've got a vague recollection that there is an open (closed) issue already, but for the life of me I can't find it. Either here, or on the raspberrypi/pico-feedback repo.

[lurch]

I wanted to get macros for register definitions

The register definitions are all nicely self-contained in https://github.com/raspberrypi/pico-sdk/tree/master/src/rp2040

re: FreeBSD

I've got a vague recollection that there is an open (closed) issue already, but for the life of me I can't find it.

I can't find it either: https://github.com/search?o=desc&q=org%3Araspberrypi+freebsd&s=updated&type=Issues
Maybe it was something that only got mentioned during the closed beta-testing? Or maybe it was mentioned on the forums but never got as far as a GitHub issue? 🤷

[aallan]

Maybe it was something that only got mentioned during the closed beta-testing? Or maybe it was mentioned on the forums but never got as far as a GitHub issue?

Ah ha! It was a forum post.