Extensibility guidance - customizing build flags for various embedded platforms

[paul-demo]

This is not a bug, but rather a request for information and/or documentation for your project. I'm a big fan, by the way, and the use of the long awaited platforms capabilities of bazel looks very promising. Last time I looked at this 3-5 years ago bazel was in an awkward transition from crosstool-style to platforms-style and everything was sort of half baked and not read yet.

Can you comment, here or in the project README, about what would be a good strategy for extending your project to customize it for a particular hardware platform? To be more specific I mean:

-> a set of custom copts/defines/includes for a given platform (ie, one Cortex-M MCU versus another; one board versus another, whatever).
-> customized copts/defines/includes for .S, .c, and .cc files, possibly also customized per target/library, and extensible to other target platforms should the need arise to switch microcontrollers

To get something working quickly, and because I'm not a bazel/skylark expert, I've started by wrapping the compatible-with cc_binary and cc_library you have set up with macros, adding in copt/include/define flags as necessary to match the recommended flags used by the target hardware and third party libraries. I tacked on an objcopy genrule to generate binary files from the .elf. Things like that. This works ok but has a number of unfortunate consequences that worry me long term:

• wrapping cc_library in a my_cc_library macro is quick and easy but is obviously not extensible to other devices. While this is probably acceptable for drivers which could only run on one target platform, for higher level [c++] code it would be nice if I stuck with the native library/binary and instead extended the rules or toolchain in bazel-embedded. I don't know how to do this, but just like you customize M4F -> fpu/cpu copt flags, there might be a level where you select {Vendor, Product} -> copt flags, or even {Project, Vendor, Product} -> copt flags. For example, maybe one project uses no RTTI and no exceptions but another project does use them. Maybe the two share some pieces of a monorepo and maybe they run on the same type of device or maybe on distinct devices.

• a lot of bazel target attributes (copts, defines, etc) are inherited/propagated: the copts of dependencies show up in the copts of libraries/binaries which depend upon them, producing an ever increasing chain of duplicated flags the more hierarchy there is. This works ok for something quick, but it's super verbose on the command line to have a copt flag duplicated for every level of hierarchy, and a more serious case I ran into is that gcc doesn't allow certain flags to be duplicated (like -specs=nano.specs) even if they're identical in each duplicate.

I guess what I'm asking is: what is the recipe for extending this nice set of features you have in bazel-embedded to customize it for a hardware platform? I don't particularly understand your STM32 examples with makefiles...why are there makefiles anyway if we're using bazel? Do I somehow add more "features" like you have for cpu/fpu? Do I do something else? I have an NXP driver library building in bazel, as well as startup code and various linker scripts, and the BUILD targets are super clean thanks to bazel and bazel-embedded. But, my issue is more about whether macros is the best way to do this, and the above issues with managing copt flags.

Could you show an example strategy that includes a set of compilation flags for .S files, a different set for .c files, a different set for .cc files? Ideally this would be specific to a hardware platform like stm_xyz or nxp_xyz, and not as generic as "arm-v7-m" or whatever architecture or core is used. I already have something like this somewhat working with my macros for a single target device, but for the aforementioned reasons, I suspect I may be doing it wrong!

[nathaniel-brough]

No worries, there is definitely room for improvement in the documentation for this project. I'll answer questions inline and then convert that to more complete docs later.

a set of custom copts/defines/includes for a given platform (ie, one Cortex-M MCU versus another; one board versus another, whatever).

There are a couple of ways of doing this at the moment. By far the simplest is to do this in your .bazelrc file. e.g.

# .bazelrc
# ...
# STM32H747XI config
build:stm32h747xi --platforms=@bazel_embedded//platforms:cortex_m7_fpu
build:stm32h747xi --copt=-D__your_c_only_define__ --cxxopt=-D__your_cc_defines

# STM32F4 config
build:stm32f4 --platforms=@bazel_embedded//platforms:cortex_m4_fpu
build:stm32f4 --copt=-D__your_c_only_m4_define__ --cxxopt=-D__your_cc_m4_defines

# etc.

These configs can be directly run from the command line e.g. bazel build //your_target --config stm32f4.

The other option to explore is as you mentioned using macros. This is the approach that the pigweed project has taken, a few examples of this here. There is also some documentation that I wrote for the pigweed project here and here that I can likely partially duplicate in this repo regarding configurations with Bazel and more advanced information on platforms.

wrapping cc_library in a my_cc_library macro is quick and easy but is obviously not extensible to other devices. While this is probably acceptable for drivers which could only run on one target platform, for higher level [c++] code it would be nice if I stuck with the native library/binary and instead extended the rules or toolchain in bazel-embedded. I don't know how to do this, but just like you customize M4F -> fpu/cpu copt flags, there might be a level where you select {Vendor, Product} -> copt flags, or even {Project, Vendor, Product} -> copt flags. For example, maybe one project uses no RTTI and no exceptions but another project does use them. Maybe the two share some pieces of a monorepo and maybe they run on the same type of device or maybe on distinct devices.

In terms of configuring flags and default libraries (e.g. libc++/libc) per project this is an area of active improvement and I am working on a far more flexible solution here though that toolchain currently only supports host x86_64 Linux, embedded ARM and RISCV are a WIP.

In terms of selecting {Project, Vendor, Product} -> copt flags this is currently possible. I would recommend reading through Pigweeds docs on this for now, until I add them here as well.

In terms of RTTI and exceptions, it is currently possible to disable these on all targets from the command line or per target using the set of toolchain features. Though I have just found a bug, filed as #47, in that these should really be inverted e.g. noexceptions. Using these features from the command line;

bazel build //your_target --features=-exceptions

e.g. from a target;

cc_library(
  name = "main",
  features = ["-exceptions"],
)

a lot of bazel target attributes (copts, defines, etc) are inherited/propagated: the copts of dependencies show up in the copts of libraries/binaries which depend upon them, producing an ever increasing chain of duplicated flags the more hierarchy there is. This works ok for something quick, but it's super verbose on the command line to have a copt flag duplicated for every level of hierarchy, and a more serious case I ran into is that gcc doesn't allow certain flags to be duplicated (like -specs=nano.specs) even if they're identical in each duplicate.

It is certainly possible to de-duplicate the flags on the command line, I did try this at some stage but found it quite challenging as I ran into ordering problems. This is certainly something that I exploring but as a lower priority.

Just to clarify, using the copts attribute in a cc_library is non-transitive so dependant libraries will not use inherit the copts flags. This can be somewhat unintuitive but it means that you can actually add non-transitive defines as well e.g. copts = ["-D__non_transitive_define__"]. However, you are right in saying that the defines attribute is transitive.

As a workaround for the -specs=nano.specs issue you can disable the specs toolchain feature e.g. --features=-sys_spec. Or per target by using the features attribute e.g. features=["-sys_spec"].

I guess what I'm asking is: what is the recipe for extending this nice set of features you have in bazel-embedded to customize it for a hardware platform? I don't particularly understand your STM32 examples with makefiles...why are there makefiles anyway if we're using Bazel? Do I somehow add more "features" like you have for cpu/fpu? Do I do something else? I have an NXP driver library building in bazel, as well as startup code and various linker scripts, and the BUILD targets are super clean thanks to Bazel and bazel-embedded. But, my issue is more about whether macros is the best way to do this, and the above issues with managing copt flags.

The makefiles themselves are actually created by STM32Cubemx when generating code. They are however unused in the build. I mostly just use them as a reference to know which files need to be included in the Bazel build and what defines I need. However recently I have been moving towards an approach that doesn't use Cubemx at all. So I can add examples in that regard. I would definitely peruse the Pigweed repo as that is probably the best open-source example that I could point you towards that is using this toolchain.

Could you show an example strategy that includes a set of compilation flags for .S files, a different set for .c files, a different set for .cc files? Ideally, this would be specific to a hardware platform like stm_xyz or nxp_xyz, and not as generic as "arm-v7-m" or whatever architecture or core is used. I already have something like this somewhat working with my macros for a single target device, but for the aforementioned reasons, I suspect I may be doing it wrong!

I will work towards providing better examples. If you have a specific query on open source code I'd be happy to point you in the right direction in the meantime.

[paul-demo]

Thanks for the detailed response. I do like the .bazelrc approach and I'll probably use that to keep it simple (rather than wade into an unfinished Google project like Pigweed).

I think one of the things that was throwing me off as I dug through a lot of subcommands output from bazel is that C libraries have fewer flags than CC libraries, and libraries / binaries have different flags as well. I was trying to force this with my own macros, but it appears to happen anyway from bazel/bazel-embedded. In retrospect, that makes sense since RTTI isn't relevant for C (for example). I noticed that -mthumb and -specs=... are only included for my [C++] binary, so I guess they are not relevant for compiling [C] libraries. I suppose this too makes sense since libraries just go to object file format and the whole thing doesn't get converted to [thumb] machine code until you make a binary out of it.

Anyway, I'm trying it out using a more elaborate .bazelrc instead of macros, and it seems like this approach will work for what I need to do. Basically I wasn't aware of the strategy of writing down --copts / --cxxopt / --conlyopt / --linkopt flags in a .bazelrc, so I was trying to get the same behavior by writing macros.

I did notice that passing --features=exceptions in the .bazelrc file doesn't seem to actually activate or deactivate the feature; that is: -fno-exceptions is included in both cases...do features only work when listed in a BUILD file? Or do I need to pass something like --features=@bazel-embedded//<path_to_exceptions_feature> ?

[nathaniel-brough]

I did notice that passing --features=exceptions in the .bazelrc file doesn't seem to actually activate or deactivate the feature; that is: -fno-exceptions is included in both cases...do features only work when listed in a BUILD file? Or do I need to pass something like --features=@bazel-embedded//<path_to_exceptions_feature> ?

By default, this toolchain disables exceptions as it is very rare to have an embedded RTOS that will correctly handle exceptions. I think the only RTOS that even attempts this is distortOS. As mentioned the features are misnamed and should be more descriptive as described in #47. Also to clarify, the noexceptions feature is enabled by default, do disable it you need to put a '-' in front of the command line. e.g. --features=-exceptions which despite the poor naming will disable the no-exceptions flags. Note that the minus sign in front of a feature will disable it.

[paul-demo]

Gotcha, I didn’t realize the minus sign was part of the semantics. Yeah, I wasn’t planning on enabling exceptions I was just using that as an example to understand how to use features. Thanks for the help!