Add and use esp32_hwdef.py to allow creation of hwdef.h from hwdef.dat

[peterbarker]

Tested on a VM with esp32buzz.dat. The single line moved has effect (I added #errors to the code to make sure).

Modeled on similar patches made for the Linux HAL.

[tpwrules]

Could you try your hand at the esp32s3m5stampfly? That one I own and can flight test.

[peterbarker]

Could you try your hand at the esp32s3m5stampfly? That one I own and can flight test.

Yes indeed. Bare in mind I was really only looking to move very minor things into here, so don't expect everything to move over!

I'm hoping the builds are repeatable in esp32-land. I don't suppose you know if they are? That would make the process of moving everything into the hwdef.dat files much cleaner/faster...

[peterbarker]

Could you try your hand at the esp32s3m5stampfly? That one I own and can flight test.

I've pushed a branch moves a single define over. Seems to work:

vagrant@vagrant:/vagrant(pr/esp32-hwdef)$ ./Tools/scripts/extract_features.py /tmp/tmp75i6smwf/out-branch-esp32s3m5stampfly/esp32s3m5stampfly/esp-idf_build/ardupilot.elf --nm=xtensa-esp32-elf-nm | grep AIR
!AP_AIRSPEED_ANALOG_ENABLED
!AP_AIRSPEED_ASP5033_ENABLED
!AP_AIRSPEED_AUAV_ENABLED
!AP_AIRSPEED_DLVR_ENABLED
!AP_AIRSPEED_DRONECAN_ENABLED
!AP_AIRSPEED_ENABLED
!AP_AIRSPEED_MS4525_ENABLED
!AP_AIRSPEED_MS5525_ENABLED
!AP_AIRSPEED_MSP_ENABLED
!AP_AIRSPEED_NMEA_ENABLED
!AP_AIRSPEED_SDP3X_ENABLED
!AP_PERIPH_AIRSPEED_ENABLED
vagrant@vagrant:/vagrant(pr/esp32-hwdef)$ 

A bunch of stuff isn't working (size_compare_branches.py) which I'll probably want working before moving a bunch of stuff over.

... something changed, however:

vagrant@vagrant:/vagrant(pr/esp32-hwdef)$ cksum ./Tools/scripts/extract_features.py /tmp/tmp75i6smwf/out-*-esp32s3m5stampfly/esp32s3m5stampfly/esp-idf_build/ardupilot.elf
2593568385 28606 ./Tools/scripts/extract_features.py
1159079373 35391844 /tmp/tmp75i6smwf/out-branch-esp32s3m5stampfly/esp32s3m5stampfly/esp-idf_build/ardupilot.elf
1111509765 35397696 /tmp/tmp75i6smwf/out-master-esp32s3m5stampfly/esp32s3m5stampfly/esp-idf_build/ardupilot.elf
vagrant@vagrant:/vagrant(pr/esp32-hwdef)$ ./Tools/scripts/extract_features.py /tmp/tmp75i6smwf/out-branch-esp32s3m5stampfly/esp32s3m5stampfly/esp-idf_build/ardupilot.elf --nm=xtensa-esp32-elf-nm >/tmp/new
vagrant@vagrant:/vagrant(pr/esp32-hwdef)$ ./Tools/scripts/extract_features.py /tmp/tmp75i6smwf/out-master-esp32s3m5stampfly/esp32s3m5stampfly/esp-idf_build/ardupilot.elf --nm=xtensa-esp32-elf-nm >/tmp/old
divagrant@vagrant:/vagrant(pr/esp32-hwdef)$ diff -u /tmp/old /tmp/new
vagrant@vagrant:/vagrant(pr/esp32-hwdef)$ 

... probably need to make sure that board was built with the s3 compiler. Hmmm.

[tpwrules]

I am reasonably sure the whole env.py thing can go entirely. waf already handles that. It always bugged me that it wasn't a Python file and that's what my partial investigation concluded.

Concretely I wouldn't suggest putting in the effort to factor it out.

[peterbarker]

I am reasonably sure the whole env.py thing can go entirely. waf already handles that. It always bugged me that it wasn't a Python file and that's what my partial investigation concluded.

Concretely I wouldn't suggest putting in the effort to factor it out.

I'd be very happy to rebase on top of a patch in master that removes it :-)

... in the meantime, it seriously irked me I was making a third copy of that method :-)

[peterbarker]

@tpwrules any testing as yet?

[tpwrules]

What does this do? It is wrong for esp32s3. But the build works for it anyway.

[peterbarker]

So this is a default value for which toolchain to use for the things in that hwdef dir. The vast majority of our Linux boards use arm-linux-gnueabihf, so you can see that default in there. However, several boards override with TOOLCHAIN entry in their hwdefs - navigator64 is an example of that. Moving the toolchain requirement out of the waf stuff and into the hwdefs is definitely something to look forward to - we'll probably want something close to the ChibiOS hwdef's MCU line or something as an abstraction instead of specifying TOOLCHAIN in the hwdef... but we'll see.

This entry isn't used for compilation, only for working out which nm to run, for example.

[tpwrules]

Thanks for the patience. This does indeed build and fly on the Stamp Fly. I just tried a clean tree, did not try to change options or break anything.

I will come back and attack env.py later, I looked back at my notes and it was more complicated than I had remembered. So factoring it out is fine for now.

[peterbarker]

@tpwrules are you happy enough with this to approve it? It's a foot in the door, at least.

[davidbuzz]

just be aware that 9 months ago, as part of this mega-esp32- pr that never got merged I implemented a full version of this, mixed-in with the rest of the PR: #28514 . If I was doing it, I'd be pulling as many of the relevant bits from that PR as I could and/or perhaps just rebasing it and reducing the number of things it touches ... but i'm not, so this probably a step in the right direction for master.
that old mega PR has important things - its got a classic/s3 split, its got ap_periph on esp32s3 , its got hwdef.dat implemented enough to build targets with it ( while concurrently supporting the old-style build on a board-by-board basis, its got esp CAN drivers that at one point in the past worked , it also added --esp32 as a flag for configure_all.py please do get as much inspiration from there as u can/like.

[davidbuzz]

I'm just an interested watcher from this point, dont wait for me to respond or anything.

[peterbarker]

just be aware that 9 months ago, as part of this mega-esp32- pr that never got merged I implemented a full version of this, mixed-in with the rest of the PR: #28514 . If I was doing it, I'd be pulling as many of the relevant bits from that PR as I

I did look at pulling that stuff in. Unfortunately a lot has happened since then, notably factoring chibios_hwdef.py into a new hwdef.py. That allows this PR to have a pretty minimal extra footprint in terms of Python code to maintain, whereas the mega-PR was just chibios_hwdef.py copied sideways.

I can't commit to grabbing stuff in from the mega-PR - this is already some pretty epic yak-shaving as I'm trying to fix clang analysis warnings by removing ownptr, and the build system is making that problematic, this linux_hwdef.py and now esp32_hwdef.py.

[tpwrules]

Seems reasonable enough. I disagree with the toolchain selection for esp32s3 in one case but haven't dug in to see what it breaks if anything; maybe the nm is compatible.

I have also not had a chance to look through the mega PR branch though I do agree it would be nice to take lessons from.