Add support for DFU upload methods (#443)

* Implement dfu-util support

* Fix terminal progress bar

* DFU support: add STM32CUBE DFU

* Quick doc fixes

Author: Jamie Smith

connectivity/drivers/emac/CompositeEMAC.md

@@ -1,6 +1,6 @@
 # Composite EMAC
 
-The original Mbed OS EMAC API was added in Mbed OS 5.9. That API grouped all EMAC functionality into a single abstract class that targets/vendors had to implement, the `EMAC` class. Since then, dozens of Mbed targets have received EMAC drivers, and the strengths and weaknesses of this API have become clear. The general structure is good, and the idea of abstracting the memory manager and the network stack from the EMAC driver works well.
+The original Mbed OS Ethernet driver (aka EMAC driver) API was added in Mbed OS 5.9. That API grouped all Ethernet MAC functionality into a single abstract class that targets/vendors had to implement, the `EMAC` class. Since then, dozens of Mbed targets have received EMAC drivers, and the strengths and weaknesses of this API have become clear. The general structure is good, and the idea of abstracting the memory manager and the network stack from the EMAC driver works well.
 
 However, the [EMAC interface](https://github.com/mbed-ce/mbed-os/blob/0553c111850997d847dc6a3189ac0b7048304e57/connectivity/netsocket/include/netsocket/EMAC.h#L33) is difficult to implement, especially for people not intimately familiar with Mbed and its IP stacks. It requires EMAC implementations to implement the specifics of memory management, MAC address tracking, and DMA ring usage themselves, even though quite a bit of this logic is common to all MAC drivers. This has led to duplicated code, and quite often to half-assed code as well as chip vendors have struggled to conform to the (in some ways not very well defined) EMAC API. 
 
@@ -20,11 +20,13 @@ Before we can get into the details of how CompositeEMAC works, we need to go ove
 
 To run an ethernet connection, two chips need to work together*: the microcontroller and an external Ethernet PHY (PHYsical layer transceiver) chip. The microcontroller sends and receives logic level Ethernet packets, while the PHY transforms those into Ethernet signals, which are decidedly *not* logic level (and actually have a lot in common with radio signals). The Ethernet signals, called MDI (Media Dependent Interface) pairs, are sent through an isolation transformer, which removes common mode interference and provides electrical isolation (e.g. so that the two ends of the connection can have different ground voltage levels). Then, they go into the ethernet jack and across the ethernet cable to the link partner!
 
+*though some MCUs, e.g. TI's TM4C129 line, do have the Ethernet PHY integrated into the MCU.
+
 The PHY and the MCU are connected via a standard called [Reduced Media Independent Interface](https://en.wikipedia.org/wiki/Media-independent_interface#RMII) (RMII), which transfers the Ethernet packets as serialized bytes. This is an 8-wire bus with a 50MHz clock, four receive lines, and three transmit lines. The clock is traditionally either supplied by the PHY or by a dedicated clock generator chip, though some MCUs support supplying this clock as well. In addition to RMII, there's also a two-wire command and control bus called [Management Data IO](https://en.wikipedia.org/wiki/Management_Data_Input/Output) (MDIO) (though it can also be referred to Station Management Interface (SMI) or even "MiiM" for some reason). MDIO is used for talking directly to the PHY, not for sending Ethernet packets. MDIO is an open-drain bus similar to I2C, but with 16-bit words instead of bytes and a specific frame format (referred to as "Clause 22"). Unlike RMII, MDIO is a multi-drop bus, so you can actually connect up to 15 PHYs or other devices to one set of MDIO lines as long as they have different addresses!
 
 Inside the microcontroller, the bridge between the CPU and Ethernet is a peripheral called the Ethernet MAC. MAC stands for "Media Access Control" and refers to the second layer of the Ethernet protocol stack, the logic which encodes Ethernet packets and decides when to send them across the wire. The MAC has a number of moving parts inside, which are shown in the diagram above. The simplest is the block of configuration registers, which is accessible at a specific memory address and sets up operation of the MAC (e.g. what MAC addresses the hardware should accept and which checksums should be inserted/checked by the MAC). There is also an MDIO master interface, which controls the MDIO lines to talk to the PHY. And then, we have the DMA.
 
-Every Ethernet MAC I've seen also has DMA functionality. This means that the Ethernet peripheral can transmit and receive packets without direct CPU intervention. This is very important because it means your device can hit high network speeds without needing to have your CPU blocked for lots of time waiting on Ethernet packets to move through the hardware! For transmit, there will be a Tx DMA module which fetches data from the main RAM, and then enqueues the packet bytes plus control information into a FIFO (which is usually at least a couple thousand bytes long). Then, another block in the MAC, sometimes called the MTL (MAC Translation Layer) takes these bytes, applies any needed Ethernet framing, and shifts them out of the RMII Tx port.
+Every Ethernet MAC I've seen also has DMA functionality. This means that the Ethernet peripheral can transmit and receive packets without direct CPU intervention. This is very important because it means your device can hit high network speeds without needing to have your CPU blocked for lots of time waiting on Ethernet packets to move through the hardware! For transmit, there will be a Tx DMA module which fetches data from the main RAM, and then enqueues the packet bytes plus control information into a FIFO (which is usually at least a couple thousand bytes long). Then, another block in the MAC operating on its own timing, sometimes called the MTL (MAC Translation Layer) takes these bytes, applies any needed Ethernet framing, and shifts them out of the RMII Tx port.
 
 For reception, the process works the same but in reverse: the decoder and shifter block takes in packets and enqueues their bytes into the Rx FIFO. Then, the Rx DMA dequeues the packets and stores them into RAM at the right location.
 
@@ -139,7 +141,7 @@ The Tx interrupt* will detect that the descriptor pointed to by `txReclaimIndex`
 *Actually, interrupts cannot deallocate memory in Mbed, so the Tx interrupt handler actually signals the MAC thread to wake up and process the completed descriptor.
 
 ##### Final State
-Eventually, after a couple hundred microseconds, the second packet will be transmitted, and its descriptors will also be reclaimed. The Tx descriptor ring is now empty, except that the descriptor pointers are now both pointing to Desc 3.
+Eventually, after a couple hundred microseconds, the second packet will be transmitted, and its descriptors will also be reclaimed. The Tx descriptor ring is now empty, and matches the initial state except that the descriptor pointers are now both pointing to Desc 3.
 
 ![DMA ring after packet 1 deallocated](doc/tx-ring-step-5.svg)
 
@@ -196,7 +198,7 @@ Anyway, when the packet is received, the MAC will write it into the next three a
 
 The Rx ISR checks the DMA ring to make sure we have at least one complete packet. Then, it signals the MAC thread to dequeue the packet and pass it off to the IP stack.
 
-When the packet is dequeued, the buffers will be removed from the descriptors and, for now, they stay in the "owned by application" region of the descriptor ring (between `rxNextIndex` and `rxBuildIndex`).
+When the packet is dequeued, the buffers will be removed from the descriptors and, for now, the descriptors stay in the "owned by application" region of the descriptor ring (between `rxNextIndex` and `rxBuildIndex`).
 
 ![Rx ring after dequeue](doc/rx-ring-after-dequeue.svg)
 

tools/cmake/UploadMethodManager.cmake

@@ -98,7 +98,7 @@ endif()
 # Load the upload method.
 # Upload methods are expected to refer to the following variables:
 # - MBED_UPLOAD_SERIAL_NUMBER - USB serial number of the mbed board or of the programmer
-# - MBED_UPLOAD_BASE_ADDR - Base address of the flash where the bin file will be updated
+# - MBED_UPLOAD_BASE_ADDR - Base address of the flash where the bin file will be written to
 #
 # Upload methods are expected to set the following variables:
 # - UPLOAD_${UPLOAD_METHOD}_FOUND - True iff the dependencies for this upload method were found

tools/cmake/upload_methods/Finddfu_util.cmake

@@ -0,0 +1,47 @@
+# Copyright (c) 2025 Jamie Smith
+# SPDX-License-Identifier: Apache-2.0
+
+# ----------------------------------------------
+# CMake finder for the dfu_util tool
+#
+#
+# This module defines:
+# dfu_util_PATH - full path to dfu-util executable
+# dfu_util_FOUND - whether or not the ArduinoBossac executable was found
+
+set(dfu_util_PATHS "")
+
+# try to figure out where dfu_util may be installed.
+# We will look for it both from the Arduino IDE, and from a manual install.
+if("${CMAKE_HOST_SYSTEM_NAME}" STREQUAL "Windows")
+
+    # on Windows, assume that the user extracted the binaries to Program Files
+    # if the host is 64 bit, there will be a Program Files (x86) folder, this covers both
+    file(GLOB dfu_util_PATHS "C:/Program Files*/dfu-util*/")
+
+    # On my computer the path is C:\Users\jamie\AppData\Local\Arduino15\packages\arduino\tools\dfu-util\0.10.0-arduino1\dfu-util.exe
+    file(GLOB dfu_util_arduino_PATHS "$ENV{LocalAppData}/Arduino*/packages/arduino/tools/dfu-util/0*")
+    list(APPEND dfu_util_PATHS ${dfu_util_arduino_PATHS})
+else()
+
+    # Linux / Mac
+    # a possible path would be $HOME/.arduino15/packages/arduino/tools/dfu-util/0.10.0-arduino1/dfu-util
+    file(GLOB dfu_util_PATHS "$ENV{HOME}/.arduino*/packages/arduino/tools/dfu-util/0*")
+
+endif()
+
+find_program(dfu_util_PATH NAMES dfu-util HINTS ${dfu_util_PATHS} DOC "Path to the dfu-util executable")
+
+if(EXISTS "${dfu_util_PATH}")
+    # Detect version
+    execute_process(COMMAND ${dfu_util_PATH} --version
+            OUTPUT_VARIABLE dfu_util_VERSION_OUTPUT)
+
+    # The output looks like "dfu-util 0.11\n", so use a regex to grab the version
+    string(REGEX REPLACE "dfu-util ([^\n]+).+" "\\1" dfu_util_VERSION ${dfu_util_VERSION_OUTPUT})
+endif()
+
+find_package_handle_standard_args(dfu_util REQUIRED_VARS dfu_util_PATH VERSION_VAR dfu_util_VERSION)
+
+
+

tools/cmake/upload_methods/UploadMethodDFU_UTIL.cmake

@@ -0,0 +1,35 @@
+# Copyright (c) 2025 Jamie Smith
+# SPDX-License-Identifier: Apache-2.0
+
+### dfu_util upload method
+### This method can be used for chips with DFU bootloaders that talk to the dfu-util program.
+### The chip must be rebooted into bootloader mode for the upload method to work.
+# This method creates the following parameters:
+# DFU_UTIL_TARGET_VID_PID - VID:PID pair of the target device being programmed.
+# DFU_UTIL_TARGET_INTERFACE - Interface number of the interface on the target that should be programmed.
+
+set(UPLOAD_SUPPORTS_DEBUG FALSE)
+
+### Check if upload method can be enabled on this machine
+find_package(dfu_util)
+set(UPLOAD_DFU_UTIL_FOUND ${dfu_util_FOUND})
+
+### Function to generate upload target
+function(gen_upload_target TARGET_NAME BINARY_FILE)
+
+	set(DFU_UTIL_SERIAL_ARGS "")
+	if(NOT "${MBED_UPLOAD_SERIAL_NUMBER}" STREQUAL "")
+		list(APPEND DFU_UTIL_SERIAL_ARGS --serial ${MBED_UPLOAD_SERIAL_NUMBER})
+	endif()
+
+	add_custom_target(flash-${TARGET_NAME}
+		COMMAND ${dfu_util_PATH}
+			--device ${DFU_UTIL_TARGET_VID_PID}
+			--download ${BINARY_FILE}
+			--alt ${DFU_UTIL_TARGET_INTERFACE}
+			${DFU_UTIL_SERIAL_ARGS}
+			--dfuse-address ${MBED_UPLOAD_BASE_ADDR}:leave
+		VERBATIM
+		USES_TERMINAL)
+
+endfunction(gen_upload_target)

tools/cmake/upload_methods/UploadMethodSTM32CUBE.cmake

@@ -34,7 +34,9 @@ function(gen_upload_target TARGET_NAME BINARY_FILE)
 		${STM32CUBE_CONNECT_COMMAND}
 		${STM32CUBE_UPLOAD_PROBE_ARGS} # probe arg must be immediately after -c command as it gets appended to -c
 		-w ${BINARY_FILE} ${MBED_UPLOAD_BASE_ADDR}
-		-rst)
+		-rst
+		VERBATIM
+		USES_TERMINAL)
 
 endfunction(gen_upload_target)
 

tools/cmake/upload_methods/UploadMethodSTM32CUBE_DFU.cmake

@@ -0,0 +1,29 @@
+# Copyright (c) 2025 Jamie Smith
+# SPDX-License-Identifier: Apache-2.0
+
+### STM32Cube DFU Upload Method
+# This method needs the following parameters:
+# STM32CUBE_DFU_CONNECT_COMMAND - "Connect" (-c) command to pass to the programmer
+
+### Check if upload method can be enabled on this machine
+find_package(STLINKTools COMPONENTS STM32CubeProg)
+set(UPLOAD_STM32CUBE_DFU_FOUND ${STLINKTools_FOUND})
+
+set(UPLOAD_SUPPORTS_DEBUG FALSE)
+
+### Function to generate upload target
+
+function(gen_upload_target TARGET_NAME BINARY_FILE)
+
+	set(STM32CUBE_UPLOAD_PROBE_ARGS sn=${MBED_UPLOAD_SERIAL_NUMBER})
+
+	add_custom_target(flash-${TARGET_NAME}
+		COMMENT "Flashing ${TARGET_NAME} with STM32CubeProg..."
+		COMMAND ${STM32CubeProg_COMMAND}
+		-c ${STM32CUBE_DFU_CONNECT_COMMAND}
+		${STM32CUBE_UPLOAD_PROBE_ARGS} # probe arg must be immediately after -c command as it gets appended to -c
+		-w ${BINARY_FILE} ${MBED_UPLOAD_BASE_ADDR}
+		VERBATIM
+		USES_TERMINAL)
+
+endfunction(gen_upload_target)