USB Custom Descriptors

[thestumbler]

I have a project with 20 or so RP2350 modules in a single system. The USB interface isn't used in the application, but for development and testing. It is becoming a pain to sort out which com port is which module.

I found a helpful discussion here by user Hippy. This substantially works, and I understand it well. At the end of that discussion it is noted that version 1.23 now includes dynamic USB drivers. Hippy points out: "Crafting initial versions of those is somewhat more complicated than copying what the MicroPython defaults are." I found some examples of dynamic USB code, but they are for changing more advanced aspects of the interface. I can't find any examples, complicated or not, of how to just change the USB descriptors (and serial number).

With that background, here is what I'd like to accomplish. By assigning a custom product descriptor, I can easily grab the desired com port from scripts on the PC. For example, "proto_meter_00, _01, _02, _03", etc. Here's an example I built last night:

Found port  /dev/cu.usbmodem101
   vid/pid       2e8a:0005
   description   proto_meter04
   manufacturer  TheStumbler
   product       proto_meter04
   serial        d589db7044d42859

Taking this approach, however, means that I would have to build a separate UF2 firmware file for each module. A smoother approach would be to make the product descriptor constant, say "proto_meter", and tweak the serial number by appending the meter number to the existing serial number (note that the meter number is read from DIP switches on each module). Ideally I'd like to be able to generate USB descriptors like below and only have a single UF2 firmware build (manually tweaked example of the above):

Found port  /dev/cu.usbmodem101
   vid/pid       2e8a:0005
   description   proto_meter
   manufacturer  TheStumbler
   product       proto_meter
   serial        d589db7044d42859-04

Questions:

(1) [NEW WAY] Are there any examples of changing USB descriptors using the dynamic USB device feature of the latest 1.23 release?

(2) [OLD WAY]

(2a) Applying the approach suggested by Mr Hippy raises a few build issues outside my previous MicroPython experiences. Other than replacing the CMakeLists.txt file in the rp2 directory, what's the usual process for dealing with custom cmake files? I tried to put CMakeLists.txt in my custom build folder, no joy. Then I extracted just the necessary bits into the .cmake file in the custom board folder, but that didn't work either.

(2b) Reading the documentation for the rp2 port closer, it states that the main way to build things is to use cmake, and the make commands are merely a convenience. But I couldn't figure out how to do a cmake-only build, and there are no examples of such that I could find. Perhaps if I knew that, I could have solved (2a) on my own.

(2c) Unlike Hippy, I put my changed USB files in my custom board folder rather than the main rp2 directory. But otherwise I still had to modify the main CMakeLists.txt file in the rp2 folder. But when I looked into changing the serial number, it seems that I have to tweak mp_usbd_port_get_serial_number() in file usbd.c. Is there an accepted way to localize all these changes to only the custom board folder? I'd like to not touch the main MicroPython code at all, and limit the changes to the custom board folder. But I don't know how to do that.

[mattytrentini]

You might want to watch the demo Damien presented on using USBDevice. You're right that the 'new way' allows you to modify the descriptors dynamically, though you'll probably find it much better to use an FTDI cable to connect to the REPL (otherwise, if your REPL is accessed over the same USB, the connection will drop when you update the descriptors).

If you're not already aware, there are some good examples in micropython-lib/usb.

There should be no need to build firmware to modify descriptors.

[thestumbler]

I'm getting much closer. I figured out how to make the strings, but updating the USB descriptor (as opposed to creating a whole new one) is eluding me so far. But I only want to update strings for the default CDC descriptor. There doesn't seem to be a method to only update the string dictionary (nor even examine the current dictionary). But it's after midnight and maybe things will be clearer in the morning.

Here's my start so far:

import machine
import sys
sys.path.extend(['/lib'])

# from shared/tinyusb/tusb_config.h
#    USBD_STR_MANUF (0x01)
#    USBD_STR_PRODUCT (0x02)
#    USBD_STR_SERIAL (0x03)
# from shared/tinyusb/mp_usbd_descriptor.c
#   .iManufacturer = USBD_STR_MANUF,
#   .iProduct = USBD_STR_PRODUCT,
#   .iSerialNumber = USBD_STR_SERIAL,

# string indices in the default descriptor
i_str_manuf = 0x01
i_str_product = 0x02
i_str_serial = 0x03

# module ID (will be read from DIP switches)
module_id = 0x03

# build some strings
str_module_id = f'{module_id:02x}'
str_unique_id = f'{int.from_bytes(machine.unique_id()):x}'
str_serial = f'{str_unique_id}_{str_module_id}'
str_product = f'proto_meter_{module_id:02x}'
str_manuf = 'TheStumbler'

# make the strings dictionary
mystrings = {}
mystrings[0] = None # need this for language
mystrings[i_str_manuf] = str_manuf
mystrings[i_str_product] = str_product
mystrings[i_str_serial] = str_serial

# for s in mystrings.items(): print(s)

desc_dev = machine.USBDevice.BUILTIN_CDC.desc_dev
desc_cfg = machine.USBDevice.BUILTIN_CDC.desc_cfg

# this isn't working, manually or in boot.py
# machine.USBDevice.active(False)
# machine.USBDevice.config(
#  desc_dev, desc_cfg, \
#  desc_strs=mystrings \
# )
# machine.USBDevice.active(True)

When it fails, I get this message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: argument has wrong type

[thestumbler]

BTW, even though I h ent succeeded yet, I completely prefer this new dynamic method over rebuilding the firmware for each different module ID in my system. I did that last night for just four meters, so I could have some working firmware to begin integration testing. That was a pain in the neck!

I also need to finish watching the presentation you linked. I got about 15 minutes in and said to myself, "Ah ha. I got this" and started trying to implement it. Maybe I missed some key points later on in the video.

[thestumbler]

@mattytrentini so, I stumbled around over the weekend and arriving at a workable but not quite correct solution. The presentation you linked above was indeed helpful. But last night, I finally realized you were pointing me in the direction of the higher-level usb drivers solution found in the micropython-lib, so I had a look and tried that approach.

I can appreciate how this driver makes the task of creating a new USB device much easier. But in the case of just changing the descriptors of the main CDC serial port REPL, I'm not sure it has any advantages.

The CDC example works as promised in the notes. I even found the arguments one passes for customizing the strings. When you run this example, you end up with two identical CDC serial ports connected to the REPL. Good.

Now let's say I want the UART REPL for testing, as advised. I could not figure out how to have three REPLs running. No matter what I tried, there seemed to be a hard-baked limit of two REPLs. The index parameter to os.dupterm() would only accept 0, weird since if there are two REPL "slots", I'd think the index could be 0 and 1.

But on the USB front, I don't want two CDC serial ports. My goal is simply to change the descriptor strings. I could not figure out how to disable the builtin CDC driver once I created the second one. If I set machine.USBDevice.active(False), that killed both CDC ports. Creating the new CDC with builtin_driver = False didn't do what I expected, there were still two CDC ports.

Strangely, the two ports have identical descriptor strings. I would expect the original port to remain as "Board in FS mode", and the new port to have the custom strings. Note -- the two serial port name on my PC are distinguishable because underscores are appended, like _1 and _3.

So, I could create custom descriptor strings using this method. But no matter what I did, it always resulted in two ports with identical descriptor strings. Furthermore, using the usb drivers did not solve the mpremote or rshell soft reset problem. In other words, I still had to include the resume option in all my mpremote commands.

So while this library makes usb device drivers easier, in the very simple case of just changing basic CDC device descriptors, I think it causes more confusion than the direct USBDevice() approach, because it creates two ports instead of one, and no way to also have a UART REPL for debugging.

[scruss]

It looks like you're using MacOS, so I don't know how easily this is done there. But on Linux, the board with machine.unique_id() == 0ecb4f7dccb7027b appears as serial port /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_0ecb4f7dccb7027b-if00

[thestumbler]

Yeah, for some reason their BSD style doesn't implement that technique. But even if it did, I'd still have to manually sort out the unique ID for each of 20-some modules and map them in an initialization script. I already do this, because even without that /dev folder, I can see the serial number using various utilities like lsusb.

By making my own custom string descriptors, I can completely automate the init script that scans the usb serial ports and initializes some variables for ease of use.

This init script assigns a shell variable to all 20-some modules in an assembly, and then I have a bunch of aliases to the mpremote and rshell tools. End result, I can simply type mpr0 reset or rs18, etc., to work with any desired meter.

[thestumbler]

This seems to have fixed it.
Key changes:
(1) instance vs type
WAS: machine.USBDevice
IS: machine.USBDevice()
(2) adding usbdev.builtin_driver = usbdev.BUILTIN_CDC
(3) REPL on UART for testing, not needed for final
Thanks to folks on the Discord channel for help.

import os
import machine
import sys
sys.path.extend(['/lib'])

# this is useful for testing but not necessary
# once everything is working
#uart = machine.UART(0,115200,tx=16,rx=17)
#os.dupterm(uart)

# from shared/tinyusb/tusb_config.h
#    USBD_STR_MANUF (0x01)
#    USBD_STR_PRODUCT (0x02)
#    USBD_STR_SERIAL (0x03)
# from shared/tinyusb/mp_usbd_descriptor.c
#   .iManufacturer = USBD_STR_MANUF,
#   .iProduct = USBD_STR_PRODUCT,
#   .iSerialNumber = USBD_STR_SERIAL,

# string indices in the default descriptor
i_str_manuf = 0x01
i_str_product = 0x02
i_str_serial = 0x03

# module ID (will be read from DIP switches)
module_id = 0x11

# make strings
str_module_id = f'{module_id:02x}'
str_unique_id = f'{int.from_bytes(machine.unique_id()):x}'
str_serial = f'{str_unique_id}_{str_module_id}'
str_product = f'proto_meter_{module_id:02x}'
str_manuf = 'TheStumbler'

mystrings = {}
mystrings[0] = None # need this for language
mystrings[i_str_manuf] = str_manuf
mystrings[i_str_product] = str_product
mystrings[i_str_serial] = str_serial

for s in mystrings.items(): print(s)

usbdev = machine.USBDevice()
desc_dev = usbdev.BUILTIN_CDC.desc_dev
desc_cfg = usbdev.BUILTIN_CDC.desc_cfg
usbdev.active(False)
usbdev.builtin_driver = usbdev.BUILTIN_CDC
usbdev.config( 
    desc_dev, desc_cfg, \
    desc_strs=mystrings \
)
usbdev.active(True)
usbdev.active(True)

[thestumbler]

Thanks for taking the time to look at my problem @projectgus For the moment it's comforting to know that I've stumbled on the most correct solution.

I would like to suggest that consideration be given to the use case of custom descriptor strings only. It seems that this is something lots of people struggle with when connecting to USB serial ports of all sorts of embedded devices, not just MicroPython. My case with a system of 20-some modules is an extreme example, but being able to clearly identify one's project by customizing USB descriptors is a common enough request from what I've seen, and provides real benefits to the users.

I am not well versed in the bowels of the USB specification and its implementation here, but based on my limited experimentation the last few days, it sure feels like implementing this feature wouldn't be a difficult thing.

Whether or not dynamic descriptor strings can be implemented in an easier way, I think there is also some benefit to making it easier to change the default strings upon building the firmware, just by adding entries to the mpconfigboard file. Currently it's a dance (though well explained by user Hippy mentioned way at the top of this discussion) requiring tweaking the main CMake file and copy / editing a couple of tinyusb modules.

[thestumbler]

Oh -- one more observation. When I used the direct USBDevice() dynamic method, or "old style" building custom MicroPython firmware per module serial number, the resulting serial ports on my system were,let's say, "normal".

/dev/cu.usbmodem1101

If I make the port, well, pair of ports, using the mip-installed usb library, the usb serial port shows up as

/dev/cu.usbmodemmodule_141
/dev/cu.usbmodemmodule_143

I note that the text "module_14" is a place holder I put in the descriptor string for serial number. But on the other custom ports above, the serial number, custom or not, doesn't get appended to the PC name for the /dev serial port. I don't really care one way or another, I just thought it was interesting that they behaved differently

[thestumbler]

Looks like USBDevice has been removed from MicroPython. That's sad.

[Josverl]

As replied on Discord, i still see it in the nightly esp32 s3 and pico firmwares.

What board-variant do you use?

[thestumbler]

I had a brain fart. I had two board definition directories, WAVESHARE_..... and WAAVESHARE_....._CUSTOM_USB

Not having rebuilt it for several months, I just assumed the CUSTOM_USB was the one that dealt with my custom usb descriptors. Wrong. That build was one where I had been testing hard-coding the descriptors in the tusb* files, not dynamically.

Once I got that sorted, it all works again at 1.26. That's the problem with working on code at 1 AM 8-)

[thestumbler]

Well, all is not quite well here. I have a boot.py that changes the usb descriptors. I have used it on other MicroPython RP2350 based boards no problem. On this Pimoroni Pico Plus2 module, I haven't been as concerned before (there's only one of them in my system vs 20 of the other boards). I got to digging deeper, because something just wasn't right. It seems that if I run boot.py manually, e.g., import boot, the descriptors get changed. But doing a normal machine reset, be it hard or soft, the descriptors are not being set.

I added a temporary block of code to write a file at the end of boot.py, just to see if the script was even being executed. I learned that nothing happens when the board boots "naturally" via reset, apparently. But as before, if I force boot.py to run by importing it at the REPL, it does indeed change the USB descriptors and writes an entry into the test file.

It seems that the Pimoroni build of MicroPython does not run boot.py?

import sys
import os
import machine

# Let's modify the USB descriptor strings...
# ----------------------------------------
# from shared/tinyusb/tusb_config.h
#    USBD_STR_MANUF (0x01)
#    USBD_STR_PRODUCT (0x02)
#    USBD_STR_SERIAL (0x03)
# from shared/tinyusb/mp_usbd_descriptor.c
#   .iManufacturer = USBD_STR_MANUF,
#   .iProduct = USBD_STR_PRODUCT,
#   .iSerialNumber = USBD_STR_SERIAL,
# string indices in the default descriptor
i_str_manuf = 0x01
i_str_product = 0x02
i_str_serial = 0x03

# note: in the meter MicroPython project
# the USB serial number was determined by
# config DIP switches.
# This interface board, there is only one
# in each system. So hard code it.

# make the strings...
str_serial = f'{int.from_bytes(machine.unique_id()):016x}'
str_product = f'proto_front_panel'
str_manuf = 'TheStumbler'

# put then in a dictionary
mystrings = {}
mystrings[0] = None # need this for language
mystrings[i_str_manuf] = str_manuf
mystrings[i_str_product] = str_product
mystrings[i_str_serial] = str_serial
# for s in mystrings.items(): print(s)

# grab the CDC USB device, change its strings
usbdev = machine.USBDevice()
desc_dev = usbdev.BUILTIN_CDC.desc_dev
desc_cfg = usbdev.BUILTIN_CDC.desc_cfg

usbdev.active(False)
usbdev.builtin_driver = usbdev.BUILTIN_CDC
usbdev.config( 
    desc_dev, desc_cfg, \
    desc_strs=mystrings \
)
usbdev.active(True)

rtc=machine.RTC()
t=rtc.datetime()
boot_time = f"booting @ {t[0]:04d}-{t[1]:02d}-{t[2]:02d} {t[3]:02d}:{t[4]:02d}:{t[5]:02d}\n"
with open("boot.txt", "a") as myfile:
  myfile.write(boot_time)