CircuitPython now shipping with a second CDC serial device

[dhalbert]

This is a heads-up about a new feature added to CircuitPython that might affect Mu.

As of CircuitPython 6.2.0-beta.3, CIrcuitPython now includes a second USB CDC serial device in many builds. This is supported by the new usb_cdc module. usb_cdc.serials[0] is the REPL (still available in its previous forms, like input(), sys.stdin, etc.). usb_cdc.serials[1] is the new device.

The new serial device is like a plain UART over USB: you can send arbitrary bytes in either direction. There is no escaping or special handling of the bytes. So it provides a very convenient way for a board to communicate arbitrary data with a connected host computer, without running into ctrl-C problems, interference from the REPL, etc. The API is more or less a subset of pyserial: there are calls to see if data is ready to be read, etc.

There may be some potential issues with Mu because now a board may have two ports, but there also solutions. Further details are below.

On the host computer, you will see two serial ports, usually /dev/ttyACM0 and /dev/ttyACM1 on Linux; COM7 and COM8, for example, on Windows; something like /dev/cu.usbmodem141301 and /dev/cu.usbmodem141303, say, on MacOS. When I did initial testing on this on all the platforms, I found that the first port ended up being the REPL port, and the second was the new serial port. But have since had one user whose Windows box assigned them in the opposite order. (I asked them to try Uwe SIeber's USB cleanup tool and try again.)

When I tested Mu initially, it seemed to look for the ports in sorted order, and its choice of port was always the REPL. However, it's possible this might not always be true. So Mu might want to vet that a particular port is the REPL port. It could be done by the interface name (see below), or perhaps by interaction with port if that isn't available. For instance, if there's output when the board is first connected, it's likely to be the REPL port.

These devices have different USB Interface names:

• REPL port (these are the same names as before)
  • CircuitPython CDC control
  • CircuitPython CDC data
• new secondary port:
  • CircuitPython CDC2 control
  • CircuitPython CDC2 data

It is possible to retrieve these interface names via pyserial or other means. There is a bug in pyserial on MacOS which causes the same name to appear for both ports. There is a PR to fix the issue, developed by one of our community members. In addition that code is pretty standalone, so it could be incorporated elsewhere easily.

I looked for a way to get interface names via Qt, but could not find any such capability. If someone knows how, let us know.

In the future, Mu could provide a second optional Serial window to capture and interact with the second port, but that could easily be beyond the scope of Mu, and we certainly don't see it as something vital. It depends on the use cases and popularity of this second port.

We welcome your comments and thoughts. We do not want this to be a support headache, but we do anticipate people wondering about this and perhaps their Mu getting confused. We need to see what will happen and what we need to do to deal with it. In the future, we will probably make this toggleable in boot.py, but that is not a trivial implementation task right now. Thanks!

[dybber]

Thanks for the heads up. Currently we use QSerialPortInfo from Qt to retrieve info about devices:

mu/mu/modes/base.py

Line 420 in f672193

available_ports = QSerialPortInfo.availablePorts()

But yes it seems like interface names is not part of their API: https://doc.qt.io/qt-5/qserialportinfo.html

We already need pySerial for one other small thing, where Qt isn't enough, so we wouldn't have to add an extra dependency, if we need to use pySerial to make this query.

CircuitPython mode derives from MicroPythonMode, and to do this, you would have to override the compatible_board function: https://github.com/mu-editor/mu/blob/master/mu/modes/base.py#L384

It should then first check the interface string, and only if it's for the REPL-port call the parents compatible_board function.

[carlosperate]

Thanks for the heads up @dhalbert.

The next release has a drop down menu at the bottom right corner where different CDCs can be chosen:

This was originally introduced to be able to select between multiple devices connected to the PC, but looks like it works for this case as well.

Certainly if we can detect the right CDC and automatically select that version that would be better, so we should look into that as well.

[dhalbert]

Excellent! I am literally at this moment working on a Python library/helper that will identify the correct port for the REPL, based on the interface name. It will use the pyserial comports() API, and for the moment include the fix mentioned above for MacOS.

[carlosperate]

That's great, let us know when it's out and we'll check it out!

[dhalbert]

We have decided to back off leaving the second CDC device on by default, starting in the next beta. Several groups of users found it interfered with their workflow, and we have seen Windows assign the COM ports in opposite order, probably if there was a free COM port lower down, and the REPL had originally been assigned to a higher one. The feature will still be available with a custom build of CircuitPython. In 7.x we would like to allow it to be enabled in boot.py, but be off by default. It will still be necessary to find the right port if it's on, so I'm still working on the helper library I described above, but the need is less urgent.

[ntoll]

👋 Hey hey @dhalbert - thanks for the heads up (and subsequent heads down). Looks like Mu is in a good place to support the work you've been doing. If/when things arrive, please do give us a collective kick and we'll check Mu is working with the way you have things set up. Hope you're well and big hugs from the UK..! 👍

[dhalbert]

I am working on a PR for this, which will use this new small library, which will be in pypi.org: https://github.com/adafruit/Adafruit_Board_Toolkit. It works for Linux and MacOS (tested on Big Sur 11.2.2). I'm still working on Windows, which does not return the interface name currently. I will either add that feature to pyserial tools.list_ports() or else guess based on the interface number.

Note that I need pyserial==3.5 to get this to work. Currently Mu is pinning pyserial==3.4. I don't know if this version is required for other reasons.

[carlosperate]

It was just the latest version at the time Mu started pinning dependencies: 803153f

I'd like to relax the version pins in setup.py, but there are advantages/disadvantages and we still need to discuss that.
We can probably set up 3.4 as the min version, I found some issues with earlier v3 releases (if I remember correctly while debugging issues on an old Raspian version back when the pyserial default was an outdated deb pacakge version), so I wouldn't go lower.

We have also separated the dependencies between "Mu core" and "modes", so PySerial is also in

mu/mu/wheels/__init__.py

Line 25 in bbe7e97

("serial", "pyserial>=3.0"),

We need to look into maybe move it to just be in one of the two places, but haven't got around that yet.