Maximum PCIe lanes for the Pi Compute module 4?

[Hiruks]

Hi Jeff Geerling, I am a long term viewer and a fan of you for a long time,

I checked out multiple videos of you integrating and testing out different PCIe based devices in your compute module,
I am currently developing a board that carries data in a Gen 2 PCIe lane for an M.2 NVMe storage device, and I want to integrate another PCIe based WiFi module / card into my board, is there a specific limit to which I can include PCIe based device and a threshold of types of PCIe Gen types to use?

Thank you for your attention.
Best Regards!

[TheGuyDanish]

Hey Hiruks,

The CM4 only exposes a single PCI-e Gen 2 lane. Normally, this would mean that you would really only be able to attach one device to the Pi, as the lane is really a point to point connection, rather than a bus connection. The receive and transmit pairs cannot just be connected to multiple devices without some serious side effects.

To get around this, a lot of boards being made right now use PCI-e switches. Think of these the same way you'd think of an Ethernet switch with all-gigabit ports. You still only have one lane of bandwidth, but the switch can process all the different information and make the most possible use out of that x1 link to your upstream port on the CM4.

You can look into ICs like the ASMedia 1806 or the Diodes/Pericom PI7C9X2G404SL. This also does come with some limitations though. A board that uses NVMe through one of these switches are not currently able to use the beta 'boot from NVMe' feature, as the NVMe is hidden behind the PCIe switch until the right driver is initialized in the OS.

[Hiruks]

yeah it seems like a PCIe switch is an option but it sounds like there could be a performance cost, like would it effect the transfer speeds of the wifi and the NVMe? the wifi module and NVMe will share data between them, thus the data from the NVMe is shared from the WiFi module

[TheGuyDanish]

Theoretically, yes. To my knowledge, there's no easily available switch that implements a way for data packets to traverse from one PCI-e device to another (I could wrong about this) without first passing to the CPU. So you would be "sharing" the bandwidth between the two devices. If one device was to use the full x1 lane bandwidth, there would be little to none available for the other device.

If you'd be looking to try and build a board with this in mind, I'd suggest looking at the PI7C9X2G303EL. It works using one lane to the upstream device, then two separate lanes for downstream devices. It is obviously a decently complicated chip, but I find it quite interesting to go through the datasheet. I've also previously spoken to Diodes and almost immediately, they were quite happy to provide a reference design using one of their chips to help me get going on my own design.

[Hiruks]

yeah at this point I am considering reverse engineering a compute module like a board with the help of an experienced development organization, if I am custom developing one I might be able to traverse data directly towards both PCIe lanes, not sure about this approach but I might consider it, as a use of a switch may severely affect the both wireless performance and the data transmission performance to the NVMe drive

[TheGuyDanish]

Before you delve into developing any board, you could get all the bare pre-made boards you can (A PCI-e switch, the WiFi card and a PCI-e to NVMe adapter) and put them all together on a compute module and test that theory. See how the performance really is. Developing a new board is expensive, so testing the theory first without spinning your own PCB is probably the best way to go.