Skip to content

Repository files navigation

Work-In-Progress: Notes need updating. v1.1 Gerbers need to be manufactured and tested.

PCIe to Innova-2 MNV303212A RevA2 SlimSAS8x

This is a variant of the OpenCAPI-to-PCIe project optimized for the Innova-2 MNV303212A Rev:A2.

The Open Coherant Accelerator Processor Interface (OpenCAPI) was a standard that had FPGA-based Advanced Accelerated Cable (AAC) Add-In cards such as ADM-PCIE-9H3, ADM-PCIE-9H7, ADM-PCIE-9V3, ADM-PCIE-9V5, BittWare XUP-VV4, BittWare XUP-VVH, and Nvidia Innova-2 Flex SmartNIC.

The OpenCAPI interface is based on PCI-Express and uses a SlimSAS 8X (SFF-8654) Connector. This adapter enables connecting an OpenCAPI FPGA board to a host using PCIe over a SlimSAS cable.

OpenCAPI-to-PCIe Board

PCB Layout

4-Layer PCB. Inner 2 layers are GND planes. Differential pair are matched to a length of 61mm +/- 1mm both inter-pair and intra-pair (N-to-P).

OpenCAPI to PCIe x8 PCB Layout

Schematic

Note the reverse channel ordering and swapped _P and _N for RX3, 4, and 7. This is done to optimize for the MNV303212A-ADLT Rev:A2 board. Technically it should not matter. PCIe features automatic _P-to-_N swapping and the FPGA supports arbitrary lane assignment. I did this to eliminate all possible conflating problems while tracking down reliability issues.

OpenCAPI to PCIe x8 Schematic

Design Notes

Only a single component is needed for the adapter, a U10A474200T/U10A474240T SlimSAS 8x Right-Angle SMD Connector.

A SlimSAS 8x Cable such as the 3M 8ES8-1DF21(Datasheet) is required to use the adapter with an OpenCAPI FPGA Board.

Resistor R1 is shorted to connect nPRSNT1 to nPRSNT2_x8. The trace can be scratched off and nPRSNT1 can be connected to nPRSNT2_x1 or nPRSNT2_x4 to limit the PCIe lane width.

PCB Stackup

I am using values from JLCPCB.

4-Layer Stackup

Trace Impedance Control

OpenCAPI uses 85ohm impedance cables. I played with the values until I got the loosest differential pair coupling that is manufacturable with larger tolerances.

85ohm Differential Impedance in DigiKey Calculator

Testing

The innova2_xdma_opencapi project is designed to test this OpenCAPI-to-PCIe for MNV303212A RevA2 Adapter using an Innova2 Flex SmartNIC Rev:A2 board.

The Innova2 SmartNIC's XCKU15P FPGA does not have its Configuration Block in the same column as the OpenCAPI GTY transceivers so it is impossible to configure the FPGA within the PCIe Specification's 100ms time limit.

Innova2 XCKU15P Bank Diagram

Motherboard boot must therefore be delayed to allow the FPGA to configure itself before the OpenCAPI-to-PCIe device is enumerated by the host system. This can be accomplished by toggling the POWER button, then pressing and holding the RESET button for a second before releasing it. Or, connect a capacitor across the reset pins of an ATX motherboard's Front Panel Header:

Delay Boot Using Capacitor across Front Panel Header Reset Pins

Cable Performance

Using a 3M 8ES8-1DF21-0.75 cable:

System with 3M 8ES8-1DF21-0.75 Cable

PCIe Link Status is usually excellent:

lspci Link Status

Using an SFPCables.com SFF-8654 to SFF-8654 8i cable:

System with SFPCables SFF-8654 8i 85Ohm Cable

PCIe Link Status is downgraded:

lspci Link Status

Additional OpenCAPI Signals

Additional useful signals from the OpenCAPI connector are routed to a 6x1 0.1" Header (DigiKey). The pinout matches a TC74 I2C Temperature Sensor (DigiKey) for testing I2C. I2C pull-ups are on the AAC Add-in card (Innova-2). Note 3.3V is from the PCIe connector. PRE is a Presence Detect pin which is connected to GND via a 50-Ohm resistor on the OpenCAPI AAC Add-In card (Innova-2). RST is connected to PCIe/OpenCAPI RESET.

TC74A0-3.3VAT in OpenCAPI-to-PCIe Adapter

The innova2_xdma_opencapi project features the ability to test a TC74Ax-3.3VAT in an OpenCAPI-to-PCIe Adapter.

TC74A0-3.3VAT Testing in a System

About

OpenCAPI-Compatible SlimSAS 8x to PCIe Endpoint Adapter for the Innova-2 SmartNIC MNV303212A-ADLT Rev:A2

Topics

Resources

Stars

9 stars

Watchers

1 watching

Forks

Releases

Contributors