How are high-speed board-to-board connectors used in ML and AI systems?

High-speed board-to-board connectivity is evolving to meet the increasing performance demands and need for scalability of machine learning (ML) and artificial intelligence (AI) systems. The basic PCIe connectors and Card Electromechanical (CEM) form factor are still widely used to support scalability and high performance, but there are also new contenders.

This article reviews the basics of CEM cards and then looks at how they are being supplemented and replaced by newer open accelerator modules (OAMs) from the Open Compute Project and the CXL 3.1 specification from the Compute Express Link Consortium. OAMs and CXL 3.1 build upon the physical and electrical interfaces of PCIe.

CEM cards sometimes referred to as “add-in cards” (AICs), support a wide range of functions, such as graphics accelerators, memory expansion, and network interfaces. They are available in a range of sizes, lane counts, and corresponding connector sizes, including x1, x2, x4, x8, and x16.

CEM cards can support speeds from 2.5 GT/s to 16 GT/s and higher. They support powerful solutions for AI and ML applications with power ratings from 75 W to 600 W. Some CEM cards support external cable attachments to link multiple servers or storage boxes. Because of their 75 W minimum power rating, CEM solutions cannot always support multiple cards per system, limiting scalability for AI and ML solutions.

As a result, CEM cards are being supplemented and replaced by alternatives like OAMs. OAMs support design options including a choice of convection or forced air cooling and liquid cooling. These modules use high-density connectors that combine reduced signal loss, low levels of crosstalk, and higher data rates with sufficient pins to support AI accelerators, memory, and more.

The v1.0 OAM standard includes a mezzanine connector that can handle 56 Gbps for adding graphics processing unit (GPU) AI accelerators. One implementation of the v1.0 standard is a high-density hermaphroditic connector system available with 107 to 115 differential pairs per square inch that meets the signal integrity needs of 56 Gbps PAM4 signaling (Figure 1).

Figure 1. Stackable, hermaphroditic mezzanine connectors for OAMs. (Image: *Molex*)

CXL for memory and more

CXL adds to PCIe’s physical and electrical interfaces. It’s built around a PCIe 5 feature that enables other protocols to use the PCIe physical layer (PHY). CXL supports three device types: Type 1 includes smart NICs and accelerators, Type 2 are accelerators with cache, and Type 3 are memory devices.

When a CXL device plugs into a PCIe x16 slot, the device negotiates with the host processor. The CXL transaction protocols are used if both sides support CXL, otherwise, they default to the PCIe protocol. CXL versions include:

CXL 1.1 and 2.0 use the PCIe 5.0 PHY and support speeds up to 64 Gbps in each direction over a 16-lane link.
CXL 3.1 uses the PCIe 6.1 PHY and supports speeds up to 128 Gbps over an x16 link. CXL 3.1 replaces nonreturn to zero (NRZ) with PAM4 signaling found in the PCIe 6.1 specification and doubles the data speed compared to CXL 1.1 and 2.0.

EDSFF, AICs and CXL

The Compute Express Link organization has integrated the enterprise and data center standard form factors (EDSFF) E1.S, E3.S, E3.L, and AIC form factors. While AICs are part of the PCIe ecosystem, the Storage Networking Industry Association (SNIA) maintains EDSFF.

EDSFF supports x4, x8, and x16 PCIe lane configurations. The various EDSFF form factors can also be used to implement CXL devices. The SFF-TA-1016 specification defines vertical and right-angle connectors used for EDSFF devices (Figure 2). The specification supports 38-, 74-, 124-, and 148-pin implementations and power levels from 25 W up to 70 W. The standard supports both NRZ and PAM4 signaling, including PCIe 6.0.

Figure 2. EDSFF E3.S (short) and E3.L (long) module outlines and power ratings. (Image: *Compute Link Express*)

Summary

Board-to-board connectors, including mezzanine and card edge configurations, support the scalability and flexibility needed by AI/ML systems. There are several industry-standard expansion card formats based on the PCIe PHY that support high-performance applications, including CEMs, OAMs, and CXL and EDSFF devices.