MTL RDMA Library#

Introduction#

The Media Transport Library (MTL) was developed to streamline the process of sending and receiving media over standard network interfaces, specifically focusing on the ST2110 standard. ST2110 is crucial for transmitting high-quality, uncompressed video and audio over IP networks. Traditionally, cameras used Serial Digital Interface (SDI) to send footage for processing. However, with the emergence of ST2110, media data can now be encapsulated in RTP packets and transmitted via Ethernet, making the process more straightforward and less reliant on specialized hardware. With MTL, you can transmit or receive ST2110 standard-compliant streams, eliminating the need for specialized and expensive ST2110 hardware. This technology can also transport raw video/audio data between data center machines.

One challenge with MTL’s ST2110 stack is its reliance on DPDK (Data Plane Development Kit), which demands dedicated CPU resources for network packet handling. This is where RDMA comes into play, offering a solution to offload these tasks from the CPU. Specifically, the Intel E810 Network Interface Card (NIC) supports RoCEv2, an RDMA implementation that uses the UDP protocol. By leveraging RDMA and its interrupt mode, significant CPU resources can be conserved, which is particularly advantageous in cloud computing environments where resource optimization is critical.

Enable IRDMA driver#

To enable RoCEv2 support on Intel E810 NIC, first you need to make sure the irdma kernel module is installed and loaded.

lsmod | grep rdma
irdma                 430080  0
i40e                  638976  1 irdma
ib_uverbs             196608  1 irdma
ib_core               516096  2 irdma,ib_uverbs
ice                  1081344  1 irdma

For some kernel version, RoCEv2 is not enabled by default. You need to reload the module.

sudo rmmod irdma
sudo modprobe irdma roce_ena=1

If you are using out of tree ice driver (see e810.md), you may need to install the out of tree irdma too.

Go to irdma to download the latest tarball and follow the readme to install.

Install RDMA dependencies#

sudo apt update
sudo apt install infiniband-diags rdma-core ibverbs-utils perftest

Configure RDMA devices#

Configure network interfaces#

For example the E810 interfaces are ens8np0 and ens9np0.

sudo ip addr add 192.168.99.110/24 dev ens8np0
sudo ip addr add 192.168.99.111/24 dev ens9np0
sudo ip link set dev ens8np0 up
sudo ip link set dev ens9np0 up
sudo sysctl net.ipv4.conf.ens8np0.accept_local=1
sudo sysctl net.ipv4.conf.ens9np0.accept_local=1

Check the connectivity.

ping -I ens8np0 192.168.99.111

Check active RDMA devices#

ibv_devices
    device                 node GUID
    ------              ----------------
    irdma0              6a05cafffec1b900
    irdma1              6a05cafffec1b901
ibstat
CA 'irdma0'
        CA type:
        Number of ports: 1
        Firmware version: 1.71
        Hardware version:
        Node GUID: 0x6a05cafffec1b900
        System image GUID: 0x6a05cafffec1b900
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 25
                Base lid: 1
                LMC: 0
                SM lid: 0
                Capability mask: 0x00050000
                Port GUID: 0x6a05cafffec1b900
                Link layer: Ethernet
CA 'irdma1'
        CA type:
        Number of ports: 1
        Firmware version: 1.71
        Hardware version:
        Node GUID: 0x6a05cafffec1b901
        System image GUID: 0x6a05cafffec1b901
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10 (FDR10)
                Base lid: 1
                LMC: 0
                SM lid: 0
                Capability mask: 0x00050000
                Port GUID: 0x6a05cafffec1b901
                Link layer: Ethernet

Run RDMA perftest#

In one shell, start the server with device 0.

ib_write_bw -d irdma0

In another shell, start the client with device 1, specify the IP of device 0. The test result will show the max write bandwidth with single queue pair. You can specify 8 queue pairs by adding -q 8 to both sides.

ib_write_bw -d irdma1 192.168.99.110
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : irdma1
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : OFF
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x01 QPN 0x0003 PSN 0xc30267 RKey 0x69540273 VAddr 0x007e4ef245e000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:99:111
 remote address: LID 0x01 QPN 0x0003 PSN 0x2ce2c7 RKey 0xbfde4945 VAddr 0x0075bee2e6f000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:99:110
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             1103.39            1103.38            0.017654
---------------------------------------------------------------------------------------

Build MTL RDMA library#

You can build entire MTL project with build.sh, or use meson to build RDMA library only.

./build.sh
# or
cd <mtl>/rdma
meson setup build
sudo meson install -C build
# check package installed
pkg-config --libs mtl_rdma
# -L/usr/local/lib/x86_64-linux-gnu -lmtl_rdma

Run TX sample app#

Prepare a file (test.yuv) of 1920 * 1080 UYVY frames to send. You can refer to run.md.

./build/app/RdmaVideoTxSample 192.168.99.110 20000 test.yuv

Run RX sample app#

You need SDL library to display the received frame.

./build/app/RdmaVideoRxSample 192.168.99.111 192.168.99.110 20000