Changing Mellanox ConnectX VPI Ports to Ethernet or InfiniBand in Linux

3
Dual Mellanox ConnectX 5 In Inspur NF5468M5
Dual Mellanox ConnectX 5 In Inspur NF5468M5

In our recent Mellanox ConnectX-5 VPI 100GbE and EDR IB Review, we showed a unique feature of the Mellanox VPI cards: they can run in InfiniBand or Ethernet modes. Indeed, one can have a single adapter and use either protocol which is handy when you have a server with limited PCIe slots, but a need to access both types of high-speed networks. We have been using Mellanox VPI cards since the ConnectX-3 generation and even had an old guide around how to Change Mellanox ConnectX-3 VPI Cards between Infiniband and Ethernet in Windows. During the process of an upcoming server review, we had the opportunity to do the same using ConnectX-5 and Linux. In this short guide, we are going to show you how to change a Mellanox ConnectX VPI port to either Ethernet or Infiniband in Linux.

Changing Mellanox VPI Ports Prerequisites

There are a few things you will need to get started.

First, you will need a VPI capable cards. Mellanox makes three main types of cards: Ethernet only, Infiniband only, and VPI cards capable of both. You need the VPI versions and you may need to check a model number against a spec sheet to ensure you have a capable part.

Next, you need Mellanox drivers and tools installed. Here we suggest, that even if your OS has native support for Mellanox networking drivers you should download and install current drivers.

This is also a Linux guide. As we showed in the above-linked Windows article, you can do the same in Windows via the GUI.

Now that those prerequisites are settled, the process is very quick and easy.

Changing Mellanox VPI Ports from Ethernet to InfiniBand

Swapping from Infiniband to Ethernet or back on a Mellanox ConnectX-5 VPI card is really simple. There are tools to help you do this, but we have a simple three-step process in the lab. First, we see what devices are installed.

In this case, we have two Mellanox ConnectX-5 VPI cards (CX556A’s) installed in the GPU server.

Mellanox Mst Status
Mellanox Mst Status

We can then select the port configuration. Here, 1 is for InfiniBand and 2 is for Ethernet.

Mellanox ConnectX 5 Set One Port IB And One Port Ethernet
Mellanox ConnectX 5 Set One Port IB And One Port Ethernet

The third step is to reboot. That is it. Here is a video with this in action:

We may pull this section out to be its own piece for others to reference later.

Final Words

This is an extremely useful feature if you need to access both 40GbE / FDR 56Gbps InfiniBand (ConnectX-3) or 100GbE / EDR 100Gbps InfiniBand (ConnectX-4 / ConnectX-5.) You can have one port set for each protocol and access both. This is one of the key reasons why we gave the Mellanox ConnectX-5 VPI card a coveted STH Editor’s Choice award in our recent review. Mellanox is the only vendor currently supporting both on the same card.

3 COMMENTS

  1. As far as I remember a reboot is not necessary, the changes apply immediately. I’m not sure if the cards even revert back to their original setting.

  2. Very good article.
    I have enabled an Ethernet connection between a ConnectX-4 card and a SFP+ switch using a breakout cable and it works fine.
    But, may you indicate the difference between infiniband and ethernet modes? I assume that the infiniband does not support connections on TCP/IP. But can it help to directly connect two devices? ConnectX-4 and SFP+ NIC on a NAS

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.