How To Configure IPoIB with Mellanox HCAs – Ubuntu 12.04.1 LTS

It has been a little more than a decade since the introduction of the first consumer computer with gigabit Ethernet and there have been drastic increases in CPU processing power and storage throughput. Gigabit Ethernet has become an obvious bottleneck for high throughput applications. 10Gbps is the next step for Ethernet, but is not the only option when looking for greater network speed. A potentially faster and less expensive alternative to 10GbE is InfiniBand. This guide shows how to configure IPoIB on an Ubuntu 12.04.1 or similar host.

Overview

Ubuntu 12.04.1 LTS loads the ib_mthca and mlx4_core modules by default when a corresponding host channel adapter (HCA) is detected at boot. The ib_mthca module provides support for older Mellanox InfiniHost III HCAs and the mlx4_* set of modules provides support for ConnectX and newer Mellanox HCAs. When using ConnectX and newer HCAs, two modules are required, mlx4_core and mlx4_ib. With Ubuntu the latter modules is not loaded by default so it must be loaded manually. The last required module for an IPoIB node is the ib_ipoib module itself. Another useful module to load on every node is the user space management datagram module, ib_umad. This module lets you use scripts like ibstat and ibhosts from the infiniband-diags package. Last, a subnet manager is required for any Infiniband fabric. OpenSM is a cross platform open source subnet manager and under Linux requires the ib_umad module to function.

[toggle_box title=”Required Modules (click to expand)” width=”600″]ib_mthca – InfiniHost III and older HCAs

mlx4_core and mlx4_ib – ConnectX and newer HCA

ib_ipoib – the IPoIB module

ib_umad – required by OpenSM, useful on clients for diagnostics (ibstat, ibhosts)[/toggle_box]

Ephemeral Ubuntu Mellanox IPoIB Setup

1. Load the required modules using modprobe.

If using a ConnectX or newer HCA, load mlx4_ib.

[box_light]max@IBClient:~ $ sudo modprobe mlx4_ib[/box_light]

Load ib_ipoib and ib_umad.

[box_light]max@IBClient:~ $ sudo modprobe ib_umad

max@IBClient:~ $ sudo modprobe ib_ipoib[/box_light]

Confirm the modules have been loaded.

[box_light]max@IBClient:~ $ lsmod | egrep ‘mlx4|mthca|ib_ipoib|ib_umad'[/box_light]

1a. On one node, install OpenSM.

[box_light]max@IBClient:~ $ sudo apt-get install opensm[/box_light]

2. Once multiple nodes are up, check the link with ibstat. The example output below is from a dual port ConnectX HCA.

[box_light]max@IBClient:~ $ ibstat

CA ‘mlx4_0’

CA type: MT25418

Number of ports: 2

Firmware version: 2.9.1000

Hardware version: a0

Node GUID: 0x0008f104039a07fc

System image GUID: 0x0008f104039a07ff

Port 1:

State: Initializing

Physical state: LinkUp

Rate: 20

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x02510868

Port GUID: 0x0008f104039a07fd

Port 2:

State: Down

Physical state: Polling

Rate: 28

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x02510868

Port GUID: 0x0008f104039a07fe[/box_light]

3. Check for the IPoIB interface with ifconfig. The example output below is from dual port ConnectX HCA.

[box_light]max@IBClient:~ $ ifconfig -a

…

ib0           Link encap:UNSPEC HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00

BROADCAST MULTICAST MTU:2044 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:256

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

 

ib1           Link encap:UNSPEC HWaddr 80-00-00-49-FE-80-00-00-00-00-00-00-00-00-00-00

BROADCAST MULTICAST MTU:2044 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:256

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

…[/box_light]

4. Configure the IPoIP interface ib0 like an regular Ethernet interface.

[box_light]max@IBClient:~ $ sudo ifconfig ib0 10.10.10.1 netmask 255.255.255.0[/box_light]

For connected mode, also run the following two commands.

[box_light]max@IBClient:~ $ echo ‘connected’ | sudo tee /sys/class/net/ib0/mode > /dev/null[/box_light]

To change the MTU in connected mode run the following, where X is the desired MTU <= 65520.

[box_light]max@IBClient:~ $ sudo ifconfig ib0 mtu X[/box_light]

Repeat the procedure on other nodes using different IP addresses and without installing OpenSM.

Permanent Ubuntu Mellanox IPoIB Setup

0. For the Infiniband network to function, a subnet manager is needed on one of the nodes. When setting up a point to point network or using an Infiniband switch without a subnet manager, install OpenSM on one of the nodes.

[box_light]max@IBClient:~ $ sudo apt-get install opensm[/box_light]

1. Add required modules to /etc/modules. If using a ConnectX or newer HCA, also add mlx4_ib.

[box_light]max@IBClient:~ $ sudo -s

root@IBClient:~ # printf “mlx4_ib” >> /etc/modules

root@IBClient:~ # printf “ib_umad” >> /etc/modules

root@IBClient:~ # printf “ib_ipoib” >> /etc/modules[/box_light]

Here is an example /etc/modules for a machine with a ConnectX-2 HCA.

[box_light]root@IBClient:~ # cat /etc/modules

# /etc/modules: kernel modules to load at boot time.

#

# This file contains the names of kernel modules that should be loaded

# at boot time, one per line. Lines beginning with “#” are ignored.

 

loop

lp

rtc

mlx4_ib

ib_umad

ib_ipoib[/box_light]

2. Edit /etc/network/interfaces adding the IPoIB interface and the desired IP and subnetmask. It is always a good idea to make a backup, /etc/network/interfaces.bak, to revert to if anything goes wrong.

[box_light]root@IBClient:~ # cp /etc/network/interfaces{,.bak}[/box_light]

To make it simple, printf(1) is used to add the new lines to /etc/network/interfaces. Replace XXX.XXX.XXX.XXX with the IP and YYY.YYY.YYY.YYY with the subnetmask.

[box_light]root@IBClient:~ # printf “\n\nauto ib0” >> /etc/network/interfaces

root@IBClient:~ # printf “iface ib0 inet static” >> /etc/network/interfaces

root@IBClient:~ # printf “\taddress XXX.XXX.XXX.XXX” >> /etc/network/interfaces

root@IBClient:~ # printf “\tnetmask YYY.YYY.YYY.YYY” >> /etc/network/interfaces[/box_light]

For connected mode, also add the following two lines where Z is the desired MTU(<= 65520).

[box_light]root@IBClient:~ # printf “post-up echo connected > /sys/class/net/ib0/mode” >> /etc/network/interfaces

root@IBClient:~ # printf “post-up /sbin/ifconfig $IFACE mtu Z >> /etc/network/interfaces[/box_light]

3. Check your /etc/network/interfaces file with cat(1). Below is an example with ib0 configured in connected mode with max MTU.

[box_light]root@IBClient:~ # cat /etc/network/interaces

…

auto ib0

iface ib0 inet static

address 10.0.0.1

netmask 255.255.255.0

post-up echo connected > /sys/class/net/ib0/mode

post-up /sbin/ifconfig $IFACE mtu 65520

…

[/box_light]

Reboot for everything to take effect. Repeat steps 1 through 3 on the other node with different IP address.

Conclusion:

IPoIB configuration nicely mimics regular Ethernet except the extra required modules. The mainline kernel has superb support for Mellanox InfiniBand hardware and helps make set up uneventful with a vanilla Ubuntu Server install. The next installment will compare the performance of multiple HCA generations.