Equalizer logo
Collage logo
GPU-SD logo

InfiniBand

Author: [email protected]
State: Implemented in 0.3

Overview

InfiniBand (IB) is supported using the SDP layer. This layer is transparent to socket-based applications, such as the Equalizer network layer. This document describes the setup to use IB hardware with Equalizer applications.

Windows XP

Overview

The usage of InfiniBand requires a Mellanox card, since Mellanox is the only vendor providing a SDP implementation. Often cards from other vendors are Mellanox cards with a different firmware and can be 'converted' by flashing a Mellanox firmware. The Mellanox Windows SDP implementation is described in great detail in the Architecture and Implementation of Sockets Direct Protocol in Windows paper.

By default, the Mellanox Win32 implementation does not allow using SDP and TCP/IP sockets concurrently. That means that all machines have to be connected using InfiniBand, and only SDP IP addresses are to be used for all connections. See the next section for SDP/TCP Coexistence.

Installation and Configuration

The following steps are necessary to use SDP on Windows:

SDP and TCP Coexistence

Starting with WinIB 1.4, it is possible to use both SDP and TCP sockets in one application process. The applications have to explicitly specify the socket type, which allows mixing of TCP and SDP sockets. Please refer to the Mellanox documentation on how to activate the 'mixed SDP applications' mode. Revisions 1001 to 1010 implement support for multiple node connections, which means that if a node (or the server) has multiple connections specified in the configuration file, they now listen to all of them, allowing to use mixed connection types. When a node connects to another node, the connections are tried in the order they are specified. Therefore connections should be configured in the fastest-to-slowest order.

Transparent SDP support, as described above, still works the same way. Explicit SDP support is used by not setting the variable SdpApplications, and explicitly configure SDP socket connections:

Implementation

The current (1.3.0) Mellanox Windows SDP implementation only supports a subset of the Windows socket interface. Most notably, the eq::net socket connection was rewritten to use overlapped (asynchronous) IO with svn revision 951. TCP/IP connections also exhibit a performance increase due to the use of overlapped IO.

The --eq-listen parsing was extended to support a simplified connection description in the form hostname(:port)(:type). Port is an unsigned short value, type is either SDP or TCPIP.

The interpretation of the node configuration has changed. When multiple connections are described, they are all set up as a listening connection during Node::initLocal(). Therefore, nodes can now listen on multiple sockets for incoming connections.

The connection setup code between two nodes has been improved.

The config file now accepts connection descriptions for the server section.

Related SVN changes: 951 952 953 1001 1002 1006 1007 1009 1010

Known Bugs

When using SDP, the client process hangs upon exit. We are investigating this issue. Defining EQ_WIN32_SDP_JOIN_WAR in definesWin32.h enables a workaround.

Linux

The Linux usage of SDP is very similar to the Windows usage described above. For the implicit, transparent mode just LD_PRELOAD the SDP library (typically libsdp.so) instead of setting SdpApplications. For explicit support, follow the same configuration as for Windows.