-
Notifications
You must be signed in to change notification settings - Fork 102
InfiniBandSDPLinuxWindowsInteroperability
Stefan Eilemann edited this page Nov 15, 2010
·
13 revisions
NOTE: For other operating systems, e.g., Fedora, Ubuntu, use the OFED distributions that is available through www.openfabrics.org.
The ibstatus
tool is useful for debugging.
Note: Install the Mellanox drivers directly after the base installation, otherwise you will run into issues with the Mellanox installer script.
- download and install Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED), then:
$ mount -o -ro,loop MLNX_OFED_LINUX-1.5.1-rhel5.4.iso /mnt
$ cd /mnt
$ ./mlnxofedinstall
* configure IPoIB:
$ ifconfig ib0 10.0.0.1 netmask 255.255.255.0
$ ifconfig ib1 10.1.1.1 netmask 255.255.255.0
* test with netperf:
$
* Start subnet manager:
$ opensm start
* Create libsdp.conf:
$ cat > $HOME/libsdp.conf << EOF
> use sdp server * *:*
> use sdp client * *:*
> EOF
* Test netperf:
$ LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -s 10.0.0.1:SDP
$ LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -c 10.0.0.1:SDP
* Verify traffic connections: ```$ sdpnetstat -S```
## Installation on Windows XP64
* install MLNX_WinOF_MLNX_WinOF_2-0_wnet_x64_fre_2_0_7_4832. Later
drivers break the interoptability with Linux.
* select custom installation.
* activate SDP and SRP.
* at the end, activate all the services.
* set environnement variable : SdpApplications=netPerf.exe
* set IP directly on the two connections: IP 11.0.0.2/24 IP 10.1.1.2/24
## Test with netperf
The two computers are connected with only one cable IB hit "Opensm
start" on Windows and Linux
Test 1
- on linux
LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -S 11.4.3.175:2001:SDP
- on windows
netperf -C 11.4.3.175:2001:SDP
speed result 600MB/s
Test 2
- on windows
netperf -C 11.4.3.171:2001:SDP
- on linux
LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -C 11.4.3.171:2001:SDP
speed result 400MB/s
=======
-
configure IPoIB:
$ ifconfig ib0 10.0.0.1 netmask 255.255.255.0
-
Start subnet manager:
$ opensm start
-
Test netperf locally:
$ netperf -s 10.0.0.1:SDP
$ netperf -c 10.0.0.1:SDP
- Verify traffic on the connection:
$ sdpnetstat -S
- install WinOF 2.0. The last version compatible with XP64 is
MLNX_WinOF_MLNX_WinOF_2-0_wnet_x64_fre_2_0_7_4832.
- select custom installation.
- activate SDP and SRP.
- at the end, activate all the services.
- Note that the Linux OFED drivers need to be at least at version 1.5 (see above)
- set IP directly on the connection: IP 10.0.0.2/24
- Test netperf locally:
$ netperf -s 10.0.0.2:SDP
$ netperf -c 10.0.0.2:SDP
- on Linux:
netperf -s 10.0.0.1:2001:SDP
- on Windows:
netperf -c 10.0.0.1:2001:SDP
The speed should be above 500MB/s. And now the other way around:
- on Windows:
netperf -s 10.0.0.2:2001:SDP
- on Linux:
netperf -c 10.0.0.2:2001:SDP