InfiniBandSDPLinuxWindowsInteroperability

Interopability Linux - Windows with Infiniband SDP

Installation Guide

NOTE: For other operating systems, e.g., Fedora, Ubuntu, use the OFED distributions that is available through www.openfabrics.org.

Installation on CentOS 5.5

The ibstatus tool is useful for debugging.

Note: Install the Mellanox drivers directly after the base installation, otherwise you will run into issues with the Mellanox installer script.

• download and install Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED), then:

    $ mount -o -ro,loop MLNX_OFED_LINUX-1.5.1-rhel5.4.iso /mnt
    $ cd /mnt 
    $ ./mlnxofedinstall

* configure IPoIB:

    $ ifconfig ib0 10.0.0.1 netmask 255.255.255.0
    $ ifconfig ib1 10.1.1.1 netmask 255.255.255.0

* test with netperf:

    $ 

* Start subnet manager:

    $ opensm start

  * Create libsdp.conf:

    $ cat > $HOME/libsdp.conf << EOF
    > use sdp server * *:*
    > use sdp client * *:*
    > EOF

  * Test netperf:

    $ LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -s 10.0.0.1:SDP
    $ LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -c 10.0.0.1:SDP
           
  * Verify traffic connections: ```$ sdpnetstat -S```

## Installation on Windows XP64

* install MLNX_WinOF_MLNX_WinOF_2-0_wnet_x64_fre_2_0_7_4832. Later
  drivers break the interoptability with Linux. 
  * select custom installation.
  * activate SDP and SRP.
  * at the end, activate all the services.
* set environnement variable : SdpApplications=netPerf.exe
* set IP directly on the two connections: IP 11.0.0.2/24 IP 10.1.1.2/24   
      
## Test with netperf 

The two computers are connected with only one cable IB hit "Opensm
start" on Windows and Linux

   Test 1
   - on linux 
        LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -S 11.4.3.175:2001:SDP
   - on windows 
        netperf -C 11.4.3.175:2001:SDP
        
      speed result 600MB/s 
        
   Test 2   
   - on windows 
        netperf -C 11.4.3.171:2001:SDP
   - on linux 
        LD_PRELOAD=libsdp.so LIBSDP_CONFIG_FILE=$HOME/libsdp.conf netperf -C 11.4.3.171:2001:SDP

      speed result 400MB/s 
=======

• configure IPoIB: $ ifconfig ib0 10.0.0.1 netmask 255.255.255.0

• Start subnet manager: $ opensm start

• Test netperf locally:

    $ netperf -s 10.0.0.1:SDP
    $ netperf -c 10.0.0.1:SDP

• Verify traffic on the connection: $ sdpnetstat -S

Installation on Windows XP64

• install WinOF 2.0. The last version compatible with XP64 is MLNX_WinOF_MLNX_WinOF_2-0_wnet_x64_fre_2_0_7_4832.
  • select custom installation.
  • activate SDP and SRP.
  • at the end, activate all the services.
• Note that the Linux OFED drivers need to be at least at version 1.5 (see above)
• set IP directly on the connection: IP 10.0.0.2/24
• Test netperf locally:

    $ netperf -s 10.0.0.2:SDP
    $ netperf -c 10.0.0.2:SDP

Test with netperf

• on Linux: netperf -s 10.0.0.1:2001:SDP
• on Windows: netperf -c 10.0.0.1:2001:SDP

The speed should be above 500MB/s. And now the other way around:

• on Windows: netperf -s 10.0.0.2:2001:SDP
• on Linux: netperf -c 10.0.0.2:2001:SDP