Sync with PSM2 master up to correction of URL for GitHub PSM2 repo

This code surface corresponds to version 10.3.58 of PSM2. In addition
to GitHub URL fixes, the change also includes-
  - PSM2 GDR Copy feature and tuning of thresholds to fully utilize the feature
  - Fix uniformity of process mapping to HFIs
  - PSM2 specific perf benchmarks
  - Fix for issue 27: Null termination of string
  - Update README

Signed-off-by: Aravind Gopalakrishnan <[email protected]>

COMMIT

@@ -1 +1 @@
-8b6ba42b45df1815a1da540ebb088b10cc8d88ea
+8a6c3e5b8d873b8ff4375a4967610d5931691ec2

Makefile

@@ -489,7 +489,8 @@ ${TARGLIB}-objs := ptl_am/am_reqrep_shmem.o	\
 		   ptl_self/ptl.o		\
 		   opa/*.o			\
 		   psm_diags.o 			\
-		   psmi_wrappers.o
+		   psmi_wrappers.o              \
+		   psm_gdrcpy.o
 
 ${TARGLIB}-objs := $(patsubst %.o, ${OUTDIR}/%.o, ${${TARGLIB}-objs})
 

README

@@ -213,34 +213,31 @@ RELATED SOFTWARE TO PSM2
 
 MPI Libraries supported
 -----------------------
-A large number of open source (OpenMPI, MVAPICH2) and Vendor MPI
+A large number of open source (Open MPI, MVAPICH2) and Vendor MPI
 implementations support PSM2 for optimized communication on HCAs. Vendor MPI
 implementations (HP-MPI, Intel MPI 4.0 with PMI, Platform/Scali MPI)
 require that the PSM2 runtime libraries be installed and available on
 each node. Usually a configuration file or a command line switch to mpirun
 needs to be specified to utilize the PSM2 transport.
 
-OpenMPI support
+Open MPI support
 ---------------
-It is recommended to use the v1.10.4 or newer version of OpenMPI.
-Prior versions of OpenMPI have an issue with support PSM2 network transports
-mixed with standard Verbs transport (BTL openib). This prevents an OpenMPI
-installation with network modules available for PSM2 and Verbs to work
-correctly on nodes with no HFI hardware. This has been fixed in the
-latest development branch allowing a single OpenMPI installation to target
-HFI hardware via PSM2 or Verbs as well as alternate transports seamlessly.
-
-If NVIDIA CUDA support is desired, you can use the OpenMPI build
-(v1.10.4-cuda-hfi) provided by Intel in the IFS installer v10.4.X or newer.
-The changes have also been accepted into v3.0.x branch of upstream OpenMPI
-repository. Therefore subsequent v3.0.x versions of OpenMPI should carry the
-required OpenMPI support for PSM2 GPUDirect feature.
-
-PSM2 header and runtime files need to be installed on a node where the OpenMPI
+If using a version of Open MPI that is not packaged within IFS release, it
+is required to use at least v1.10.4. Older versions are not supported. Since
+v1.10.4 is not in active development, it is further recommended to use upstream
+versions v2.1.2 or newer.
+
+If NVIDIA* CUDA* support is desired, you can use Open MPI built with CUDA*
+support provided by Intel in the IFS installer 10.4 or newer. This Open MPI
+build is identified with the "-cuda-hfi" tag to the Open MPI base version
+name.  The NVIDIA* CUDA* support changes have also been accepted into v2.1.3,
+v3.0.1 and v3.1.0 branches of upstream Open MPI repository.
+
+PSM2 header and runtime files need to be installed on a node where the Open MPI
 build is performed. All compute nodes additionally should have the PSM2 runtime
-libraries available on them. OpenMPI provides a standard configure, make and
+libraries available on them. Open MPI provides a standard configure, make and
 make install mechanism which will detect and build the relevant PSM2 network
-modules for OpenMPI once the header and runtime files are detected.
+modules for Open MPI once the header and runtime files are detected.
 
 MVAPICH2 support
 ----------------
@@ -251,30 +248,25 @@ compute nodes should also have the PSM2 runtime libraries available on them.
 
 For building and installing MVAPICH2 with OPA support, refer to MVAPICH2
 user guides here:
-http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.2rc1-userguide.html
+http://mvapich.cse.ohio-state.edu/userguide/
 
-(Note: Support for PSM2 is currently on v2.2rc1 of OSU MVAPICH2 code base.
-The above link might change when a stable v2.2 is released.)
+(Note: Support for PSM2 is included in v2.2 and newer)
 
 OFED Support
 ------------
 Intel OPA is not yet included within OFED. But the hfi1 driver is available
-publicly at kernel.org. Please do pull the driver from either kernel.org or
-the github page for opa-hfi1 driver (https://github.com/01org/opa-hfi1)
+publicly at kernel.org.
 
 SUPPORTING DOCUMENTATION
 ------------------------
 PSM2 Programmer's Guide is published along with documentation for "Intel® Omni-Path
 Host Fabric Interface PCIe Adapter 100 Series"
-(http://www.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/000016242.html)
+(https://www.intel.com/content/www/us/en/support/articles/000016242/network-and-i-o/fabric-products.html)
 
 Refer to this document for description on APIs and environment variables that
 are available for use. For sample code on writing applications leveraging the
 PSM2 APIs, refer to Section 5.
 
-Link to latest (as of Sep 2017) PSM2 Programmer's Guide:
-https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_PSM2_PG_H76473_v7_0.pdf
-
 PSM Compatibility Support
 ------------
 

include/opa_service.h

@@ -75,6 +75,11 @@
 
 /* base name of path (without unit #) for qib driver */
 #define HFI_DEVICE_PATH "/dev/hfi1"
+
+#ifdef PSM_CUDA
+#define GDR_DEVICE_PATH "/dev/hfi1_gdr"
+#endif
+
 #define HFI_CLASS_PATH "/sys/class/infiniband/hfi1"
 
 /* Commands used to communicate with driver. */

libpsm2.spec.in

@@ -53,10 +53,10 @@ Name: @RPM_NAME@
 Version: @VERSION@
 Release: 1@SPEC_FILE_RELEASE_DIST@
 License: BSD or GPLv2
-URL: https://github.com/01org/opa-psm2/
+URL: https://github.com/intel/opa-psm2/
 
 # The tarball can be created by:
-# git clone https://github.com/01org/opa-psm2
+# git clone https://github.com/intel/opa-psm2
 # cd opa-psm2
 # git checkout @DIST_SHA@
 # make dist

makesrpm.sh

@@ -79,12 +79,16 @@ function usage()
     echo "           Sets PSM_CUDA=1, creating -cuda based spec and rpms"
     echo "     -d <path>, -dir <path>"
     echo "           Optionally sets output folder for rpmbuild to use"
+    echo "     -h <hal_gen>, -hal_gen <hal_gen>"
+    echo "           Optional, default is includes all HAL generations"
+    echo "           Sets hal generations for rpmbuild to use"
     echo "     Examples:"
     echo "           $0 b"
     echo "           $0 s -cuda"
     echo "           $0 -cuda"
     echo "           $0 -d ./temp"
     echo "           $0 b -cuda -dir output"
+    echo "           $0 -h gen1"
     exit 1
 }
 
@@ -98,6 +102,8 @@ OUTDIR=build_release
 # It can be set the same as OUTDIR, and work just fine if desired.
 TEMPDIR=temp.$$
 
+HAL_GENS=""
+
 while [ "$1" != "" ]; do
     case $1 in
         -d | -dir)      shift
@@ -116,6 +122,9 @@ while [ "$1" != "" ]; do
                         RPM_NAME_BASEEXT="$1"
                         export RPM_NAME_BASEEXT="$1"
                         ;;
+        -h | -halgen)   shift
+                        HAL_GENS="$1 $HAL_GENS"
+	                ;;
         -r | -rpmname)  shift
                         if [ -z "$1" ]; then
                             usage
@@ -132,11 +141,15 @@ while [ "$1" != "" ]; do
     shift
 done
 
+if [ "$HAL_GENS" = "" ]; then
+    HAL_GENS="*"
+fi
+
 # Generic cleanup, build, and tmp folder creation
 make distclean OUTDIR=$OUTDIR
-make RPM_NAME=$RPM_NAME RPM_NAME_BASEEXT=$RPM_NAME_BASEEXT dist OUTDIR=$OUTDIR
+make RPM_NAME=$RPM_NAME RPM_NAME_BASEEXT=$RPM_NAME_BASEEXT "PSM_HAL_ENABLE=$HAL_GENS" dist OUTDIR=$OUTDIR
 mkdir -p ./$TEMPDIR/{BUILD,RPMS,SOURCES,SPECS,SRPMS,BUILDROOT}
-# Differnet paths based on RPM_EXT
+# Different paths based on RPM_EXT
 cp ${OUTDIR}/$RPM_NAME-*.tar.gz $TEMPDIR/SOURCES
 make RPM_NAME=$RPM_NAME RPM_NAME_BASEEXT=$RPM_NAME_BASEEXT specfile OUTDIR=$OUTDIR
 cp ${OUTDIR}/$RPM_NAME.spec $TEMPDIR/SPECS

opa/opa_sysfs.c

@@ -374,6 +374,10 @@ static int read_page(int fd, char **datap)
 	if (ret == -1) {
 		free(data);
 	} else {
+		if (ret < sysfs_page_size)
+			data[ret] = 0;
+		else
+			data[sysfs_page_size-1] = 0;
 		*datap = data;
 	}
 

perf_test/Makefile

@@ -0,0 +1,72 @@
+#
+#  This file is provided under a dual BSD/GPLv2 license.  When using or
+#  redistributing this file, you may do so under either license.
+#
+#  GPL LICENSE SUMMARY
+#
+#  Copyright(c) 2018 Intel Corporation.
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of version 2 of the GNU General Public License as
+#  published by the Free Software Foundation.
+#
+#  This program is distributed in the hope that it will be useful, but
+#  WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#  General Public License for more details.
+#
+#  Contact Information:
+#  Intel Corporation, www.intel.com
+#
+#  BSD LICENSE
+#
+#  Copyright(c) 2018 Intel Corporation.
+#
+#  Redistribution and use in source and binary forms, with or without
+#  modification, are permitted provided that the following conditions
+#  are met:
+#
+#    * Redistributions of source code must retain the above copyright
+#      notice, this list of conditions and the following disclaimer.
+#    * Redistributions in binary form must reproduce the above copyright
+#      notice, this list of conditions and the following disclaimer in
+#      the documentation and/or other materials provided with the
+#      distribution.
+#    * Neither the name of Intel Corporation nor the names of its
+#      contributors may be used to endorse or promote products derived
+#      from this software without specific prior written permission.
+#
+#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+
+CFLAGS+=-Wall -g -I.
+LIBS=-lpsm2
+
+SRC=$(wildcard *.c)
+OBJS=$(patsubst %.c, %.o, $(SRC))
+MAIN_SRC=latency.o bi-bw-mrate.o bw-mrate.o
+BIN_OBJS=$(filter-out $(MAIN_SRC), $(OBJS))
+
+all: latency bw-mrate bi-bw-mrate
+latency: $(BIN_OBJS) latency.o
+	$(CC) $(CFLAGS) $(LIBS) -o latency latency.c $(BIN_OBJS)
+bw-mrate: $(BIN_OBJS) bw-mrate.o
+	$(CC) $(CFLAGS) $(LIBS) -o bw-mrate bw-mrate.c $(BIN_OBJS)
+bi-bw-mrate: $(BIN_OBJS) bi-bw-mrate.o
+	$(CC) $(CFLAGS) $(LIBS) -o bi-bw-mrate bi-bw-mrate.c $(BIN_OBJS)
+
+$(OBJS): %.o : %.c
+	$(CC) $(CFLAGS) -c $< $(LIBS)
+
+clean:
+	rm -f *.o bi-bw-mrate bw-mrate latency