Installing pytorch with OpenMPI support

As per the README, install the dependencies, you'll need anaconda, CUDA, CuDNN as well.

export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" # [anaconda root directory]

# Install basic dependencies
conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing
conda install -c mingfeima mkldnn

# Add LAPACK support for the GPU
conda install -c pytorch magma-cuda92 # or [magma-cuda80 | magma-cuda91] depending on your cuda version

Install openmpi, which should link automatically by runnning

conda install -c conda-forge openmpi

Next, install pytorch by cloning the repo and running

python setup.py install

If you get a gcc version error, install gcc-7 (7.3 worked for me) and run

CC=gcc-7 CXX=g++-7 python setup.py install