Configuring UCX with Omnipath

[mahendrapaipuri]

Hello,

I am trying to build OpenMPI 4.1.1 with UCX support using Spack. I have access to a cluster that is connected with Intel Omnipath 100Gb interconnect. On the Spack default spec for building UCX has all transports turned off. So, I tried to build UCX with default Spack spec, I could not find port hfi1_0 in the ucx_info -d. I tried to turn on the transports in the configuration and I got errors as some of them depends on Infiniband drivers. I could build UCX and OpenMPI on clusters with Infiniband support without any issues. I am new to Omnipath and do not know where to go. I tried to find information in the issues here and elsewhere, returned empty handed. Any help would be appreciated.

Thanks!

[jtfrey]

Open MPI defaults its pml_ucx_devices parameter to the pattern mlx*, thus it will never match the Omni-path hfi* device. If you enable extensive Open MPI and UCX verbosity

UCX_LOG_LEVEL=data mpirun … --mca base_verbose 100 --mca pml_base_verbose 100 --mca pml_ucx_verbose 100 …

you can observe UCX's finding the device and disqualifying it because a valid transport is present in the TLS list but it's not in the device list (doesn't match mlx*). Set the pml_ucx_devices to * or more specifically hfi* and UCX will use the OPA HFI.

That's not to say it will work. Right now I'm trying to debug a build of Open MPI 4.1.4 against UCX 1.14.0: when Open MPI is compiled with various versions of GCC and the traditional Intel compiler suite it works fine, but the library produced with Intel oneAPI 2023 is delivering the lower 32-bits of 64-bit integers. That's only when using the UCX PML — any other PML works fine.