warpX on polaris

[ppiot]

All: Could somebody confirm whether the script in the /Tools/machines/polaris-alcf are up to date? They somehow did not work out of the box for me (just following instruction in the warpX https://warpx.readthedocs.io/en/latest/install/hpc/polaris.html . Below are the error messages. Before proceedings in fixing them I want to check whether these scripts were used by anybody else recently. Thank you! -- Philippe.

-- the first couple of errors I have are:
source polaris_gpu_warpx.profile.example 
Lmod has detected the following error: Swap failed: "PrgEnv-nvhpc" is not loaded.

then when running the install script the default compiler goes to gcc 7.5 and the following error appears:
The C compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc - broken
CMake Error at /soft/spack/base/0.7.1/install/linux-sles15-x86_64/gcc-12.3.0/cmake-3.27.7-a435jtzvweeos2es6enirbxdjdqhqgdp/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/usr/bin/gcc"

  is not able to compile a simple test program.

[ax3l]

Hi @ppiot,

Thanks for the report!
We think the modules on Polaris probably were updated, let's overhaul the documentation together. I do not personally have access to Polaris right now but @WeiqunZhang has.

We document the last modules we tried here:
https://github.com/ECP-WarpX/WarpX/blob/development/Tools/machines/polaris-alcf/polaris_gpu_warpx.profile.example
which is what the documentation page includes.

@WeiqunZhang will take a look if there are newer modules that we can use.
From the warning I see above, probably these modules just need to use newer version and/or another CUDA module:

# ...
module load gcc-native/12.3
module load nvhpc-mixed/23.9
# ...
module load cmake/3.27.7
# ...
module load cray-hdf5-parallel/1.12.2.9

In parallle, @ppiot if you see matching modules in module list/module spider and the official ALCF Polaris docs, let's try those.

[ppiot]

@ax3l and @WeiqunZhang I think the module did not change much actually, the list of command below was sufficient to compile the "bare-bone" warpX:

module purge
module load gcc-native/12.3 nvhpc-mixed/23.9
module load craype-x86-milan 
module use /soft/modulefiles
module load spack-pe-gnu
module load cmake/3.27.7
module load boost
module load cray-hdf5-parallel
export CRAY_ACCEL_TARGET=nvidia80
export AMREX_CUDA_ARCH=8.0
export CXXFLAGS="-march=znver3"
export CFLAGS="-march=znver3"
export CC=$(which gcc-12)
export CXX=$(which g++-12)
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=${CXX}

cmake -S . -B build -DWarpX_DIMS="3" -DWarpX_COMPUTE=CUDA 
cmake --build build -j 4

note the strange way I had to set the CC and CXX flags somehow it would otherwise always default to gcc7.5.
there is a couple of issues I am not able to fix yet (hdf5 is not "seen" by cmake though I loaded the module with a sufficiently high version number) and part of the install script is still not working (but probably can be simplified as some of the things being built, e.g. adios2, are now available as module... anyway for now I am set to do a few tests...

[ppiot]

ahhrr I spoke too fast. Also it compiles it throws a

terminate called after throwing an instance of 'openPMD::error::WrongAPIUsage'
  what():  Wrong API usage: JSON backend not available in parallel openPMD.
SIGABRT

error. Although I am running an example file

/src/warpx/Examples/Physics_applications/laser_acceleration/inputs_base_3d

[ax3l]

Ah, that means neither parallel HDF5 or parallel ADIOS2 were found. Taking a look with @WeiqunZhang right now.

[ax3l]

The Cray HDF5 module did patch-break the installer logic for CMake. I had this before, reported it to Cray, the fixed it, seems they have the issue again on Polaris. This should be an issue with ALCF, reporting that CMake find_package does not work for their cray-hdf5-parallel module.

For now, we can just use the hdf5/1.14.3 module instead of cray-hdf5-parallel.

[ppiot]

Just FYI, hdf5/1.14.3 does not work, warpX compiles fine but does not run as the hdf5 module is not available on the running nodes... Anyway I reported the issue with cray-hdf5-parallel and the one with hdf5/1.14.3

[WeiqunZhang]

@ppiot Could give #5358 a try?

Preview of new docs until this PR is merged: https://warpx--5358.org.readthedocs.build/en/5358/install/hpc/polaris.html
After merge this updates: https://warpx.readthedocs.io/en/latest/install/hpc/polaris.html

[ppiot]

@WeiqunZhang I could run one of the demo examples fine and the simulation I am trying to run without hdf5 back end. However toggling hdf5 back end by adding the line

slabmovingwarpx.openpmd_backend = h5

give the error below:

TEP 1 ends. TIME = 1.444749487e-13 DT = 1.444749487e-13
Evolve time = 0.653470138 s; This step = 0.653470138 s; Avg. per step = 0.653470138 s

STEP 2 starts ...
amrex::Abort::61::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::51::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::63::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::59::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::55::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::49::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::53::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::57::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::48::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::58::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::54::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::56::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::60::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::62::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::52::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT
amrex::Abort::50::CUDA error 700 in file /home/ppiot/software/WarpX/build_pm_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 652: an illegal memory access was encountered !!!
SIGABRT

[ax3l]

In doubt, we could compile our own HDF5 for Polaris. The Cray Module is broken and this one seems to trigger this.

Here is an example to compile and integrate our own parallel HDF5 on a Cray:

• WarpX/Tools/machines/lumi-csc/install_dependencies.sh

  Lines 103 to 120 in 27181aa

  	# HDF5 (for openPMD)
  	if [ -d ${SRC_DIR}/hdf5 ]
  	then
  	cd ${SRC_DIR}/hdf5
  	git fetch --prune
  	git checkout hdf5-1_14_1-2
  	cd -
  	else
  	git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git ${SRC_DIR}/hdf5
  	fi
  	rm -rf ${build_dir}/hdf5-lu-build
  	cmake -S ${SRC_DIR}/hdf5 \
  	-B ${build_dir}/hdf5-lu-build \
  	-DBUILD_TESTING=OFF \
  	-DHDF5_ENABLE_PARALLEL=ON \
  	-DCMAKE_INSTALL_PREFIX=${SW_DIR}/hdf5-1.14.1.2
  	cmake --build ${build_dir}/hdf5-lu-build --target install --parallel 10
  	rm -rf ${build_dir}/hdf5-lu-build

• WarpX/Tools/machines/lumi-csc/lumi_warpx.profile.example

  Lines 24 to 26 in 27181aa

  	export CMAKE_PREFIX_PATH=${SW_DIR}/hdf5-1.14.1.2:$CMAKE_PREFIX_PATH
  	export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH
  	export PATH=${SW_DIR}/hdf5-1.14.1.2/bin:${PATH}

@WeiqunZhang @ppiot I pushed a PR for you to test:
#5379