NVIDIA to AMD conversion

[davidbenncsiro]

Part of the XBeach-GPU/CSIRO collaboration involved understanding how to convert the code so from CUDA to HIP so that it will run on AMD instead of NVIDIA GPUs. The following bash script carries out this conversion, then builds it.

#!/usr/bin/bash

# change these paths to suit your system
FFTW=/apps/fftw/3.3.10
NETCDF=/apps/netcdf/4.8.0
ROCM=/apps/rocm-sys/5.2.1

CC=${ROCM}/bin/hipcc
 
for CU in *.cu; do
    CPP=$(echo ${CU}.cpp | sed -e "s/\.cu/\.cpp/")
    if [ ! -f "$CPP" ]; then
        echo hipifying $CU...
        ${ROCM}/bin/hipify-perl $CU > $CPP
    fi
done
 
$CC   -Wno-unused-variable \
      -Wno-unused-value \
      -Wno-unused-but-set-variable \
      -Wno-c++11-narrowing \
      -Wno-sign-conversion \
      -Wno-sign-compare \
      -Wno-format \
      -I${FFTW}/include -I${NETCDF}/include \
      -L${FFTW}/lib64 -L${NETCDF}/lib64 \      
      -O3 \
      -g \
      -lnetcdf \
      -lfftw3 \
      -lm \
      -o XBGPU \
      Wave_gpu.cpp wave_CPU.cpp Flow_CPU.cpp Sediment_CPU.cpp \
      read_input.cpp tools.cpp writenetcdf.cpp makjonswap.cpp

[davidbenncsiro]

The following issues arose when creating/troubleshooting this script:

• The -WnoX options indicate errors/warnings to be addressed in the source code.
• Wave_gpu.cpp #includes all other CUDA/HIP code. Should instead be compiled separately with headers for function signatures.
• Wavestep.cpp
  • wavebnd() has no return statement but some code in Wave_gpu.cpp assigns to a Param variable
  • header file has different signature (void return type)
• min, max functions required std:: prefixes in a few C++ files
  • see also Adding optimisation at build time yields unresolved min/max errors  #52
• hipify-perl gets kernel invocations wrong if spaces anywhere in <<< or >>>
  • is hipify-clang any better?

[davidbenncsiro]

@CyprienBosserelle: what would you like to see done with the foregoing?

• One approach is to say: capturing the information in this issue is sufficient.
• Another is to create a GitHub Wiki page with the information.
• Another is to commit the bash script. We would probably also need to create a software collection for that (from a CSIRO perspective at least).

Also, I initially wondered about creating a Makefile for this. For build? Sure. For the conversion? Possibly.

Then there's the issue of the C++ code problems mentioned above. These could be part of the aforementioned software collection and would further justify it.