(WIP) HELINL=L (L for linker) helas mode: pre-compile templates into separate .o object files (using RDC for CUDA; still missing HIP)

[valassi]

WIP on removing template/inline from helas (related to splitting kernels)

[valassi]

The functionality is in principle completed including the backport to CODEGEN. I will run some functionality and performance tests.

[valassi]

I add here now some comments that I had started last week. I have renamed this and put this in WIP. Many features are complete but I am passing to other things and I just want to document this so far before I move elsewhere.

(1) Description so far

Below is an update and a description before I move back to other things.

I added a new HELINL=L mode. This complements the default HELINL=0 mode and the experimental HELINL=1 mode.

HELINL=0 (default) aka "templates with moderate inlining".
This has templated helas functions FFV. The templates are in the memory access classes, i.e. essentially the template specialization depends on the AOSOA format used for momenta, wavefunctions and couplings. The sigmakin and calculate_wavefunction functions in CPPProcess.cc use these templated FFV functions, which are then implemented (and possibly inlined). The build times can be long, because the same templates are reevaluated all over the place, but the runtime speed is good.

HELINL=1 aka "templates with aggressive inlining".
This is the mode that I had introduced to mimic -flto i.e. link time optimizations. The FFV functions (and others) are inlined with always_inline. This significantly increases the build times because in practice it does the equiavelent of link time optimizations (while compiling CPPProcess.o). The runtime speed can get a signifcant boost for simple processes, where data access is important, but the speedups tend to decrease for complex processes, where arithmetic operations dominate. In a realistic madevent environment, this is probably not interesting: for simple processes, it can be ineresting, but the ME calculation is outnumbered by non-ME fortran parts and so it is not interesting to have faster MEs; in complex processes, the build times become just too large.

HELINL=L aka "linked objects".
This is the new mode I introduced here. The FFV functions are pre-compiled for the appropriate templates into .o object files. A technical detail: the HelAmps.cc file is common in Subprocess, but it must be compiled in each P* subdirectory, because the memory access classes may be different: for instance, a subprocess with 3 final state particles and one with 4 particles have different AOSOA, hence different memory access classes. My tests so far show that the build times can decrease/improve by a factor two, while the runtime can increase/degrade by around 10% for complex processes. (More detailed studies should show if it is the cuda or c++ build times that improve, or both). This is work that goes somewhat in the direction of splitting kernels and that I imagined in that context, but it is not exactly the same. It may become interesting for users especially for complex processes, and especially as long as the non-ME part is still important (eg DY+3j where cuda ME becomes 25% and sampling non-ME is over 50%, there having a ME that is 10% slower is acceptable).

(2) To do (non exhaustive list)

This is a non exhaustive list of things pending (unfortunately I was interrupted last week while writing this so I may be forgetting things)

• move the ixxx templated functions to linked mode too
• perform a more systematic study of build times, BACKEND one by one (now I only now the 'bldall' speedups)
• [edited] in particular, measure the build times of HelAmps.o and CppProcess.o separately
• test this mode on HIP (what is the rdc equivalent?)
• (consider some mixed HELINL Mode where for instance C++ uses the standard mode 0, but cuda uses mode L? not sure)
• and then the whole splitting kernel ideas, separate color from Feynman, separate FFVs individually etc

[valassi]

I updated this with the latest master as I am doing on all PRs

• test this mode on HIP (what is the rdc equivalent?

I had some LUMI shell running and I tried this (after also merging in #1007 with various AMD things)

There is a -fgpu-rdc which succeeds compilation but the issues come at link time.

• Using gfortran to link (as I do now due to Segfault in fgcheck.exe on LUMI (should we link hip, c++, fortran using hipcc or the fortran compiler?) #802) I am unable to link '__hip_fatbin'
• If I go back to hipcc for linking and add -fgpu-rdc --hip-link then it links, but it fails at runtime with Segfault in fgcheck.exe on LUMI (should we link hip, c++, fortran using hipcc or the fortran compiler?) #802
• I also tried to compile .f files with hipcc (I guess flang? Port to flang and F90 compliance of madgraph fortran codebase #804) and this succeeds, but then the 'main' cannot be found at link time, only the 'MAIN_'

Note that #802 is actually a 'shared object initialization failed' error

So the status is

• HELINL=L works ok for C++ and (with rdc) for CUDA
• HELINL=L does not work for HIP yet

[valassi]

Now including upstream/master with v1.00.00 and also the AMD and v1.00.01 patches #1014 and #1012