Support for multi-GPU nodes (1: choose one of many available GPUs)

[valassi]

Just a reminder, we should do something about nodes with multiple GPUs.

This was for instance asked by Jin for CMS, as he found that one of our condor nodes has 4 GPUs (b9g57n8656.cern.ch).

Presently we simply do cudaSetDevice(0) and hipSetDevice(0) (actually both as gpuSetDevice(0). In any case for the moment the code is meant to only use one GPU at a time, so one option is to keep the code the same, but then have the CUDA/etc specific env variables to select one single GPU as the visible GPU (I believe this is CUDA_VISIBLE_DEVICES, while instead NVIDIA_VISIBLE_DEVICES is for visibility inside docker containers?)

Some interesting reads on multi-GPU

• https://events.prace-ri.eu/event/989/sessions/3096/
• https://developer.nvidia.com/blog/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/
• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

[valassi]

I opened two new issues

Note:

• part 1 in Support for multi-GPU nodes (1: choose one of many available GPUs) #836 is simply about choosing a single GPU (should be easy, and useful)
• part 2 in Support for multi-GPU nodes (2: send MEs to more than one GPU from the same madevent executable?) #989 is about sending MEs to more than one GPU from the same executable (complex, and not necessarily useful?)
• part 3 in Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs) #990 is about orchestrating from python/bash the different madevent executables to target different GPUs (not to ocomplex, and sounds very useful?)