GPU device file permission issue in rootless Docker containers

[hexclover]

Hello,

Our server 2 GPU devices located at /dev/nvidia{0,1}. We create a group nvidia0 and set the permission of the device file nvidia0 to root:nvidia0 and 660 to selectively allow some users to use this GPU.

The problem is, with rootless Docker, for some user foo who already belongs to the nvidia0 group, they cannot see both GPUs unless we apply two workarounds at the same time, namely file ACL and --privileged:

• With the file permissions intact, with docker run --runtime=nvidia --gpus=all ubuntu nvidia-smi they cannot see the nvidia0 GPU, only nvidia1. Note in this case only one device file is present in /dev of the container.
• With the file permissions intact, with docker run --runtime=nvidia --gpus=all --privileged ubuntu nvidia-smi (adding --privileged), they cannot see the GPU either.
• After adding u:foo:rw to the ACL of the device file, with the --privileged command, they are able to see both GPUs.

I wonder if there is a way to remove the need for --privileged. The ACL is also something I'd like to remove but if it's impossible we can live with it.

Thank you for your time.