How-to for FUSE device mount inside container with Singularity 4 and OCI mode

[marcodelapierre]

Hi @dtrudg ,

we actually have a current topic where we might benefit from some advice from you folks: we need to be able to mount /dev/fuse inside the container.

Quoting my colleague @jordeu from an internal chat:

I understood that when you enable OCI mode on singularity 4.0 then they also use the standard CDI to mount devices.

This is something similar to our Kubernetes Device Plugin, but not exactly the same.

So, we need to figure out how to create/install this CDI driver and then it should be possible to run with something like:

singularity exec  --device nextflow.io/fuse=1 --oci [..] <img> <cmd>

Do you have any suggestions/guidelines/tips in this space?
What would be the smoothest way to bind mount a FUSE device in a OCI container with Singularity V4?

Thank you,

[marcodelapierre]

Actually, found this straight after writing this issue:

would the --fusemount option be the best fit ?

[dtrudg]

Hi @marcodelapierre - converted to a discussion as this is a question rather than an issue.

If you use Singularity in non-oci mode then the default is to bind all of /dev so you get /dev/fuse there...

singularity run ubuntu_latest.sif 
Singularity> ls /dev/fuse
/dev/fuse

However, you probably won't be able to do a lot with it, unless you also add --fakeroot to have the container running in a user namespace with a root-user mapping and capabilities granted inside the user namespace.

In OCI-mode then you could use CDI, and you could also do a bind...

 $ singularity run -B /dev/fuse --oci docker://ubuntu 
INFO:    Using cached OCI-SIF image
dtrudg-sylabs@mini:~$ ls -lah /dev/fuse
crw-rw-rw-. 1 nobody nogroup 10, 229 Nov  6 11:14 /dev/fuse

The --fusemount flag is a something a bit different. It is for the native mode (non-OCI-mode). It creates a specific FUSE mount associated with a file descriptor at the source end... allowing the FUSE binary providing the fs to then be run on the host or from the container.

What's the overall goal here?

[marcodelapierre]

Thanks for the quick reply Dave.

@jordeu can you have a look here, maybe we can identify a good solution for Fusion vs Singularity OCI in this context?

[jordeu]

Hi @dtrudg, we are just trying to mount a FUSE tool inside the container as we were doing before but now with OCI mode enabled.

Here a reproducible example of what we do without OCI:

1. Download gocryptfs on the host

$ wget https://github.com/rfjakob/gocryptfs/releases/download/v2.4.0/gocryptfs_v2.4.0_linux-static_amd64.tar.gz
$ tar -xvzf gocryptfs_v2.4.0_linux-static_amd64.tar.gz

2. Run singularity with fake root but without OCI, and mount a gocryptfs volume.

$ singularity run --fakeroot docker://ubuntu

Singularity> mkdir cipher_nonoci plain_nonoci
Singularity> ./gocryptfs -init cipher_nonoci
Singularity> ./gocryptfs cipher_nonoci plain_nonoci
Singularity> mount | grep cipher
/home/jordi/fusetest/cipher_nonoci on /home/jordi/fusetest/plain_nonoci type fuse.gocryptfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,max_read=1048576)

3. Now try the same but with OCI enabled

$ singularity run --fakeroot -B $(pwd):/root -B /dev/fuse --oci docker://ubuntu

Singularity> mkdir cipher_oci plain_oci
Singularity> ./gocryptfs -init cipher_oci
Singularity> ./gocryptfs cipher_oci plain_oci

fs.Mount failed: exec: "/bin/fusermount": stat /bin/fusermount: no such file or directory

It is always failing to mount. Is there any seccomp profile that blocks mounting enabled when running on OCI mode?

[jordeu]

@dtrudg I've tested current main branch, and now it works as expected. Thank you!

[dtrudg]

Great, thanks. 4.0.2 release will happen next week (hopefully) or the week after.

[marcodelapierre]

Fantastic Dave, thank you!

[dtrudg]

This is now fixed in the CE 4.0.2 release that went out yesterday.

[marcodelapierre]

Tested and merged by the team, works successfully with Nextflow+Fusion - thank you!