Can't start nextPYP: java.nio.file.FileSystemException: /mnt/data0: Function not implemented

[asarnow]

I'm trying to update nextPYP on my standalone workstation, but I get this error when I try to start up the service. I've also tried reinstalling from scratch. I'm not sure where it's reading the problematic path, it doesn't seem to matter if I change the binds or scratch directory. (I also don't know why it's problematic, /mnt/data0 is just a ext4 filesystem on a large SSD, and I haven't changed anything since nextPYP was working last).

Using configuration at /mnt/data0/applications/nextpyp/config.toml
Reading config.toml using CLI tool ...
WARNING: /etc/singularity/ exists, migration to apptainer by system administrator is not complete
Exception in thread "main" java.lang.ExceptionInInitializerError
        at edu.duke.bartesaghi.micromon.linux.CudaKt.countCudaGpus(cuda.kt:20)
        at edu.duke.bartesaghi.micromon.Config$Standalone.<clinit>(Config.kt:137)
        at edu.duke.bartesaghi.micromon.Config.<init>(Config.kt:225)
        at edu.duke.bartesaghi.micromon.Config$Companion.fromCanon(Config.kt:34)
        at edu.duke.bartesaghi.micromon.CliKt.main(cli.kt:12)
Caused by: java.nio.file.FileSystemException: /mnt/data0: Function not implemented
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
        at java.base/java.nio.file.Files.createDirectories(Files.java:793)
        at edu.duke.bartesaghi.micromon.IoKt.createDirsIfNeeded(io.kt:190)
        at edu.duke.bartesaghi.micromon.Config.initDirs(Config.kt:280)
        at edu.duke.bartesaghi.micromon.Backend.<clinit>(Backend.kt:34)
        ... 5 more

[cuchaz]

Guessing from the information in that error, that /mnt/data0 path is coming from either your web.localDir or web.sharedDir setting. For some reason, nextpyp isn't allowed to create a folder there. The underlying filesystem is returning a "Function not implemented" error. I don't know what could cause that particular filesystem error (maybe the mount is using a userspace filesystem driver, like something from FUSE land?), but maybe take a look at the folders in your web config and see if anything looks out of place there.

[cuchaz]

That fine, it will exist inside the container. You're missing basically all the binds though. That would explain the issues you're getting. Nothing is visible inside the container. Something has gone horribly wrong with the startup script, but I can't tell what.

[cuchaz]

Oh wait, you might be echoing $binds too early. Try echoing it at the bottom of the script, just before the singularity command.

[cuchaz]

wait a minute... you're getting java exceptions in the systemd log?? Not the micromon log? That's very weird. I missed that detail earlier. We almost never get java exceptions there, so I was just assuming the exception was happening during the JVM server process startup. But you're actually not getting to the bottom of the startup script, right? The exception is happening inside of the $($cli ...) subshell? I'll have to think for a minute about what went wrong since that's a very different scenario than was I was thinking about before.

[cuchaz]

Wow, ok. Found the cause of the issue. It's complicated.

Turns out a bug in my code was causing the server startup routines to be called during the CLI processing of the config file, which is waaay too early. That's why it looked like the server process was starting, because it was actually trying to do that, but that's not the right time to do it. I've already fixed the bug in the code, but you'll have to wait for the next release to get the fix.

In the meantime, you can work around the issue by putting the local and shared folders somewhere under your home directory. Through sheer dumb luck, that takes a code path that avoids the bug. If you need more space than your home folder has, you should be able to symlink your /mnt/data0 folder into your home folder and use that. eg. /home/me/symlink-to-mnt-0data0/local.

[asarnow]

Great, that was it! I'm using the symlink workaround for now. Thanks!

[asarnow]

I think this caused something bad to happen. There were some jobs running last night but this morning the home directory links are real, and the /mnt/data0/.../shared contains a new link called shared that points to itself (which ofc. is not allowed).

[asarnow]

Local too.

[cuchaz]

Weird. I have no idea what would cause what you're seeing. But now I can't get the workaround working again either (different errors though). Maybe it was a fluke it worked the first time at all.

Maybe it's best to either wait for the next version that has the fix for this issue already, or temporarily downgrade back to the old version.