setting up a TMPDIR in directory /run

[JuhaHuiskonen]

Hi,

On my HPC environment, local SSD disk on compute nodes is at $LOCAL_SCRATCH=/run/nvme/job_<jobid>/data. This directory is created on the compute node at run time.

I am trying to add /run/nvme to the binds in config.toml. When running nextpyp-start I get the following error

Reading config.toml using CLI tool ...
Exception in thread "main" java.lang.IllegalArgumentException: The following binds are not allowed:
	/run/nvme
	at edu.duke.bartesaghi.micromon.Config$Pyp.<init>(Config.kt:91)
	at edu.duke.bartesaghi.micromon.Config.<init>(Config.kt:187)
	at edu.duke.bartesaghi.micromon.Config$Companion.fromCanon(Config.kt:34)
	at edu.duke.bartesaghi.micromon.CliKt.main(cli.kt:12)

I guess there is /run inside the container. How can I bind my SSD disk?

[abartesaghi]

You don't need to manually bind the scratch folder, nextPYP will do that automatically.

Some paths are not allowed in the binds because they can interfere with important files in the container(s).

The workaround is to create a symlink in your network storage pointing to the location of your local scratch:

ln -s /run/nvme /nfs/somefolder

You should then configure the scratch directory in the configuration as follows:

scratch = '/nfs/somefolder/job_$SLURM_JOB_ID/data'

[JuhaHuiskonen]

Thanks for the suggestion! I have made the link as you suggested, modified the config file and restarted nextPYP. It seems this does not work because launch jobs do not have gres field, so the scratch disk cannot be requested for this step. And as the scratch disk does not exist at the launch step, the job will fail. I am getting the following error at when Running the job at launch step:

1 | mkdir: cannot create directory ‘/scratch/project_2009485/nextPYP/tmp’: File exists
-- | --
2 | WARNING: skipping mount of /scratch/project_2009485/nextPYP/tmp/job_20639341/data: stat /scratch/project_2009485/nextPYP/tmp/job_20639341/data: no such file or directory
3 | FATAL:   container creation failed: mount hook function failure: mount /scratch/project_2009485/nextPYP/tmp/job_20639341/data->/scratch/project_2009485/nextPYP/tmp/job_20639341/data error: while mounting /scratch/project_2009485/nextPYP/tmp/job_20639341/data: mount source /scratch/project_2009485/nextPYP/tmp/job_20639341/data doesn't exist
4

[abartesaghi]

OK, since this was a minor change and we weren't planning on making another release before the workshop, I just patched the current release to include this feature. If you re-install 0.6.3 it should solve this problem. Let us know if it doesn't.