How to set SLURM Account variable?

[JuhaHuiskonen]

My HPC cluster allows submitting jobs to SLURM using sbatch only when SBATCH --account=XXXXXX has been set in the submission script or alternatively when the account is given as a parameter sbatch -A XXXXXX. How can I set this variable when running pyp from command line?

[abartesaghi]

We don't currently support specifying arbitrary sbatch options, unfortunately. However, if you are using the CLI, there is a way to workaround this limitation by bundling the -A option together with another option, for example: -slurm_queue="quick -A XXXXXX". Going forward, we will consider implementing a proper mechanism to support the specification of arbitrary slurm options. Thanks for the feedback.

[JuhaHuiskonen]

The solution worked for CLI. However when I use the web interface, the job dies immediately with a warning internal server error.

Examining the log files, I found out that local/logs/micromon shows the following error:

sbatch: error: invalid partition specified: small -A project_2007321
sbatch: error: Batch job submission failed: Invalid partition name specified

[abartesaghi]

Yes, like I said on my response above, the workaround only works if you are on the CLI.

[JuhaHuiskonen]

I have an idea for a temporary solution for web interface. Create a script called sbatch

#!/bin/bash
# Wrapper script for sbatch to include default account
/usr/bin/sbatch -A project_2007321 "$@"

Then edit config.toml and set the path to the corresponding directory. I also made links to other SLURM programs, such as scancel etc to be safe.

[abartesaghi]

The new version 0.6.1 allows you to specify a slurm account. This has to be set for each job in the Resources tab. You will need to use Slurm account to project_2007321 in your case.

[JuhaHuiskonen]

I am trying to run tomography tutorial and the first job (B1). For some reason the job does not start now at all. Message Internal server error is displayed in the Web interface. I found the created batch file and if I run it from the command line, the micrograph thumbnail appears. I assume there is a problem with SLURM configuration. But as the job does not get submitted, I am not sure how to debug this.

[abartesaghi]

You can look at the server logs as described here to see if there are any errors.

[JuhaHuiskonen]

This may have been due to an incompatible SLURM parameters (number of threads and queue) in my system. After more testing, it seems to work now.

[JuhaHuiskonen]

The SLURM account parameter works in Projects. Should it already be functional in Sessions? I am asking because I am trying to start a session with the same parameters that work in a project but to job fails with the error

[2024-02-01 19:48:23,352]-[eventLoopGroupProxy-4-17] ERROR Service - Service error
edu.duke.bartesaghi.micromon.cluster.ClusterJob$LaunchFailedException: unable to read job id from sbatch output:
sbatch: error: AssocMaxSubmitJobLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

This could mean that the account parameter is not read correctly from the Session setup. Any suggestions how to debug this, thanks!

[abartesaghi]

Indeed, this is a bug, it will be fixed in the next patch release.

[abartesaghi]

These issues are now addressed in version 0.6.2.