sbatch script for running inference

[sarazimmerman0]

Hi,

I'm using SLURM for the first time and trying to run inference on a few videos using the SDSC cluster, but keep receiving an error that the process is out of memory. Any recommendations for adjusting the settings of my sbatch parameters? I am able to run inference locally on a Mac (Apple M3 Max chip and 64GB of memory) and allocating much more memory when submitting the job. The error log and sbatch commands are both attached.

I noticed the error discussed in #1918 in the log as well, but not sure if this is related.

Thanks for the help!
Sara

edited_log_file (1).txt

[github-actions[bot]]

Thank you for starting a new discussion!

We appreciate your input and will review it soon.

Warning

A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards,
The Team

[gitttt-1234]

Hi @sarazimmerman0,

I can see that there are 2 errors in your logs. Yes, you are right, the first error Multiple nodes named "name" is addressed in #1918. Could you please check if the solutions provided here work for you?

Regarding the OOM error, I'm still looking into it, it would be great if you could share your training_config.json here. Meanwhile, this is a useful resource on OOM error.

Thanks,

Divya

[talmo]

Just copying my response from email thread:

A few suggestions/observations:

1. SLEAP currently does not support multi-GPU acceleration, so requesting 4 GPUs will confer no advantage over just 1.

2. These OOM errors are really weird:

2024-09-04 23:04:43.777087: W [tensorflow/core/common_runtime/bfc_allocator.cc:462](http://tensorflow/core/common_runtime/bfc_allocator.cc:462)] Allocator (gpu_host_bfc) ran out of memory trying to allocate 8.58MiB (rounded to 9000192)requested by op 
2024-09-04 23:04:43.779106: I [tensorflow/core/common_runtime/bfc_allocator.cc:1010](http://tensorflow/core/common_runtime/bfc_allocator.cc:1010)] BFCAllocator dump for gpu_host_bfc
2024-09-04 23:04:43.779128: I [tensorflow/core/common_runtime/bfc_allocator.cc:1017](http://tensorflow/core/common_runtime/bfc_allocator.cc:1017)] Bin (256): 	Total Chunks: 30584, Chunks in use: 30576. 7.47MiB allocated for chunks. 7.46MiB in use in bin. 834.8KiB client-requested in use in bin.
You have pretty large memory GPUs, so it's strange that it's running out of memory right away. I'm guessing that this has something to do with the shared computing environment that you're on and that the GPUs are perhaps being shared across jobs? Either way, it'll be easier to troubleshoot once you crank down the number of GPUs requested to just 1 instead of 4.

3. I see you're getting this ValueError: Found multiple nodes named (name). error. This is something that came up when a SLEAP dependency got updated and broke how we serialize skeletons (described in this issue). The short term solution is to update to SLEAP v1.3.4 which has a hotfix for this bug.

4. In your run script, I see that you're limiting the number of tracks to 1 using these flags: --tracking.tracker simplemaxtracks --tracking.max_tracking 1 --tracking.max_tracks 1. You can actually just replace all of those with just --max_instances 1 which will limit the maximum number of predicted instances to just 1 without actually applying the tracker (which is much slower).