Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exonerate() raised: [Errno 24] Too many open files #110

Open
Alicia-Talavera opened this issue Feb 7, 2023 · 11 comments
Open

exonerate() raised: [Errno 24] Too many open files #110

Alicia-Talavera opened this issue Feb 7, 2023 · 11 comments

Comments

@Alicia-Talavera
Copy link

Hello,

I ran "hybpiper assemble" (HybPiper v. 2.0.1) command with my last batch of samples and I get the following error for several genes:

For gene LOC100267692 exonerate() raised: [Errno 24] Too many open files:
T399/LOC100267692/LOC100267692_2023-02-07-01_08_33.log'
error.traceback is: Traceback (most recent call last):
File "/bioinformatics/hybpiper/2.0.1/lib/python3.9/site-packages/pebble/common.py", line 174, in process_execute
return function(*args, **kwargs)
File "/bioinformatics/hybpiper/2.0.1/lib/python3.9/site-packages/hybpiper/assemble.py", line 868, in exonerate
worker_configurer_func(gene_name) # set up process-specific logging to file
File "/bioinformatics/hybpiper/2.0.1/lib/python3.9/site-packages/hybpiper/utils.py", line 358, in worker_configurer
File "/bioinformatics/hybpiper/2.0.1/lib/python3.9/logging/init.py", line 1146, in init
File "/bioinformatics/hybpiper/2.0.1/lib/python3.9/logging/init.py", line 1175, in _open
OSError: [Errno 24] Too many open files: '/T399/LOC100267692/LOC100267692_2023-02-07-01_08_33.log'

I couldn´t find any information related to this problem and Hybpiper has worked fine with other samples.

Could you please help me out with this?

Thank you

Best wishes

Alicia
@chrisjackson-pellicle
Copy link
Collaborator

Hi Alicia,

I haven't seen this particular issue with HybPiper before. It suggests that the number of log files being opened by the Exonerate stage of the pipeline is greater than the number allowed by your operating system. However, unless the limit on you system is very small, and you're running HybPiper with hundreds of cpus/threads, this shouldn't be the case.

Can you tell me:

  1. What your operating system is (macOS and version, or Linux distribution and version?)
  2. What the output of the command ulimit -a is?

Also, can you send the me HybPiper log file for this sample, found in the sample directory?

Cheers,

Chris

@Alicia-Talavera
Copy link
Author

Alicia-Talavera commented Feb 9, 2023 via email

@chrisjackson-pellicle
Copy link
Collaborator

Hi Alicia,

I can't see your log file attached - could you re-upload it, please?

Your open file limit (131072) looks fine, to me, as far as HybPiper goes. To confirm, you're running this on a local machine (not a remote cluster), and the ulimit -a output is from that local machine?

Is the problem repeatable? That is, if you run the same sample with the same targetfile and settings, do you get the same error?

Also, do you mean that HybPiper ran fine for different samples when using the same target file? Are you able to upload the target file here? I'll load up a CentOS 7 VM and see if I can replicate the problem.

Cheers,

Chris

@Alicia-Talavera
Copy link
Author

Alicia-Talavera commented Feb 15, 2023 via email

@chrisjackson-pellicle
Copy link
Collaborator

Hi Alicia,

I just noticed that you're replying to this issue thread via email. Attachments you send via email replies won't appear here. Can you attach both:

  1. your log file
  2. your target file

...via this issue thread on the git repository webpage, directly?

Ah, you're using a HPC. In that case, we need to check output of ulimit -a on the worker nodes of the HPC. Is that what you posted above? I would presume that HPC worker nodes would have a high limit set by default, but it's worth double checking.

Cheers,

Chris

@Alicia-Talavera
Copy link
Author

Hi Chris!

Yes, I was replying via email, sorry. The output of ulimit -a is:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515138
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 131072
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Please find attached the files.

Thank you!

AT255_hybpiper_assemble_2023-02-05-12_15_21.log
Target_file-20190502-1013.fas.txt

@chrisjackson-pellicle
Copy link
Collaborator

Hi Alicia,

Thanks for those files.

Can you confirm that the ulimit -a output you posted above is from one of the HPC worker nodes, rather than e.g. your personal computer, or the HPC login node?

Regardless, I see from your log file that you ran HybPiper with a single thread/cpu (using --cpu 1 - I assume you were doing this for testing purposes?), which means that only a single gene should be processed at a time during the exonerate_contigs step of the pipeline. Therefore, I'm not sure why you're getting a "OSError: [Errno 24] Too many open files" error - processing a single gene at a time shouldn't produce a lot of open files, and hence shouldn't produce an error even if the ulimit -n (open files) value is low.

When you've re-run this sample, is it always the same genes that cause this error? The gene list can be seen at the end of the log file:

2023-02-05 16:51:12,826 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO - [INFO]:    The exonerate_contigs step of the pipeline failed for the following genes:
2023-02-05 16:51:12,826 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100263869
2023-02-05 16:51:12,826 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100853312
2023-02-05 16:51:12,826 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100264328
2023-02-05 16:51:12,826 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100244984
2023-02-05 16:51:12,826 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100267053
2023-02-05 16:51:12,827 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100267692
2023-02-05 16:51:12,827 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100262954
2023-02-05 16:51:12,827 - assemble.py - hybpiper.assemble - exonerate_multiprocessing - INFO -            LOC100263053

Also, can you please send me the *_contigs.fasta and *_target.fasta files for a couple of the genes that failed? These are found in the corresponding gene subdirectory within the sample directory. For example, for gene LOC100267692 they'll be located at sample_directory -> LOC100267692 -> LOC100267692_contigs.fasta and sample_directory -> LOC100267692 -> LOC100267692_target.fasta.

Lastly, I had a look at your target file, which contains 1,086 sequences for 1,013 unique genes. It looks like the sequences are all transcript sequences (i.e., with 5' and 3' UTR), rather than CDS sequences that can be correctly translated from the first reading frame to produce the corresponding protein sequence. I recommend reading this part of the wiki to see how this can reduce your locus recovery.

Cheers,

Chris

@Alicia-Talavera
Copy link
Author

Hi Chris, 

Thank you again for your reply, 

Sorry, I misunderstand it. The ulimit -a output that I posted is from my HPC login node.

When I re-run this sample or another one from the same batch always the same genes cause the error. Attached are the contigs.fasta and target.fasta files.

Thank you for your time and help!

Best wishes

Alicia

LOC100263869_contigs.fasta.txt
LOC100263869_target.fasta.txt
LOC100264328_contigs.fasta.txt
LOC100264328_target.fasta.txt
LOC100853312_contigs.fasta.txt
LOC100853312_target.fasta.txt

 

@chrisjackson-pellicle
Copy link
Collaborator

Thanks! I should've also asked for the *_interleaved.fasta files for these genes - can you upload them, please?

@Alicia-Talavera
Copy link
Author

Sure thing!

Thank you! I will check your recommendation about the target file!

LOC100264328_interleaved.fasta.txt
LOC100263869_interleaved.fasta.txt
LOC100853312_interleaved.fasta.txt

@chrisjackson-pellicle
Copy link
Collaborator

Hi Alicia,

I've run just the Exonerate contigs step of the pipeline on the *.contigs.fasta, *.target.fasta and *.interleaved.fasta files you provided, and they all complete successfully.

Given that, I'm still not sure what could be causing the error in your case. Can you please:

  1. Check your usage limits on your HPC, including your inode limits, and post them here.

  2. Upload the job submission script you used to submit the sample(s) that failed.

Thanks,

Chris

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Alicia-Talavera @chrisjackson-pellicle