addsv Error missing bwatmp file

[fre335]

Hi there!
im trying to simulate SVs but I keep running into an Error:

FileNotFoundError: [Errno 2] No such file or directory: 'addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam'

It seems that the error already appears in a previous step as the file "bwatmp.7875ea41-f81d-41b7-bf1b-ea54dfe6490a.sam" is empty.

This is the command I used:

python3 ../bamsurgeon/bin/addsv.py -v SV_del.bed -f NA24143.wgs.hg38.chr22.bam -r variant_sim/hg38_canonical.fa.fasta -l 100 --aligner mem --debug -o NA24143.wgs.chr22_simulated.bam &>NA24143_wgs_del.log

This is how my bed file looks:

chr22 33871043 33874754 DEL 0.75

This is the output:

`�����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������WARNING 2021-02-09 15:22:58,856 cannot find mate for read marked paired: D00360:94:H2YT5BCXX:1:2207:10181:21772
INFO 2021-02-09 15:23:10,989 found mates for 1000 reads, 0.00 discordant.
INFO 2021-02-09 15:23:12,722 found 1065 reads in region.
[0.000009] Reading FastA file addsv.tmp/chr22_33871043_33874754_DEL.tmpreads.71817fd1-f0b2-49e5-b2b5-fe3571f5d529.fasta;
[0.167270] 1096 sequences found
[0.167287] Done
[0.170336] Reading read set file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/Sequences;
[0.183409] 1096 sequences found
[0.192503] Done
[0.192517] 1096 sequences in total.
[0.197740] Writing into roadmap file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/Roadmaps...
[0.201799] Inputting sequences...
[0.201817] Inputting sequence 0 / 1096
[0.415459] === Sequences loaded in 0.213761 s
[0.416716] Done inputting sequences
[0.416731] Destroying splay table
[0.439146] Splay table destroyed
[0.000010] Reading roadmap file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/Roadmaps
[0.019355] 1096 roadmaps read
[0.019439] Creating insertion markers
[0.019598] Ordering insertion markers
[0.022015] Counting preNodes
[0.022173] 2390 preNodes counted, creating them now
[0.036117] Adjusting marker info...
[0.036300] Connecting preNodes
[0.039974] Cleaning up memory
[0.039991] Done creating preGraph
[0.040001] Concatenation...
[0.040699] Renumbering preNodes
[0.040711] Initial preNode count 2390
[0.040784] Destroyed 1313 preNodes
[0.040795] Concatenation over!
[0.040805] Clipping short tips off preGraph
[0.040907] Concatenation...
[0.041089] Renumbering preNodes
[0.041102] Initial preNode count 1077
[0.041153] Destroyed 339 preNodes
[0.041164] Concatenation over!
[0.041173] 178 tips cut off
[0.041184] 738 nodes left
[0.045900] Writing into pregraph file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/PreGraph...
[0.069851] Reading read set file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/Sequences;
[0.070397] 1096 sequences found
[0.076816] Done
[0.082083] Reading pre-graph file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/PreGraph
[0.084935] Graph has 738 nodes and 1096 sequences
[0.091187] Scanning pre-graph file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/PreGraph for k-mers
[0.093176] 16125 kmers found
[0.096812] Sorting kmer occurence table ...
[0.103009] Sorting done.
[0.103018] Computing acceleration table...
[0.145077] Computing offsets...
[0.145194] Ghost Threading through reads 0 / 1096
[0.169069] === Ghost-Threaded in 0.023874 s
[0.169093] Threading through reads 0 / 1096
[0.188364] === Threaded in 0.019272 s
[0.202754] Correcting graph with cutoff 0.200000
[0.202809] Determining eligible starting points
[0.203444] Done listing starting nodes
[0.203453] Initializing todo lists
[0.203509] Done with initilization
[0.203521] Activating arc lookup table
[0.203560] Done activating arc lookup table
[0.217556] Concatenation...
[0.217584] Renumbering nodes
[0.217595] Initial node count 738
[0.217610] Removed 581 null nodes
[0.217621] Concatenation over!
[0.217632] Clipping short tips off graph, drastic
[0.217656] Concatenation...
[0.217705] Renumbering nodes
[0.217717] Initial node count 157
[0.217729] Removed 14 null nodes
[0.217740] Concatenation over!
[0.217750] 143 nodes left
[0.223701] Writing into graph file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/Graph2...
[0.258715] Measuring median coverage depth...
[0.258773] Median coverage depth = 14.542781
[0.260745] Removing contigs with coverage < 7.271390...
[0.260863] Concatenation...
[0.261068] Renumbering nodes
[0.261079] Initial node count 143
[0.261089] Removed 135 null nodes
[0.261099] Concatenation over!
[0.261109] Concatenation...
[0.261119] Renumbering nodes
[0.261129] Initial node count 8
[0.261138] Removed 0 null nodes
[0.261148] Concatenation over!
[0.261158] Clipping short tips off graph, drastic
[0.261168] Concatenation...
[0.261178] Renumbering nodes
[0.261187] Initial node count 8
[0.261197] Removed 0 null nodes
[0.261206] Concatenation over!
[0.261216] 8 nodes left
[0.261225] Read coherency...
[0.261235] Identifying unique nodes
[0.261245] Done, 0 unique nodes counted
[0.261254] Trimming read tips
[0.261264] Renumbering nodes
[0.261273] Initial node count 8
[0.261282] Removed 0 null nodes
[0.261292] Confronted to 0 multiple hits and 0 null over 0
[0.261479] Read coherency over!
[0.261498] Starting pebble resolution...
[0.261513] Computing read to node mapping array sizes
[0.261534] Computing read to node mappings
[0.261564] Estimating library insert lengths...
[0.261582] Done
[0.261592] Computing direct node to node mappings
[0.261611] Scaffolding node 0
[0.261629] === Nodes Scaffolded in 0.000018 s
[0.261641] Preparing to correct graph with cutoff 0.200000
[0.261672] Cleaning memory
[0.261683] Deactivating local correction settings
[0.261697] Pebble done.
[0.261707] Starting pebble resolution...
[0.261723] Computing read to node mapping array sizes
[0.261740] Computing read to node mappings
[0.261759] Estimating library insert lengths...
[0.261777] Done
[0.261787] Computing direct node to node mappings
[0.261805] Scaffolding node 0
[0.261824] === Nodes Scaffolded in 0.000019 s
[0.261834] Preparing to correct graph with cutoff 0.200000
[0.261859] Cleaning memory
[0.261869] Deactivating local correction settings
[0.261883] Pebble done.
[0.261893] Concatenation...
[0.261903] Renumbering nodes
[0.261912] Initial node count 8
[0.261922] Removed 0 null nodes
[0.261932] Concatenation over!
[0.261941] Removing reference contigs with coverage < 7.271390...
[0.261952] Concatenation...
[0.261962] Renumbering nodes
[0.261971] Initial node count 8
[0.261981] Removed 0 null nodes
[0.261990] Concatenation over!
[0.268407] Writing contigs into addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/contigs.fa...
[0.277561] Writing into stats file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/stats.txt...
[0.284527] Writing into graph file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/LastGraph...
[0.295381] Writing into AMOS file addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/velvet_asm.afg...
[0.595490] Printing unused reads into addsv.tmp/chr22_33871043_33874754_DEL.df07c15a/UnusedReads.fa
[0.601631] Estimated Coverage = 14.542781
[0.601652] Estimated Coverage cutoff = 7.271390
Final graph has 8 nodes and n50 of 1551, max 1662, total 4868, using 1091/1096 reads
INFO 2021-02-09 15:23:14,640 chr22_33871043_33874754_DEL best contig length: 1692
INFO 2021-02-09 15:23:14,723 chr22_33871043_33874754_DEL alignment result: ['SUMMARY', '6415', '0', '1292', '2708', '4000']
INFO 2021-02-09 15:23:14,726 chr22_33871043_33874754_DEL trimmed contig length: 1292
INFO 2021-02-09 15:23:14,728 chr22_33871043_33874754_DEL start: 33870898, end: 33874898, tgtstart: 2708, tgtend: 4000, refstart: 33873606, refend: 33874898
INFO 2021-02-09 15:23:14,729 chr22_33871043_33874754_DEL action: DEL 0.75 DEL
INFO 2021-02-09 15:23:14,732 chr22_33871043_33874754_DEL final VAF accounting for copy number 1.000000: 0.750000
WARNING 2021-02-09 15:23:14,734 chr22_33871043_33874754_DEL contig does not cover user start
INFO 2021-02-09 15:23:14,735 chr22_33871043_33874754_DEL set paired end mean distance: 300.000000
INFO 2021-02-09 15:23:14,737 chr22_33871043_33874754_DEL set paired end distance stddev: 70.000000
INFO 2021-02-09 15:23:14,746 chr22_33871043_33874754_DEL paired reads: 134
INFO 2021-02-09 15:23:14,748 chr22_33871043_33874754_DEL single reads: 35
INFO 2021-02-09 15:23:14,750 chr22_33871043_33874754_DEL discard reads: 0
INFO 2021-02-09 15:23:14,752 chr22_33871043_33874754_DEL total reads: 303
INFO 2021-02-09 15:23:14,754 chr22_33871043_33874754_DEL old ctg len: 1292
INFO 2021-02-09 15:23:14,756 chr22_33871043_33874754_DEL new ctg len: 200
INFO 2021-02-09 15:23:14,758 chr22_33871043_33874754_DEL adj. factor: 0.154799
INFO 2021-02-09 15:23:14,759 chr22_33871043_33874754_DEL num. sim. reads: 17
INFO 2021-02-09 15:23:14,760 chr22_33871043_33874754_DEL PE mean outer distance: 300.000000
INFO 2021-02-09 15:23:14,762 chr22_33871043_33874754_DEL PE outer distance SD: 70.000000
INFO 2021-02-09 15:23:14,763 chr22_33871043_33874754_DEL rerror rate: 0.000000
INFO 2021-02-09 15:23:14,766 ['wgsim', '-e', '0.0', '-d', '300.0', '-s', '70.0', '-N', '17', '-1', '250', '-2', '250', '-r', '0', '-R', '0', 'addsv.tmp/chr22_33871043_33874754_DEL.wgsimtmp.9052dfd2-9255-431b-899f-47f40fd712ed.fasta', 'addsv.tmp/chr22_33871043_33874754_DEL.wgsimtmp.9052dfd2-9255-431b-899f-47f40fd712ed.1.fq', 'addsv.tmp/chr22_33871043_33874754_DEL.wgsimtmp.9052dfd2-9255-431b-899f-47f40fd712ed.2.fq']
[wgsim] seed = 1612884194
[wgsim_core] calculating the total length of the reference sequence...
[wgsim_core] 1 sequences, total length: 200
[wgsim_core] skip sequence 'target' as it is shorter than 510!
INFO 2021-02-09 15:23:14,836 chr22_33871043_33874754_DEL aligning addsv.tmp/chr22_33871043_33874754_DEL.wgsimtmp.9052dfd2-9255-431b-899f-47f40fd712ed.1.fq,addsv.tmp/chr22_33871043_33874754_DEL.wgsimtmp.9052dfd2-9255-431b-899f-47f40fd712ed.2.fq with bwa mem
[E::bwa_idx_load_from_disk] fail to locate the index files
INFO 2021-02-09 15:23:14,887 chr22_33871043_33874754_DEL writing bwatmp.7875ea41-f81d-41b7-bf1b-ea54dfe6490a.sam to BAM...
[main_samview] fail to read the header from "bwatmp.7875ea41-f81d-41b7-bf1b-ea54dfe6490a.sam".
INFO 2021-02-09 15:23:14,926 chr22_33871043_33874754_DEL deleting SAM: bwatmp.7875ea41-f81d-41b7-bf1b-ea54dfe6490a.sam
INFO 2021-02-09 15:23:14,930 chr22_33871043_33874754_DEL sorting output: samtools sort -@ 1 -T bwatmp.7875ea41-f81d-41b7-bf1b-ea54dfe6490a.sorted.bam -o bwatmp.7875ea41-f81d-41b7-bf1b-ea54dfe6490a.sorted.bam addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam
[E::hts_open_format] Failed to open file "addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam" : No such file or directory
samtools sort: can't open "addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam": No such file or directory
INFO 2021-02-09 15:23:14,976 chr22_33871043_33874754_DEL remove original bam:addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "../bamsurgeon/bin/addsv.py", line 945, in makemut
outreads = aligners.remap_fastq(args.aligner, fq1, fq2, args.refFasta, outbam_mutsfile, alignopts, mutid=mutid, threads=int(args.alignerthreads))
File "/usr/local/lib/python3.8/dist-packages/bamsurgeon-1.2-py3.8.egg/bamsurgeon/aligners.py", line 718, in remap_fastq
return remap_bwamem_fastq(fq1, fq2, threads, fastaref, outbam, deltmp=deltmp, mutid=mutid)
File "/usr/local/lib/python3.8/dist-packages/bamsurgeon-1.2-py3.8.egg/bamsurgeon/aligners.py", line 757, in remap_bwamem_fastq
os.remove(outbam)
FileNotFoundError: [Errno 2] No such file or directory: 'addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "../bamsurgeon/bin/addsv.py", line 1360, in
main(args)
File "../bamsurgeon/bin/addsv.py", line 1121, in main
tmpbam, exclfn, mutinfo = result.get()
File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'addsv.tmp/chr22_33871043_33874754_DEL.7b6f5b07-0251-4666-b0ff-a63998cff574.muts.bam'`

If anyone has an idea what is wrong, I would be very happy if you could help me with this.

[adamewing]

Hi there,

Are you able to run the python setup.py install command successfully or does it complain about missing dependencies?

Also, are you able to run test_sv.sh from the test/ directory?

[fre335]

Hi Adam,
Thanks for your reply. I am running the script in the Docker image and I am successfully able to run the test script: test_sv.sh.
So it shouldn't be a problem with the dependencies.