outbam

[pengsl-lab]

No error is reported, but the final bam file is not generated, only some log files.

[mrmilhaven]

I have this same issue.

[sfragkoul]

I am facing this bug as well...

[adamewing]

Hi, sorry I've been way behind on addressing bamsurgeon issues. Are you able to run the tests e.g. from within the bamsurgeon repository:

cd test
./test_snv path/to/picard.jar

(it should run for awhile and yield a few snv calls in the end)

If that doesn't work, please reply with the full output. If it does work, please post the full output of the command that didn't work. Thanks!

[sfragkoul]

Hello and thanks for the quick response!

When running the script as it is, the initial output prompted me to change a command, here follows only the last part of the output:

Note that using "samtools mpileup" to generate BCF or VCF files has been removed. To output these formats, please use "bcftools mpileup" instead. ./test_snv.sh: line 19: bcftools: command not found

And then I installed bcftools in my environment and changed samtools mpileup to bcftools mpileup, so here is the second output, when running ./test_snv.sh:

`./test_snv.sh ../picard.jar
INFO 2022-12-13 10:17:13,174 starting ../bin/addsnv.py called with args: ../bin/addsnv.py -v ../test_data/random_snvs.txt -f ../test_data/testregion_realign.bam -r ../test_data/Homo_sapiens_chr22_assembly19.fasta -o ../test_data/testregion_mut.bam -n 5 -c ../test_data/test_cnvlist.txt.gz --picardjar ../picard.jar --aligner mem --seed 1234
INFO 2022-12-13 10:17:13,372 haplo_22_33714965_33714965 creating tmp bam: addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam
INFO 2022-12-13 10:17:21,836 haplo_22_33714965_33714965 len(readlist): 49
INFO 2022-12-13 10:17:21,837 haplo_22_33714965_33714965 copy number in snp region: 22 33714965 33714965 = 3.0
INFO 2022-12-13 10:17:21,837 haplo_22_33714965_33714965 adjusted VAF: 0.166667
INFO 2022-12-13 10:17:21,837 haplo_22_33714965_33714965 picked: 8
INFO 2022-12-13 10:17:21,837 haplo_22_33714965_33714965 wrote: 49, mutated: 8
INFO 2022-12-13 10:17:21,895 haplo_22_33714965_33714965 converting addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam to fastq
INFO 2022-12-13 10:17:21,896 converting BAM addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam to FASTQ

10:17:23.834 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/d/sfragkoul/Synth_Data/Synthesizers/bamsurgeon/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Tue Dec 13 10:17:23 EET 2022] SamToFastq INPUT=addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam FASTQ=addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.fastq INTERLEAVE=true INCLUDE_NON_PRIMARY_ALIGNMENTS=false VALIDATION_STRINGENCY=SILENT OUTPUT_PER_RG=false COMPRESS_OUTPUTS_PER_RG=false RG_TAG=PU RE_REVERSE=true INCLUDE_NON_PF_READS=false CLIPPING_MIN_LENGTH=0 READ1_TRIM=0 READ2_TRIM=0 VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Tue Dec 13 10:17:23 EET 2022] Executing as sfragkoul@ws-70-84 on Linux 5.10.102.1-microsoft-standard-WSL2 amd64; OpenJDK 64-Bit Server VM 17.0.3-internal+0-adhoc..src; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.18.4-SNAPSHOT
[Tue Dec 13 10:17:24 EET 2022] picard.sam.SamToFastq done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=427819008
INFO 2022-12-13 10:17:24,125 haplo_22_33714965_33714965 aligning addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.fastq with bwa mem
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 98 sequences (9800 bp)...
[M::process] 0 single-end sequences; 98 paired-end sequences
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 43, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (469, 502, 524)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (359, 634)
[M::mem_pestat] mean and std.dev: (496.95, 38.40)
[M::mem_pestat] low and high boundaries for proper pairs: (304, 689)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 98 reads in 0.008 CPU sec, 0.013 real sec
[main] Version: 0.7.17-r1188
[main] CMD: bwa mem -Y -t 1 -M -p ../test_data/Homo_sapiens_chr22_assembly19.fasta addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.fastq
[main] Real time: 0.768 sec; CPU: 0.117 sec
INFO 2022-12-13 10:17:24,902 haplo_22_33714965_33714965 writing addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam.realign.sam to BAM...
INFO 2022-12-13 10:17:24,980 haplo_22_33714965_33714965 deleting SAM: addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam.realign.sam
INFO 2022-12-13 10:17:24,985 haplo_22_33714965_33714965 sorting output: samtools sort -@ 1 -T addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam.realign.sorted.bam -o addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam.realign.sorted.bam addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam
INFO 2022-12-13 10:17:25,058 haplo_22_33714965_33714965 remove original bam: addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam
INFO 2022-12-13 10:17:25,062 haplo_22_33714965_33714965 rename sorted bam: addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam.realign.sorted.bam to original name: addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam
INFO 2022-12-13 10:17:25,069 haplo_22_33714965_33714965 indexing: samtools index addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam
INFO 2022-12-13 10:17:25,184 haplo_22_33714965_33714965 removing addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.fastq
INFO 2022-12-13 10:17:25,226 haplo_22_33714965_33714965 avgincover: 49.500000, avgoutcover: 49.000000
INFO 2022-12-13 10:17:25,268 haplo_22_33769483_33769483 creating tmp bam: addsnv.tmp/haplo_22_33769483_33769483.tmpbam.95a0ee0a-10c2-4058-9eb7-ccb2b090a310.bam
INFO 2022-12-13 10:17:29,319 haplo_22_33769483_33769483 len(readlist): 0
WARNING 2022-12-13 10:17:29,319 haplo_22_33769483_33769483 too few reads in region (0) skipping...
INFO 2022-12-13 10:17:29,466 haplo_22_33908770_33908770 creating tmp bam: addsnv.tmp/haplo_22_33908770_33908770.tmpbam.a6f0f000-f67f-4e51-991c-01de5f251327.bam
INFO 2022-12-13 10:17:36,732 haplo_22_33908770_33908770 len(readlist): 0
WARNING 2022-12-13 10:17:36,732 haplo_22_33908770_33908770 too few reads in region (0) skipping...
INFO 2022-12-13 10:17:36,882 haplo_22_34166720_34166720 creating tmp bam: addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam
INFO 2022-12-13 10:17:42,852 haplo_22_34166720_34166720 len(readlist): 56
INFO 2022-12-13 10:17:42,852 haplo_22_34166720_34166720 copy number in snp region: 22 34166720 34166720 = 3.0
INFO 2022-12-13 10:17:42,852 haplo_22_34166720_34166720 adjusted VAF: 0.333333
INFO 2022-12-13 10:17:42,852 haplo_22_34166720_34166720 picked: 18
INFO 2022-12-13 10:17:42,853 haplo_22_34166720_34166720 wrote: 56, mutated: 18
INFO 2022-12-13 10:17:42,914 haplo_22_34166720_34166720 converting addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam to fastq
INFO 2022-12-13 10:17:42,915 converting BAM addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam to FASTQ

10:17:44.678 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/d/sfragkoul/Synth_Data/Synthesizers/bamsurgeon/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Tue Dec 13 10:17:44 EET 2022] SamToFastq INPUT=addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam FASTQ=addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.fastq INTERLEAVE=true INCLUDE_NON_PRIMARY_ALIGNMENTS=false VALIDATION_STRINGENCY=SILENT OUTPUT_PER_RG=false COMPRESS_OUTPUTS_PER_RG=false RG_TAG=PU RE_REVERSE=true INCLUDE_NON_PF_READS=false CLIPPING_MIN_LENGTH=0 READ1_TRIM=0 READ2_TRIM=0 VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Tue Dec 13 10:17:44 EET 2022] Executing as sfragkoul@ws-70-84 on Linux 5.10.102.1-microsoft-standard-WSL2 amd64; OpenJDK 64-Bit Server VM 17.0.3-internal+0-adhoc..src; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.18.4-SNAPSHOT
[Tue Dec 13 10:17:44 EET 2022] picard.sam.SamToFastq done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=427819008
INFO 2022-12-13 10:17:44,943 haplo_22_34166720_34166720 aligning addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.fastq with bwa mem
[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::process] read 112 sequences (11200 bp)...
[M::process] 0 single-end sequences; 112 paired-end sequences
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 56, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (469, 504, 539)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 679)
[M::mem_pestat] mean and std.dev: (503.70, 52.15)
[M::mem_pestat] low and high boundaries for proper pairs: (259, 749)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 112 reads in 0.008 CPU sec, 0.014 real sec
[main] Version: 0.7.17-r1188
[main] CMD: bwa mem -Y -t 1 -M -p ../test_data/Homo_sapiens_chr22_assembly19.fasta addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.fastq
[main] Real time: 0.881 sec; CPU: 0.152 sec
INFO 2022-12-13 10:17:45,832 haplo_22_34166720_34166720 writing addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam.realign.sam to BAM...
INFO 2022-12-13 10:17:45,909 haplo_22_34166720_34166720 deleting SAM: addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam.realign.sam
INFO 2022-12-13 10:17:45,913 haplo_22_34166720_34166720 sorting output: samtools sort -@ 1 -T addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam.realign.sorted.bam -o addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam.realign.sorted.bam addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam
INFO 2022-12-13 10:17:45,990 haplo_22_34166720_34166720 remove original bam: addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam
INFO 2022-12-13 10:17:45,995 haplo_22_34166720_34166720 rename sorted bam: addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam.realign.sorted.bam to original name: addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam
INFO 2022-12-13 10:17:46,002 haplo_22_34166720_34166720 indexing: samtools index addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam
INFO 2022-12-13 10:17:46,118 haplo_22_34166720_34166720 removing addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.fastq
INFO 2022-12-13 10:17:46,158 haplo_22_34166720_34166720 avgincover: 56.000000, avgoutcover: 56.000000
INFO 2022-12-13 10:17:46,209 haplo_22_34166721_34166721 creating tmp bam: addsnv.tmp/haplo_22_34166721_34166721.tmpbam.ccf70db7-4a52-4ad3-82be-82017a8a6fe8.bam
INFO 2022-12-13 10:17:52,259 haplo_22_34166721_34166721 len(readlist): 0
WARNING 2022-12-13 10:17:52,259 haplo_22_34166721_34166721 too few reads in region (0) skipping...
INFO 2022-12-13 10:17:52,347 len(bamlist): 2
INFO 2022-12-13 10:17:52,347 merging, cmd: samtools merge -f tmp.merging.0.addsnv.165f836b-f4c1-4f63-9105-695ee6714412.muts.bam addsnv.tmp/haplo_22_33714965_33714965.tmpbam.4b7e8c65-379b-4a7f-a029-f85565fbdc9c.bam addsnv.tmp/haplo_22_34166720_34166720.tmpbam.b937ce8f-5d64-455b-bfae-f0c539e2db3e.bam
INFO 2022-12-13 10:17:52,423 merge finished, renaming: tmp.merging.0.addsnv.165f836b-f4c1-4f63-9105-695ee6714412.muts.bam --> addsnv.165f836b-f4c1-4f63-9105-695ee6714412.muts.bam
INFO 2022-12-13 10:17:52,470 done making mutations, merging mutations into ../test_data/testregion_realign.bam --> ../test_data/testregion_mut.bam
[E::idx_find_and_load] Could not retrieve index file for 'addsnv.165f836b-f4c1-4f63-9105-695ee6714412.muts.bam'
INFO 2022-12-13 10:17:52,600 loading donor reads into dictionary...

INFO 2022-12-13 10:17:52,605 secondary reads count:0
INFO 2022-12-13 10:17:52,605 supplementary reads count:0
INFO 2022-12-13 10:17:52,605 loaded 210 reads, (0 excluded, 0 null or secondary or supplementary--> ignored)

INFO 2022-12-13 10:17:57,684 replaced 210 reads (0 excluded )

INFO 2022-12-13 10:17:57,684 kept 0 secondary reads.

INFO 2022-12-13 10:17:57,685 kept 0 supplementary reads.

INFO 2022-12-13 10:17:57,685 ignored 0 non-primary reads in target BAM.

INFO 2022-12-13 10:17:58,000 vcf output written to testregion_mut.addsnv.random_snvs.vcf
Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
mpileup: invalid option -- 'u'
Invalid option: '?'
Failed to read from standard input: unknown file type`

[surajraj99]

Any update on how to fix this? Facing this issue as well.

[gailrosen]

Yeah --doesn't work for me too. Here's my full output (at the end i try to copy over the output from scratch.. but it's not there.) This is run from a container i made. at the end i list the temp directory and then the directory where i output the output bam.. but it's not there.

INFO 2023-05-13 10:11:02,017 starting /bamsurgeon/bin/addsnv.py called with args: /bamsurgeon/bin/addsnv.py -p 48 -f /nistdata/HG002/SRX936335
2_chr22.bam -v /nistdata/HG002/SRX9363352_chr22.bed --reference /nistdata/reference_set/GRCh38_masked_v2_decoy_excludes_GPRIN2_DUSP22_FANCD2.f
asta -n 10 --ignoresnps --tagreads --ignorepileup --tmpdir /tmp --aligner mem --minmutreads 1 --picardjar /picard.jar --requirepaired -o /tmpg
lr/randomsnv_chr22_9363352_max10_rp.bam --seed 1234
INFO 2023-05-13 10:11:06,154 haplo_chr22_50807974_50807974 creating tmp bam: /tmp/haplo_chr22_50807974_50807974.tmpbam.2dc3ef10-55d7-4323-9800
-a9febcf75599.bam
INFO 2023-05-13 10:11:06,154 haplo_chr22_50808016_50808016 creating tmp bam: /tmp/haplo_chr22_50808016_50808016.tmpbam.fd9f0d14-5041-4f6f-82b3
-096fc05c8146.bam
INFO 2023-05-13 10:11:06,155 haplo_chr22_50807971_50807971 creating tmp bam: /tmp/haplo_chr22_50807971_50807971.tmpbam.5363ebf3-7ed3-4c2a-8613
-30375cc7e850.bam
INFO 2023-05-13 10:11:06,155 haplo_chr22_50807976_50807976 creating tmp bam: /tmp/haplo_chr22_50807976_50807976.tmpbam.19b2dc48-3839-4404-bb2e
-0db56944fa65.bam
INFO 2023-05-13 10:11:06,155 haplo_chr22_50808040_50808040 creating tmp bam: /tmp/haplo_chr22_50808040_50808040.tmpbam.f7865779-6e52-4473-a743
-1a3c9a0a419b.bam
INFO 2023-05-13 10:11:06,156 haplo_chr22_50808091_50808091 creating tmp bam: /tmp/haplo_chr22_50808091_50808091.tmpbam.657bba96-a407-4410-b377
-9a8984e0a2fb.bam
INFO 2023-05-13 10:11:06,156 haplo_chr22_50808150_50808150 creating tmp bam: /tmp/haplo_chr22_50808150_50808150.tmpbam.55092b12-7748-4f78-97bd
-da4b87708b2c.bam
INFO 2023-05-13 10:11:06,156 haplo_chr22_50808453_50808453 creating tmp bam: /tmp/haplo_chr22_50808453_50808453.tmpbam.672cde53-6549-40ff-84f4
-a5d7fd88ff35.bam
INFO 2023-05-13 10:11:06,156 haplo_chr22_50808462_50808462 creating tmp bam: /tmp/haplo_chr22_50808462_50808462.tmpbam.a9966c55-df15-4c02-95ac
-c7032fcbd2e5.bam
INFO 2023-05-13 10:11:06,157 haplo_chr22_50808436_50808436 creating tmp bam: /tmp/haplo_chr22_50808436_50808436.tmpbam.8ba75f0e-09fd-4cae-b515
-d03d0ed00145.bam
WARNING 2023-05-13 10:11:42,116 chr22_50807934_50808085_None_None warning: no mate for SRR12898339.143113464
WARNING 2023-05-13 10:11:42,117 chr22_50807934_50808085_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,120 chr22_50808382_50808443_None_None warning: no mate for SRR12898339.470590902
WARNING 2023-05-13 10:11:42,120 chr22_50808382_50808443_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,135 chr22_50807955_50808106_None_None warning: no mate for SRR12898339.143105968
WARNING 2023-05-13 10:11:42,135 chr22_50807955_50808106_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,135 chr22_50807900_50808024_None_None warning: no mate for SRR12898339.143113464
WARNING 2023-05-13 10:11:42,136 chr22_50807900_50808024_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,136 chr22_50808378_50808458_None_None warning: no mate for SRR12898339.470588912
WARNING 2023-05-13 10:11:42,136 chr22_50808378_50808458_None_None skipped mutation due to --requirepaired
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/concurrent/futures/process.py", line 239, in process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/bamsurgeon/bin/addsnv.py", line 119, in makemut
mutfail, hasSNP, maxfrac, outreads, mutreads, mutmates = mutation.mutate(args, log, bamfile, bammate, chrom, min(mutpos_list), max(mutpos
list)+1, mutpos_list, avoid=avoid, mutid_list=mutid_list, is_snv=True, mutbase_list=mutbase_list, reffile=reffile)
ValueError: not enough values to unpack (expected 6, got 5)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/bamsurgeon/bin/addsnv.py", line 483, in
run()
File "/bamsurgeon/bin/addsnv.py", line 480, in run
main(args)
File "/bamsurgeon/bin/addsnv.py", line 390, in main
tmpbamlist = result.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
ValueError: not enough values to unpack (expected 6, got 5)
WARNING 2023-05-13 10:11:42,155 chr22_50808102_50808168_None_None warning: no mate for SRR12898339.143111018
WARNING 2023-05-13 10:11:42,155 chr22_50808102_50808168_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,162 chr22_50807950_50808029_None_None warning: no mate for SRR12898339.143113464
WARNING 2023-05-13 10:11:42,163 chr22_50807950_50808029_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,168 chr22_50807932_50808029_None_None warning: no mate for SRR12898339.143113464
WARNING 2023-05-13 10:11:42,168 chr22_50807932_50808029_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,222 chr22_50808020_50808094_None_None warning: no mate for SRR12898339.143117872
WARNING 2023-05-13 10:11:42,222 chr22_50808020_50808094_None_None skipped mutation due to --requirepaired
WARNING 2023-05-13 10:11:42,309 chr22_50808378_50808468_None_None warning: no mate for SRR12898339.143113335
WARNING 2023-05-13 10:11:42,310 chr22_50808378_50808468_None_None skipped mutation due to --requirepaired
total 0
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50807971_50807971.tmpbam.5363ebf3-7ed3-4c2a-8613-30375cc7e850.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50807974_50807974.tmpbam.2dc3ef10-55d7-4323-9800-a9febcf75599.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50807976_50807976.tmpbam.19b2dc48-3839-4404-bb2e-0db56944fa65.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808016_50808016.tmpbam.fd9f0d14-5041-4f6f-82b3-096fc05c8146.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808040_50808040.tmpbam.f7865779-6e52-4473-a743-1a3c9a0a419b.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808091_50808091.tmpbam.657bba96-a407-4410-b377-9a8984e0a2fb.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808150_50808150.tmpbam.55092b12-7748-4f78-97bd-da4b87708b2c.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808436_50808436.tmpbam.8ba75f0e-09fd-4cae-b515-d03d0ed00145.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808453_50808453.tmpbam.672cde53-6549-40ff-84f4-a5d7fd88ff35.bam
-rw-rw-r-- 1 gailr gailr 0 May 13 10:11 haplo_chr22_50808462_50808462.tmpbam.a9966c55-df15-4c02-95ac-c7032fcbd2e5.bam
total 0
cp: cannot stat '/scratch/gailr/randomsnv_chr22_9363352_max10_rp.bam': No such file or directory

[juliawiggeshoff]

Having the same problem here... I see all the tmpbam files created in the temporary directory, but the final mutated bam file is not being written.

I cannot really perform the tests inside of test because I'm running everything from a singularity container I've built. However, when I do a test run, all expected files are there, so I don't understand why I'm not getting a bam file with my real data.

singularity exec --pid -B local_tmp_dir/:/outdir/ bamsurgeon.simg python3 -O /opt/bamsurgeon/bin/addsv.py -p 1 -v /opt/bamsurgeon/test_data/test_sv.txt -f /opt/bamsurgeon/test_data/testregion_realign.bam -r /opt/bamsurgeon/test_data/Homo_sapiens_chr22_assembly19.fasta -o /outdir/testregion_sv_mut.bam --vcf /outdir/ --tmpdir /outdir/tmp --aligner mem --keepsecondary --seed 1234 --inslib /opt/bamsurgeon/test_data/test_inslib.fa

As expected, if I were to list the contents of the folder local_tmp_dir I see the bam and the vcf files testregion_sv_mut.bam testregion_sv_mut.addsv.test_sv.vcf, though I am not seeing any tmp files inside of local_tmp_dir/tmp, I don't know if that is the excepted behaviour. When I tried to rum addsnv.py with my actual data, for which no mutated bam file was created, I see that the tmp folder was still full of tmpbam and bai files.

The BAM is one of the references from Genome in a Bottle, WGS, very high coverage (300x) and large (550 GB), could it be related to that (maybe related to #217)? If so, how to mitigate this issue? @adamewing

This is how I attempted to execute bamsurgeon with my real data

mydir="/my/path"
bamsurgeon_img="/my/path/bamsurgeon.simg"

mycmd=$(echo "python3 -O /opt/bamsurgeon/scripts/randomsites.py \
      --genome /mydir/reference/GCA_000001405.15_GRCh38_no_alt_plus_hs38d1_analysis_set.fasta \
      --bed /mydir/reference/padded_20bp_exons_GCA_000001405.15_GRCh38_full_analysis_set.refseq_annotation.bed \
      --seed 9876 --numpicks 5000 --avoidN --minvaf 0.05 --maxvaf 1.0 snv | sort -k1,1V -k2,2n -k3,3n \
      > /mydir/seed_9876_5000snvs.bed; \
      mkdir -p /mydir/addsnv_tmp; \
      python3 -O /opt/bamsurgeon/bin/addsnv.py \
      -v /mydir/seed_9876_5000snvs.bed \
      -f /mydir/HG001.GRCh38_full_plus_hs38d1_analysis_set_minus_alts.300x.bam \
      -r /mydir/reference/GCA_000001405.15_GRCh38_no_alt_plus_hs38d1_analysis_set.fasta \
      --tmpdir /mydir/addsnv_tmp \
      -o /mydir/unsorted.HG001.GRCh38_full_plus_hs38d1_analysis_set_minus_alts.300x.seed_1234.MUTATED.SNV.bam \
      --vcf /mydir/ \
      -p 8 --aligner mem --seed 1234 --alignerthreads 8 --minmutreads 3 --mindepth 20 --picardjar /usr/local/bin/picard.jar; \
      samtools sort /mydir/unsorted.HG001.GRCh38_full_plus_hs38d1_analysis_set_minus_alts.300x.seed_1234.MUTATED.SNV.bam \
      -o /mydir/sorted.HG001.GRCh38_full_plus_hs38d1_analysis_set_minus_alts.300x.seed_1234.MUTATED.SNV.bam; \
      samtools index /mydir/sorted.HG001.GRCh38_full_plus_hs38d1_analysis_set_minus_alts.300x.seed_1234.MUTATED.SNV.bam")

singularity exec --pid -B ${mydir}/:/mydir/ ${bamsurgeon_img} bash -c "${mycmd}"

after the last mutation has been added, i see from the log file the mutated bam cannot be sorted by samtools because it is simply not there