racon_wrapper: Overflow and corrupting overlaps?

[EinarBaldvin]

Hi Robert,

I am having some interesting issues with both split and subsample in the wrapper. I am using Racon v1.4.21.

1st: When I use subsample, it seems that my genome size could be causing a 32bit overflow in the subsampler.

2nd: Using split results in "[racon::Overlap::find_breaking_points] error: overlap is not transmuted!" error.

Below is an example with both parameters set:

racon_wrapper -m 8 -x -6 -g -8 -w 500 -t 2 –subsample 4400000000 70 –split 500000000 –cudapoa-batches 50 –cudaaligner-batches 10 raw_readID_integer_whitespaceRemoved.fastq default_intRead_int_reads2asm_SecNo.paf default.pol2.fasta
[RaconWrapper::run] preparing data with rampler
[RaconWrapper::run] total number of splits: 8
[RaconWrapper::run] processing data with racon
[racon::Polisher::initialize] loaded target sequences 1.991097 s
[racon::Polisher::initialize] loaded sequences 12.319651 s
[racon::Polisher::initialize] loaded overlaps 71.611163 s
[racon::Overlap::find_breaking_points] error: overlap is not transmuted!

Fastq is 600 GB
paf ~100 GB
Assembly ~ 3.7 Gbp

We could get Racon running without the wrapper, with the same datasets, same compilation and on the same hardware. (Currently still running... )

Any ideas to what is happening in the wrapper?

Best regards,
Einar

[rvaser]

Probably bugs in the sampler, will fix it :)

Best regards,
Robert

[rvaser]

Can you please try the latest commit?

[fbemm]

I run into the same problem. The latest commit did not help. Racon works, racon_wrapper not. Long reads don't have duplicate ids. PAF contains only IDs that are also in the long reads. Any chance to check which reads/alignments are causing this?

Bests,
Felix

[rvaser]

Did you do a clean git clone or just git pull? If the latter, did you run git submodule update --recursive? If all of this is fine, you can send me the data and I'll investigate.

[fbemm]

Fresh start from 378dd81 using git clone within a nightly container build.
Sending data is unfortunately not possible. I have another suspicion I am following now. I'll keep you updated.

[fbemm]

My idea was a dead end. Thought that the chunk size somehow influenced the whole thing but that is not the case. I is there a way to report the alignments / reads causing this?

[rvaser]

Your error is overlap not transmutted while using split? Can you please paste all commands you used (including mapping) and tell me the file sizes? I'll try to recreate it with something similar.

[fbemm]

Yes, overlap not transmutted happens when I do this:

minimap2 -x map-ont -t 64 asm.fa reads.fq > paf.0.paf
racon_wrapper -m 8 -x -6 -g -8 -w 500 --split 36839382 -t 64 reads.fq paf.0.paf asm.0.fa > asm.1.fa

• asm.0.fa is 2086842457
• paf.0.paf is 3130012015
• reads.fq is 240679143440

[EinarBaldvin]

I finally could give it a go, but now I am running into a new error:

[RaconWrapper::run] preparing data with rampler
[RaconWrapper::run] total number of splits: 8
[RaconWrapper::run] processing data with racon
[racon::createPolisher] error: file has unsupported format extension (valid extensions: .fasta, .fasta.gz, .fna, .fna.gz, .fa, .fa.gz, .fastq, .fastq.gz, .fq, .fq.gz)!

When I run the same data with Racon directly it is able to load the files.

[rvaser]

@EinarBaldvin, please paste the whole command you were using (plus mapping).

@fbemm, I'll investigate shortly. Sorry for the delay.

[rvaser]

@fbemm, shouldn't asm.0.fa be also in minimap2 command? Is that a typo or?

[fbemm]

minimap2 -x map-ont -t 64 asm.0.fa reads.fq > paf.0.paf
racon_wrapper -m 8 -x -6 -g -8 -w 500 --split 36839382 -t 64 reads.fq paf.0.paf asm.0.fa > asm.1.fa

Correct! Just a type <- Kinda this stuff ;) ... typo

[rvaser]

@fbemm, can you please run head -n 1 asm.0.fa, head -n 1 reads.fq and head -n 1 paf.0.paf?

[fbemm]

==> asm.0.fa <==
>utg000001l

==> paf.0.paf <==
12fa85bb-b2a3-4864-ac43-d499ffe80337    100934  31      76572   -       utg000223l      8558942 4976222 5052868 21164   78033   60      tp:A:P  cm:i:2398       s1:i:20740      s2:i:1929       dv:f:0.1159     rl:i:6408

==> reads.fq <==
@12fa85bb-b2a3-4864-ac43-d499ffe80337 runid=95685f66382cea7d3558a2d2270d69c65ac9f6bb sampleid=28546 read=10042 ch=1357 start_time=2021-03-23T18:41:08Z

[EinarBaldvin]

minimap2 -ax map-ont -L -I 120G -t $NCPUS asm.fasta reads.fq > reads2asm.sam

Then kept only primary mappings before sam2paf with paftools.js in minimap2.

I am currently rerunning the mapping with: minimap2 -ax map-ont -I 80G -t $NCPUS asm.fasta reads.fq > reads2asm.sam

However, same dataset two behaviors:

racon_wrapper -m 8 -x -6 -g -8 -w 500 -t 96 --subsample 4400000000 30 --split 500000000 --cudapoa-batches 50 --cudaaligner-batches 10 reads.fastq reads2asm.paf asm.fasta > asm_polished.fasta

[RaconWrapper::run] preparing data with rampler
[RaconWrapper::run] total number of splits: 8
[RaconWrapper::run] processing data with racon
[racon::createPolisher] error: file has unsupported format extension (valid extensions: .fasta, .fasta.gz, .fna, .fna.gz, .fa, .fa.gz, .fastq, .fastq.gz, .fq, .fq.gz)!

racon -m 8 -x -6 -g -8 -w 500 -t 64 --cudapoa-batches 50 --cudaaligner-batches 10 reads.fastq reads2asm.paf asm.fasta > asm_polished.fasta

Using 1 GPU(s) to perform polishing
Initialize device 0
[CUDAPolisher] Constructed.
[racon::Polisher::initialize] loaded target sequences 14.976488 s
[racon::Polisher::initialize] loaded sequences 1397.237132 s
[racon::Polisher::initialize] loaded overlaps 96.867616 s
GPU 0: Aligning with band width 1432
[racon::CUDAPolisher::initialize] allocated memory on GPUs for alignment 2.285585 s
[racon::CUDAPolisher::initialize] aligning overlaps [=> ] 2.353224 s
[racon::CUDAPolisher::initialize] aligning overlaps [==> ] 419.621921 s

head -n 1 reads2asm.paf
1 278 36 276 + contig_1047 66049974 3064035830640603 225 252 43 NM:i:27 ms:i:316 AS:i:316 nn:i:0 tp:A:P cm:i:6 s1:i:77 s2:i:46 de:f:0.0964 rl:i:0 cg:Z:35M2D32M1D5M1I31M1D20M1I9M1I3M2D3M1D6M1D14M1I13M1I15M1I6M1I15M2D13M1D3M1D10M

head -n 1 asm.fasta

contig_10006

head -n 1 reads.fastq
@1

@rvaser Should I drop the -c/a flag, base-level alignment, when mapping?

[rvaser]

@fbemm, looks fine.

@EinarBaldvin, if you are converting to paf anyway, then do not run minimap2 with -a/-c.

[rvaser]

I managed to use the wrapper on chm13 draft genome and some reads, tried split with size of the largest contig and that size divided by ten, in both instances wrapper is running as intended. The error "overlap not transmuted" means that one or both sequences in the overlap are missing from the sequence/target file.

@fbemm, @EinarBaldvin, please comment out https://github.com/lbcb-sci/racon/blob/master/scripts/racon_wrapper.py#L53-L56 and put pass in the __exit__ function, run cmake again and then please inspect the working folder created by the wrapper (after it errors out again).

[rvaser]

Actually, I get the same error at the last chunk. @fbemm, does the error on your data occur immediately on the first chunk or?

[fbemm]

From looking at the run time I would rather say it is a later one, if not the last. Restarted with the edited wrapper now. Can take a while.

[rvaser]

You do not have to run it again, I found the bug and will resolve it soon.

[rvaser]

Please try the latest commit.

[fbemm]

Running. Will take a while. Thx!

[fbemm]

Died again but without the transmute. There is no further error message.

[rvaser]

Do you have a time limit on the server? Weird that there is nothing printed afterwards.

[fbemm]

No, the same thing happens when executed manually. Should I comment in the pass procedure and rerun?

[rvaser]

How many chunks are created and finished successfully? You can put pass and then just run Racon on the chunk that breaks.

[fbemm]

How do I figure that out? There is a racon workdir created but with only the chunked fasta files inside.

[rvaser]

You can count number of processing data with racon lines in the log.

[fbemm]

Just a suspicion, it might be that the error only occurs if --split is empty or zero. Seems as racon than splits in n chunks and n corresponds to the number of sequences in the assembly. Or it actually happens only if split number equals sequence number.

[fbemm]

Suspicion is wrong.

[rvaser]

Did you run it again and have the working folder? Did you run the chunk that breaks?

[rvaser]

Similar/same issue at isovic/racon#194, I think I fixed it with the latest commit. Please try.