Use racon for de novo variant discovery

[mbhall88]

TODO

• Sort out the merge conflicts
• Define SAM file format (SAM file discussion #300)
• Finish producing the debugging extra files (we create four extra files when mapping for each sample: a minimizer PAF file, a .minimatches file, and two files describing the clusters definition and filtering. These are debugging files, they seem very useful, but there is no need to thorough unit test. Their production will be hidden by default, and can be activate just with an env variable);
• Check new index param: indexing_upper_bound;
• Check and test racon discovery;
• Test out the rectified version for clear runtime bugs
• Write any further unit tests
• Update documentation

[leoisl]

Hey @mbhall88 , sorry, I just corrected two big bugs in the two most recent commits.

Bug 1: In this commit we started outputting SAM lines with flag 16, where the read maps as reverse complemented. However, when running discovery, we would just pull out reads where the flag was 0, i.e. I introduced a bug in this commit and de novo would just consider reads that mapped in the forward strand, effectively losing around half of the reads. This was fixed in this commit. This probably affects your results.

Bug 2: In this commit, I changed the order of them SAM extra fields so that is more pleasant to look at, so that very long extra fields (i.e. the flanks and the PRG path) are the last fields. Unfortunately, de novo racon code would find the left and the right flank fields by using predefined indexes, and not looking by the LF:Z and RF:Z tag. This was just fixed in this commit. This severely impacts your results, as you basically lost the left and right flank, but this commit was on 27th October, maybe you had not pulled these changes when you ran pandora...

[leoisl]

Finished memfd implementation (see #302) and improved the performance on some critical regions, which can yield improved runtime if pandora discover is ran with several threads. Just realised I have to do at least a few tests with different number of threads to see if the current code works as expected with several threads and how it scales with more threads

[mbhall88]

pandora-e164973.gz

[mbhall88]

Ooops I assume the tip of this PR is not functional yet? With that binary I got the following error

[racon::createPolisher] error: file /proc/874910/fd/4 has unsupported format extension (valid extensions: .fasta, .fasta.gz, .fna, .fna.gz, .fa, .fa.gz, .fastq, .fastq.gz, .fq, .fq.gz)!

Will retry with the commit you suggested in #303 9bb67c6

[mbhall88]

pandora-9bb67c6.gz

[mbhall88]

Ooops I assume the tip of this PR is not functional yet? With that binary I got the following error

[racon::createPolisher] error: file /proc/874910/fd/4 has unsupported format extension (valid extensions: .fasta, .fasta.gz, .fna, .fna.gz, .fa, .fa.gz, .fastq, .fastq.gz, .fq, .fq.gz)!

Will retry with the commit you suggested in #303 9bb67c6

Actually I get the same error with 9bb67c6 as well

[leoisl]

Sorry, forgot to say, I updated the racon fork to not error out on this, please run git submodule update and it should work

…

On Tue, 1 Nov 2022, 02:43 Michael Hall, ***@***.***> wrote: Ooops I assume the tip of this PR is not functional yet? With that binary I got the following error [racon::createPolisher] error: file /proc/874910/fd/4 has unsupported format extension (valid extensions: .fasta, .fasta.gz, .fna, .fna.gz, .fa, .fa.gz, .fastq, .fastq.gz, .fq, .fq.gz)! Will retry with the commit you suggested in #303 <#303> 9bb67c6 <9bb67c6> Actually I get the same error with 9bb67c6 <9bb67c6> as well — Reply to this email directly, view it on GitHub <#299 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACACJD64ZVVQG3WXFA2RJSLWGB7VRANCNFSM6AAAAAARGWTRMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[leoisl]

Hey @mbhall88 , I am also starting to run this code on the paper data for benchmarking purposes. You can find a precompiled binary for the latest commit attached.
pandora-linux-precompiled-glibc.gz

[leoisl]

Small update: started adding unit tests

[mbhall88]

Are you happy with the testing I've done on this now @leoisl?

Anything else that needs to be done before merging?

[leoisl]

Sorry for the delay, but I think you were on vacations this week anyway, right?

Are you happy with the testing I've done on this now @leoisl?

Happy with your tests! Not happy with my lack of unit tests!

Anything else that needs to be done before merging?

Well, there are some misc stuff like fixing the example script, changelog, etc... Should we do for this alpha release?