Faster FASTA and FASTQ metadata setting

[bernt-matthias]

So far the Fasta datatype just used the set_meta method of Sequence which is quite general and allows for things like comments and unnecessarily strips spaces and checks for empty lines. By a more strict interpretation of the FASTA "standard" we can be nearly 50% faster (see).

Implemented the same for FASTQ. The sniffer here is really strict (e.g. expects strictly 4 line blocks) so I think we can also simplify the code here...

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[nsoranzo]

According to https://en.wikipedia.org/wiki/FASTA_format , empty lines are allowed and I don't think they should be counted as data_lines.

[bernt-matthias]

Hmm. Right.

But it might be a less intrusive change if we would keep counting empty lines as data lines, or:

galaxy/lib/galaxy/datatypes/sequence.py

Line 125 in 6b52d6d

data_lines += 1

, or am I wrong.

Also one might argue that also an empty sequence is a sequence. Edit: but our sniffer would not accept it :)

According to https://blast.ncbi.nlm.nih.gov/doc/blast-topics/ empty lines are not allowed:

Blank lines are not allowed in the middle of FASTA input.

[nsoranzo]

In the middle, but they are allowed between the end of a sequence and the header of the next one, as in the example on the Wikipedia page.

[bernt-matthias]

I'm completely agnostic wrt this, i.e. I don't care (and can't imagine a case where this matters .. but probably there is). I guess if we add such a check (strip + the check) we loose 50% of the improvement.

[bernt-matthias]

How about 24950a7? Surprisingly line[0] == ">" is faster than line.startswith(">") and then it comes handy to check for empty lines first (overall the runtime stays approx the same compared to the previous commit).

[mvdbeek]

Not a fan of these microoptimizations, that seems like something you'd have to evaluate on every python version, given that that seems so optimizable. If this isn't dominating the runtime I wouldn't do it ?

[nsoranzo]

Not a fan of these microoptimizations, that seems like something you'd have to evaluate on every python version, given that that seems so optimizable. If this isn't dominating the runtime I wouldn't do it ?

It does actually make a difference on a 3 Gb FASTA file as it avoids a function call (even on Python 3.12).

[bernt-matthias]

Thanks for checking.

[martenson]

nice, more of these! Thanks @bernt-matthias