test: speed up unit tests by using smaller datasets

[fengelniederhammer]

part of #560

Summary

Local results (debug build):

• database.test.cpp: Use a smaller dataset. 7s -> 800ms
• preprocessor.test.cpp: The RSV dataset took 45s -> shift it to the new e2e tests and delete here
• preprocessor.test.cpp: Use smaller datasets for some tests: 1.8s -> ~500ms

On CI, time of the unit tests:

• last run on main: 16s
• first build on this branch: 2s

PR Checklist

- [ ] All necessary documentation has been adapted or there is an issue to do so.

• The implemented feature is covered by an appropriate test.

[github-actions]

This is a preview of the changelog of the next release:

0.2.18 (2024-09-11)

Features

• do not abort on assertment failures (4e5f598)
• panic: add ASSERT, UNREACHABLE, UNIMPLEMENTED, safer env var (867d08f)

[Taepper]

I added GenSpectrum/LAPIS-SILO-e2e#34 to not drop our covered input data

[fengelniederhammer]

I added GenSpectrum/LAPIS-SILO-e2e#34 to not drop our covered input data

IMO the dataset is not of immediate importance. In these tests here, it was just some dataset. We already have some other dataset in other tests.
In the longterm of course still good to have.

[Taepper]

Exactly. I just wanted to have the issue such that we remember adding it at some point. Not blocking in any way :)

[pflanze]

Cool. I haven't looked into all of the details, but looks good to me.

(It's good to have a test suite that covers most of the potential issues when coding, and then have other more expensive tests/test suites that give more certainty for a higher cost. Even in unit tests, could have expensive ones if there are 2 classes of tests, important-but-quick and more-extensive-but-expensive ones.)

[Taepper]

We could also learn from other codebases. e.g. duckdb marks suffixes some test-files with .test_slow:
https://github.com/duckdb/duckdb/tree/main/test/sql/copy

[Taepper]

And then their test runner can execute them based on the arguments given by the developer running the tests