Allow loading a "dataset"

[ctron]

Discussed in #772

^{Originally posted by ctron September 9, 2024}
In v1 we had the idea of a "dataset", a curated set of files that get ingested into the system in order to have something available for the user to play with. Ingesting the dataset should:

• be quick
• easy
• lead to the same state of the system for every import (given the same code/commit/version is being used)

In v1, this was orchestrated through Helm charts, and the process wasn't pretty. In v2, we don't really have that, although we could use the same Helm chart approach.

However, I think we could do better. What about having a .zip file structure (yes, I come from the world or .jar, .war, .ear, …) which includes all the files, and which can be uploaded via the Rest API with our generic "upload anything" endpoint? That should also make it possible to upload this via the UI.

In the background, the "loader" demuxes this archive into the different actual loaders.

Given the system was empty before, it should lead to the same state once the upload is complete.

[ctron]

Closed by #783