feat: allow importing complete datasets #783
Conversation
|
@carlosthe19916 It might make sense to provide a UI for that too. So that a user interested in testing the system can be instructed to just upload |
ctron
force-pushed
the
feature/dataset_import_1
branch
4 times, most recently
from
7f984aa
to
2baecaf
Compare
ctron
force-pushed
the
feature/dataset_import_1
branch
2 times, most recently
from
13d2a0a
to
5bc3ced
Compare
| @@ -302,7 +302,7 @@ impl InitData { | |||
| fn configure( | |||
| svc: &mut web::ServiceConfig, | |||
| db: db::Database, | |||
| storage: impl Into<DispatchBackend>, | |||
| storage: impl Into<DispatchBackend> + Clone, | |||
There was a problem hiding this comment.
oh interesting, I did not know one could do this
| let stream = self | ||
| .storage | ||
| .retrieve(result.key()) | ||
| .await | ||
| .map_err(Error::Storage)? | ||
| .ok_or_else(|| Error::Storage(anyhow!("file went missing during upload")))?; | ||
|
|
There was a problem hiding this comment.
We were intentionally retrieving the doc after storing it to ensure that we could. If we don't find out now, we'll never know. Are we sure we want to eliminate this failsafe?
There was a problem hiding this comment.
The reason for retrieving it was actually because the stream was consumed at that point. From my side, it was never intended as a check.
Since none of the internal APIs makes actual use of that stream, it didn't seem reasonable to keep that pattern. We anyway load the full document in memory. So why store megabytes of SBOMs, and then load them again? If the don't trust S3 when it says "ok, I stored it" I think we have a different problem.
This adds an endpoint for ingesting an archive of documents. The idea is to have a a simpler version of ingesting "datasets", like the "dataset 1", that we had from Trustification.
This also adds a "dataset 3", which consists of "dataset 1", but adding the matching CVE files, so that all information in the UI is present.
The original idea was to ingest this using the existing "detect format" code path. However, it doesn't fix that pattern:
IngestResult, and doesn't have IDs or digestsIngestResult, one for each file in the archive. That information should be reported back.That's why this PR adds a new endpoint, for this specific use case. In the future, if we have more endpoints with a similar pattern, which make this more generic.