feat: add ability to distribute artifacts through archivista

[mikhailswift]

What this PR does / why we need it

Support use case of private archivista deployments where getting witness
may be difficult. Also support use case of folks having custom builds of
witness to distribute. Extended to be a more generic artifact store for
additional tools.

Which issue(s) this PR fixes (optional)

(optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged)*

Fixes #

Acceptance Criteria Met

• Docs changes if needed
• Testing changes if needed
• All workflow checks passing (automatically enforced)
• All review conversations resolved (automatically enforced)
• DCO Sign-off

Special notes for your reviewer:

[jkjell]

Probably not needed for now but, we should consider how to upload/update for new versions of Witness. Depending on where Archivista is running, it could be difficult to access it's storage to update config files or add witness binaries. Also, updating config file will require a file system watcher or a restart of archivista.

[mikhailswift]

Yeah, this is very true on all accounts

[jkjell]

Other than the obvious desire to have the binaries distributed with TUF metadata, I wonder if there could be value of an added a check against the go binary metadata?

For example:

go version -m /usr/local/bin/witness | grep build
	build	-buildmode=exe
	build	-compiler=gc
	build	-trimpath=true
	build	CGO_ENABLED=0
	build	GOARCH=arm64
	build	GOOS=darwin
	build	vcs=git
	build	vcs.revision=aa35c1f4b1d41c87e139c2d333f09117fd0daf4f
	build	vcs.time=2023-08-10T14:25:28Z
	build	vcs.modified=true

[mikhailswift]

Yeah, I think in the future we probably want to expand the checks we can do here

[mikhailswift]

This should be good for final review now. Tests are added and docs are updated.

[kriscoleman]

I like this approach of providing a mechanism for distributing witness in archivista, a dev can provide a config and what versions they want and the distro will make sure it gets the versions, checks the checksums, then expose them through API

[colek42]

I like this! This would also be a great pattern to distribute other artifacts like a tar of the admission controller image, or witness plugins

[jkjell]

As long as @kairoaraujo gives it a 👍 I think we're good to merge.

[kairoaraujo]

As long as @kairoaraujo gives it a 👍 I think we're good to merge.

I had a chat with @mikhailswift, and I will summarize what we discussed

• Make the endpoints /v1/witness more generic (/v1/artifacts) so we can include other artifacts, such as archivistactl

  • Example: /v1/artifacts/witness/v0.2.0 or /v1/artifacts/archivistactl

• Reuse the endpoint /v1/download to download artifacts /v1/download/{any}/{version}/{distribution} where without {version} and {distribution} it is a gitoid

  • Another options are: /v1/download/artifact/name/{distribution}

The changes are to achieve a more generic endpoint to avoid breaking compatibility in the future.

The future

As shared in the future, we can use TUF to protect downloads of anything that goes on Archivista (attestations, policies, and artifacts)
It will allow us to download an artifact and map all attestation and policies.

I will file an issue about all my suggestions to protect the users and enable more features such as updating witness and archivistactl safety.

[jkjell]

Make the endpoints /v1/witness more generic (/v1/artifacts) so we can include other artifacts

We should probably have some bounds on this. ☺️ But, with that said, we have talked about storing "evidence" in Archivista's backend storage as well. That evidence would be reference by hash. Example: a tracing log of a large build (multiple GBs). Then we could asynchronously apply SCAI policy to that evidence, creating additional attestations based on those attributes. However, I don't think we would need TUF to secure that set of evidence.

[kairoaraujo]

We should probably have some bounds on this. ☺️ But, with that said, we have talked about storing "evidence" in Archivista's backend storage as well. That evidence would be reference by hash. Example: a tracing log of a large build (multiple GBs). Then we could asynchronously apply SCAI policy to that evidence, creating additional attestations based on those attributes. However, I don't think we would need TUF to secure that set of evidence.

The idea of having /v1/artifacts
Following your idea we could have for example /v1/"evidencies"
As TUF protects by the path from the base URL, it gives flexibility in what protects or not.

[mikhailswift]

@kairoaraujo i can real quick adjust the download handler to be in line with your comment versus what I have here.

i did /v1/artifacts/{name}/{version}/{distribution}, adding the /artifacts. oops

[mikhailswift]

missing {version} parameter

/v1/download/artifacts/....
or just
/v1/download/.... ?

[kairoaraujo]

I don't have a strong opinion; both work well.

[kairoaraujo]

A minor fix for the API doc, splitting it into specific tags.

It requires make docs after adding.