Question: Best practice for storage and cleanup in a Tekton CI pipeline

[djfinnoy]

I've patched together a pipeline using tasks from the catalog, and I'm noticing some areas where I'm probably not "doing it right".

The pipeline gets triggered via Github, and it simply clones the triggering repo, builds and pushes its dockerimage, and runs Helm to update an ArgoCD application resource.

The idea is that we'll have dozens of github repositories that call on this pipeline to deploy themselves into k8s.
Right now, I'm using a volumeClaimTemplate in the PipelineRun, and im noticing that this leaves a bit of clutter in the cluster;
Each PipelineRun leaves a PersistentVolumeClaim, after it completes; that's bad. Would it make more sense to have a permanent pvc, and if so, how to I ensure that it doesn't run out of space?

I'm also noticing that the PipelineRuns and their associated resources stick around indefinitely after completion; this is handy with respect to the Tekton Dashboard, but I'm thinking I'll need some sort of cleanup job as well.

Can anyoine point me toward resources / writeups on how to deal with storage and cleaning when using Tekton for CI?

[ghost]

/kind question

Thanks for the issue! I'm going to convert this over to a discussion thread so that it doesn't get chewed up by our issue automation.

Each PipelineRun leaves a PersistentVolumeClaim, after it completes; that's bad. Would it make more sense to have a permanent pvc, and if so, how to I ensure that it doesn't run out of space?

Worth noting: the PVCs created from a volumeClaimTemplate will be cleaned up when the PipelineRun is deleted. As you mentioned the runs get kept around because their logs and other info is useful for the dashboard. The Tekton Results project (https://github.com/tektoncd/results) has been started with the goal of moving some of that historical info into a database so that runs + their dependencies can be safely deleted without losing a bunch of data.

Using a permanent PVC would be another alternative way to go but Pipelines doesn't help out much with that approach - storage management is largely left up to the user because requirements around persistentce can vary dramatically from one org to another.

A totally different approach would be to "co-locate" the Tasks that share storage into a single Pod so that they can use an emptyDir to shuttle data around quickly. We're looking at that through a couple different ongoing designs but the one most closely related is tektoncd/community#318

I'm also noticing that the PipelineRuns and their associated resources stick around indefinitely after completion; this is handy with respect to the Tekton Dashboard, but I'm thinking I'll need some sort of cleanup job as well.

The Tekton Results project is designing a cleanup feature. In a nutshell: It'll observe your runs and consider them candidates for deletion after they complete (+ a grace period). The design doc is here: https://github.com/tektoncd/community/blob/main/teps/0052-tekton-results-automated-run-resource-cleanup.md

[gdisanto]

Hi @sbwsg,
Sorry if I'm replying to this old thread but I think that the information I need is related.
I basically have the same problem described by the author of the topic. I know that you have replied mentioning the Tekton results project but it looks like it is still not mature enough?.
For the moment I don't mind leaving the pipelineruns and the pods, but I would like to delete the PVC at least... but it seems that it is impossible unless the linked pod is deleted as well... which I do not want to delete to preserve the logs.
Is there a way to delete the pvc and keep the pods for the pipelineruns that have been already executed?.

[ghost]

Once k8s has tied a pvc to a pod i don't think there's a way to untie them without deleting the pod or really hacking at the objects: A quick search leads me to believe it might be possible by nulling out the finalizer on the PV/C, then using the --force flag when deleting them. Caveat: I haven't tried this & don't know the consequences for the volume or the pod. Recommend being cautious here.

An alternative would be Tekton Results or, if that isn't suitable for your use case right now, you could write your own CronJob to pull the logs from completed taskruns, persist them somewhere (e.g. cloud storage) and then delete the runs.

[gdisanto]

Thanks a lot @sbwsg, I'll try to follow your suggestions.