feat: use tekton pipeline remote resolution

[JordanGoasdoue]

The goal is to create our own Jx3-Git Resolver from Tekton Pipeline Remote Resolution to be able to use Remote Pipeline / Task from Git inside our PipelineRuns with the image: uses:sourceURI inheritance working. All this logic could then be removed from the WebHook/ChatOps Lighthouse that could be more focused on Webhook / ChatOps things

What do you think ? @msvticket @ankitm123 @tomhobson @babadofar @rajatgupta24

[keskad]

Lighthouse is definitely a big blob and splitting it could be a good direction.

By the way regarding splitting up the Lighthouse I finally started a PoC to create a SCM feedback (ChatOps) integration, based on Jenkins X's go-scm.

Having a plan to stablize the core API, freeze it, document well, so it could be used as a library to build integrations with Tekton, Argo Workflows etc.
I'm sure it will be soon available for Tekton. It is so universal it could be adapted also to Argo Workflows and Jenkins X if somebody would like to create a controller basing on that library.

[msvticket]

It does sound like a good proposal. Do I understand you correctly that you think it is possible to make this new jx3-git resolver so that t will work as a drop in replacement (with new features) to the implementation in lighthouse? Without any changes to the pipelines or the pipeline catalog I mean.

Could you move the markdown file in the proposal so that the file structure matches the one for the existing proposals?

[keskad]

To check: if the Tekton Controller validation webhook would allow to submit image: uses:sourceURI

[JordanGoasdoue]

@msvticket, done it's moved to proposals.

The use case I had in mind when writing this proposal was the remote pipeline
Now, I'm thinking about just a PipelineRun, without any remote ref, we would still need to be able to use a jx3-git resolver in order to replace the image: uses:sourceURI with real steps fetched remotely.
But I'm not sure it's possible to add a resolver without specifying a taskRef / pipelineRef, which has no sense if we are just using a PipelineRun as is, with a bunch of image: uses:sourceURI

If all of this is possible, we would still need to add resolver: jx3-git everywhere on the jx3-catalog

So also to check after what proposed @keskad , is the possibility to run PipelineRun with image: uses:sourceURI with a resolver, without remote pipelineRef / taskRef.

I will ask in the tekton slack

---

If it's not possible, then we can discuss with Tekton and see the best approach to just make this work officially from Tekton Controller.

[keskad]

I was thinking about it, regarding the possibilities, I see now that:

• Lighthouse would still need to search for PipelineRun in a repository, when a webhook is handled. So this mechanism would stay as it is
• Only the difference is the remote resolution in that discovered PipelineRun - remote Pipelines catalog could be possibly resolved with a remote resolver

Is this a correct scope?

[JordanGoasdoue]

Yes exactly Lighthouse would just search PipelineRun through the TriggerConfig, and then just apply it without resolving anything.

Then the PipelineRun would be resolved by the jx3-git resolver, even if it's a single PipelineRun or with remote ref to Pipeline / Task

But if it's not possible to have on a PipelineRun a resolver without specifying pipelineRef / taskRef, we would be stuck

[tomhobson]

jenkins-x/lighthouse#1539

I think that removing all of the uses: spec is required for 0.41.0, as it creates a lot of problems as the way that jenkins-x does uses, does not pass the validation that tekton has now added.

Instead of adding a bunch of fluff to pass the validation, I figured let's just go ahead with this.

We're going to build a migrator cli or something similar within jx-pipeline so that we can convert uses to tektons TaskRefs. But we'll have to move away from steps to separated individual tasks.

I believe that this drops a lot of maintaince from jenkins x and means that we can utilise all of the upstream features from Tekton without massive modification of jenkins x in future

[JordanGoasdoue]

Yes this is too much of a headache to maintain this tekton fork, difficult to maintain, the best is to design our Pipelines another way in order to use all the tekton features without trouble.

the atomicity would be on task and not step, this is not that bad, maybe some copy paste between tasks but it's fine.
Thanks for looking at all of this @tomhobson

[msvticket]

But tasks are executed as separate pods so using a shared workspace becomes more complicated, right? But as long as the migration tool you are working on Tom handles this I suppose this is fine.

[tomhobson]

@msvticket, there is a shared workspace still, it's just passed around as a PVC. They've got some affinity magic to make it quick to mount

[msvticket]

@tomhobson @JordanGoasdoue Is this still the route you want to continue on?