This pipeline example demonstrates that within a pipeline, you can call another pipeline during execution, also known as nested pipeline. This example is originated from the kubeflow pipeline's compose.py example that is running with the KFP API. We have modified this example to run with the Tekton compiler and submit the pipeline directly to the Tekton engine.
This example contains two pipelines, save-most-frequent and download-and-save-most-frequent. save-most-frequent will take two inputs, one is a message string which you want to find the most frequent word in it, the other is the location/path you want to save in GCS, if you don't have GCS setup, this parameter will be ignored, and the result will not be saved. Both input fields are required. download-and-save-most-frequent will take two inputs, one is an URL with plain text, the second one is the location/path which you want to save the result in GCS. This pipeline will first download the text from the input URL, then call save-most-frequent pipeline. Both inputs are required.
- Install KFP Tekton prerequisites
- Once you have completed all the prerequisites for this example, you can first try out the
save-most-frequentpipeline
dsl-compile-tekton --py compose.py --output pipeline.yaml --function save_most_frequent_word
Next, upload the pipeline.yaml file to the Kubeflow pipeline dashboard to run this pipeline.
- To tryout the nested pipeline
download-and-save-most-frequent:
dsl-compile-tekton --py compose.py --output pipeline.yaml --function download_save_most_frequent_word
Next, upload the pipeline.yaml file to the Kubeflow pipeline dashboard with Tekton Backend to run this pipeline.
The original nested pipeline from kubeflow/pipeline project.