Skip to content

Commit

Permalink
Adds example for tuning a kfp v1 pipeline with Katib
Browse files Browse the repository at this point in the history
This example illustrates how a full kfp pipeline can
be tuned using Katib.

It is based on a metrics collector to collect kubeflow
pipeline metrics (kubeflow#2019). This is used as a Custom Collector.

Addresses: kubeflow#1914, kubeflow#2019
  • Loading branch information
votti committed Feb 15, 2023
1 parent 03b8795 commit cae257d
Show file tree
Hide file tree
Showing 2 changed files with 1,095 additions and 3 deletions.
14 changes: 11 additions & 3 deletions examples/v1beta1/kubeflow-pipelines/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
The following examples show how to use Katib with
[Kubeflow Pipelines](https://github.com/kubeflow/pipelines).

Two different aspects are illustrated here:
A) How to orchestrate Katib experiments from Kubeflow pipelines using the Katib Kubeflow Component (Example 1 & 2)
B) How to use Katib to tune parameters of Kubeflow pipelines

You can find the Katib Component source code for the Kubeflow Pipelines
[here](https://github.com/kubeflow/pipelines/tree/master/components/kubeflow/katib-launcher).

Expand All @@ -13,6 +17,8 @@ You have to install the following Python SDK to run these examples:
- [`kfp`](https://pypi.org/project/kfp/) >= 1.8.12
- [`kubeflow-katib`](https://pypi.org/project/kubeflow-katib/) >= 0.13.0

In order to run parameter tuning over Kubeflow pipelines, additionally Katib needs to be setup to run with Argo workflow tasks. The setup is described within the example notebook (3).

## Multi-User Pipelines Setup

The Notebooks examples run Pipelines in multi-user mode and your Kubeflow Notebook
Expand All @@ -25,10 +31,12 @@ to give an access Kubeflow Notebook to run Kubeflow Pipelines.

The following Pipelines are deployed from Kubeflow Notebook:

- [Kubeflow E2E MNIST](kubeflow-e2e-mnist.ipynb)
1) [Kubeflow E2E MNIST](kubeflow-e2e-mnist.ipynb)

2) [Katib Experiment with Early Stopping](early-stopping.ipynb)

- [Katib Experiment with Early Stopping](early-stopping.ipynb)
3) [Tune parameters of a `MNIST` kubeflow pipeline with Katib](pipeline-parameters.ipynb)

The following Pipelines have to be compiled and uploaded to the Kubeflow Pipelines UI:
The following Pipelines have to be compiled and uploaded to the Kubeflow Pipelines UI for examples 1 & 2:

- [MPIJob Horovod](mpi-job-horovod.py)
Loading

0 comments on commit cae257d

Please sign in to comment.