Name	Name	Last commit message	Last commit date
parent directory ..
README.md	README.md
kustomization.yaml	kustomization.yaml

Deploying Kubeflow on EKS

This guide describes how to deploy vanilla Kubeflow on AWS EKS. This vanilla version has very minimal changes to the upstream Kubeflow manifests. Here are the changes

Kubeflow Notebook configuration has been modified to include some custom container images built on AWS DLC
Usage Tracking

Prerequisites

This guide assumes that you have:

Installed the following tools on the client machine
- AWS CLI - A command line tool for interacting with AWS services.
- eksctl - A command line tool for working with EKS clusters.
- kubectl - A command line tool for working with Kubernetes clusters.
- yq - A command line tool for YAML processing. (For Linux environments, use the wget plain binary installation)
- jq - A command line tool for processing JSON.
- kustomize version 3.2.0 - A command line tool to customize Kubernetes objects through a kustomization file.
  - ⚠️ Kubeflow is not compatible with the latest versions of of kustomize 4.x. This is due to changes in the order resources are sorted and printed. Please see kubernetes-sigs/kustomize#3794 and kubeflow/manifests#1797. We know this is not ideal and are working with the upstream kustomize team to add support for the latest versions of kustomize as soon as we can.

Created an EKS cluster

If you do not have an existing cluster, run the following command to create an EKS cluster. More details about cluster creation via eksctl can be found here.

Substitute values for the CLUSTER_NAME and CLUSTER_REGION in the script below

export CLUSTER_NAME=$CLUSTER_NAME
export CLUSTER_REGION=$CLUSTER_REGION
eksctl create cluster \
--name ${CLUSTER_NAME} \
--version 1.19 \
--region ${CLUSTER_REGION} \
--nodegroup-name linux-nodes \
--node-type m5.xlarge \
--nodes 5 \
--nodes-min 1 \
--nodes-max 10 \
--managed

Clone the awslabs/kubeflow-manifest repo, kubeflow/manifests repo and checkout the release branches.
- Substitute the value for KUBEFLOW_RELEASE_VERSION(e.g. v1.4.1) and AWS_RELEASE_VERSION(e.g. v1.4.1-aws-b1.0.0) with the tag or branch you want to use below. Read more about releases and versioning policy if you are unsure about what these values should be.
```
export KUBEFLOW_RELEASE_VERSION=<>
export AWS_RELEASE_VERSION=<>
git clone https://github.com/awslabs/kubeflow-manifests.git && cd kubeflow-manifests
git checkout ${AWS_RELEASE_VERSION}
git clone --branch ${KUBEFLOW_RELEASE_VERSION} https://github.com/kubeflow/manifests.git upstream
```

Build Manifests and Install Kubeflow

There two options for installing Kubeflow official components and common services with kustomize.

Single-command installation of all components under apps and common
Multi-command, individual components installation for apps and common

Option 1 targets ease of deployment for end users.
Option 2 targets customization and ability to pick and choose individual components.

⚠️ In both options, we use a default email ([email protected]) and password (12341234). For any production Kubeflow deployment, you should change the default password by following the relevant section.

NOTE

kubectl apply commands may fail on the first try. This is inherent in how Kubernetes and kubectl work (e.g., CR must be created after CRD becomes ready). The solution is to simply re-run the command until it succeeds. For the single-line command, we have included a bash one-liner to retry the command.

Install with a single command

You can install all Kubeflow official components (residing under apps) and all common services (residing under common) using the following command:

while ! kustomize build docs/deployment/vanilla | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

Once, everything is installed successfully, you can access the Kubeflow Central Dashboard by logging in to your cluster.

Congratulations! You can now start experimenting and running your end-to-end ML workflows with Kubeflow.

Install individual components

In this section, we will install each Kubeflow official component (under apps) and each common service (under common) separately, using just kubectl and kustomize.

If all the following commands are executed, the result is the same as in the above section of the single command installation. The purpose of this section is to:

Provide a description of each component and insight on how it gets installed.
Enable the user or distribution owner to pick and choose only the components they need.

cert-manager

cert-manager is used by many Kubeflow components to provide certificates for admission webhooks.

Install cert-manager:

kustomize build common/cert-manager/cert-manager/base | kubectl apply -f -
kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -

Istio

Istio is used by many Kubeflow components to secure their traffic, enforce network authorization and implement routing policies.

Install Istio:

kustomize build common/istio-1-9/istio-crds/base | kubectl apply -f -
kustomize build common/istio-1-9/istio-namespace/base | kubectl apply -f -
kustomize build common/istio-1-9/istio-install/base | kubectl apply -f -

Dex

Dex is an OpenID Connect Identity (OIDC) with multiple authentication backends. In this default installation, it includes a static user with email [email protected]. By default, the user's password is 12341234. For any production Kubeflow deployment, you should change the default password by following the relevant section.

Install Dex:

kustomize build common/dex/overlays/istio | kubectl apply -f -

OIDC AuthService

The OIDC AuthService extends your Istio Ingress-Gateway capabilities, to be able to function as an OIDC client:

kustomize build common/oidc-authservice/base | kubectl apply -f -

Knative

Knative is used by the KFServing official Kubeflow component.

Install Knative Serving:

kustomize build common/knative/knative-serving/base | kubectl apply -f -
kustomize build common/istio-1-9/cluster-local-gateway/base | kubectl apply -f -

Optionally, you can install Knative Eventing which can be used for inference request logging:

kustomize build common/knative/knative-eventing/base | kubectl apply -f -

Kubeflow Namespace

Create the namespace where the Kubeflow components will live in. This namespace is named kubeflow.

Install kubeflow namespace:

kustomize build common/kubeflow-namespace/base | kubectl apply -f -

Kubeflow Roles

Create the Kubeflow ClusterRoles, kubeflow-view, kubeflow-edit and kubeflow-admin. Kubeflow components aggregate permissions to these ClusterRoles.

Install kubeflow roles:

kustomize build common/kubeflow-roles/base | kubectl apply -f -

Kubeflow Istio Resources

Create the Istio resources needed by Kubeflow. This kustomization currently creates an Istio Gateway named kubeflow-gateway, in namespace kubeflow. If you want to install with your own Istio, then you need this kustomization as well.

Install istio resources:

kustomize build common/istio-1-9/kubeflow-istio-resources/base | kubectl apply -f -

Kubeflow Pipelines

Install the Multi-User Kubeflow Pipelines official Kubeflow component:

kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user | kubectl apply -f -

KFServing

Install the KFServing official Kubeflow component:

kustomize build apps/kfserving/upstream/overlays/kubeflow | kubectl apply -f -

Katib

Install the Katib official Kubeflow component:

kustomize build apps/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -

Central Dashboard

Install the Central Dashboard official Kubeflow component:

kustomize build apps/centraldashboard/upstream/overlays/istio | kubectl apply -f -

Admission Webhook

Install the Admission Webhook for PodDefaults:

kustomize build apps/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -

Notebooks

Install the Notebook Controller official Kubeflow component:

kustomize build apps/jupyter/notebook-controller/upstream/overlays/kubeflow | kubectl apply -f -

Install the Jupyter Web App official Kubeflow component:

kustomize build apps/jupyter/jupyter-web-app/upstream/overlays/istio | kubectl apply -f -

Profiles + KFAM

Install the Profile Controller and the Kubeflow Access-Management (KFAM) official Kubeflow components:

kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -

Volumes Web App

Install the Volumes Web App official Kubeflow component:

kustomize build apps/volumes-web-app/upstream/overlays/istio | kubectl apply -f -

Tensorboard

Install the Tensorboards Web App official Kubeflow component:

kustomize build apps/tensorboard/tensorboards-web-app/upstream/overlays/istio | kubectl apply -f -

Install the Tensorboard Controller official Kubeflow component:

kustomize build apps/tensorboard/tensorboard-controller/upstream/overlays/kubeflow | kubectl apply -f -

Training Operator

Install the Training Operator official Kubeflow component:

kustomize build apps/training-operator/upstream/overlays/kubeflow | kubectl apply -f -

MPI Operator

Install the MPI Operator official Kubeflow component:

kustomize build apps/mpi-job/upstream/overlays/kubeflow | kubectl apply -f -

AWS Telemetry

Install the AWS Kubeflow telemetry component. This is an optional component. See the usage tracking documentation for more information

kustomize build awsconfigs/common/aws-telemetry | kubectl apply -f -

User Namespace

Finally, create a new namespace for the the default user (named kubeflow-user-example-com).

kustomize build common/user-namespace/base | kubectl apply -f -

Connect to your Kubeflow Cluster

After installation, it will take some time for all Pods to become ready. Make sure all Pods are ready before trying to connect, otherwise you might get unexpected errors. To check that all Kubeflow-related Pods are ready, use the following commands:

kubectl get pods -n cert-manager
kubectl get pods -n istio-system
kubectl get pods -n auth
kubectl get pods -n knative-eventing
kubectl get pods -n knative-serving
kubectl get pods -n kubeflow
kubectl get pods -n kubeflow-user-example-com

Port-Forward

To get started quickly you can access Kubeflow via port-forward. Run the following to port-forward Istio's Ingress-Gateway to local port 8080:

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

After running the command, you can access the Kubeflow Central Dashboard by doing the following:

Open your browser and visit http://localhost:8080. You should get the Dex login screen.
Login with the default user's credential. The default email address is [email protected] and the default password is 12341234.

Exposing Kubeflow over Load Balancer

In order to expose Kubeflow over an external address you can setup AWS Application Load Balancer. Please take a look at the load-balancer guide to set it up.

Change default user password

For security reasons, we don't want to use the default password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password before deploying. To define a password for the default user:

Pick a password for the default user, with email [email protected], and hash it using bcrypt:

python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))'

Edit upstream/common/dex/base/config-map.yaml and fill the relevant field with the hash of the password you chose:
```
...
  staticPasswords:
  - email: [email protected]
    hash: <enter the generated hash here>
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vanilla

vanilla

README.md

Deploying Kubeflow on EKS

Prerequisites

Build Manifests and Install Kubeflow

Install with a single command

Install individual components

cert-manager

Istio

Dex

OIDC AuthService

Knative

Kubeflow Namespace

Kubeflow Roles

Kubeflow Istio Resources

Kubeflow Pipelines

KFServing

Katib

Central Dashboard

Admission Webhook

Notebooks

Profiles + KFAM

Volumes Web App

Tensorboard

Training Operator

MPI Operator

AWS Telemetry

User Namespace

Connect to your Kubeflow Cluster

Port-Forward

Exposing Kubeflow over Load Balancer

Change default user password

Files

vanilla

Directory actions

More options

Directory actions

More options

Latest commit

History

vanilla

Folders and files

parent directory

README.md

Deploying Kubeflow on EKS

Prerequisites

Build Manifests and Install Kubeflow

Install with a single command

Install individual components

cert-manager

Istio

Dex

OIDC AuthService

Knative

Kubeflow Namespace

Kubeflow Roles

Kubeflow Istio Resources

Kubeflow Pipelines

KFServing

Katib

Central Dashboard

Admission Webhook

Notebooks

Profiles + KFAM

Volumes Web App

Tensorboard

Training Operator

MPI Operator

AWS Telemetry

User Namespace

Connect to your Kubeflow Cluster

Port-Forward

Exposing Kubeflow over Load Balancer

Change default user password