README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLOps with Seldon and Jenkins Classic\n",
    "\n",
    "This repository shows how you can build a Jenkins Classic pipeline to enable Continuous Integration and Continuous Delivery (CI/CD) on your Machine Learning models leveraging Seldon for deployment.\n",
    "This CI/CD pipeline will allow you to:\n",
    "\n",
    "- Run unit tests using Jenkins Classic.\n",
    "- Run end-to-end tests for your model with KIND (Kubernetes in Docker).\n",
    "- Promote your model as a across multiple (staging / prod) environments.\n",
    "\n",
    "To showcase these features we will implement add continuous integration and delivery to three different models. \n",
    "You can find these under the `/models` folder.\n",
    "As we shall see, each of them will require a [different approach to deployment](#Use-Cases)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CI/CD Pipeline\n",
    "\n",
    "The diagram below provides a high level overview of the CI/CD pipeline.\n",
    "It includes an overview of all the different types of repositories, together with the stakeholders that are the primary contributors of each, as well as the Kubernetes environments in which the applications are deployed.\n",
    "\n",
    "The key pieces to note on the diagram are:\n",
    "\n",
    "- There are different types of environments with different restrictions and behaviours, e.g. staging and production.\n",
    "- It’s possible to have more than one environment for each type (as the type is just what would give it a specific type of config/behaviour).\n",
    "- The environments are by default in the same cluster (as namespaces), however it’s also possible to configure them across different clusters.\n",
    "- Each of the green boxes is a single repository, but it can also have a mono-repo approach, whereby each of the white boxes is a folder within a repo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![CI/CD Pipeline](./images/pipeline-architecture.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model implementation repository\n",
    "\n",
    "From a high-level point of view, when a model implementation repository is updated by a Data Scientist or ML Engineer, the Jenkins CI will push changes to the [GitOps repository](#gitops-repository). This enables the following workflow:\n",
    "\n",
    "1. A Data Scientist or ML Engineer trains a new model.\n",
    "2. The Data Scientist or ML Engineer pushes the updated configuration to the model implementation repository.\n",
    "3. The CI tool automatically builds and tests the model implementation.\n",
    "4. The CI tool automatically pushes the change into the GitOps staging repository.\n",
    "5. The CI tool automatically opens a PR into the GitOps production repository.\n",
    "\n",
    "One key point to highlight which may not be obvious by just looking at the diagram is that in this phase of model implementation, the example above showcases how we can leverage a re-usable model server - that is, reusing a pre-built docker image instead of building one every time.\n",
    "If there are more custom requirements, the user is in full control of the steps performed by the CI Platform Jenkins.\n",
    "This means that it is also possible to build s2i wrapped components which may require training the image every time.\n",
    "\n",
    "To gain a better understanding of how the CI/CD pipeline is implemented on each model implementation repository you can check the documented [deep dive](./docs/deep-dive.md).\n",
    "\n",
    "#### Why a new repo for every model?\n",
    "\n",
    "A new model implementation repo is currently created because it provides us with a way to separate the “Model Deployment” phase and the “Model Training/Experimentation” phase, and allows us to use the repo as the integration between any frameworks that can serve as sources of models (MLFlow, Kubeflow, Spark, etc).\n",
    "The repo is able to store any metadata, IDs, and configuration files required, and is processed through the CI pipeline every time it is modified. \n",
    "\n",
    "#### Building a docker image in model implementation repository\n",
    "\n",
    "Whilst most of the times users of this approach will be leveraging re-usable model servers such as the SKLearn model server, it is also possible to build a docker image every single time (i.e. build a non-reusable model every time a model changes).\n",
    "This can be be done by adding the relevant steps which would most often include the s2i utility.\n",
    "This may be desired if there are non-standard linux libraries or non-standard depdencies that need to be re-installed every time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GitOps repository\n",
    "\n",
    "The state of each of our environments (e.g. production or staging) is stored on a GitOps repository.\n",
    "This repository contains all the different Kubernetes resources that have been deployed to each cluster.\n",
    "It is linked through [ArgoCD](#ArgoCD) to each of our Kubernetes clusters (or namespaces) so that a change in the repository triggers an update of our environment.\n",
    "\n",
    "When the deployment configuration of a machine learning model implementation is updated, this will automatically make the changes available through a PR to the respective manager/tech-lead/approver.\n",
    "This step will enable the end to end machine learning model promotion to be reviewed and approved by the respective individual.\n",
    "\n",
    "The manager/tech-lead will have to approve the PR before it can be merged.\n",
    "Once it’s approved, it will be merged into the GitOps repo, which will immediately trigger the update in the production namespace/cluster.\n",
    "\n",
    "You can see an example of a GitOps repository in the [SeldonIO/seldon-gitops](https://github.com/SeldonIO/seldon-gitops) repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-usable model server repository\n",
    "\n",
    "If there is a need for a new reusable model server, then it’s possible to do so by creating a repository which would follow a different path.\n",
    "This would be different to the model implementation repository as it would only be built once in a while, whilst the model server would be built multiple times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up\n",
    "\n",
    "As a pre-requisite you need to ensure that have access to a Kubernetes cluster.\n",
    "In particular, this guide requires the following pre-requisites:\n",
    "\n",
    "- A Kubernetes cluster running v1.13+.\n",
    "- Jenkins Classic installed in your cluster. You can find instructions on how to install and configure it on the [Installing Jenkins on your K8s cluster](#Installing-Jenkins-on-your-K8s-cluster) section.\n",
    "- Seldon Core v0.5.1 installed in your cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true,
    "toc-nb-collapsed": true
   },
   "source": [
    "## Use cases\n",
    "\n",
    "This guide goes through three different methods to build and deploy your model.\n",
    "Each of these can be found under the `./models/` of this repository.\n",
    "\n",
    "- Using Seldon pre-built re-usable model servers (`./models/news_classifier`). \n",
    "- Using custom re-usable servers (`./models/images_classifier`).\n",
    "- Using custom servers with an embedded model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diving into our CI/CD Pipeline\n",
    "\n",
    "On this section we will dive into the internals of the CI/CD pipeline for our [model implementation repositories](README.md#model-implementation-repository).\n",
    "This includes a detailed description of the `Jenkinsfile`, as well as a look into our suggested testing methodology.\n",
    "\n",
    "Note that this will cover a generic example.\n",
    "However, as we shall see, specialising this approach into any of our [three main use cases](README.md#use-cases) will be straightforward.\n",
    "\n",
    "We leverage [Jenkins Pipelines](https://jenkins.io/doc/book/pipeline/) in order to run our continous integration and delivery automation.\n",
    "From a high-level point of view, the pipeline configuration will be responsible for:\n",
    "\n",
    "- Define a **replicable** test and build environment.\n",
    "- Run the unit and integration tests (if applicable).\n",
    "- Promote the application into our staging and production environments.\n",
    "  \n",
    "We can see a `Jenkinsfile` below taken from the [`news_classifier`](./models/news_classifier) example.\n",
    "This `Jenkinsfile` defines a pipeline which takes into account all of the points mentioned above.\n",
    "The following sections will dive into each of the sections in a much higher detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./models/news_classifier/Jenkinsfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./models/news_classifier/Jenkinsfile\n",
    "pipeline {\n",
    "  agent {\n",
    "    kubernetes {\n",
    "      defaultContainer 'core-builder'\n",
    "      yamlFile 'models/news_classifier/podTemplate.yaml'\n",
    "    }\n",
    "  }\n",
    "\n",
    "  stages {\n",
    "    stage('Test') {\n",
    "      steps {\n",
    "        sh '''\n",
    "          cd models/news_classifier\n",
    "          make install_dev test\n",
    "        '''\n",
    "      }\n",
    "    }\n",
    "\n",
    "    stage('Test integration') {\n",
    "      steps {\n",
    "        sh '''\n",
    "          cd models/news_classifier\n",
    "          ./integration/kind_test_all.sh\n",
    "        '''\n",
    "      }\n",
    "    }\n",
    "\n",
    "    stage('Promote application') {\n",
    "      steps {\n",
    "        withCredentials([[$class: 'UsernamePasswordMultiBinding',\n",
    "              credentialsId: 'github-access',\n",
    "              usernameVariable: 'GIT_USERNAME',\n",
    "              passwordVariable: 'GIT_PASSWORD']]) {\n",
    "          sh '''\n",
    "            cd models/news_classifier\n",
    "            ./promote_application.sh\n",
    "          '''\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "\n",
    "  }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./models/news_classifier/podTemplate.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./models/news_classifier/podTemplate.yaml\n",
    "apiVersion: v1\n",
    "kind: Pod\n",
    "metadata:\n",
    "  name: test-pod\n",
    "spec:\n",
    "  containers:\n",
    "  - name: core-builder\n",
    "    image: seldonio/core-builder:0.8\n",
    "    resources:\n",
    "      limits:\n",
    "        cpu: 500m\n",
    "        memory: 1500Mi\n",
    "        ephemeral-storage: \"15Gi\"\n",
    "      requests:\n",
    "        cpu: 200m\n",
    "        memory: 1500Mi\n",
    "        ephemeral-storage: \"15Gi\"\n",
    "    securityContext:\n",
    "      privileged: true\n",
    "    tty: true\n",
    "    volumeMounts:\n",
    "      - mountPath: /lib/modules\n",
    "        name: modules\n",
    "        readOnly: true\n",
    "      - mountPath: /sys/fs/cgroup\n",
    "        name: cgroup\n",
    "      - mountPath: /var/lib/docker\n",
    "        name: dind-storage\n",
    "  volumes:\n",
    "  - name: modules\n",
    "    hostPath:\n",
    "      path: /lib/modules\n",
    "  - name: cgroup\n",
    "    hostPath:\n",
    "      path: /sys/fs/cgroup\n",
    "  - name: dind-storage\n",
    "    emptyDir: {}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replicable test and build environment\n",
    "\n",
    "In order to ensure that our test environments are versioned and replicable, we make use of the [Jenkins Kubernetes plugin](https://github.com/jenkinsci/kubernetes-plugin).\n",
    "This will allow us to create a Docker image with all the necessary tools for testing and building our models.\n",
    "Using this image, we will then spin up a separate pod, where all our build instructions will be ran.\n",
    "We will use the `podTemplate()` object in the Jenkins Pipeline configuration to define the requirements of this pod\n",
    "\n",
    "Since it leverages Kubernetes underneath, this also ensure that our CI/CD pipelines are easily scalable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integration tests\n",
    "\n",
    "Now that we have a model that we want to be able to deploy, we want to make sure that we run end-to-end tests on that model to make sure everything works as expected.\n",
    "For this we will leverage the same framework that the Kubernetes team uses to test Kubernetes itself: [KIND](https://kind.sigs.k8s.io/).\n",
    "\n",
    "KIND stands for Kubernetes-in-Docker, and is used to isolate a Kubernetes environent for end-to-end tests.\n",
    "In our case, we will use this isolated environment to test our model.\n",
    "\n",
    "The steps we'll have to carry out include:\n",
    "\n",
    "1. Enable Docker within your CI/CD pod.\n",
    "2. Add an integration test stage.\n",
    "3. Leverage the `kind_test_all.sh` script that creates a KIND cluster and runs the tests.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add integration stage to Jenkins\n",
    "\n",
    "We can leverage Jenkins Pipelines to manage the different stages of our CI/CD pipeline.\n",
    "In particular, to add an integration stage, we can use the `stage()` object:\n",
    "\n",
    "```groovy\n",
    "    stage('Test integration') {\n",
    "      steps {\n",
    "        sh '''\n",
    "          cd models/news_classifier\n",
    "          ./integration/kind_test_all.sh\n",
    "        '''\n",
    "      }\n",
    "    }\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enable Docker\n",
    "\n",
    "To test our models, we will need to build their respective containers, for which we will need Docker."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to do so, we will first need to mount a few volumes into the CI/CD container.\n",
    "These basically consist of the core components that docker will need to be able to run.\n",
    "To mount them we will add these entries into the `podTemplate.yaml` file.\n",
    "\n",
    "Please also note that we set container to run in `privileged` mode.\n",
    "\n",
    "\n",
    "```yaml\n",
    "ApiVersion: v1\n",
    "...\n",
    "spec:\n",
    "  containers:\n",
    "  - name: core-builder\n",
    "    ...\n",
    "    securityContext:\n",
    "      privileged: true\n",
    "    ...\n",
    "    volumeMounts:\n",
    "      - mountPath: /lib/modules\n",
    "        name: modules\n",
    "        readOnly: true\n",
    "      - mountPath: /sys/fs/cgroup\n",
    "        name: cgroup\n",
    "      - mountPath: /var/lib/docker\n",
    "        name: dind-storage\n",
    "  volumes:\n",
    "  - name: modules\n",
    "    hostPath:\n",
    "      path: /lib/modules\n",
    "  - name: cgroup\n",
    "    hostPath:\n",
    "      path: /sys/fs/cgroup\n",
    "  - name: dind-storage\n",
    "    emptyDir: {}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run tests in Kind \n",
    "\n",
    "The `kind_run_all.sh` may seem complicated at first, but it's actually quite simple. \n",
    "All the script does is set-up a kind cluster with all dependencies, deploy the model and clean everything up.\n",
    "Let's break down each of the components within the script."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first start the docker daemon and wait until Docker is running (using `docker ps q` for guidance.\n",
    "\n",
    "```bash\n",
    "# FIRST WE START THE DOCKER DAEMON\n",
    "service docker start\n",
    "# the service can be started but the docker socket not ready, wait for ready\n",
    "WAIT_N=0\n",
    "while true; do\n",
    "    # docker ps -q should only work if the daemon is ready\n",
    "    docker ps -q > /dev/null 2>&1 && break\n",
    "    if [[ ${WAIT_N} -lt 5 ]]; then\n",
    "        WAIT_N=$((WAIT_N+1))\n",
    "        echo \"[SETUP] Waiting for Docker to be ready, sleeping for ${WAIT_N} seconds ...\"\n",
    "        sleep ${WAIT_N}\n",
    "    else\n",
    "        echo \"[SETUP] Reached maximum attempts, not waiting any longer ...\"\n",
    "        break\n",
    "    fi\n",
    "done\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we're running a docker daemon, we can run the command to create our KIND cluster, and install all the components.\n",
    "This will set up a Kubernetes cluster using the docker daemon (using containers as Nodes), and then install Ambassador + Seldon Core.\n",
    "\n",
    "\n",
    "```bash\n",
    "#######################################\n",
    "# AVOID EXIT ON ERROR FOR FOLLOWING CMDS\n",
    "set +o errexit\n",
    "\n",
    "# START CLUSTER \n",
    "make kind_create_cluster\n",
    "KIND_EXIT_VALUE=$?\n",
    "\n",
    "# Ensure we reach the kubeconfig path\n",
    "export KUBECONFIG=$(kind get kubeconfig-path)\n",
    "\n",
    "# ONLY RUN THE FOLLOWING IF SUCCESS\n",
    "if [[ ${KIND_EXIT_VALUE} -eq 0 ]]; then\n",
    "    # KIND CLUSTER SETUP\n",
    "    make kind_setup\n",
    "    SETUP_EXIT_VALUE=$?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now run the tests; for this we run all the dev installations and kick off our tests (which we'll add inside of the integration folder).\n",
    "\n",
    "```bash\n",
    "    # BUILD S2I BASE IMAGES\n",
    "    make build\n",
    "    S2I_EXIT_VALUE=$?\n",
    "\n",
    "    ## INSTALL ALL REQUIRED DEPENDENCIES\n",
    "    make install_integration_dev\n",
    "    INSTALL_EXIT_VALUE=$?\n",
    "    \n",
    "    ## RUNNING TESTS AND CAPTURING ERROR\n",
    "    make test\n",
    "    TEST_EXIT_VALUE=$?\n",
    "fi\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Finally we just clean everything, including the cluster, the containers and the docker daemon.\n",
    "\n",
    "```bash\n",
    "# DELETE KIND CLUSTER\n",
    "make kind_delete_cluster\n",
    "DELETE_EXIT_VALUE=$?\n",
    "\n",
    "#######################################\n",
    "# EXIT STOPS COMMANDS FROM HERE ONWARDS\n",
    "set -o errexit\n",
    "\n",
    "# CLEANING DOCKER\n",
    "docker ps -aq | xargs -r docker rm -f || true\n",
    "service docker stop || true\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Promote your application\n",
    "\n",
    "After running our integration tests, the last step is to promote our model to our staging and production environments.\n",
    "For that, we will leverage our [GitOps repository](./README.md#gitops-repository) where the state of each environment is stored.\n",
    "\n",
    "In particular, we will:\n",
    "\n",
    "- Push a change to the staging GitOps repository, which will update the staging environment instantly.\n",
    "- Submit a PR to the production GitOps repository, which will wait for a Tech Lead / Manager approval.\n",
    "\n",
    "This will be handled by the `promote_application.sh` script, which can be seen below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./models/news_classifier/promote_application.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./models/news_classifier/promote_application.sh\n",
    "#!/bin/bash\n",
    "\n",
    "# ENSURE WE ARE IN THE DIR OF SCRIPT\n",
    "cd -P -- \"$(dirname -- \"$0\")\"\n",
    "# SO WE CAN MOVE RELATIVE TO THE ACTUAL BASE DIR\n",
    "\n",
    "export GITOPS_REPO=\"seldon-gitops\"\n",
    "export GITOPS_ORG=\"adriangonz\"\n",
    "export STAGING_FOLDER=\"staging\"\n",
    "export PROD_FOLDER=\"production\"\n",
    "\n",
    "# This is the user that is going to be assigned to PRs\n",
    "export GIT_MANAGER=\"adriangonz\"\n",
    "\n",
    "export UUID=$(cat /proc/sys/kernel/random/uuid)\n",
    "\n",
    "git clone https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/${GITOPS_ORG}/${GITOPS_REPO}\n",
    "\n",
    "cd ${GITOPS_REPO}\n",
    "cp -r ../charts/* ${STAGING_FOLDER}/.\n",
    "ls ${STAGING_FOLDER}\n",
    "\n",
    "# Check if any modifications identified\n",
    "git add -N ${STAGING_FOLDER}/\n",
    "git --no-pager diff --exit-code --name-only origin/master ${STAGING_FOLDER}\n",
    "STAGING_MODIFIED=$?\n",
    "if [[ $STAGING_MODIFIED -eq 0 ]]; then\n",
    "  echo \"Staging env not modified\"\n",
    "  exit 0\n",
    "fi\n",
    "\n",
    "# Adding changes to staging repo automatically\n",
    "git add ${STAGING_FOLDER}/\n",
    "git commit -m '{\"Action\":\"Deployment created\",\"Message\":\"\",\"Author\":\"\",\"Email\":\"\"}'\n",
    "git push https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/${GITOPS_ORG}/${GITOPS_REPO}\n",
    "\n",
    "# Add PR to prod\n",
    "cp -r ../charts/* production/.\n",
    "\n",
    "# Create branch and push\n",
    "git checkout -b ${UUID}\n",
    "git add ${PROD_FOLDER}/\n",
    "git commit -m '{\"Action\":\"Moving deployment to production repo\",\"Message\":\"\",\"Author\":\"\",\"Email\":\"\"}'\n",
    "git push https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/${GITOPS_ORG}/${GITOPS_REPO} ${UUID}\n",
    "\n",
    "# Create pull request\n",
    "export PR_RESULT=$(curl \\\n",
    "  -u ${GIT_USERNAME}:${GIT_PASSWORD} \\\n",
    "  -v -H \"Content-Type: application/json\" \\\n",
    "  -X POST -d \"{\\\"title\\\": \\\"SeldonDeployment Model Promotion Request - UUID: ${UUID}\\\", \\\"body\\\": \\\"This PR contains the deployment for the Seldon Deploy model and has been allocated for review and approval for relevant manager.\\\", \\\"head\\\": \\\"${UUID}\\\", \\\"base\\\": \\\"master\\\" }\" \\\n",
    "  https://api.github.com/repos/$GITOPS_ORG/$GITOPS_REPO/pulls)\n",
    "export ISSUE_NUMBER=$(echo \\\n",
    "  $PR_RESULT |\n",
    "  python -c 'import json,sys;obj=json.load(sys.stdin);print(obj[\"number\"])')\n",
    "\n",
    "# Assign PR to relevant user\n",
    "curl \\\n",
    "  -u ${GIT_USERNAME}:${GIT_PASSWORD} \\\n",
    "  -v -H \"Content-Type: application/json\" \\\n",
    "  -X POST -d \"{\\\"assignees\\\": [\\\"${GIT_MANAGER}\\\"] }\" \\\n",
    "  https://api.github.com/repos/$GITOPS_ORG/$GITOPS_REPO/issues/$ISSUE_NUMBER\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a CI/CD pipeline\n",
    "\n",
    "In order to add a pipeline to Jenkins, you just have to go to the \"Manage Jenkins\" configuration dashboard, and click on \"New Item\" to create a new pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![New Item](./images/new-item.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In the first menu, we'll add a name.\n",
    "For example, we can create a new pipeline with name `news_classifier`.\n",
    "We will then be able to add the specific details.\n",
    "Most of these will remain on \"default\", but we will need to change a couple of them to add a GitHub trigger, Docker access and to point to the right folder within the repository.\n",
    "\n",
    "Firstly, we will change the following:\n",
    "\n",
    "* GitHub hook trigger for GITScm polling. \n",
    "* Tick \"This project is parameterised\", and then when you see the next dialog:\n",
    "    * Click on the \"Add parameter\" dropdown, and select \"Credential Parameter\".\n",
    "    * This will open yet another box, where you want to provide the following details:\n",
    "        * name: `docker-access`\n",
    "        * Credential type \"Username and Password\"\n",
    "        * Tick: required\n",
    "        * Default value: Click on the \"Add\" dropdown, and then on \"Jenkins provider\":\n",
    "            * This has opened another dialog box, where you want to add your docker credentials.\n",
    "            * For this you need to make sure that the current selected option is \"Username and Password\".\n",
    "            * There you have to enter your Docker username, and for password it's advised to use a Docker API Key."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Pipeline Config](./images/pipeline-config.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we will need to point to the right `Jenkinsfile`.\n",
    "Note that since we are working with a monorepository, where multiple model implementations are tracked, we will need to point our pipeline to the `./models/news_classifier` folder.\n",
    "If we were working with a single model implementation repository, we would only need to point to the global repo.\n",
    "\n",
    "* Select \"Pipeline script from SCM\" from dropdown.\n",
    "* Add the repository as SCM (in this case https://github.com/SeldonIO/sig-mlops-jenkins-classic/)\n",
    "* Point to the right `Jenkinsfile` under \"Script Path\". In this case, `models/news_classifier/Jenkinsfile`.\n",
    "* If needed, add credentials that will allow to access private repos."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SCM Config](./images/scm-config.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running pipeline\n",
    "\n",
    "In order to trigger a new build, we can do it manually by clicking on \"Build with Parameters\" and then on \"Build\" or we can just push a new change to our GitHub repo.\n",
    "This will take us to a view where we can see some details about each of the stages of the latest builds. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Pipeline Stages](./images/pipeline-stages.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Installing Jenkins on your K8s cluster\n",
    "\n",
    "If you already have access to a cluster but which doesn't have Jenkins installed, you can do so easily using Helm.\n",
    "In particular, you will need to run the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "helm install \\\n",
    "    --name \"jenkins\" stable/jenkins \\\n",
    "    --namespace \"jenkins\" \\\n",
    "    --set \"rbac.create=true\" \\\n",
    "    --set \"master.adminUser=admin\" \\\n",
    "    --set \"master.adminPassword=admin\" \\\n",
    "    --set \"master.serviceType=LoadBalancer\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will install Jenkins and all the required services in the cluster.\n",
    "To get the Load Balancer where it can be accessed you can run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "kubectl get svc -n jenkins | grep jenkins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further configuration \n",
    "\n",
    "If you wish to set up automated pipeline triggers, you will have to install the \"GitHub\" plugin (there are quite a few github related ones but the one you want is the one called plainly \"GitHub\", which then will allow for triggering pipelines automatically on commit.\n",
    "\n",
    "- Install the GitHub Plugin [(for automated webhook triggers)](https://support.cloudbees.com/hc/en-us/articles/115003015691-GitHub-Webhook-Non-Multibranch-Jobs).\n",
    "- Provide a GitHub token with read access so it can clone relevant repositories.\n",
    "- Set-up webhooks so that GitHub can send push requests.\n",
    "\n",
    "Additionally, you will need to configure your Git's `name` and `email` as part of Jenkins settings.\n",
    "\n",
    "![Git user config](./images/git-user.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make sure plugins are updated\n",
    "\n",
    "If you try to run a pipeline and you get an error such as \"No Such DSL Method\", or any strange Java exception when running a pipeline, the most probably reason is due to current plugins not being up to date. \n",
    "\n",
    "Updating your plugins can be done by going to \"Manage Jenkins\" -> \"Plugins\", and then selecct all the plugins and click \"Update and load after restart\". This will take you to another screen - there you should tick the checkbox that reads \"restart after plugins are downloaded and installed\".\n",
    "\n",
    "Once you update our plugins you should be ready to go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ArgoCD\n",
    "\n",
    "A key point of this approach to MLOps relies on having a GitOps repository which gets synced with our Kubernetes cluster.\n",
    "To achieve this we leverage [ArgoCD](https://argoproj.github.io/argo-cd/), which will take care of setting up webhooks with your GitOps repository so that on every change it triggers a synchronisation between the resources you've pushed and what's deployed on the cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "If you don't have it already, you can install ArgoCD following the [official documentation](https://argoproj.github.io/argo-cd/getting_started/#1-install-argo-cd):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "kubectl create namespace argocd\n",
    "kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, you will need to install the accompanying CLI tool.\n",
    "This tool will allow you to easily link your GitOps repository taking care of the entire process.\n",
    "The instructions to install it will vary between different platforms.\n",
    "The official documentation shows the [recommended method](https://argoproj.github.io/argo-cd/cli_installation/) on each of the major ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up GitOps repository\n",
    "\n",
    "To set up the GitOps repository so that it's tracked by ArgoCD we will use the `argocd` CLI tool.\n",
    "We will assume that the `GITHUB_ORG` and `REPONAME` environment variables have been created and that the repository has already been created and can be found in the `https://github.com/$GITHUB_ORG/$REPONAME` url."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export GITHUB_ORG=SeldonIO\n",
    "export REPONAME=seldon-gitops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Private repositories (optional)\n",
    "\n",
    "If your repository is private, we will first need to provide the right credentials for ArgoCD to use.\n",
    "We can do so either using a [user / password login](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#https-username-and-password-credential) or using [SSH keys](https://argoproj.github.io/argo-cd/user-guide/private-repositories/#tls-client-certificates-for-https-repositories).\n",
    "Note that, for the former, we can also use a [personal access token](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) instead of the password.\n",
    "\n",
    "As an example, we will add our GitOps repository using a personal access token.\n",
    "We will assume that the environment variables `GITHUB_USER` and `GITHUB_TOKEN` are set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "export GITHUB_USER=john.doe\n",
    "export GITHUB_TOKEN=12341234\n",
    "\n",
    "argocd repo add https://github.com/$GITHUB_ORG/$REPONAME --username $GITHUB_USER --password $GITHUB_TOKEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create ArgoCD projects\n",
    "\n",
    "The next step is to create two projects within ArgoCD to manage the staging and production environments respectively.\n",
    "Each of them will be linked to a folder within our GitOps repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "argocd app create seldon-staging \\\n",
    "    --repo https://github.com/$GITHUB_ORG/$REPONAME \\\n",
    "    --path staging \\\n",
    "    --dest-namespace staging\n",
    "argocd app create seldon-production \\\n",
    "    --repo https://github.com/$GITHUB_ORG/$REPONAME \\\n",
    "    --path production \\\n",
    "    --dest-namespace production"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we could also sync our `staging` and `production` environment differently.\n",
    "For example, we could have them on separate repositories or separate branches.\n",
    "In this case we would also need to update the `promote_application.sh` script so that it knows how it should promote the respective model between environments."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}