getting-started-guides/csp/databricks/init-notebook-for-rapids-spark-xgboost-on-databricks-gpu-5.3-5.4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download latest Jars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.mkdirs(\"dbfs:/FileStore/jars/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "cd ../../dbfs/FileStore/jars/\n",
    "wget -O cudf-0.9.1.jar https://search.maven.org/remotecontent?filepath=ai/rapids/cudf/0.9.1/cudf-0.9.1.jar\n",
    "wget -O xgboost4j_2.11-1.0.0-Beta2.jar https://search.maven.org/remotecontent?filepath=ai/rapids/xgboost4j_2.11/1.0.0-Beta2/xgboost4j_2.11-1.0.0-Beta2.jar\n",
    "wget -O xgboost4j-spark_2.11-1.0.0-Beta2.jar https://search.maven.org/remotecontent?filepath=ai/rapids/xgboost4j-spark_2.11/1.0.0-Beta2/xgboost4j-spark_2.11-1.0.0-Beta2.jar\n",
    "ls -ltr\n",
    "\n",
    "# Your Jars are downloaded in dbfs:/FileStore/jars directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Directory for your init script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.mkdirs(\"dbfs:/databricks/init_scripts/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.put(\"/databricks/init_scripts/init.sh\",\"\"\"\n",
    "#!/bin/bash\n",
    "sudo cp /dbfs/FileStore/jars/xgboost4j_2.11-1.0.0-Beta2.jar /databricks/jars/spark--maven-trees--ml--xgboost--ml.dmlc--xgboost4j--ml.dmlc__xgboost4j__0.81.jar\n",
    "sudo cp /dbfs/FileStore/jars/cudf-0.9.1.jar /databricks/jars/\n",
    "sudo cp /dbfs/FileStore/jars/xgboost4j-spark_2.11-1.0.0-Beta2.jar /databricks/jars/spark--maven-trees--ml--xgboost--ml.dmlc--xgboost4j-spark--ml.dmlc__xgboost4j-spark__0.81.jar\"\"\", True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confirm your init script is in the new directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "cd ../../dbfs/databricks/init_scripts\n",
    "pwd\n",
    "ls -ltr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the Mortgage Dataset into your local machine and upload Data using import Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "dbutils.fs.mkdirs(\"dbfs:/FileStore/tables/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "cd /dbfs/FileStore/tables/\n",
    "wget -O mortgage.zip https://rapidsai-data.s3.us-east-2.amazonaws.com/spark/mortgage.zip\n",
    "ls\n",
    "unzip mortgage.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "%sh\n",
    "pwd\n",
    "cd ../../dbfs/FileStore/tables\n",
    "ls -ltr mortgage/csv/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps\n",
    "\n",
    "1. Edit your cluster, adding an initialization script from `dbfs:/databricks/init_scripts/init.sh` in the \"Advanced Options\" under \"Init Scripts\" tab\n",
    "2. Reboot the cluster\n",
    "3. Go to \"Libraries\" tab under your cluster and install `dbfs:/FileStore/jars/xgboost4j-spark_2.11-1.0.0-Beta2.jar` in your cluster by selecting the \"DBFS\" option for installing jars\n",
    "4. Import the mortgage example notebook from `https://github.com/rapidsai/spark-examples/blob/master/examples/notebooks/python/mortgage-gpu.ipynb`\n",
    "5. Inside the mortgage example notebook, update the data paths\n",
    "  `train_data = GpuDataReader(spark).schema(schema).option('header', True).csv('dbfs:/FileStore/tables/mortgage/csv/train/mortgage_train_merged.csv')`\n",
    "  `eval_data = GpuDataReader(spark).schema(schema).option('header', True).csv('dbfs:/FileStore/tables/mortgage/csv/test/mortgage_eval_merged.csv')`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "name": "Init Scripts_demo",
  "notebookId": 2585487876834579
 },
 "nbformat": 4,
 "nbformat_minor": 1
}