DS107-Big-Data/DS107L5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "50e6ddbb-52d3-4a02-9b69-100a4181e3b0",
   "metadata": {},
   "source": [
    "# DS107 Big Data : Lesson Five Companion Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1bd4bfc-9b15-4b80-87b4-b8b60eb56371",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS107L5_toc\"></a>\n",
    "\n",
    "* [Table of Contents](#DS107L5_toc)\n",
    "    * [Page 1 - Introduction](#DS107L5_page_1)\n",
    "    * [Page 2 - Spark](#DS107L5_page_2)\n",
    "    * [Page 3 - Running Spark in Hadoop](#DS107L5_page_3)\n",
    "    * [Page 4 - Spark Data Storage](#DS107L5_page_4)\n",
    "    * [Page 5 - Introduction to Scala](#DS107L5_page_5)\n",
    "    * [Page 6 - Using Spark 2.0](#DS107L5_page_6)\n",
    "    * [Page 7 - Using Spark SQL](#DS107L5_page_7)\n",
    "    * [Page 8 - Spark Shell](#DS107L5_page_8)\n",
    "    * [Page 9 - Decision Trees in Spark MLLib](#DS107L5_page_9)\n",
    "    * [Page 10 - Decision Trees and Accuracy](#DS107L5_page_10)\n",
    "    * [Page 11 - Hyperparameter Tuning](#DS107L5_page_11)\n",
    "    * [Page 12 - Best Fit Model](#DS107L5_page_12)\n",
    "    * [Page 13 - Key Terms](#DS107L5_page_13)\n",
    "    * [Page 14 - Lesson 5 Practice Hands-On](#DS107L5_page_14)\n",
    "    * [Page 15 - Lesson 5 Practice Hands-On Solution](#DS107L5_page_15)\n",
    "    * [Page 16 - Lesson 5 Practice Hands-On Solution - Alternative Assignment](#DS107L5_page_16)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef5a8c65-ed48-4354-a257-e01b0fd733d4",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 - Overview of this Module<a class=\"anchor\" id=\"DS107L5_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "88ad2c13-4f38-416a-8e0f-71843a6750b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/388865681\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7fce205e9e80>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Spark 2.0 and Zeppelin\n",
    "VimeoVideo('388865681', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6ad526-95fa-43c4-bc43-1fe807708ed3",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO107L05overview.zip)**.\n",
    "\n",
    "# Introduction\n",
    "\n",
    "Probably the most useful and versatile big data program you could utilize is *Spark*. It has wide-reaching functionality, including a SQL and a machine learning module, and can be used in many different languages.  In this lesson, you will learn about: \n",
    "\n",
    "* Different components of Spark\n",
    "* Ways to run Spark on your Hadoop cluster\n",
    "* Apache Zeppelin, a notebook interface for Hadoop\n",
    "* Three ways to store data in Spark\n",
    "* Scala basics\n",
    "* Using Spark 2.0\n",
    "* Using Spark SQL\n",
    "* Launching the Spark Shell\n",
    "* Decision Trees in Spark using Scala\n",
    "\n",
    "This lesson will culminate in a hands-on in which you perform your own decision tree machine learning model in Spark and Scala.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You may want to watch this <a href=\"https://vimeo.com/458401059\"> recorded live workshop on the concepts in this lesson. </a> </p>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7990ac7a-5a2f-4744-af14-1b7e385ea484",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Spark<a class=\"anchor\" id=\"DS107L5_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612429df-e4ef-4d85-8460-5bc8914079c3",
   "metadata": {},
   "source": [
    "# Spark\n",
    "\n",
    "*Spark* is a data processing program built on top of MapReduce in the Hadoop ecosystem.  Among many, many other things, it allows you to perform machine learning, data mining, and data streaming. It is powerful, fast, and scalable.  How fast, you ask? 100 times faster than MapReduce, which is why MapReduce has pretty much become obsolete as an actual tool, though it remains an incredibly important and foundational big data concept. It uses the same kind of framework as TEZ to work backwards and find the fastest solution to your queries.  Spark is actually written in *Scala*, but you can code in Spark using Python, Scala, or Java, with Python and Scala being the most popular languages. When using Spark, there are a lot of similarities between Python and Scala.\n",
    "\n",
    "---\n",
    "\n",
    "## Spark Components \n",
    "\n",
    "Spark has come a long way in the last several years, and now has a lot of different components that you can utilize within Spark to get varying data science tasks completed.  These components include:\n",
    "\n",
    "* **Spark Core:** This is the base program for Spark.   It is also known as *Spark 1.0*. \n",
    "* **Spark Streaming:** Allows you to feed in real-time data and provide real-time output. \n",
    "* **Spark SQL:** Write SQL queries and use SQLite functions in Spark. This is part of *Spark 2.0.* \n",
    "* **MLLib:** A library of machine learning and data mining tools you can use in Spark. This is also part of *Spark 2.0*.\n",
    "* **GraphX:** Allows you to create social network graphs and determine the degrees of separation in your data.\n",
    "\n",
    "Of these Spark components, you will be introduced to all but GraphX, which is quite specialized.\n",
    "\n",
    "---\n",
    "\n",
    "## Basic Programming Steps in Spark\n",
    "\n",
    "Although you will be using Spark to do all kinds of big data work, there are a few general steps that you will most likely always take when working in Spark:\n",
    "\n",
    "* Run transformations on the input data set\n",
    "* Run actions on the transformed data that can be stored or used\n",
    "* Work further with the results in a distributed fashion and see where to go next.\n",
    "\n",
    "---\n",
    "\n",
    "## Spark SQL\n",
    "\n",
    "Spark SQL allows you to transform your Spark DataFrames and DataSets into SQL tables, so that you can run SQL queries in Spark. With Spark SQL, you have the ability to read and write to a variety of file types, including JSON, Hive, and Parquet, and you can communicate with database connectors and with Tableau. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832358fd-55fa-420a-8aa2-46fbf2097986",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Running Spark in Hadoop<a class=\"anchor\" id=\"DS107L5_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f564e3e7-3a7f-4936-b3a3-2e0a9ab78512",
   "metadata": {},
   "source": [
    "# Running Spark in Hadoop\n",
    "\n",
    "There are three ways you can interact with Spark on your Hadoop cluster: \n",
    "\n",
    "1. Write files in a text editor and then run them through the command prompt. \n",
    "2. Use the Spark Shell.\n",
    "3. Interact with Spark in *Zeppelin*, which is a notebook system similar to Jupyter Notebook that you can access on Hadoop.\n",
    "\n",
    "You will make use of Apache Zeppelin over the next few pages, then you'll switch to using the Spark Shell.\n",
    "\n",
    "---\n",
    "\n",
    "## Apache Zeppelin\n",
    "\n",
    "Luckily for you, Zeppelin comes pre-installed on your Hortonworks instance of Hadoop, and Zeppelin even comes with interaction to Spark 1.0 and 2.0, so that you can hit the ground running! You will access your Zeppelin notebook by typing **[http://127.0.0.1:9995](http://127.0.0.1:9995)** into your browser. \n",
    "\n",
    "Here's what it will look like when you get there:\n",
    "\n",
    "![A browser displays a webpage for zeppelin. The screen has a welcome message, a search box, and a dropdown list box labeled anonymous. It also displays hyperlinks for import note, create new note, and help.](Media/zeppelin1.png)\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You will learn the basics here, but if you want more tutorials, Zeppelin just happens to have them for you here - in particular look at the ones labeled Lab... and the ones labeled Zeppelin Tutorial if you want to go into more detail!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "You can get started by clicking on `Create new note`. When you do, it will ask you to give it a name:\n",
    "\n",
    "![NoteNameA window for creating a new note. A text field for note name is provided and the text field is labeled untitled note 2. A button labeled create note is place at the bottom of the window.](Media/zeppelin2.png)\n",
    "\n",
    "Once you do, you'll get to this section here:\n",
    "\n",
    "![A browser displays a webpage for zeppelin. The screen has a search box and a dropdown list box labeled anonymous. It displays an empty box labeled burn baby burn.](Media/zeppelin3.png)\n",
    "\n",
    "You will be able to type code into the cell, and just like Jupyter Notebook, either pressing the arrow button or typing `shift + enter` will run the cell.\n",
    "\n",
    "---\n",
    "\n",
    "### Ensure it Interprets Spark and Markdown\n",
    "\n",
    "Click the little black gear icon in the upper right hand corner of your Zeppelin notebook, which will bring up a menu looking something like this:\n",
    "\n",
    "![A window for settings displays a message that reads, interpreter binding. Bind interpreter for this note. Click to bind or unbind interpreter. Drag and drop to reorder interpreters. The first interpreter on the list becomes default. To create or remove interpreters, go to interpreter menu. Two buttons labeled save and cancel are placed at the bottom of the window.](Media/zeppelin4.png)\n",
    "\n",
    "You will want to ensure that `spark` is first on that list, followed by `md`, for Markdown.  Changing these settings makes sure that the Notebook processes the right type of languages or programs.\n",
    "\n",
    "---\n",
    "\n",
    "### Tell Zeppelin the Language\n",
    "\n",
    "The `%` sign tells Zeppelin what language you want to use for each cell.  Try out something in Markdown, just to get a feel for it.  Type in:\n",
    "\n",
    "```\n",
    "%md\n",
    "# Testing out Markdown\n",
    "This is text\n",
    "```\n",
    "\n",
    "And hit `shift + enter`.  It should start running for you, and when it's done, the information you have should be output in Markdown, which can make web text much prettier. You can use Markdown for all your notes, just like you do in Jupyter.\n",
    "\n",
    "![A page displays percentage m d. Hashtag testing out markdown. This is text. Testing out markdown. This is text.](Media/zeppelin5.png)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1aade2-59af-4cbc-b5ed-da31e39f266d",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Spark Data Storage<a class=\"anchor\" id=\"DS107L5_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba2ed6e-98dd-43f7-8761-6ecc9a13af08",
   "metadata": {},
   "source": [
    "# Spark Data Storage\n",
    "\n",
    "There are three main ways that data can be stored in Spark:\n",
    "\n",
    "* Resilient Distributed Datasets (RDDs)\n",
    "* DataSets\n",
    "* DataFrames\n",
    "\n",
    "RDDs are an artifact of Spark 1.0, and they will covered in more detail later on.  This lesson will cover both DataSets and DataFrames, which are part of Spark 2.0's architecture. *RDDs* are a type of data storage that distribute your data across your Hadoop cluster, but they are somewhat slow. Mostly, you will only utilize them in Spark 1.0. \n",
    "\n",
    "*DataSets* are similar to RDDs, but they speed up the process, by utilizing more efficient memory representation in Spark.  \n",
    "\n",
    "A *DataFrame* is a subclass of a DataSet, and they are specifically meant for relational data. A DataFrame keeps ahold of rows and columns in Spark, which allows you to do more with your data.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>If you ever need to really optimize something, and pick up the speed of your work, use DataSets instead of DataFrames, since they don't have to store a structure.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead4ad3c-2fe6-457e-8924-1410e951bebd",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Introduction to Scala<a class=\"anchor\" id=\"DS107L5_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbbc06a2-e702-47a1-942f-cbcbe31221a7",
   "metadata": {},
   "source": [
    "# Introduction to Scala\n",
    "\n",
    "Although you can and will utilize Spark through Python in later lessons, which together is called *PySpark*, this lesson will make use of Scala.  That means you can check another language off your bucket list! There are a few advantages to using Scala when it comes to Spark, since Spark is natively written in Scala.  Those advantages include:\n",
    "\n",
    "* Your program is more likely to run as intended, because your code does not have to get translated between languages or environments.\n",
    "* Access to the most recent updates. It takes time to translate the most recent changes into other languages, so you might be a step or two behind in technology without using Scala.\n",
    "* You will understand Spark better, because it is natively written in Scala.\n",
    "* You save time and are more efficient because you will never need to switch between languages when using Spark.\n",
    "\n",
    "<div class=\"panel panel-success\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Additional Info!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Here is a list of <a href=\"https://alvinalexander.com/scala/scala-data-types-bits-ranges-int-short-long-float-double\"> all the different data types in Scala! </a></p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Values and Variables in Scala\n",
    "\n",
    "Often you will see in Scala code the designations of `val` and `var`.  `val` stands for a value, and it is something that cannot be changed once it has been assigned.  `var` stands for variable, and you can change it.  When creating objects, you will need to designate them as either `val` or `var`, with `val` being by far the most common.\n",
    "\n",
    "---\n",
    "\n",
    "## How to Comment in Scala\n",
    "\n",
    "You can comment out code in scala for a single line using `//` at the beginning. It would look something like this:\n",
    "\n",
    "```scala\n",
    "//This is a comment! Scala won't read it as code.\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9e3b8a-9224-447f-8cf7-5035cae86421",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Using Spark 2.0<a class=\"anchor\" id=\"DS107L5_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a81e2098-28ef-4ff5-a155-afe824de4323",
   "metadata": {},
   "source": [
    "# Using Spark 2.0\n",
    "\n",
    "Now that you've been introduced to Zeppelin, and thus have the environment you'll use to interact with Spark, you will really start to get into Spark 2.0 and Scala! You will do the work on this page in Zeppelin.\n",
    "\n",
    "---\n",
    "\n",
    "## Reading in Data\n",
    "\n",
    "The first thing you will do is read in your data. For learning Spark 2.0 and Spark SQL, you will be looking at **[data with movie ratings](https://repo.exeterlms.com/documents/V2/DataScience/Big-Data/u.data.zip)**.  There is also a file that has the **[movie names](https://repo.exeterlms.com/documents/V2/DataScience/Big-Data/u.item.zip)** contained in it. Make sure you upload these files to the `files` view, which will allow you to follow along with the lesson.\n",
    "\n",
    "You'll start by creating a line that identifies the structure that your data will have:\n",
    "\n",
    "```scala\n",
    "final case class Table(movieID: Int, rating: Int)\n",
    "```\n",
    "\n",
    "`Table` is just the name you will give to your data; it could be anything you like.  Within the parentheses, you have key-value pairs that represents the header and data type of each column.  In this case, the data only has two rows, so that keeps it nice and simple! Further, both are integers (`Int`), making this even easier.\n",
    "\n",
    "Then, you will need to transform your file into an RDD, by mapping.  In Scala, you will always start this with `val`.  Next, you'll give your RDD a name; in this case, `MappedTable`.  Then you will call `sc` for `Spark Context`, which just tells Zeppelin you are using Spark, and use the `textFile.map` command to map a flat text file into a usable RDD. You will put the file pathway in for the data.  Since you uploaded it earlier to your `maria_dev` section of your `Files View` in Ambari, the pathway below should work for you.\n",
    "\n",
    "Next, call the `map` function.  Here, you are basically separating your data into columns. `x =>` basically tells it that you are going to define the way to split things next, and in the curly brackets `{}`, you will find that map of how to split, based on `val fields`.  This uses the `.split` function to break at the delimiter specified in the parentheses - in this case, tab: `\\t`. You could also split by other things, including, but not limited to, commas.  Then, you will call the `Table` structure you created earlier, and map each field to the appropriate data structure. Each field will be numbered in parentheses (i.e.`fields(1)`), and should be followed by `.to` the data type.  Since these were both `Int`, they are going to `Int`. The data type you specified above for `Table` must match what you call here.\n",
    "\n",
    "```scala\n",
    "val MappedTable = sc.textFile(\"hdfs:///user/maria_dev/u.data\").map(x => {val fields = x.split(\"\\t\"); Table(fields(1).toInt, fields(2).toInt)})\n",
    "```\n",
    "\n",
    "When you run those two lines together up above, you will most likely get a warning looking something like this:\n",
    "\n",
    "```text\n",
    "warning: there were 1 unchecked warning(s); re-run with -unchecked for details\n",
    "defined class Rating\n",
    "lines: org.apache.spark.rdd.RDD[Rating] = MapPartitionsRDD[85] at map at <console>:43\n",
    "```\n",
    "\n",
    "That is perfectly fine; just a warning and something you can ignore.\n",
    "\n",
    "---\n",
    "\n",
    "## Making Data into a Spark 2.0 DataFrame\n",
    "\n",
    "You now have your data in an RDD, which is serviceable, but slow and clunky compared to Spark 2.0's concept of a DataFrame.  So, from this RDD, you will transfer to a DataFrame.  Start by importing `sqlContext.implicits._`, which is what tells Zeppelin that you are now working with Spark 2.0.\n",
    "\n",
    "Then, you will use the function `.toDF()` to turn the `MappedTable` from above into a DataFrame. You will need to start the line with `val` and give it a name; in this case `DFtable` is being used.\n",
    "\n",
    "Lastly, you can call the `printSchema()` function on your newly created DataFrame, which will provide you with the structure of the data.\n",
    "\n",
    "```scala\n",
    "import sqlContext.implicits._\n",
    "val DFtable = MappedTable.toDF()\n",
    "\n",
    "DFtable.printSchema()\n",
    "```\n",
    "\n",
    "Here is the end result: \n",
    "\n",
    "```text\n",
    " |-- movieID: integer (nullable = false)\n",
    " |-- rating: integer (nullable = false)\n",
    "```\n",
    "\n",
    "As you can see, they are both integers and they do not allow null values.\n",
    "\n",
    "---\n",
    "\n",
    "## Find the Most Popular Movies\n",
    "\n",
    "Now, you can leverage the power of Spark 2.0 and Scala to find the highest rated movies! You'll make use of the functions `groupBy()`, `count()` and `orderBy()` to produce some meaningful results: \n",
    "\n",
    "```scala\n",
    "val MostPopularMovies = DFtable.groupBy(\"movieID\").count().orderBy(desc(\"count\")).cache()\n",
    "MostPopularMovies.show()\n",
    "```\n",
    "\n",
    "The code above first creates a table named `MostPopularMovies`.  Then it uses the `DFtable` you created in the previously code line to group the data by `movieID` and get a count of how many times each `movieID` was referenced. Then, it orders the data by that `count` that was just created.  Finally, this newly created table is cached (`.cache()`), so that it will stay in memory and will make referencing it faster.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">To Cache or Not to Cache?</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>That 'tis the question! Typically, if you are going to use data more than once and it is relatively small, you will save more memory and data usage on your cluster than if you didn't cache.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "With all that done, it is easy to use the `.show()` command, which is much like printing something, to display your results:\n",
    "\n",
    "```text\n",
    "topMovieIDS: org.apache.spark.sql.DataFrame = [movieID: int, count: bigint]\n",
    "+-------+-----+\n",
    "|movieID|count|\n",
    "+-------+-----+\n",
    "|     50|  583|\n",
    "|    258|  509|\n",
    "|    100|  508|\n",
    "|    181|  507|\n",
    "|    294|  485|\n",
    "|    286|  481|\n",
    "|    288|  478|\n",
    "|      1|  452|\n",
    "|    300|  431|\n",
    "|    121|  429|\n",
    "|    174|  420|\n",
    "|    127|  413|\n",
    "|     56|  394|\n",
    "|      7|  392|\n",
    "|     98|  390|\n",
    "|    237|  384|\n",
    "|    117|  378|\n",
    "|    172|  367|\n",
    "|    222|  365|\n",
    "|    313|  350|\n",
    "+-------+-----+\n",
    "only showing top 20 rows\n",
    "```\n",
    "\n",
    "And there you have it! Of course, without a key for what the `movieID`s are, this is not incredibly useful, but you should still be proud of your first execution in Spark 2.0!\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42cb7d8a-aced-42e5-ac24-32acf8c95473",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Using Spark SQL<a class=\"anchor\" id=\"DS107L5_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf0debaf-5fae-4035-8e7d-ae62e62456cb",
   "metadata": {},
   "source": [
    "# Using Spark SQL\n",
    "\n",
    "Instead of using SQL through Pig or Hive, you could also use SQL through Spark! In a lovely, in-line interface like Zeppelin! Keep reading to get in on the fun.\n",
    "\n",
    "---\n",
    "\n",
    "## Create a Temporary Table\n",
    "\n",
    "First, you need to create a temporary table from the data frame you had created on the last page, using the function `registerTempTable`.  You'll give this new table a name called `MovieRatings`.\n",
    "\n",
    "```scala\n",
    "DFtable.registerTempTable(\"MovieRatings\")\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Run SQL Queries\n",
    "\n",
    "Then you're all set to run SQL queries! \n",
    "\n",
    "---\n",
    "\n",
    "### Show the First Ten Movies\n",
    "\n",
    "These can be as simple as just showing the first ten movie ratings:\n",
    "\n",
    "```sql\n",
    "%sql\n",
    "\n",
    "select * from MovieRatings limit 10\n",
    "```\n",
    "\n",
    "In order to use Spark SQL, you'll need to include the `%sql` in your notebook, so that Zeppelin knows what type of code it is!  And here is the result:\n",
    "\n",
    "![A table with two columns and nine entries. The entries for rows are as follows. 242, 3. 302, 3. 377, 1. 51, 2. 346, 1. 474, 4. 265, 2. 465, 5.](Media/Spark1.png)\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>In Spark SQL, the only thing that differs is that you DON'T end your lines with a semi-colon! It won't run if you do.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "### Show the Frequency of Ratings\n",
    "\n",
    "Now, try and step up the complexity a notch. How about using the `count` function to get the frequency of each rating type? Ratings range from 1-5. \n",
    "\n",
    "```sql\n",
    "%sql\n",
    "\n",
    "select rating, count(*) as count from MovieRatings group by rating\n",
    "```\n",
    "\n",
    "And here is that result:\n",
    "\n",
    "![A table has two columns and five entries. The entries are as follows. 1, 6,110. 2, 11,370. 3, 27,145. 4, 34,174. 5, 21,201.](Media/Spark2.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Visualize your Data\n",
    "\n",
    "If your mind isn't blown already, get ready to hang on to something! Because Zeppelin in conjunction with Spark 2.0 SQL can automatically visualize your results! You just need to press a button! \n",
    "\n",
    "![A bar chart has five bars labeled 1, 2, 3, 4, and 5 on the x-axis. The y-axis has three parts labeled 10,000, 20,000, and 30,000.](Media/Spark3.png)\n",
    "\n",
    "The buttons from left to right show the following:\n",
    "\n",
    "* Table\n",
    "* Bar graph (like above)\n",
    "* Pie chart\n",
    "* Area graph\n",
    "* Line plot\n",
    "* Scatter plot\n",
    "\n",
    "You can play with them here, but with frequencies and a category, bar chart is definitely the appropriate visual for this data.\n",
    "\n",
    "---\n",
    "\n",
    "## Find the Most Popular Movie\n",
    "\n",
    "What if you wanted to join some data, so you can actually find out the name of the most popular movie? Well, you can bring in data from the **[u.item]()** file as well, which contains movie IDs and titles, and merge it with your current data!\n",
    "\n",
    "The first step is to read in that file:\n",
    "\n",
    "```scala\n",
    "final case class Movie(movieID: Int, title: String)\n",
    "\n",
    "val lines = sc.textFile(\"hdfs:///user/maria_dev/u.item\").map(x => {val fields = x.split('|'); Movie(fields(0).toInt, fields(1))})\n",
    "\n",
    "import sqlContext.implicits._\n",
    "val moviesDF = lines.toDF()\n",
    "\n",
    "moviesDF.show()\n",
    "```\n",
    "\n",
    "You should get something like this:\n",
    "\n",
    "```text\n",
    "ines: org.apache.spark.rdd.RDD[Movie] = MapPartitionsRDD[7] at map at <console>:34\n",
    "import sqlContext.implicits._\n",
    "moviesDF: org.apache.spark.sql.DataFrame = [movieID: int, title: string]\n",
    "+-------+--------------------+\n",
    "|movieID|               title|\n",
    "+-------+--------------------+\n",
    "|      1|    Toy Story (1995)|\n",
    "|      2|    GoldenEye (1995)|\n",
    "|      3|   Four Rooms (1995)|\n",
    "|      4|   Get Shorty (1995)|\n",
    "|      5|      Copycat (1995)|\n",
    "|      6|Shanghai Triad (Y...|\n",
    "|      7|Twelve Monkeys (1...|\n",
    "|      8|         Babe (1995)|\n",
    "|      9|Dead Man Walking ...|\n",
    "|     10|  Richard III (1995)|\n",
    "|     11|Seven (Se7en) (1995)|\n",
    "|     12|Usual Suspects, T...|\n",
    "|     13|Mighty Aphrodite ...|\n",
    "|     14|  Postino, Il (1994)|\n",
    "|     15|Mr. Holland's Opu...|\n",
    "|     16|French Twist (Gaz...|\n",
    "|     17|From Dusk Till Da...|\n",
    "|     18|White Balloon, Th...|\n",
    "|     19|Antonia's Line (1...|\n",
    "|     20|Angels and Insect...|\n",
    "+-------+--------------------+\n",
    "only showing top 20 rows\n",
    "```\n",
    "\n",
    "Then the next step is to create a temporary table:\n",
    "\n",
    "```scala\n",
    "moviesDF.registerTempTable(\"MovieTitles\")\n",
    "```\n",
    "\n",
    "And then lastly, you can do your SQL query:\n",
    "\n",
    "```sql\n",
    "%sql \n",
    "\n",
    "select t.title, count(*) count from MovieRatings r join MovieTitles t on r.movieID = t.movieID group by t.title order by count desc limit 20\n",
    "```\n",
    "\n",
    "You will give the table `MovieRatings` an alias of `r` and the `MovieTitles` table an alias of `t` to make this a little easier to join.  Then you'll count the ratings, group by the title, and order by the count.  Results should look like this:\n",
    "\n",
    "![A table has two columns and nine entries. The entries are as follows. Star wars 583. contact, 509. Fargo, 508. Return of the Jedi, 507. Liar Liar, 485. English patient, 481. Scream, 478. Toy story, 452. Air Force one, 431.](Media/Spark4.png)\n",
    "\n",
    "Note that you can't see all 20 results, but that there is a scroll bar on the right that can help.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7b3f800-0a14-4d13-8c32-390965c537ca",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Spark Shell<a class=\"anchor\" id=\"DS107L5_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4597e4b4-ce22-448d-843e-eab5ba74a9c4",
   "metadata": {},
   "source": [
    "# Spark Shell\n",
    "\n",
    "Next, you will move into using the Spark MLLib, and for best results, this must be done in the Spark Shell and in Scala.  The next several pages contain directions for how run decision trees in Spark MLLib, using Scala.  Get your swagger on, because this officially makes you cool! Or geeky.  Do you find the line is so fine?\n",
    "\n",
    "---\n",
    "\n",
    "## Specify the Spark Version\n",
    "\n",
    "Your Hadoop Cluster comes with both versions of Spark, 1.0 and 2.0, so you will need to specify which version you want to use.  If you don't, it will run 1.0 by default, which won't be any help to you, since Spark MLLib is contained within Spark 2.0. \n",
    "\n",
    "```bash\n",
    "export SPARK_MAJOR_VERSION=2\n",
    "```\n",
    "\n",
    "Nothing should happen when you run this code, so if you get a clean line, you are good to go.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You will need to run the export line every time you run the shell - it is not a permanent setting that sticks around.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Start Spark Shell Locally\n",
    "\n",
    "Then you can open up the Spark Shell, using ths command.  This has you opening Spark on your local machine, and not through your Hadoop cluster, because you don't actually have a real cluster with multiple nodes here.\n",
    "\n",
    "```bash\n",
    "spark-shell --master local[*]\n",
    "```\n",
    "\n",
    "This may take a little while - so don't be alarmed if you have enough time to grab a cup o' tea!\n",
    "\n",
    "---\n",
    "\n",
    "### Start Spark through YARN\n",
    "\n",
    "If you were using this in a real big data situation, in which you had multiple nodes, you would use a command like this, which has you open Spark through YARN:\n",
    "\n",
    "```bash\n",
    "spark-shell --master yarn --deploy-mode client\n",
    "```\n",
    "\n",
    "Now that you are in the Spark Shell, there are all sorts of things you can do to interact with Spark. You will know you are in and ready to roll when you see the `scala>` prompt.\n",
    "\n",
    "---\n",
    "\n",
    "## Exiting Spark Shell\n",
    "\n",
    "To exit the Spark Shell, use `Ctrl + C`. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36943bcd-7e16-4af1-97d3-c09e81ae5e99",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Decision Trees in Spark MLLib<a class=\"anchor\" id=\"DS107L5_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf2cdde-9447-49b5-8936-17d80f46b45e",
   "metadata": {},
   "source": [
    "# Decision Trees in Spark MLLib\n",
    "\n",
    "Now that you're into Spark Shell, you will need to do a fair amount of data wrangling and prep work before you can actually launch into your decision tree model.  You'll need to read in your data, change the outcome variable data type, split your data up into testing and training data sets, and create a feature vector which contains all of your predictor variables.\n",
    "\n",
    "---\n",
    "\n",
    "## Read in Data\n",
    "\n",
    "First, you will need to read in your data. Spark 2.0 allows you to easily read-in CSVs, with options to bring with it the schema, or structure, of the data, and the headers.  Brilliant! You'll be using **[a dataset](https://repo.exeterlms.com/documents/V2/DataScience/Big-Data/glass1.zip)** that will allow you to predict what kind of glass you have based on the component elements in it. \n",
    "\n",
    "The `Type` variable is the type of glass, and the options are:\n",
    "\n",
    "* 1: Building Windows Float Processed\n",
    "* 2: Building Windows Non-Float Processed\n",
    "* 3: Vehicle Windows Float Processed\n",
    "* 4: Vehicle Windows Non-Float Processed\n",
    "* 5: Containers\n",
    "* 6: Tableware\n",
    "* 7: Headlamps\n",
    "\n",
    "For those of you who care, float processed glass is made by floating molten glass along on a bed of molten metal.  Hot stuff! \n",
    "\n",
    "Make sure you add this dataset to your `Files` view in Ambari in the `user/maria_dev` folder, so that you can use the code below to read in your data.\n",
    "\n",
    "```scala\n",
    "val data = spark.read.\n",
    "option(\"inferSchema\", true).\n",
    "option(\"header\", true).\n",
    "csv(\"hdfs:///user/maria_dev/glass1.csv\")\n",
    "```\n",
    "\n",
    "The code above tells Spark to pull in the structure of your data with the `inferSchema` option, and that your data has headers (`header` option).  You'll also use the argument `csv` because your data is a CSV file.\n",
    "\n",
    "If this works for you, Scala should spit out some basic information about your structure:\n",
    "\n",
    "```text\n",
    "data: org.apache.spark.sql.DataFrame = [RI: double, Na: double ... 8 more fields]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Convert Outcome Variable to Double\n",
    "\n",
    "When you actually get to running your decision trees, all of the variables must be doubles.  So, you'll need to convert any that aren't. You can do that by *casting* the variable to `double`. \n",
    "\n",
    "```scala\n",
    "val data1 = data.\n",
    "withColumn(\"Type\", $\"Type\".cast(\"double\"))\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Train Test Split\n",
    "\n",
    "Then, you need to split your data into training and testing data. In this case, you are keeping 90% of the data for training, and 10% for testing. Of the 90% of the training data, you will actually later be reserving 10% for additional testing, so if the high percentage of training data surprised you, it's actually really an 80-20 split rather than 90-10; it just doesn't look like it here.\n",
    "\n",
    "```scala\n",
    "val Array(trainData, testData) = data1.randomSplit(Array(0.9, 0.1))\n",
    "trainData.cache()\n",
    "testData.cache()\n",
    "```\n",
    "\n",
    "If that has worked, Scala will echo back the fields for the training and the testing data:\n",
    "\n",
    "```text\n",
    "res0: trainData.type = [RI: double, Na: double ... 8 more fields]\n",
    "res1: testData.type = [RI: double, Na: double ... 8 more fields]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Create a Feature Vector\n",
    "\n",
    "Next, you will prep your data for machine learning in Spark MLLib.  When you feed in data, it does not allow more than one column, so you will need to arrange all your data into only one column, which has a value of vector. Luckily, there is a function for this: `VectorAssembler`. \n",
    "\n",
    "In the second line of the code below, you will state that you want to utilize all columns except for `Type`, which is what you are going to predict - the type of glass. The `_!=` is what specifies the exception. Then, in line 3, you will make use of the `VectorAssembler()` function to put the rest of the columns altogether in one vector. You'll then actually run this on your `trainData`, and then `show()` it, so you know it worked. \n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.feature.VectorAssembler\n",
    "val inputCols = trainData.columns.filter(_ != \"Type\")\n",
    "val assembler = new VectorAssembler().\n",
    "setInputCols(inputCols).\n",
    "setOutputCol(\"featureVector\")\n",
    "val assembledTrainData = assembler.transform(trainData)\n",
    "assembledTrainData.select(\"featureVector\").show(truncate = false)\n",
    "```\n",
    "\n",
    "You should get output looking like this back, showing your vector:\n",
    "\n",
    "```text\n",
    "19/11/15 04:09:05 WARN Executor: 1 block locks were not released by TID = 4:\n",
    "[rdd_9_0]\n",
    "+--------------------------------------------------+\n",
    "|featureVector                                     |\n",
    "+--------------------------------------------------+\n",
    "|[1.51115,17.38,0.0,0.34,75.41,0.0,6.65,0.0,0.0]   |\n",
    "|[1.51131,13.69,3.2,1.81,72.81,1.76,5.43,1.19,0.0] |\n",
    "|[1.51215,12.99,3.47,1.12,72.98,0.62,8.35,0.0,0.31]|\n",
    "|[1.51299,14.4,1.74,1.54,74.55,0.0,7.59,0.0,0.0]   |\n",
    "|[1.51316,13.02,0.0,3.04,70.48,6.21,6.96,0.0,0.0]  |\n",
    "|[1.51321,13.0,0.0,3.02,70.7,6.21,6.93,0.0,0.0]    |\n",
    "|[1.51409,14.25,3.09,2.08,72.28,1.1,7.08,0.0,0.0]  |\n",
    "|[1.51508,15.15,0.0,2.25,73.5,0.0,8.34,0.63,0.0]   |\n",
    "|[1.51514,14.01,2.68,3.5,69.89,1.68,5.87,2.2,0.0]  |\n",
    "|[1.51514,14.85,0.0,2.42,73.72,0.0,8.39,0.56,0.0]  |\n",
    "|[1.51531,14.38,0.0,2.66,73.1,0.04,9.08,0.64,0.0]  |\n",
    "|[1.51545,14.14,0.0,2.68,73.39,0.08,9.07,0.61,0.05]|\n",
    "|[1.51556,13.87,0.0,2.54,73.23,0.14,9.41,0.81,0.01]|\n",
    "|[1.51567,13.29,3.45,1.21,72.74,0.56,8.57,0.0,0.0] |\n",
    "|[1.51569,13.24,3.49,1.47,73.25,0.38,8.03,0.0,0.0] |\n",
    "|[1.51571,12.72,3.46,1.56,73.2,0.67,8.09,0.0,0.24] |\n",
    "|[1.51574,14.86,3.67,1.74,71.87,0.16,7.36,0.0,0.12]|\n",
    "|[1.5159,12.82,3.52,1.9,72.86,0.69,7.97,0.0,0.0]   |\n",
    "|[1.5159,13.24,3.34,1.47,73.1,0.39,8.22,0.0,0.0]   |\n",
    "|[1.51593,13.09,3.59,1.52,73.1,0.67,7.83,0.0,0.0]  |\n",
    "+--------------------------------------------------+\n",
    "only showing top 20 rows\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a0d02d5-04fa-4220-9637-d02c3b32dce5",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Decision Trees and Accuracy<a class=\"anchor\" id=\"DS107L5_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f9813d-f4c0-4f35-814b-a1802c030aba",
   "metadata": {},
   "source": [
    "# Decision Trees and Accuracy\n",
    "\n",
    "Now that you've done all the prep work, it's time to actually run your decision tree and see how accurate it is! A decision tree is a type of machine learning model in which the computer finds the best way to differentiate between outcomes.\n",
    "\n",
    "---\n",
    "\n",
    "## Decision Tree Classifier\n",
    "\n",
    "It's time to actually run the decision tree classifier! You'll save it in a `val` named `model`, and if you print those lines out (`println`), you can see the different branches of the decision tree.\n",
    "\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.classification.DecisionTreeClassifier\n",
    "import scala.util.Random\n",
    "val classifier = new DecisionTreeClassifier().\n",
    "setSeed(Random.nextLong()).\n",
    "setLabelCol(\"Type\").\n",
    "setFeaturesCol(\"featureVector\").\n",
    "setPredictionCol(\"prediction\")\n",
    "val model = classifier.fit(assembledTrainData)\n",
    "println(model.toDebugString)\n",
    "```\n",
    "\n",
    "Here are the branching results, meaning all the steps that the algorithm took to separate out the different types of glass:\n",
    "\n",
    "```text\n",
    "DecisionTreeClassificationModel (uid=dtc_5e57af65a40f) of depth 5 with 37 nodes\n",
    "  If (feature 7 <= 0.27)\n",
    "   If (feature 3 <= 1.38)\n",
    "    If (feature 2 <= 3.25)\n",
    "     If (feature 0 <= 1.5202)\n",
    "      If (feature 1 <= 13.78)\n",
    "       Predict: 2.0\n",
    "      Else (feature 1 > 13.78)\n",
    "       Predict: 6.0\n",
    "     Else (feature 0 > 1.5202)\n",
    "      Predict: 2.0\n",
    "    Else (feature 2 > 3.25)\n",
    "     If (feature 0 <= 1.5167)\n",
    "      If (feature 0 <= 1.51567)\n",
    "       Predict: 1.0\n",
    "      Else (feature 0 > 1.51567)\n",
    "       Predict: 3.0\n",
    "     Else (feature 0 > 1.5167)\n",
    "      If (feature 2 <= 3.61)\n",
    "       Predict: 1.0\n",
    "      Else (feature 2 > 3.61)\n",
    "       Predict: 1.0\n",
    "   Else (feature 3 > 1.38)\n",
    "    If (feature 2 <= 1.88)\n",
    "     If (feature 1 <= 13.44)\n",
    "      If (feature 0 <= 1.52172)\n",
    "       Predict: 5.0\n",
    "      Else (feature 0 > 1.52172)\n",
    "       Predict: 2.0\n",
    "     Else (feature 1 > 13.44)\n",
    "      If (feature 0 <= 1.519)\n",
    "       Predict: 6.0\n",
    "      Else (feature 0 > 1.519)\n",
    "       Predict: 2.0\n",
    "    Else (feature 2 > 1.88)\n",
    "     If (feature 5 <= 0.0)\n",
    "      Predict: 6.0\n",
    "     Else (feature 5 > 0.0)\n",
    "      If (feature 6 <= 8.31)\n",
    "       Predict: 2.0\n",
    "      Else (feature 6 > 8.31)\n",
    "       Predict: 2.0\n",
    "  Else (feature 7 > 0.27)\n",
    "   If (feature 1 <= 14.01)\n",
    "    If (feature 4 <= 71.76)\n",
    "     If (feature 0 <= 1.51567)\n",
    "      Predict: 5.0\n",
    "     Else (feature 0 > 1.51567)\n",
    "      If (feature 0 <= 1.5202)\n",
    "       Predict: 1.0\n",
    "      Else (feature 0 > 1.5202)\n",
    "       Predict: 2.0\n",
    "    Else (feature 4 > 71.76)\n",
    "     Predict: 7.0\n",
    "   Else (feature 1 > 14.01)\n",
    "    Predict: 7.0\n",
    "```\n",
    "\n",
    "You'll notice that the features are just numbered, which makes this a little difficult to interpret, but since you have neither fed it a codebook nor could use one in a decision tree, since they all have to be doubles, this makes sense.\n",
    "\n",
    "---\n",
    "\n",
    "## Assess the Importance of Features\n",
    "\n",
    "Next, you can use the `model` you created to assess the importance of the features, or variables, in your decision tree.  You'll use the function `featureImportances` to do so, and then can sort them in reverse order and print, so you get the most important feature first on your list!\n",
    "\n",
    "```scala\n",
    "model.featureImportances.toArray.zip(inputCols).\n",
    "sorted.reverse.foreach(println)\n",
    "```\n",
    "\n",
    "Here are the results:\n",
    "\n",
    "```text\n",
    "(0.2694959533428122,Mg)\n",
    "(0.22949708599977559,Ba)\n",
    "(0.17868395220045094,Al)\n",
    "(0.1575592774341163,RI)\n",
    "(0.08972321273293513,Na)\n",
    "(0.03538920228307928,K)\n",
    "(0.022147581913725102,Ca)\n",
    "(0.0175037340931054,Si)\n",
    "(0.0,Fe)\n",
    "```\n",
    "\n",
    "The higher the number, the better, which is why they have been printed in reverse order.  The weight is printed first for the feature, and then the feature name.  So, you can see up above that the most important feature is whether Magnesium (Mg) is present in the glass, followed by whether Barium (Ba) is present in the glass, etc. The least important feature is Iron (Fe). \n",
    "\n",
    "---\n",
    "\n",
    "## See the Accuracy of the Training Data\n",
    "\n",
    "Now you can start investigating how well your model is doing. The first thing to do is to examine the training data, and see if the actual glass `Type` matches the `prediction` that the decision tree made:\n",
    "\n",
    "```scala\n",
    "val predictions = model.transform(assembledTrainData)\n",
    "predictions.select(\"Type\", \"prediction\", \"probability\").\n",
    "show(truncate = false)\n",
    "```\n",
    "\n",
    "Here are the results from the first 20 rows:\n",
    "\n",
    "```text\n",
    "19/11/15 06:53:07 WARN Executor: 1 block locks were not released by TID = 24:\n",
    "[rdd_24_0]\n",
    "+----+----------+-------------------------------------------------------------------------------+\n",
    "|Type|prediction|probability                                                                    |\n",
    "+----+----------+-------------------------------------------------------------------------------+\n",
    "|6.0 |6.0       |[0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0]                                              |\n",
    "|1.0 |1.0       |[0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0]                                              |\n",
    "|6.0 |6.0       |[0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0]                                              |\n",
    "|5.0 |5.0       |[0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0]                                              |\n",
    "|5.0 |5.0       |[0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0]                                              |\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|7.0 |7.0       |[0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0]                                              |\n",
    "|5.0 |5.0       |[0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0]                                              |\n",
    "|7.0 |7.0       |[0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0]                                              |\n",
    "|7.0 |7.0       |[0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0]                                              |\n",
    "|7.0 |7.0       |[0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0]                                              |\n",
    "|1.0 |1.0       |[0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0]                                              |\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|1.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|2.0 |2.0       |[0.0,0.14285714285714285,0.5714285714285714,0.2857142857142857,0.0,0.0,0.0,0.0]|\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "|2.0 |2.0       |[0.0,0.08571428571428572,0.9142857142857143,0.0,0.0,0.0,0.0,0.0]               |\n",
    "+----+----------+-------------------------------------------------------------------------------+\n",
    "only showing top 20 rows\n",
    "```\n",
    "\n",
    "The `Type` column is the actual data, and the `prediction` column is what the decision tree predicted based on the model. The `probability` column shows the likelihood that each `Type` is correct.  So you can read the first row as: \n",
    "\n",
    "```text\n",
    "Index - ignore\n",
    "0% chance that the glass type is 1\n",
    "0% chance that the glass type is 2\n",
    "0% chance that the glass type is 3\n",
    "0% chance that the glass type is 4\n",
    "0% chance that the glass type is 5\n",
    "100% chance that the glass type is 6\n",
    "0% chance that the glass type is 7\n",
    "```\n",
    "\n",
    "You'll notice that there are actually eight numbers, not seven, even though there are only seven glass types.  This is because the first probability value is just the zero index, and it will always show a probability of zero.\n",
    "\n",
    "Looking at just the first 20 rows, it looks like you have created a decently accurate decision tree, but you'll also want to take a look at the accuracy values as well. \n",
    "\n",
    "---\n",
    "\n",
    "## Print Total Model Accuracy Values\n",
    "\n",
    "Just looking at the first twenty rows is a good eyeball check, but doesn't give you the total accuracy for the model.  Good thing the function `MulticlassClassificationEvaluator` has your back! \n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator\n",
    "val evaluator = new MulticlassClassificationEvaluator().\n",
    "setLabelCol(\"Type\").\n",
    "setPredictionCol(\"prediction\")\n",
    "evaluator.setMetricName(\"accuracy\").evaluate(predictions)\n",
    "```\n",
    "\n",
    "Here is the result:\n",
    "\n",
    "```text\n",
    "res25: Double = 0.8201058201058201\n",
    "```\n",
    "\n",
    "This shows that the model accurately predicts the glass type 82% of the time.  That's not bad...but it could be much higher! \n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Your accuracy value may come out slightly different than what is here, because by default, the decisions in a decision tree are random, and your computer may have done it slightly differently than the instructor's!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Examine the Classification Matrix\n",
    "\n",
    "Another way you can examine accuracy, which will give you a little more detail about what is going well and what isn't, is to look at a confusion matrix. This will compare the predictions to the actual data, so that you can see where the decision tree got it right, and if it didn't, as what type of glass it was misclassified.\n",
    "\n",
    "```scala\n",
    "val confusionMatrix = predictions.\n",
    "groupBy(\"Type\").\n",
    "pivot(\"prediction\", (1 to 7)).\n",
    "count().\n",
    "na.fill(0.0).\n",
    "orderBy(\"Type\")\n",
    "confusionMatrix.show()\n",
    "```\n",
    "\n",
    "The output shows the predicted versus actual types.  Along the diagonal (54, 56, etc.) you will find the number that was correctly classified.  So in reading this matrix, you find that for type 1, 54 pieces of glass were correctly classified as 1s and 8 were incorrectly classified as type 2. You want to see lots of zeros for things not on the diagonal, so at a glance, this it looking pretty decent.\n",
    "\n",
    "```text\n",
    "+----+---+---+---+---+---+---+---+\n",
    "|Type|  1|  2|  3|  4|  5|  6|  7|\n",
    "+----+---+---+---+---+---+---+---+\n",
    "| 1.0| 54|  8|  0|  0|  0|  0|  0|\n",
    "| 2.0| 11| 56|  2|  0|  0|  0|  0|\n",
    "| 3.0|  6|  4|  6|  0|  0|  0|  0|\n",
    "| 5.0|  0|  1|  0|  0| 11|  0|  0|\n",
    "| 6.0|  0|  0|  0|  0|  0|  7|  0|\n",
    "| 7.0|  1|  1|  0|  0|  0|  0| 21|\n",
    "+----+---+---+---+---+---+---+---+\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Is Your Accuracy Better than Random?\n",
    "\n",
    "It is pretty difficult to benchmark accuracy.  Is 82% good? Bad? Ugly? One way to determine at first blush whether your accuracy is any good at all is to find out what the accuracy would be like if you were random guessing.  Here's the code to attempt that:\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.sql.DataFrame\n",
    "def classProbabilities(data: DataFrame): Array[Double] = {\n",
    "val total = data.count()\n",
    "data.groupBy(\"Type\").count().\n",
    "orderBy(\"Type\").\n",
    "select(\"count\").as[Double].\n",
    "map(_ / total).\n",
    "collect()\n",
    "}\n",
    "\n",
    "val trainPriorProbabilities = classProbabilities(trainData)\n",
    "val testPriorProbabilities = classProbabilities(testData)\n",
    "trainPriorProbabilities.zip(testPriorProbabilities).map {\n",
    "case (trainProb, cvProb) => trainProb * cvProb\n",
    "}.sum\n",
    "```\n",
    "\n",
    "And here is the result:\n",
    "\n",
    "```text\n",
    "res33: Double = 0.2452910052910053\n",
    "```\n",
    "\n",
    "Looks like random guessing will give you an accuracy of 25%, so 82% accuracy with your decision tree is looking spectacular now, isn't it?! \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e8d1ec-4e4b-44c7-9fe0-5966ac031d1c",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Hyperparameter Tuning<a class=\"anchor\" id=\"DS107L5_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "730c1691-c343-41d4-88e9-415857aecf9a",
   "metadata": {},
   "source": [
    "# Hyperparameter Tuning\n",
    "\n",
    "The decision tree was fun, and it was decently accurate.  But why stop there? Don't you want to be the best you possibly can? Well, you can probably improve things by playing with the *hyperparameters*. Hyperparameters are the components of the way your model has been created, and by changing them, you can get a better model fit. A decision tree has the following hyperparameters:\n",
    "\n",
    "* **Maximum depth:** Limits the number of decisions you can make in a decision tree.  Sometimes having too many can lead to overfitting of data.\n",
    "* **Maximum bins:** Limits the number of decision rules the decision tree can have. A lot will probably make your decision tree more accurate, but could take up too much processing power.\n",
    "* **Impurity measure:** *Purity* is how good you are at classifying accurately.  If you have two categories, and each group only contains the appropriate data for that category, then you have complete purity.  If you have some mix-up in there, then you have *impurity*. You want to have low impurity.\n",
    "* **Minimum information gain:** If you include a decision rule that does not make the data more pure, then what good is it? Including a hyperparameter of minimum information gain allows you to only keep levels that will actually add to the accuracy of your data, and it can help ensure you don't overfit the model.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Create a Pipeline for Hyperparameter Tuning\n",
    "\n",
    "Now that you know what the hyperparameters are for decision trees, you can start playing with them! The code below will create a *pipeline* for your decision tree data, and then the next set of code after that will be for trying all different varieties of these hyperparameters, to see which fits the best.  A pipeline is when you chain two operations together, so that you don't need to keep running them individually over and over again. Using something like this can take some time and processing power on the computer's end, but will save you the trouble of having to tune everything manually, one at a time, which would take forever!\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.Pipeline\n",
    "val inputCols = trainData.columns.filter(_ != \"Type\")\n",
    "val assembler = new VectorAssembler().\n",
    "setInputCols(inputCols).\n",
    "setOutputCol(\"featureVector\")\n",
    "val classifier = new DecisionTreeClassifier().\n",
    "setSeed(Random.nextLong()).\n",
    "setLabelCol(\"Type\").\n",
    "setFeaturesCol(\"featureVector\").\n",
    "setPredictionCol(\"prediction\")\n",
    "val pipeline = new Pipeline().setStages(Array(assembler, classifier))\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Set the Hyperparameter Boundaries and How to Determine Which is Best\n",
    "\n",
    "Next, you will set the hyperparameter boundaries for impurity, maximum depth, maximum bins, and minimum information gain.  For `impurity`, you will try two different versions of impurity: `gini` and `entropy`.  For `maxDepth`, you will try the values of 1 and 20.  For `maxBins`, you will try all the values of 40 and 300, and for `minInfoGain`, you will try all the values of 0 and .05. All in all, you will be testing 16 models, since you have two values for each hyperparameter. Then, you'll get the accuracy for each of them!\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.tuning.ParamGridBuilder\n",
    "val paramGrid = new ParamGridBuilder().\n",
    "addGrid(classifier.impurity, Seq(\"gini\", \"entropy\")).\n",
    "addGrid(classifier.maxDepth, Seq(1, 20)).\n",
    "addGrid(classifier.maxBins, Seq(40, 300)).\n",
    "addGrid(classifier.minInfoGain, Seq(0.0, 0.05)).\n",
    "build()\n",
    "val multiclassEval = new MulticlassClassificationEvaluator().\n",
    "setLabelCol(\"Type\").\n",
    "setPredictionCol(\"prediction\").\n",
    "setMetricName(\"accuracy\")\n",
    "```\n",
    "\n",
    "The output you will receive back just basically acknowledges your model creation. \n",
    "\n",
    "---\n",
    "\n",
    "## Train Test Split for Hyperparameter Tuning\n",
    "\n",
    "Now that you know what all you want to run, and have it saved as `multiclassEval`, you can run train test split again, so that you have a little bit of data in reserve to test the hyperparameter tuning.  This ensures that you don't accidentally overfit your hyperparameters as well.\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.tuning.TrainValidationSplit\n",
    "val validator = new TrainValidationSplit().\n",
    "setSeed(Random.nextLong()).\n",
    "setEstimator(pipeline).\n",
    "setEvaluator(multiclassEval).\n",
    "setEstimatorParamMaps(paramGrid).\n",
    "setTrainRatio(0.9)\n",
    "val validatorModel = validator.fit(trainData)\n",
    "```\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Running this may take a while, depending on your computing power, and you may get many warnings about \"1 block locks were not released by TID =\". Just keep waiting, and know that you'll still get results, even though all those scary warnings show. Remember that you're checking 16 different models for accuracy!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "The `validatorModel` contains the best fit model, but you'll have to wait to see what it is.  The suspense builds...\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7503c0dc-0264-483b-a17a-5013b40a88ab",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 12 - Best Fit Model<a class=\"anchor\" id=\"DS107L5_page_12\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2eca1a5-31f3-47bb-9673-e5d5125fc00e",
   "metadata": {},
   "source": [
    "# Best Fit Model\n",
    "\n",
    "The result of your hyperparameter tuning is the best-fit model for your data. This page will show you what that model is and the accuracy of said model.\n",
    "\n",
    "---\n",
    "\n",
    "## Determine the Best Fit Model\n",
    "\n",
    "Now you can actually extract the best fit model and find out what it is:\n",
    "\n",
    "```scala\n",
    "import org.apache.spark.ml.PipelineModel\n",
    "val bestModel = validatorModel.bestModel\n",
    "bestModel.asInstanceOf[PipelineModel].stages.last.extractParamMap\n",
    "```\n",
    "\n",
    "And here is the output:\n",
    "\n",
    "```text\n",
    "res34: org.apache.spark.ml.param.ParamMap =\n",
    "{\n",
    "        dtc_0e1e4da37cb7-cacheNodeIds: false,\n",
    "        dtc_0e1e4da37cb7-checkpointInterval: 10,\n",
    "        dtc_0e1e4da37cb7-featuresCol: featureVector,\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-labelCol: Type,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-maxMemoryInMB: 256,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0,\n",
    "        dtc_0e1e4da37cb7-minInstancesPerNode: 1,\n",
    "        dtc_0e1e4da37cb7-predictionCol: prediction,\n",
    "        dtc_0e1e4da37cb7-probabilityCol: probability,\n",
    "        dtc_0e1e4da37cb7-rawPredictionCol: rawPrediction,\n",
    "        dtc_0e1e4da37cb7-seed: 8849016365946518463\n",
    "}\n",
    "```\n",
    "\n",
    "Looks like you want to use the `gini` method of calculating impurity, want a `maxBins` of 300, a `maxDepth` of 20, and want to use zero `minInfoGain`.\n",
    "\n",
    "---\n",
    "\n",
    "## Determine the Accuracy of the Models\n",
    "\n",
    "But you want to know what the accuracy is for your best fit model? Too crazy; best go back to bed.  Or maybe just run the code below:\n",
    "\n",
    "```scala\n",
    "val validatorModel = validator.fit(trainData)\n",
    "val paramsAndMetrics = validatorModel.validationMetrics.\n",
    "zip(validatorModel.getEstimatorParamMaps).sortBy(-_._1)\n",
    "paramsAndMetrics.foreach { case (metric, params) =>\n",
    "println(metric)\n",
    "println(params)\n",
    "println()\n",
    "}\n",
    "```\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>This may also take a minute, and you may see some of the same warnings up above.  But as the Brits say, keep calm and carry on!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "The output of the code above gives you the accuracy of the models in descending order, followed by the hyperparameters for that model: \n",
    "\n",
    "```text\n",
    "0.8571428571428571\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.8571428571428571\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.7857142857142857\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.7857142857142857\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.7142857142857143\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.6428571428571429\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: gini,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 20,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.35714285714285715\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.35714285714285715\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 40,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "\n",
    "0.35714285714285715\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.0\n",
    "}\n",
    "\n",
    "0.35714285714285715\n",
    "{\n",
    "        dtc_0e1e4da37cb7-impurity: entropy,\n",
    "        dtc_0e1e4da37cb7-maxBins: 300,\n",
    "        dtc_0e1e4da37cb7-maxDepth: 1,\n",
    "        dtc_0e1e4da37cb7-minInfoGain: 0.05\n",
    "}\n",
    "```\n",
    "\n",
    "As noted the first time, the best model is gini/300/20/0, and it yields an accuracy of 87%, which is an improvement upon the original accuracy of 82%.  You gained an extra 5% accuracy utilizing your hyperparameters! High five!\n",
    "\n",
    "---\n",
    "\n",
    "## Use Testing Data to Evaluate the Model\n",
    "\n",
    "Now that you've split the data, and trained with it, it's time to test that bad boy out! You know that you're 87% accurate when training, but does that hold up in testing? The first line below is for evaluating the 10% for hyperparameter testing, and the second line below is for evaluating the 10% for testing as a whole.\n",
    "\n",
    "```scala\n",
    "validatorModel.validationMetrics.max\n",
    "multiclassEval.evaluate(bestModel.transform(testData))\n",
    "```\n",
    "\n",
    "Here are the hyperparameter tuning results:\n",
    "\n",
    "```text\n",
    "res52: Double = 0.8571428571428571\n",
    "```\n",
    "\n",
    "And here are the overall testing results:\n",
    "\n",
    "```text\n",
    "res53: Double = 0.76\n",
    "```\n",
    "\n",
    "So it looks like tuning those hyperparameters was a good call! A 9% increase in accuracy is good to have! \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fd4909-e199-48f9-9609-e793d7d4cbda",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 13 - Key Terms<a class=\"anchor\" id=\"DS107L5_page_13\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ca07d0-565d-4694-8d31-0ecb4206fb24",
   "metadata": {},
   "source": [
    "# Key Terms\n",
    "\n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Spark</td>\n",
    "        <td>Data processing program built on top of MapReduce.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Spark Core</td>\n",
    "        <td>Spark base; also known as Spark 1.0.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Spark Streaming</td>\n",
    "        <td>Program to feed in real-time data and receive real-time output.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Spark SQL</td>\n",
    "        <td>Use SQL within Spark for a speed boost.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>MLLib</td>\n",
    "        <td>Machine learning library for Spark.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>GraphX</td>\n",
    "        <td>Social networking graphs in Spark.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Apache Zeppelin</td>\n",
    "        <td>Notebook interface for your Hadoop cluster.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Resilient Distributed Datasets (RDDs)</td>\n",
    "        <td>Data stored across your Hadoop cluster.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>DataSets</td>\n",
    "        <td>Spark 2.0 data storage that allows for efficiency.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>DataFrames</td>\n",
    "        <td>Spark 2.0 data storage that maintains data structure. For relational data.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Scala</td>\n",
    "        <td>Programming language that Spark was built in.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Hyperparameter</td>\n",
    "        <td>Components to the way you create your machine learning model.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Maximum Depth</td>\n",
    "        <td>The number of decisions you allow your decision tree to make.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Maximum Bins</td>\n",
    "        <td>The number of decision rules you allow your decision tree to have.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Impurity Measure</td>\n",
    "        <td>When you sort your outcomes with some mix-up.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Purity</td>\n",
    "        <td>Having separate groups that only contain the specified category.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Minimum Information Gain</td>\n",
    "        <td>Don't include a decision rule that won't make the data more pure.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Pipeline</td>\n",
    "        <td>Automatically chaining operations together.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "## Key Scala Code\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>val</td>\n",
    "        <td>A value that cannot be changed once it has been assigned.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>var</td>\n",
    "        <td>A variable that can be changed after it has been assigned.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>sc</td>\n",
    "        <td>Spark Context; environment in which you can use Spark.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>.map()</td>\n",
    "        <td>Provides structure for text files in Spark.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>.toDF()</td>\n",
    "        <td>Changes data from an RDD to a DataFrame.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>printSchema()</td>\n",
    "        <td>Prints the structure of your DataFrame.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>.cache()</td>\n",
    "        <td>Keeps your data in memory so you can access it faster.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>registerTempTable()</td>\n",
    "        <td>Creates a temporary table that you can use with Spark SQL.</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c846400-985d-465c-852a-a4c0953f6cf6",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 14 - Lesson 5 Practice Hands-On<a class=\"anchor\" id=\"DS107L5_page_14\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27cf5a70-a2f6-4a0e-958a-dd9341bcfe72",
   "metadata": {},
   "source": [
    "This Hands-On will **not** be graded, but you are encouraged to complete it. The best way to become a great data scientist is to practice. Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Description\n",
    "\n",
    "Using **[this data on boardgame ratings](https://repo.exeterlms.com/documents/V2/DataScience/Big-Data/boardgames3.zip)**, perform a decision tree to predict the average rating of boardgames (`average_rating`). You will need to upload this data file to your HDFS.\n",
    "\n",
    "Please copy your Scala code into a text file, and include at the bottom the answer to the following questions:\n",
    "\n",
    "* What was the best model after hyperparameter tuning? \n",
    "* What is the overall accuracy?\n",
    "\n",
    "---\n",
    "\n",
    "## Alternative Assignment if You Can't Run Hadoop and/or Ambari\n",
    "\n",
    "If your computer refuses to run Hadoop and/or Ambari, **[here](https://repo.exeterlms.com/documents/V2/DataScience/Big-Data/L5exam.zip)** is an alternative exam to test your understanding of the material. Please attach it instead.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d518cec-6dd7-4c68-a463-524fb1deba96",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 15 - Lesson 5 Practice Hands-On Solution<a class=\"anchor\" id=\"DS107L5_page_15\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e2ca60-c32d-446d-8a7e-4084119f212c",
   "metadata": {},
   "source": [
    "# Lesson 5 Practice Hands-On Solution\n",
    "\n",
    "---\n",
    "\n",
    "## Best Model After Hyperparameter Tuning\n",
    "\n",
    "The best model after hyperparamter tuning was the one that used entropy, a max of 300 bins, a depth of 20, and .05 of minimum information gain.\n",
    "\n",
    "---\n",
    "\n",
    "## Overall Accuracy\n",
    "\n",
    "The overall accuracy was 62% with the best hyperparamter model.\n",
    "\n",
    "---\n",
    "\n",
    "## Code\n",
    "\n",
    "Below you will find all the code to provide the answers above.\n",
    "\n",
    "---\n",
    "\n",
    "### Done in the Command Prompt\n",
    "\n",
    "```bash\n",
    "export SPARK_MAJOR_VERSION=2\n",
    "\n",
    "spark-shell --master local[*]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "### Done in Spark Shell\n",
    "\n",
    "```scala\n",
    "val data = spark.read.\n",
    "option(\"inferSchema\", true).\n",
    "option(\"header\", true).\n",
    "csv(\"hdfs:///user/maria_dev/boardgames3.csv\")\n",
    "\n",
    "data.printSchema()\n",
    "\n",
    "val Array(trainData, testData) = data.randomSplit(Array(0.9, 0.1))\n",
    "trainData.cache()\n",
    "testData.cache()\n",
    "\n",
    "import org.apache.spark.ml.feature.VectorAssembler\n",
    "val inputCols = trainData.columns.filter(_ != \"average_rating\")\n",
    "val assembler = new VectorAssembler().\n",
    "setInputCols(inputCols).\n",
    "setOutputCol(\"featureVector\")\n",
    "val assembledTrainData = assembler.transform(trainData)\n",
    "assembledTrainData.select(\"featureVector\").show(truncate = false)\n",
    "\n",
    "import org.apache.spark.ml.classification.DecisionTreeClassifier\n",
    "import scala.util.Random\n",
    "val classifier = new DecisionTreeClassifier().\n",
    "setSeed(Random.nextLong()).\n",
    "setLabelCol(\"average_rating\").\n",
    "setFeaturesCol(\"featureVector\").\n",
    "setPredictionCol(\"prediction\")\n",
    "val model = classifier.fit(assembledTrainData)\n",
    "println(model.toDebugString)\n",
    "\n",
    "model.featureImportances.toArray.zip(inputCols).\n",
    "sorted.reverse.foreach(println)\n",
    "\n",
    "//The best feature for prediction is users_rated.\n",
    "\n",
    "val predictions = model.transform(assembledTrainData)\n",
    "predictions.select(\"average_rating\", \"prediction\", \"probability\").\n",
    "show(truncate = false)\n",
    "\n",
    "//Eyeball analysis shows that right now it is not predicting very well.\n",
    "\n",
    "import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator\n",
    "val evaluator = new MulticlassClassificationEvaluator().\n",
    "setLabelCol(\"average_rating\").\n",
    "setPredictionCol(\"prediction\")\n",
    "evaluator.setMetricName(\"accuracy\").evaluate(predictions)\n",
    "\n",
    "//Right now the model has 57% accuracy.\n",
    "\n",
    "val confusionMatrix = predictions.\n",
    "groupBy(\"average_rating\").\n",
    "pivot(\"prediction\", (0 to 10)).\n",
    "count().\n",
    "na.fill(0.0).\n",
    "orderBy(\"average_rating\")\n",
    "confusionMatrix.show()\n",
    "\n",
    "//Looks like most ratings, both higher and lower, are getting misclassified as 5s, 6s, and 7s. There were no accurate predictions of 1-4 or 8-10.\n",
    "\n",
    "import org.apache.spark.sql.DataFrame\n",
    "def classProbabilities(data: DataFrame): Array[Double] = {\n",
    "val total = data.count()\n",
    "data.groupBy(\"average_rating\").count().\n",
    "orderBy(\"average_rating\").\n",
    "select(\"count\").as[Double].\n",
    "map(_ / total).\n",
    "collect()\n",
    "}\n",
    "\n",
    "val trainPriorProbabilities = classProbabilities(trainData)\n",
    "val testPriorProbabilities = classProbabilities(testData)\n",
    "trainPriorProbabilities.zip(testPriorProbabilities).map {\n",
    "case (trainProb, cvProb) => trainProb * cvProb\n",
    "}.sum\n",
    "\n",
    "//Random guessing accuracy is 18%, so the current model is better than just guessing.  Yay!\n",
    "\n",
    "import org.apache.spark.ml.Pipeline\n",
    "val inputCols = trainData.columns.filter(_ != \"average_rating\")\n",
    "val assembler = new VectorAssembler().\n",
    "setInputCols(inputCols).\n",
    "setOutputCol(\"featureVector\")\n",
    "val classifier = new DecisionTreeClassifier().\n",
    "setSeed(Random.nextLong()).\n",
    "setLabelCol(\"average_rating\").\n",
    "setFeaturesCol(\"featureVector\").\n",
    "setPredictionCol(\"prediction\")\n",
    "val pipeline = new Pipeline().setStages(Array(assembler, classifier))\n",
    "\n",
    "import org.apache.spark.ml.tuning.ParamGridBuilder\n",
    "val paramGrid = new ParamGridBuilder().\n",
    "addGrid(classifier.impurity, Seq(\"gini\", \"entropy\")).\n",
    "addGrid(classifier.maxDepth, Seq(1, 20)).\n",
    "addGrid(classifier.maxBins, Seq(40, 300)).\n",
    "addGrid(classifier.minInfoGain, Seq(0.0, 0.05)).\n",
    "build()\n",
    "val multiclassEval = new MulticlassClassificationEvaluator().\n",
    "setLabelCol(\"average_rating\").\n",
    "setPredictionCol(\"prediction\").\n",
    "setMetricName(\"accuracy\")\n",
    "\n",
    "import org.apache.spark.ml.tuning.TrainValidationSplit\n",
    "val validator = new TrainValidationSplit().\n",
    "setSeed(Random.nextLong()).\n",
    "setEstimator(pipeline).\n",
    "setEvaluator(multiclassEval).\n",
    "setEstimatorParamMaps(paramGrid).\n",
    "setTrainRatio(0.9)\n",
    "val validatorModel = validator.fit(trainData)\n",
    "\n",
    "import org.apache.spark.ml.PipelineModel\n",
    "val bestModel = validatorModel.bestModel\n",
    "bestModel.asInstanceOf[PipelineModel].stages.last.extractParamMap\n",
    "\n",
    "//The best model uses entropy, a max of 300 bins, a depth of 20, and .05 of minimum information gain.\n",
    "\n",
    "val validatorModel = validator.fit(trainData)\n",
    "val paramsAndMetrics = validatorModel.validationMetrics.\n",
    "zip(validatorModel.getEstimatorParamMaps).sortBy(-_._1)\n",
    "paramsAndMetrics.foreach { case (metric, params) =>\n",
    "println(metric)\n",
    "println(params)\n",
    "println()\n",
    "}\n",
    "\n",
    "validatorModel.validationMetrics.max\n",
    "multiclassEval.evaluate(bestModel.transform(testData))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a208ded6-817b-4901-9712-578e5aa6de80",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 16 - Lesson 5 Practice Hands-On Solution - Alternative Assignment<a class=\"anchor\" id=\"DS107L5_page_16\"></a>\n",
    "\n",
    "[Back to Top](#DS107L5_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a567621-044d-4648-a674-c1c3e618195e",
   "metadata": {},
   "source": [
    "# Lesson 5 Practice Hands-On Solution - Alternative Assignment\n",
    "\n",
    "This exam serves as the assessment for those students who cannot utilize the Hadoop system and/or Ambari GUI. Correct answers are show in bold.\n",
    "\n",
    "1.\tWhich of the Spark components most interests you and why?\n",
    "\n",
    "    **Spark ML seems so powerful! It also seems the most like \"regular\" programming that isn't done with big data, which makes it a little easier.**\n",
    "\n",
    "2.\tTrue or False? \"Zeppelin is very similar in structure and function to Jupyter Notebook.\"\n",
    "    **a.\tTrue**\n",
    "    b.\tFalse\n",
    "\n",
    "3.\tHow do the three types of Spark data storage differ from each other?\n",
    "\n",
    "    **RDDs are the original way to store data, but they are slow.  DataSets are more efficient.  DataFrames are meant specially for relational data and hold row and column data.**\n",
    "\n",
    "4.\tHow do you denote comments in Scala?\n",
    "    a.\t# \n",
    "    b.\tChange it to markdown\n",
    "    **c.\t//**\n",
    "    d.\t/#\n",
    "\n",
    "5.\tWhat are the four hyperparameters for decision trees? Give both names and descriptions.\n",
    "\n",
    "    **Maximum Depth: The number of decisions you can make in a tree.**\n",
    "\n",
    "    **Maximum Bins: The number of decision rules you can use in a tree.**\n",
    "\n",
    "    **Impurity Measure: How much mix-up you allow between categories.**\n",
    "\n",
    "    **MInimum Information Gain: Keep only things that add to the accuracy of your data.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}