3. Internal Details of Spark.ipynb

{"cells":[{"cell_type":"markdown","source":["## Internal Details of Spark\n### Agenda\n<hr>\n* Understanding Cluster\n* Driver & Executors\n* Partitions\n* Spark Entities - Application, Job, Stages & Task\n* Resilient Distributed Datastructure\n* Spark DataFrames\n\n<hr>"],"metadata":{}},{"cell_type":"markdown","source":["### 1. Understanding Cluster\n<hr>\n* In distributed computing environment, systems have to be inter-connected for information exchange. This inter-connected system is known as cluster. \n* Systems are connected using switch.\n* Cluster needs to be managed. \n* Spark have it's own cluster manager.\n* Other cluster managers are Mesos, YARN\n\n<img src=\"https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/23586e93bfb6846c5bbd0a5c14f2ed4a1e1ec40a/2-Figure1-1.png\" width=\"300px\">"],"metadata":{}},{"cell_type":"markdown","source":["### 2. Driver & Executors\n<hr>\n\n<img src=\"https://spark.apache.org/docs/latest/img/cluster-overview.png\">\n\n* Spark runs on cluster in deployments.\n* Spark applications run as independent sets of processes on a cluster, coordinated by the program running in driver node.\n* SparkContext object is part of driver program & controls the spark processes.\n* For controlling processes across cluster SparkContext needs to use cluster managers (Sparks' Cluster Manager, YARN, Mesos).\n* One responsibility of cluster manager is to allocate resources in executors ( nodes in which data processing & storage happens )\n* Resources like (cpu,memory) gets allocated at executor, application code is sent.\n* Task runs on executor & is controlled by driver.\n* Driver program listens to executors."],"metadata":{}},{"cell_type":"markdown","source":["### 3. Data Partitioning\n<hr>\n* Data is splitted into partitions.\n* Each machine has more than one partitions.\n* Number of partitions per machine is configurable. Too-much or too-less may not be a great idea.\n* Partitions' don't span across multiple machines.\n* Usual sizes of partitions are 64MB, 128MB or 256MB\n* By increasing number of partitioning, we can achieve higher parallelism.\n* But, network communication is expensive thing in distributed computing and manging too many tasks gets difficult.\n* Having right balance of partition maximizes usage.\n* Two ways of partitioning data\n  - HashPartition : Tries to evenly distribute data based on hash value.\n  - RangePartition : Tuples having keys of same range will appear in same partition."],"metadata":{}},{"cell_type":"markdown","source":["### 4. Spark Entities - Application, Job, Stages & Task\n<hr>\n* Application - Complete user program built on Spark consisting of driver & executor code.\n* Job - An application when subjected to execution is known as job.\n* Stages - Job gets divided into stages. Same stage can be parallelized, but different stages are sequential.\n* Task - A stage in execution is known as task. Number of tasks at a time will be equal to number of partitions."],"metadata":{}},{"cell_type":"markdown","source":["<img src=\"https://cdn-images-1.medium.com/max/1000/1*wiXLNwwMyWdyyBuzZnGrWA.png\" width=\"600px\">"],"metadata":{}},{"cell_type":"markdown","source":["### 5. Resilient Distributed Datasets (RDDs)\n<hr>\n* Fundamental & Low-level data-structure around which spark revolves.\n* RDDs are immutable in nature\n* RDD for a data is the mapping where the partition lies.\n\n<img src=\"https://github.com/awantik/machine-learning-slides/blob/master/rdd.PNG?raw=true\">"],"metadata":{}},{"cell_type":"code","source":["rdd = sc.parallelize([1,2,3,4,5],2)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":8},{"cell_type":"code","source":["rdd.glom().collect()"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"><span class=\"ansired\">Out[</span><span class=\"ansired\">2</span><span class=\"ansired\">]: </span>[[1, 2], [3, 4, 5]]\n</div>"]}}],"execution_count":9},{"cell_type":"markdown","source":["### 6. Spark DataFrames\n<hr>\n* Distributed tabular data structure.\n* Spark dataframes is scaleable unlike pandas dataframe.\n* They are immutable.\n* RDD has already gone into maintainance phase.\n* Spark Engine is responsible for generating optimized RDD from dataframes or spark-sql."],"metadata":{}},{"cell_type":"code","source":["df = spark.createDataFrame([(1,2),(2,3),(4,3),(7,8)],['A','B'])"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\"></div>"]}}],"execution_count":11},{"cell_type":"code","source":["display(df)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .table-result-container {\n    max-height: 300px;\n    overflow: auto;\n  }\n  table, th, td {\n    border: 1px solid black;\n    border-collapse: collapse;\n  }\n  th, td {\n    padding: 5px;\n  }\n  th {\n    text-align: left;\n  }\n</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>A</th><th>B</th></tr></thead><tbody><tr><td>1</td><td>2</td></tr><tr><td>2</td><td>3</td></tr><tr><td>4</td><td>3</td></tr><tr><td>7</td><td>8</td></tr></tbody></table></div>"]}}],"execution_count":12},{"cell_type":"code","source":[""],"metadata":{},"outputs":[],"execution_count":13}],"metadata":{"name":"Internal Details of Spark","notebookId":3707507431604184},"nbformat":4,"nbformat_minor":0}