pca_demo_databricks.ipynb

{"cells":[{"cell_type":"markdown","source":["#GPU Accelerated Principal Component Analysis (PCA) using RAPIDS on a Sample Dataset with CPU vs GPU comparison"],"metadata":{}},{"cell_type":"markdown","source":["#### Verifying GPUs\n\nRAPIDS requires GPUs with Pascal Architecture or better. That means any GPUs starting with K series (e.g. K80) or M series (e.g., M60) would not work. You can use the `nvidia-smi` command to verify the type of your GPU as well as the memory size which may be needed for some of the RAPIDS examples."],"metadata":{}},{"cell_type":"code","source":["%sh nvidia-smi"],"metadata":{},"outputs":[],"execution_count":3},{"cell_type":"markdown","source":["## Let's begin by importing RAPIDS and scikit learn libraries!"],"metadata":{}},{"cell_type":"code","source":["import numpy as np\nimport pandas as pd\nfrom sklearn.decomposition import PCA as skPCA\nfrom cuml import PCA as cumlPCA\nimport cudf\nimport os"],"metadata":{},"outputs":[],"execution_count":5},{"cell_type":"markdown","source":["## Downloading the data\nFor this example we are downloading a sample dataset (Mortgage.csv) from Nvidia's repository\n\n__We have already completed Data prep (ETL) and feature engineering on this dataset and the dataset is ready for Machine Learning__\n\n__Note:__ If you already have a dataset, please don't run the next command"],"metadata":{}},{"cell_type":"code","source":["%sh \n\nwget https://github.com/rapidsai/notebooks-extended/raw/master/data/mortgage/mortgage.csv.gz\n\ngunzip mortgage.csv.gz\nmkdir -p /dbfs/RAPIDS/mortgage\ncp mortgage.csv /dbfs/RAPIDS/mortgage/"],"metadata":{},"outputs":[],"execution_count":7},{"cell_type":"markdown","source":["## Loading the data with Spark"],"metadata":{}},{"cell_type":"code","source":["data = spark.read.csv(\"/RAPIDS/mortgage/mortgage.csv\", header = \"false\", inferSchema = \"true\", sep = \",\")"],"metadata":{},"outputs":[],"execution_count":9},{"cell_type":"markdown","source":["__Let's check out the dataset in the spark dataframe and count the number of rows in the dataset__"],"metadata":{}},{"cell_type":"code","source":["display(data)"],"metadata":{},"outputs":[],"execution_count":11},{"cell_type":"code","source":["dataCount = data.count() # We're storing this value for later use\nprint(dataCount)"],"metadata":{},"outputs":[],"execution_count":12},{"cell_type":"markdown","source":["__Helper Functions to compare CPU vs GPU results__"],"metadata":{}},{"cell_type":"code","source":["from sklearn.metrics import mean_squared_error\ndef array_equal(a,b,threshold=2e-3,with_sign=True):\n    a = to_nparray(a)\n    b = to_nparray(b)\n    if with_sign == False:\n        a,b = np.abs(a),np.abs(b)\n    error = mean_squared_error(a,b)\n    res = error<threshold\n    return res\n\ndef to_nparray(x):\n    if isinstance(x,np.ndarray) or isinstance(x,pd.DataFrame):\n        return np.array(x)\n    elif isinstance(x,np.float64):\n        return np.array([x])\n    elif isinstance(x,cudf.DataFrame) or isinstance(x,cudf.Series):\n        return x.to_pandas().values\n    return x    "],"metadata":{},"outputs":[],"execution_count":14},{"cell_type":"markdown","source":["__Helper Functions to remove any null values__"],"metadata":{}},{"cell_type":"code","source":["def null_workaround(df, **kwargs):\n    for column, data_type in df.dtypes.items():\n        if str(data_type) == \"category\":\n            df[column] = df[column].astype('int32').fillna(-1)\n        if str(data_type) in ['int8', 'int16', 'int32', 'int64', 'float32', 'float64']:\n            df[column] = df[column].fillna(-1)\n    return df"],"metadata":{},"outputs":[],"execution_count":16},{"cell_type":"markdown","source":["## Converting the Spark Dataframe into Pandas Dataframe"],"metadata":{}},{"cell_type":"markdown","source":["__Load data function allows you to create a user defined sample of your data and converts the spark dataframe to pandas dataframe.  Then, it removes any null values in the dataset.  If you want to to experiment with a different dataset sizes, use the random array generator to load the random data.__"],"metadata":{}},{"cell_type":"code","source":["def load_data(nrows, ncols):\n  try:\n    frac = nrows/dataCount # as sample() takes an integer, we are creating a factor by which to get the approximate number of rows \n    print(frac) # just for checks :)\n    if (frac > 1): \n      frac = 1.0\n    print(frac) # just for checks++ :)\n    X = data.sample(True, frac) \n    print(X)\n    df = X.toPandas() # we then convert the Spark Dataframe to Pandas.  \n    df = null_workaround(df)\n    print(\"everything worked\")\n  except Exception as e: \n    print(e)\n    print('use random data')\n    X = np.random.rand(nrows,ncols)\n    df = pd.DataFrame({'fea%d'%i:X[:,i] for i in range(X.shape[1])})\n    print(\"only random data\")\n  return df"],"metadata":{},"outputs":[],"execution_count":19},{"cell_type":"markdown","source":["__Setting up data in Pandas Dataframe using Load data and null workaround function__"],"metadata":{}},{"cell_type":"code","source":["%%time\nnrows = 2**20\nnrows = int(nrows * 1.5)\nncols = 400\n\nX = load_data(nrows,ncols)\nX = null_workaround(X)\n\n"],"metadata":{},"outputs":[],"execution_count":21},{"cell_type":"markdown","source":["# Brief Intro to PCA parameters"],"metadata":{}},{"cell_type":"markdown","source":["Let's take a look into all possible parameters that we can use when applying PCA: \nhttp://scikitlearn.org/stable/modules/generated/sklearn.decomposition.PCA.html\n\nWe will start here with the following :\n\n__n_components__ : int, float, None or string  \nNumber of components to keep. if n_components is not set all components are kept\n\n__whiten__ : bool, optional (default False) \nWhen True (False by default) the components_ vectors are multiplied by the square root of n_samples and then divided by the singular values to ensure uncorrelated outputs with unit component-wise variances. Whitening will remove some information from the transformed signal (the relative variance scales of the components) but can sometime improve the predictive accuracy of the downstream estimators by making their data respect some hard-wired assumptions\n\n__random_state__ : int, RandomState instance or None, optional (default None) \nIf int, random_state is the seed used by the random number generator\n\n__svd_solver__ : string {‘auto’, ‘full’, ‘arpack’, ‘randomized’} \nIf \"full\" :run exact full SVD calling the standard LAPACK solver via scipy.linalg.svd and select the components by postprocessing"],"metadata":{}},{"cell_type":"code","source":["n_components = 10\nwhiten = False\nrandom_state = 42\nsvd_solver=\"full\"\n"],"metadata":{},"outputs":[],"execution_count":24},{"cell_type":"markdown","source":["# Run PCA on CPU"],"metadata":{}},{"cell_type":"markdown","source":["Let's check the time needed to execute PCA function using standard sklearn library. \n__Note: this algorithm runs on CPU only.__"],"metadata":{}},{"cell_type":"code","source":["import multiprocessing\nprint(multiprocessing.cpu_count()) # Return the number of CPUs in the system."],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">8\n</div>"]}}],"execution_count":27},{"cell_type":"code","source":["%%time\npca_sk = skPCA(n_components=n_components,svd_solver=svd_solver, \n            whiten=whiten, random_state=random_state)\nresult_sk = pca_sk.fit_transform(X)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">CPU times: user 1min 3s, sys: 12.9 s, total: 1min 16s\nWall time: 13.2 s\n</div>"]}}],"execution_count":28},{"cell_type":"markdown","source":["# Run PCA on GPU"],"metadata":{}},{"cell_type":"markdown","source":["Now, before we execute PCA function using RAPIDS cuml library we will first read the data in GPU data format using cudf. \n\n__cudf__ - GPU DataFrame manipulation library https://github.com/rapidsai/cudf\n\n__cuml__ - suite of libraries that implements a machine learning algorithms within the RAPIDS data science ecosystem https://github.com/rapidsai/cuml"],"metadata":{}},{"cell_type":"code","source":["%%time\nXt = cudf.DataFrame.from_pandas(X) # Convert Pandas Dataframe to GPU Dataframe!\nprint(Xt)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">                  _c0                 _c1                   _c2         _c3                _c4  _c5  _c6 ... _c541\n 0 0.6666666865348816 0.19690021872520447 0.0045045046135783195 0.763671875 0.9972222447395325  0.0  0.0 ...   1.0\n 1 0.6666666865348816 0.19690021872520447 0.0045045046135783195 0.763671875 0.9972222447395325  0.0  0.0 ...   1.0\n 2 0.6666666865348816 0.19690021872520447 0.0045045046135783195 0.763671875 0.9972222447395325  0.0  0.0 ...   1.0\n 3 0.6666666865348816 0.19690021872520447  0.022522522136569023 0.755859375 0.9861111044883728  0.0  0.0 ...   1.0\n 4 0.6666666865348816 0.19690021872520447  0.022522522136569023 0.755859375 0.9861111044883728  0.0  0.0 ...   1.0\n 5 0.6666666865348816 0.19690021872520447  0.027027027681469917  0.75390625 0.9861111044883728  0.0  0.0 ...   1.0\n 6 0.6666666865348816 0.16271363198757172  0.036036036908626556        0.75 0.9777777791023254  0.0  0.0 ...   1.0\n 7 0.6666666865348816  0.1625978648662567   0.04054053872823715 0.748046875 0.9750000238418579  0.0  0.0 ...   1.0\n 8 0.6666666865348816 0.16248132288455963  0.045045044273138046  0.74609375 0.9722222089767456  0.0  0.0 ...   1.0\n 9 0.6666666865348816 0.16248132288455963  0.045045044273138046  0.74609375 0.9722222089767456  0.0  0.0 ...   1.0\n[166065 more rows]\n[534 more columns]\nCPU times: user 1.49 s, sys: 380 ms, total: 1.87 s\nWall time: 1.87 s\n</div>"]}}],"execution_count":31},{"cell_type":"code","source":["%%time\npca_cuml = cumlPCA(n_components=n_components,svd_solver=svd_solver, \n            whiten=whiten, random_state=random_state)\nresult_cuml = pca_cuml.fit_transform(Xt)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">CPU times: user 2.1 s, sys: 104 ms, total: 2.21 s\nWall time: 2.2 s\n</div>"]}}],"execution_count":32},{"cell_type":"code","source":["for attr in ['singular_values_','components_','explained_variance_',\n             'explained_variance_ratio_']:\n    passed = array_equal(getattr(pca_sk,attr),getattr(pca_cuml,attr))\n    message = 'compare pca: cuml vs sklearn {:>25} {}'.format(attr,'equal' if passed else 'NOT equal')\n    print(message)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">compare pca: cuml vs sklearn          singular_values_ equal\ncompare pca: cuml vs sklearn               components_ equal\ncompare pca: cuml vs sklearn       explained_variance_ equal\ncompare pca: cuml vs sklearn explained_variance_ratio_ equal\n</div>"]}}],"execution_count":33},{"cell_type":"code","source":["# Spark ML accelerated with RAPIDS\npassed = array_equal(result_sk,result_cuml)\nmessage = 'compare pca: cuml vs sklearn transformed results %s'%('equal'if passed else 'NOT equal')\nprint(message)"],"metadata":{},"outputs":[{"metadata":{},"output_type":"display_data","data":{"text/html":["<style scoped>\n  .ansiout {\n    display: block;\n    unicode-bidi: embed;\n    white-space: pre-wrap;\n    word-wrap: break-word;\n    word-break: break-all;\n    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n    font-size: 13px;\n    color: #555;\n    margin-left: 4px;\n    line-height: 19px;\n  }\n</style>\n<div class=\"ansiout\">compare pca: cuml vs sklearn transformed results equal\n</div>"]}}],"execution_count":34}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"mimetype":"text/x-python","name":"python","pygments_lexer":"ipython3","codemirror_mode":{"name":"ipython","version":3},"version":"3.6.6","nbconvert_exporter":"python","file_extension":".py"},"name":"pca_demo","notebookId":4448159764540247},"nbformat":4,"nbformat_minor":0}