BIDS_05_Clustering_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aim to complete as much of this tutorial on your own *before* coming to the practical session.\n",
    "\n",
    "Use the practical session to get help for any aspect you do not understand or were unable to complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q6CiSLUhhlLj"
   },
   "source": [
    "# Clustering 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jZyTDkUchlOo"
   },
   "source": [
    "Learning objectives\n",
    "1. Apply k-Means, HCA, and kNN using the popular python library [sklearn](https://scikit-learn.org/stable/)\n",
    "2. Interpret the results to learn about the data structure \n",
    "3. Visualise the clustering results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import specific packages and functions\n",
    "You should have `numpy`, `pandas`, `matplotlib` and `sklearn` already installed from the previous sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd ## pandas creates data frames/ data tables similar to an excel file, although most python functions like the fromat of a numpy array\n",
    "import numpy as np ### for working with numpy arrays\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.cluster import KMeans, AgglomerativeClustering\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.manifold import MDS\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import silhouette_score, silhouette_samples\n",
    "\n",
    "from scipy.cluster.hierarchy import dendrogram\n",
    "\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from matplotlib.colors import colorConverter\n",
    "#%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IP8bBL25j2Iy"
   },
   "source": [
    "## Load in dataset\n",
    "In this tutorial we will aim to predict the occurence of COVID-19 using the same data used in BIDS 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cYkRLm2A5OLF"
   },
   "outputs": [],
   "source": [
    "# Use pandas to load the data\n",
    "\n",
    "# Load the data\n",
    "#df = pd.read_csv('../Data-main/COVID19_proteomics.xlsx', index_col=0) ## path to file, may not need the \"index_col=0\" paramater depending on operating system\n",
    "df = pd.read_excel(\"../Data-main/COVID19_proteomics.xlsx\")\n",
    "# View the first 5 records to see what features we have in our dataset should start with first column called 'COVID19'\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 231
    },
    "id": "vT2Qht5b5O9a",
    "outputId": "93f199a3-4157-489a-8cb3-4661d5bde85d"
   },
   "source": [
    "As you can see this data set in its current format is not suitable for our algorithms. \n",
    "In python column indices start from 0, we want to select only the proteomics columns for feature scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to predict COVID19 (column 0 ) using columns 3:453"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fXZJ5J6UTg96"
   },
   "source": [
    "As we can see, the data is now in the proper format for our KNN models.\n",
    "\n",
    "Now that out data has been pre-processed, we can seperate the target from the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vRz5ZU-xAmhp"
   },
   "outputs": [],
   "source": [
    "# Create feature matrix and target vector\n",
    "X = df.iloc[:,3:]\n",
    "y = df['COVID19']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "je7uqh3STg6l"
   },
   "source": [
    "In order to test our alogrithms we need to set aside some of the data we have. This is practice for machine learning models. We will use 80% of our data to train our model, and the remaining 20% will be used to test the performance of our model. \n",
    "\n",
    "[sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) has a function ```train_test_split``` to easily do this for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter your CID here, or date of birth, or another number of your choosing to use as random state\n",
    "CID = 0\n",
    "\n",
    "# remember to check the documentation of each algorithm if setting the random_state is needed\n",
    "# for this tutorial (where it is clearly being used) and all upcoming tutorials..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "REFpFQiRBomE"
   },
   "outputs": [],
   "source": [
    "# Split the df into 80% train 20% test data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.20, random_state=CID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PNuyfPvPTg3D"
   },
   "source": [
    "### Normalise Data\n",
    "Our data has been cleaned and split into training and testing features and targets. We have one last pre-processing step to take before being able to run our data: scaling. As we mentioned before, these methods (k-Means, kNN, etc.) make predictions based on distances between data points. As such, it is crucial that all of the data it is comparing is on the same scale. In our proteomics data, most of the data is continuous. We will scale the data using the ```StandardScaler()``` shown in the previous tutorials. \n",
    "\n",
    "When scaling your data you want to fit the model to your training data, and only transform your testing data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KiNEUPPe2JwF"
   },
   "outputs": [],
   "source": [
    "# Instantiate scaler model\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit and Transform X_train\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "\n",
    "# Transform X_test\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Means Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8, 8))\n",
    "plt.scatter(X.iloc[:,0], X.iloc[:,1]) ### plotting first two proteomics features\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=3, init='k-means++', max_iter=300, n_init=10, random_state=CID) ### kmeans clustering \n",
    "kmeans.fit(X) ## fit on X and define the random state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do we pick the number of clusters? \n",
    "K-means clustering requires that the number of clusters are predefined, however we would want justify how many clusters are *optimal*. We can treat ```n_clusters``` as a hyperparamater and for loop through several number of clusters. To select the best number of clusters we will implement the elbow method, where we train multiple models using a different number of clusters and storing the value of the ```intertia_``` property, the Within Cluster Sum of Squares (WCSS), every time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WCSS is defined as the sum of the squared distance between each member of the cluster and its centroid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wcss = [] ### create a list \n",
    "\n",
    "for i in range(2, 12): ## loop through n_clusters 2:11\n",
    "    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=CID)\n",
    "    kmeans.fit(X_train) ## fit on X and define the random state\n",
    "    wcss.append(kmeans.inertia_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(range(2, 12), wcss)\n",
    "plt.title('Elbow Method')\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('WCSS')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many clusters would you pick? We've chosen 4 below, but please change it to what you think is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=4, init='k-means++', max_iter=300, n_init=10, random_state=CID)\n",
    "pred_y = kmeans.fit_predict(X_train)\n",
    "plt.scatter(X_train[:,0], X_train[:,1], c=pred_y, s=50, cmap='viridis')\n",
    "plt.scatter(kmeans.cluster_centers_[:, 0],kmeans.cluster_centers_[:, 1], s=300, c='red')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like there are 4 clusters at the shapest elbow point. However, when plotting just the first two proteomics features it is not clear and their is overlap of the clusters.\n",
    "However, we can use alternative methods to visualise these clusters. One of these is PCA from the BIDS2 tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=4)\n",
    "pca_covid = pca.fit_transform(X_train)\n",
    "# run PCA with 4 components\n",
    "# plot a scatterplot using seaborn\n",
    "# the x axis will contain the first column of the pca scores x=pca_covid[:, 0]\n",
    "plt.scatter(pca_covid[:, 0], pca_covid[:, 1], c=pred_y, s=50, cmap='viridis')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do this with other dimension reduction techniques such as MDS, t-SNE, UMAP, etc. (see tutorials and optional materials from BIDS 2-4 for code to reuse here).\n",
    "\n",
    "For example for MDS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding = MDS(n_components=4, metric=True, random_state=CID)\n",
    "embedding_covid = embedding.fit_transform(X_train)\n",
    "plt.scatter(embedding_covid[:, 0], embedding_covid[:, 1], c=pred_y, s=50, cmap='viridis')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, is there a way to plot all 4 components from the MDS instead of only the first 2...?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Reuse code from a previous tutorial here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hierarchical Clustering (or Hierarchical Cluster Analysis, HCA)\n",
    "\n",
    "This is similar to K-means although this time we wil use the ```AgglomerativeClustering``` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = AgglomerativeClustering(n_clusters=2, affinity='euclidean', linkage='ward')\n",
    "cluster.fit(X_train)\n",
    "labels = cluster.fit_predict(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "silhouette_score(X_train,labels,metric=\"euclidean\",sample_size=1000,random_state=CID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To mix things up we'll use a different colourmap ([cmap](https://matplotlib.org/stable/gallery/color/colormap_reference.html)), have a look at different ones in [Matplotlib](https://matplotlib.org/stable/) and see which one you prefer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## visualise as a scatter plot with the PCA components\n",
    "plt.scatter(pca_covid[:, 0], pca_covid[:, 1], c=cluster.labels_, s=50, cmap='rainbow')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Like in the K-means approach, defining the *optimal* number of clusters requires iterating over ```n_clusters```. This time we will implement the silhouette method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same challenge occours, where we need to define the ```n_clusters```. This time we will use the silhouette score to detemine optimal number of clusters. The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The value of the silhouette ranges between [1, -1], where a high value indicates that the object is well matched to its own cluster and poorly matched to neighbouring clusters. If most objects have a high value, then the clustering configuration is appropriate. If many points have a low or negative value, then the clustering configuration may have too many or too few clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Silhoutte plot function no need to change this it's a custom function to obtain silhouete plots\n",
    "def silhouette_plot(X, y, n_clusters):\n",
    "\n",
    "    # Compute the silhouette scores for each sample\n",
    "    silhouette_avg = silhouette_score(X, y)\n",
    "    sample_silhouette_values = silhouette_samples(X, y)\n",
    "\n",
    "    y_lower = padding = 2\n",
    "    for i in range(n_clusters):\n",
    "        ax = plt.gca()\n",
    "        # Aggregate the silhouette scores for samples belonging to\n",
    "        ith_cluster_silhouette_values = sample_silhouette_values[y == i]\n",
    "        ith_cluster_silhouette_values.sort()\n",
    "\n",
    "        size_cluster_i = ith_cluster_silhouette_values.shape[0]\n",
    "        y_upper = y_lower + size_cluster_i\n",
    "        cmap = cm.get_cmap(\"Spectral\")\n",
    "        color = cmap(float(i) / n_clusters)\n",
    "        ax.fill_betweenx(np.arange(y_lower, y_upper),\n",
    "                         0,\n",
    "                         ith_cluster_silhouette_values,\n",
    "                         facecolor=color,\n",
    "                         edgecolor=color,\n",
    "                         alpha=0.7)\n",
    "\n",
    "        # Label the silhouette plots with their cluster numbers at the middle\n",
    "        ax.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i + 1))\n",
    "\n",
    "        # Compute the new y_lower for next plot\n",
    "        y_lower = y_upper + padding\n",
    "    ax.set_xlabel(\"The silhouette coefficient values\")\n",
    "    ax.set_ylabel(\"Cluster label\")\n",
    "\n",
    "    # The vertical line for average silhoutte score of all the values\n",
    "    ax.axvline(x=silhouette_avg, c='r', alpha=0.8, lw=0.8, ls='-')\n",
    "    ax.annotate('Average',\n",
    "                xytext=(silhouette_avg, y_lower * 1.025),\n",
    "                xy=(0, 0),\n",
    "                ha='center',\n",
    "                alpha=0.8,\n",
    "                c='r')\n",
    "\n",
    "    ax.set_yticks([])  # Clear the yaxis labels / ticks\n",
    "    ax.set_xticks([0, 0.2, 0.4, 0.6, 0.8, 1])\n",
    "    ax.set_ylim(0, y_upper + 1)\n",
    "    ax.set_xlim(-0.075, 1.0)\n",
    "    plt.figure()\n",
    "    return ax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "silho = [] \n",
    "for i in range(2,11):\n",
    "    # Run the HCC algorithm\n",
    "    cluster = AgglomerativeClustering(n_clusters=i, affinity='euclidean', linkage='ward')\n",
    "    cluster.fit(X_train)\n",
    "    labels = cluster.fit_predict(X_train)\n",
    "    silhouette_plot(X_train, labels, n_clusters=i)\n",
    "    silho.append(silhouette_score(X_train,labels,metric=\"euclidean\",sample_size=1000,random_state=CID))\n",
    "plt.plot(range(2, 11), silho)\n",
    "plt.title('Silhouette Method')\n",
    "plt.xlabel('Number of clusters')\n",
    "plt.ylabel('Avg. Silhouette Coeficient')\n",
    "plt.show()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that the optimal K (the one with the highest average Silhoutte coeficient) is 2. However, often averages can be deceiving (e.g. outliers with either very low coeficients or extremely high coeficients can drastically change the average). With the silhouette method it is a good idea to examine the coeficients of each sample in each cluster independently as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did with the k-means approach we can also visualise these clusters, as a scatter plot, using various techniques. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### edit this to show the two clusters\n",
    "cluster = AgglomerativeClustering(n_clusters=3, affinity='euclidean', linkage='ward')\n",
    "cluster.fit(X_train)\n",
    "labels = cluster.fit_predict(X_train)\n",
    "\n",
    "# plot a scatterplot using seaborn\n",
    "# the x axis will contain the first column of the pca scores x=pca_covid[:, 0]\n",
    "plt.scatter(pca_covid[:, 0], pca_covid[:, 1], c=labels, s=50, cmap='viridis')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most well-known methods of visualisation for hierarchical clustering is by means of a [dendrogram](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### this helper function is required, no need to edit this, you may need to also install scipy if this is not installed yet\n",
    "def plot_dendrogram(model, **kwargs):\n",
    "    # Create linkage matrix and then plot the dendrogram\n",
    "\n",
    "    # create the counts of samples under each node\n",
    "    counts = np.zeros(model.children_.shape[0])\n",
    "    n_samples = len(model.labels_)\n",
    "    for i, merge in enumerate(model.children_):\n",
    "        current_count = 0\n",
    "        for child_idx in merge:\n",
    "            if child_idx < n_samples:\n",
    "                current_count += 1  # leaf node\n",
    "            else:\n",
    "                current_count += counts[child_idx - n_samples]\n",
    "        counts[i] = current_count\n",
    "\n",
    "    linkage_matrix = np.column_stack(\n",
    "        [model.children_, model.distances_, counts]\n",
    "    ).astype(float)\n",
    "\n",
    "    # Plot the corresponding dendrogram\n",
    "    dendrogram(linkage_matrix, **kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting distance_threshold=0 ensures we compute the full tree.\n",
    "model = AgglomerativeClustering(distance_threshold=0, n_clusters=None)\n",
    "\n",
    "model = model.fit(X)\n",
    "plt.title(\"Hierarchical Clustering Dendrogram\")\n",
    "# plot the top three levels of the dendrogram\n",
    "plot_dendrogram(model, truncate_mode=\"level\", p=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why does the plot below look different? What is the difference between `level` and `lastp`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the top three clusters of the dendrogram\n",
    "plot_dendrogram(model, truncate_mode=\"lastp\", p=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are there any outliers? How does this compare with outliers detected using PCA for example?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "grqIZnvtjyX3"
   },
   "source": [
    "## k-Nearest Neighbours (kNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e8u-GV_FOw9R"
   },
   "source": [
    "### Scikit-Learn's kNN \n",
    "\n",
    "\n",
    "Let's apply the [kNN](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) algorithm to a Covid-19 proteomics data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-jP41SdNj2F-"
   },
   "source": [
    "### Theory Recap (Optional)\n",
    "k-Nearest Neighbours is a simple machine learning model that makes predictions based off of the most similar observations in its traing data. While it can be very powerful, its predictive capability is limited to observations that are similar to what the training data it has in memory.\n",
    "\n",
    "Unlike most other models, kNN does not 'learn' from its training dataset. Instead it holds the entire training set in memory and then compares the new observation to its stored data. kNN performs no work until a prediction is required.\n",
    "\n",
    "When a prediction is required it does exactly what it name says. The model examines the new observation and finds the most similar records (nearest neighbours) that it holds in its training set. The number of neighbours (k) the model selects from its training data is defined by the user. \n",
    "\n",
    "A prediction can be made by either returning the most common outcome (classification) or by taking the average (regression).\n",
    "\n",
    "**Some Important Notes on Using kNN**\n",
    "\n",
    "- kNN is a simple model to implement, but as a result it is limited in the types of data it can take as input.\n",
    "- When working with kNN the phrase \"rubbish in, rubbish out\" is never more accurate.\n",
    "- kNN does not handle categorical variables so everything must be pre-processed to include numerical values only.\n",
    "- Additionally, as you will see in the next section, the nearest neighbours are found by calculating the distances between the new observation and the records held in memory. Those with the smallest distances are considered most similar.\n",
    "- Intuitively, you should understand the importance of scaling your data (so that they are all being measured on the same metric) before running a kNN model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TqK97yK1663t"
   },
   "source": [
    "If you have kept with us this far, it is about to pay off, it's time to run some models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uhUfSBwuHhqT"
   },
   "source": [
    "### Baseline Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aLXddWpHHj_V"
   },
   "source": [
    "When evaluating model performance we want to start with a baseline accuracy. This is the accuracy score if we were to simply guess the majority outcome everytime. It gives us a starting point to compare our models to. The baseline metric is the best we can do without models. Hopefully, our models can improve over the baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "eQaX0bkoGsBX",
    "outputId": "1feb1648-9dd3-4811-adc7-8b8bad22b415"
   },
   "outputs": [],
   "source": [
    "# Calculate the baseline accuracy\n",
    "\n",
    "# Find the majority count\n",
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "OW2gvCnS1QuK",
    "outputId": "9e7a7d00-6093-4da2-fceb-eaef8d528c48"
   },
   "outputs": [],
   "source": [
    "# COVID counts for y_test\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above numbers depend on the `random_state` of the `train_test_split` function. So you may see a difference compared to your _neighbour_ (pardon the pun).\n",
    "\n",
    "Change the cell below to the number from your test set - which group has more samples?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "8flMi2_sG3z7",
    "outputId": "1984b964-fc79-4884-8c03-3f967a9ef32a"
   },
   "outputs": [],
   "source": [
    "# If we were to guess the majority (1 or 0) for each test sample, we would get a total of ... correct\n",
    "# The baseline accuracy is the number of correct guesses divided by total guesses \n",
    "baseline = np.max(y_test.value_counts()) / np.sum(y_test.value_counts())\n",
    "baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CHkE4Hx2Tg0l"
   },
   "source": [
    "### COVID predictions using a kNN classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RJ8rWaEjDZBg"
   },
   "source": [
    "Let's start by making predictions and calculating the accuracy of Scikit-Learn's [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) model.\n",
    "\n",
    "This can be done by the following steps:\n",
    "\n",
    "- Instantiate the model\n",
    "- Fit the model with our training data\n",
    "- Make predictions based off of our test features\n",
    "- Provide the known test targets to determine the accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 123
    },
    "id": "cbxHRJd7Dapt",
    "outputId": "a9acf3b8-917c-470c-aad7-eb5365473bde"
   },
   "outputs": [],
   "source": [
    "# Instantiate the KNClassifier object\n",
    "scikit_KNN = KNeighborsClassifier(n_neighbors=9)\n",
    "\n",
    "# Fit the model with training data\n",
    "scikit_KNN.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "print(scikit_KNN.predict(X_test))\n",
    "\n",
    "# Calculate accuracy\n",
    "scikit_KNN.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Accuracy is one performance metric here we will define several alternative metrics \n",
    "\n",
    "def modelPerformance(confMat):\n",
    "    TN = confMat[0, 0]\n",
    "    TP = confMat[1, 1]\n",
    "    FP = confMat[0, 1]\n",
    "    FN = confMat[1, 0]\n",
    "    prec = TP / (TP + FP)\n",
    "    rec = TP / (TP + FN)\n",
    "    spec = TN / (TN + FP)\n",
    "    fpr = FP / (TN + FP)\n",
    "    f1 = 2 * (prec * rec) / (prec + rec)\n",
    "    acc = (TP + TN) / (TP + FP + TN + FN)\n",
    "    return (acc, prec, rec, spec, fpr, f1)\n",
    "\n",
    "def printPerformance(confMat):\n",
    "    acc, prec, rec, spec, fpr, f1 = modelPerformance(confMat)\n",
    "    print(\"Accuracy = \" \"%.4f\" % acc)\n",
    "    print(\"Precision = \" \"%.4f\" % prec)\n",
    "    print(\"Recall = \" \"%.4f\" % rec)\n",
    "    print(\"Specificity = \" \"%.4f\" % spec)\n",
    "    print(\"False positive rate = \" \"%.4f\" % fpr)\n",
    "    print(\"F1-score = \" \"%.4f\" % f1)\n",
    "    np.set_printoptions(precision=2)\n",
    "    print(\"Confusion matrix (%):\")\n",
    "    print(confMat/np.sum(confMat)*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_hat = scikit_KNN.predict(X_test)\n",
    "cmat = metrics.confusion_matrix(y_test, y_test_hat)\n",
    "printPerformance(cmat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bnuVNjFBFb88"
   },
   "source": [
    "Complete the sentence below with your results:\n",
    "\n",
    "Considering the baseline accuracy was ...%, sklearn's model is an improvement/reduction at ...%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xzShDL9iDY8u"
   },
   "source": [
    "### What's next?\n",
    "We have gone through how to implement a kNN model that will work with either classification or regression problems. kNN is included here as it assignes a cluster (class or, more broadly, a _value_) based on its similarity to other samples in the neighbourhood (which can be considered a cluster). Then we walked through a classification problem using the COVID-19 proteomics dataset.\n",
    "\n",
    "For further understanding and practice:\n",
    "- Plot kNN results like above \n",
    "- Cluster a different data set using HCA and k-means\n",
    "- Use a different dataset for a regression problem, look up: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html\n",
    "- Implement different distance metrics such as the Manhattan distance or Hamming distance and see how they change results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your turn\n",
    "Select another dataset from the `Data` folder, import and inspect it.\n",
    "\n",
    "Are the same parameters as used above optimal for the new dataset? Do different parameters changes the outcome?\n",
    "\n",
    "Can you identify some potential outliers with these methods?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import datasets...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform scaling (if needed)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying some of the clustering methods\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualise your results using some of the dimension reduction techniques (for k-Means and kNN) or via a dendrogram (HCA)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "KNN_Algorithm.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}