advanced ml.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced machine learning\n",
    "#### Decision Trees: \n",
    "Decision Trees are a flowchart-like type of Supervised Machine Learning where the data is continuously split according to a certain parameter. They are easy to understand and interpret, which is one of their biggest advantages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=7)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# Initialize the model\n",
    "clf = DecisionTreeClassifier()\n",
    "\n",
    "# Fit the model to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Predict on the test data\n",
    "predictions = clf.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest: \n",
    "Random Forest is a popular machine learning algorithm that belongs to the supervised learning technique. It can be used for both Classification and Regression problems in ML. It is based on the concept of ensemble learning, which is a process of combining multiple algorithms to solve a particular problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Initialize the model\n",
    "forest = RandomForestClassifier(n_estimators=100, random_state=42)\n",
    "\n",
    "# Fit the model\n",
    "forest.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "forest_predictions = forest.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Naive Bayes: \n",
    "Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "# Initialize the model\n",
    "nb = GaussianNB()\n",
    "\n",
    "# Fit the model\n",
    "nb.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "nb_predictions = nb.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### K-Nearest Neighbors (KNN): \n",
    "K-Nearest Neighbors is one of the most basic yet essential classification algorithms in Machine Learning. It belongs to the supervised learning domain and finds intense application in pattern recognition, data mining and intrusion detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# Initialize the model\n",
    "knn = KNeighborsClassifier(n_neighbors=5)\n",
    "\n",
    "# Fit the model\n",
    "knn.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "knn_predictions = knn.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Support Vector Machines (SVM): \n",
    "SVM is a supervised machine learning algorithm which can be used for both classification or regression challenges. However, it is mostly used in classification problems. In the SVM algorithm, we plot each data item as a point in n-dimensional space with the value of each feature being the value of a particular coordinate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "# Initialize the model\n",
    "svc = svm.SVC(kernel='linear')\n",
    "\n",
    "# Fit the model\n",
    "svc.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "svc_predictions = svc.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ensemble methods\n",
    "Ensemble methods are techniques that combine predictions from multiple machine learning algorithms to deliver more accurate predictions than a single model.\n",
    "##### Bagging\n",
    "Bagging, or Bootstrap Aggregating, involves taking multiple subsets of your original dataset, building a separate model for each subset, and then combining the output of all these models. For instance, the Random Forest algorithm is a type of bagging method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "bagging = BaggingClassifier(KNeighborsClassifier(), max_samples=0.5, max_features=0.5)\n",
    "bagging.fit(X_train, y_train)\n",
    "predictions = bagging.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Boosting\n",
    "Boosting works by training a model, identifying the mistakes it made, and then building a new model that focuses on the mistakes of the first model. This process is repeated, each time focusing on the mistakes of the last model, until a combined model with a low error rate is obtained. An example is the AdaBoost algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "adaboost = AdaBoostClassifier(n_estimators=100)\n",
    "adaboost.fit(X_train, y_train)\n",
    "predictions = adaboost.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Stacking\n",
    "Stacking involves training multiple different models and then using another machine learning model to combine their outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import StackingClassifier\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "estimators = [('rf', RandomForestClassifier(n_estimators=10, random_state=42)),\n",
    "              ('svr', SVC(random_state=42))]\n",
    "\n",
    "stacking = StackingClassifier(estimators=estimators, final_estimator=LogisticRegression())\n",
    "stacking.fit(X_train, y_train)\n",
    "predictions = stacking.predict(X_test)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Overfitting and Underfitting:\n",
    "\n",
    "Overfitting and underfitting refer to the phenomena when a machine learning model performs well on the training data but poorly on the test data (overfitting), or when it performs poorly on both the training data and the test data (underfitting).\n",
    "\n",
    "* Overfitting occurs when the model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. It means the model has learned the data \"too well\".\n",
    "* Underfitting occurs when a machine learning model cannot capture the underlying pattern of the data. These models usually have poor predictive performance.\n",
    "\n",
    "In essence, underfitting is a model with high bias (it makes strong assumptions and oversimplifies the problem), and overfitting is a model with high variance (it models the random noise in the training data, not the intended outputs).\n",
    "\n",
    "Let's demonstrate underfitting and overfitting using the decision tree algorithm. We will use the depth of the tree as our tuning parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import make_moons\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Create a moon-shaped, noisy dataset\n",
    "X, y = make_moons(n_samples=300, noise=0.25, random_state=42)\n",
    "\n",
    "# Split the dataset\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Let's set the tree depth to 1 (A stump)\n",
    "stump = DecisionTreeClassifier(max_depth=1, random_state=42)\n",
    "stump.fit(X_train, y_train)\n",
    "\n",
    "# Now let's set the tree depth to 6\n",
    "tree = DecisionTreeClassifier(max_depth=6, random_state=42)\n",
    "tree.fit(X_train, y_train)\n",
    "\n",
    "# And finally, let's not limit the tree depth\n",
    "deep_tree = DecisionTreeClassifier(max_depth=None, random_state=42)\n",
    "deep_tree.fit(X_train, y_train)\n",
    "\n",
    "# Now let's test the accuracy of each model on the training data and test data\n",
    "models = [stump, tree, deep_tree]\n",
    "for model in models:\n",
    "    y_train_pred = model.predict(X_train)\n",
    "    y_test_pred = model.predict(X_test)\n",
    "    print(f'Model with max_depth={model.max_depth}:')\n",
    "    print(f'Training accuracy: {accuracy_score(y_train, y_train_pred)}')\n",
    "    print(f'Test accuracy: {accuracy_score(y_test, y_test_pred)}\\n')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the output, you might observe that the model with a depth of 1 (the stump) performs poorly on both the training and test sets. This is a classic case of underfitting.\n",
    "\n",
    "On the other hand, the model with no limit on the tree depth might perform very well on the training set but poorly on the test set. This is a classic case of overfitting.\n",
    "\n",
    "The model with a depth of 6 might give the best results, as it might strike a balance and perform well on both the training set and the test set.\n",
    "\n",
    "One way to fine-tune the trade-off between underfitting and overfitting is to use cross-validation to find the optimal model complexity. Moreover, many models have regularization parameters that can be adjusted to avoid overfitting."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unsupervised learning\n",
    "Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses. The most common unsupervised learning method is cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data. Another common technique is dimensionality reduction, which attempts to reduce the number of features in a dataset while preserving as much statistical information as possible.\n",
    "\n",
    "#### Clustering\n",
    "##### DBSCAN\n",
    "Density-Based Spatial Clustering of Applications with Noise, or DBSCAN, is a density-based clustering algorithm, which has the concept of core samples of high density and expands clusters from them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.cluster import DBSCAN\n",
    "\n",
    "X, y = make_moons(n_samples=200, noise=0.05, random_state=0)\n",
    "\n",
    "dbscan = DBSCAN(eps=0.3, min_samples=5)\n",
    "dbscan.fit(X)\n",
    "\n",
    "# Plot the cluster assignments\n",
    "plt.scatter(X[:, 0], X[:, 1], c=dbscan.labels_)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### LDA (Latent Dirichlet Allocation): \n",
    "This is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. It's most commonly used for natural language processing tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Sample data\n",
    "data = ['This is the first document.','This document is the second document.','And this is the third one.','Is this the first document?']\n",
    "\n",
    "vectorizer = CountVectorizer()\n",
    "X = vectorizer.fit_transform(data)\n",
    "\n",
    "lda = LatentDirichletAllocation(n_components=2, random_state=0)\n",
    "lda.fit(X)\n",
    "\n",
    "# Displaying topics\n",
    "for idx, topic in enumerate(lda.components_):\n",
    "    print(\"Topic %d:\" % (idx))\n",
    "    print([(vectorizer.get_feature_names()[i], topic[i])\n",
    "                    for i in topic.argsort()[:-10 - 1:-1]])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dimensionality Reduction\n",
    "\n",
    "##### PCA (Principal Component Analysis): \n",
    "This is a technique used for feature extraction. It combines our input variables in a specific way, and we can drop the “least important” variables while still retaining the most valuable parts of all of the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Assume X is your matrix with shape (n_samples, n_features)\n",
    "pca = PCA(n_components=2)\n",
    "X_pca = pca.fit_transform(X)\n",
    "\n",
    "plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### t-SNE (t-Distributed Stochastic Neighbor Embedding): \n",
    "This is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# Again, assume X is your matrix with shape (n_samples, n_features)\n",
    "tsne = TSNE(n_components=2)\n",
    "X_tsne = tsne.fit_transform(X)\n",
    "\n",
    "plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For both PCA and t-SNE examples, the input X should be a matrix with shape (n_samples, n_features). y is only used for coloring the points in the plot, and represents the true labels of the samples."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Networks and Deep Learning\n",
    "Neural networks are a set of algorithms modeled after the human brain, that are designed to recognize patterns. They interpret sensory data through a kind of machine perception, labeling or clustering raw input. Deep Learning is a subfield of machine learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks.\n",
    "\n",
    "In more concrete terms, deep learning is the name for multilayered neural networks, which are networks composed of several \"layers\" of nodes — connected in a \"deep\" structure.\n",
    "\n",
    "Here's an example of a simple feedforward neural network built with PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# Set up a simple feedforward neural network\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(20, 50)  # 20 input units to 50 hidden units\n",
    "        self.fc2 = nn.Linear(50, 1)  # 50 hidden units to 1 output unit\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.fc1(x))  # Activation function for hidden layer\n",
    "        x = self.fc2(x)  # No activation function for output layer\n",
    "        return x\n",
    "\n",
    "# Assume we have some data in X and targets in Y\n",
    "X = torch.randn(100, 20)\n",
    "Y = torch.randn(100, 1)\n",
    "\n",
    "# Instantiate the network, loss function and optimizer\n",
    "net = Net()\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
    "\n",
    "# Train the network\n",
    "for epoch in range(100):  # loop over the dataset multiple times\n",
    "    # zero the parameter gradients\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    # forward + backward + optimize\n",
    "    outputs = net(X)\n",
    "    loss = criterion(outputs, Y)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "print('Finished Training')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this code, we define a simple network with one hidden layer and one output layer. The hidden layer uses the ReLU activation function. We're training this network to minimize the mean squared error between its outputs and the target values Y. We're using stochastic gradient descent (SGD) as our optimization algorithm.\n",
    "\n",
    "This is a simple example, but deep learning can get much more complex! In real-world scenarios, we often use much larger networks and train them on big datasets. This requires more computational resources (especially GPUs), and more sophisticated techniques for managing data and training dynamics.\n",
    "\n",
    "Additionally, you would typically want to separate your data into a training set and a validation set, so that you can monitor your network's performance on unseen data as it trains, and prevent overfitting.\n",
    "\n",
    "Finally, deep learning also includes other types of architectures, such as convolutional neural networks (CNNs) for image tasks, recurrent neural networks (RNNs) for sequential data, transformers for natural language processing, autoencoders for unsupervised learning, and many more. Deep learning is my bread and butter, so please check out other corners of my GitHub to learn more about these fascinating structures!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorchenv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.7.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}