10_Evaluating_model_performance_using_cross_validation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "10. Evaluating model performance using cross validation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPPcwHjr/4Mblc0QyvBBDZ7",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Sicily-F/cagedbirdID/blob/main/10_Evaluating_model_performance_using_cross_validation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXLgHKquXdEH"
      },
      "source": [
        "# 10. Evaluating model performance using cross validation\n",
        "\n",
        "If the performance of the single train-test split can be widely retained after repeated testing, this suggests that the model generalizes reasonably within the domain tested, which in this case was primarily wildlife markets. However, it is possible to accidentally train on a subset that does not reflect a real-world situation. The validation or test datasets could consist of easier examples than the training dataset. By training the model on randomly shuffled training, validation, and test datasets, any tendencies towards overfitting can be avoided.\n",
        "\n",
        "\n",
        "Here we show an example of cross-validation. This can be done with any of your model, though we show a vision transformer model here. Essentially it's the same code as file 8., just in one giant for loop!\n",
        "\n",
        "There are other ways to perform kfold validation using the [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html) package, and ours is certainly not the fastest. However, given that we used the TensorFlow ImageDataGenerator, we had to use this version, so we could preserve our initial folder structure.\n",
        "\n",
        "\n",
        "We wrote a *for loop* to split our intial data (a folder named 'new_all_species', which had a folder for each 37 species), into 5 different folders, each which represents a randomised complication of the original data with 70% for training, 15% for validation and 15% for testing. We used the amazing [splitfolders](https://pypi.org/project/split-folders/) package to do this!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PZwP83doahG5"
      },
      "source": [
        "import splitfolders\n",
        "\n",
        "# This runs the loop 5 times\n",
        "m = 5\n",
        "for i in range(m):\n",
        "    input_folder = 'F:/new_all_species/'\n",
        "    output = 'F:/refolds/'+ str(i)\n",
        "    splitfolders.ratio(input_folder, output=output, seed=None, ratio=(.7, .15, .15), group_prefix=None) \n",
        "\n",
        "\n",
        "  # This results in 5 folders named 0,1,2,3,4\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qqlr0ZDvXZT8"
      },
      "source": [
        "import numpy as np\t\n",
        "import pandas as pd\n",
        "from pandas import DataFrame\n",
        "import time\n",
        "import PIL.Image as Image\n",
        "import matplotlib.pylab as plt\n",
        "import tensorflow as tf\n",
        "import tensorflow_addons as tfa\n",
        "import PIL\n",
        "import pathlib\n",
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array, load_img\n",
        "from tensorflow.keras.applications import DenseNet121\n",
        "from tensorflow.keras.models import Model, Sequential, load_model\n",
        "from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Flatten, Dropout\n",
        "import os\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.callbacks import Callback, EarlyStopping, ModelCheckpoint,ReduceLROnPlateau\n",
        "\n",
        "RANDOM_SEED = 1\n",
        "IMAGE_SIZE = 224\n",
        "BATCH_SIZE = 16\n",
        "\n",
        "import numpy as np\n",
        "def get_random_eraser(p=0.5, s_l=0.02, s_h=0.4, r_1=0.3, r_2=1/0.3, v_l=0, v_h=255, pixel_level=True): # was False before, as true still doesn't help me do the pixel level implementation \n",
        "    def eraser(input_img):\n",
        "        if input_img.ndim == 3:\n",
        "            img_h, img_w, img_c = input_img.shape\n",
        "        elif input_img.ndim == 2:\n",
        "            img_h, img_w = input_img.shape\n",
        "\n",
        "        p_1 = np.random.rand()\n",
        "\n",
        "        if p_1 > p:\n",
        "            return input_img\n",
        "\n",
        "        while True:\n",
        "            s = np.random.uniform(s_l, s_h) * img_h * img_w\n",
        "            r = np.random.uniform(r_1, r_2)\n",
        "            w = int(np.sqrt(s / r))\n",
        "            h = int(np.sqrt(s * r))\n",
        "            left = np.random.randint(0, img_w)\n",
        "            top = np.random.randint(0, img_h)\n",
        "\n",
        "            if left + w <= img_w and top + h <= img_h:\n",
        "                break\n",
        "\n",
        "        if pixel_level:\n",
        "            if input_img.ndim == 3:\n",
        "                c = np.random.uniform(v_l, v_h, (h, w, img_c))\n",
        "            if input_img.ndim == 2:\n",
        "                c = np.random.uniform(v_l, v_h, (h, w))\n",
        "        else:\n",
        "            c = np.random.uniform(v_l, v_h)\n",
        "\n",
        "        input_img[top:top + h, left:left + w] = c\n",
        "\n",
        "        return input_img\n",
        "\n",
        "    return eraser\n",
        "\t\n",
        "root = 'refolds/'\n",
        "\n",
        "os.walk(root)\n",
        "\n",
        "for folder in os.listdir(root):\n",
        "    datagen = ImageDataGenerator(\n",
        "    rescale=1/255,\n",
        "    preprocessing_function=get_random_eraser(v_l=0, v_h=255, pixel_level=True),\n",
        "    rotation_range=20,\n",
        "    width_shift_range=0.2,\n",
        "    height_shift_range=0.2,\n",
        "    horizontal_flip=True)\n",
        "    \n",
        "\n",
        "    train_gen = datagen.flow_from_directory(\n",
        "       root + folder + '/train', \n",
        "        target_size=(IMAGE_SIZE , IMAGE_SIZE), \n",
        "        batch_size=BATCH_SIZE,\n",
        "        seed=RANDOM_SEED\n",
        "    )\n",
        "\n",
        "    testgen = ImageDataGenerator(\n",
        "        rescale=1/255)\n",
        "        \n",
        "    val_gen = testgen.flow_from_directory(\n",
        "        root + folder + '/val', \n",
        "        target_size=(IMAGE_SIZE, IMAGE_SIZE), \n",
        "        batch_size=BATCH_SIZE,\n",
        "        seed=RANDOM_SEED\n",
        "    )\n",
        "    \n",
        "    test_gen = testgen.flow_from_directory(\n",
        "        root + folder + '/test', \n",
        "        target_size=(IMAGE_SIZE, IMAGE_SIZE), \n",
        "        batch_size= BATCH_SIZE,\n",
        "        shuffle=False,\n",
        "        seed=RANDOM_SEED\n",
        "    )\n",
        "    \n",
        "    # create the base pre-trained model\n",
        "    from vit_keras import vit\n",
        "\n",
        "    vit_model = vit.vit_b32(\n",
        "            image_size = IMAGE_SIZE,\n",
        "            activation = 'softmax',\n",
        "            pretrained = True,\n",
        "            include_top = False,\n",
        "            pretrained_top = False,\n",
        "            classes = 37)\n",
        "    \t\t    \n",
        "    model = tf.keras.Sequential([\n",
        "            vit_model,\n",
        "            tf.keras.layers.Flatten(),\n",
        "            tf.keras.layers.BatchNormalization(),\n",
        "            tf.keras.layers.Dense(11, activation = 'relu'), #was tfa.activations.gelu\n",
        "            tf.keras.layers.BatchNormalization(),\n",
        "            tf.keras.layers.Dense(37, 'softmax')\n",
        "        ],\n",
        "        name = 'vision_transformer')\n",
        "    \n",
        "    model.summary()\n",
        "    \n",
        "    # training the model\n",
        "    learning_rate = 1e-4\n",
        "    \n",
        "    optimizer = Adam(learning_rate = learning_rate) \n",
        "    \n",
        "    model.compile(optimizer = optimizer, \n",
        "                  loss = tf.keras.losses.CategoricalCrossentropy(), \n",
        "                  metrics = ['accuracy'])\n",
        "    \n",
        "    STEP_SIZE_TRAIN = train_gen.n // train_gen.batch_size\n",
        "    STEP_SIZE_VAL = val_gen.n // val_gen.batch_size\n",
        "     \n",
        "    \n",
        "    earlystopping = tf.keras.callbacks.EarlyStopping(monitor = 'val_accuracy',\n",
        "                                                     min_delta = 1e-4,\n",
        "                                                     patience = 3, \n",
        "                                                     mode = 'max',\n",
        "                                                     restore_best_weights = True,\n",
        "                                                     verbose = 1)\n",
        "    \n",
        "    \n",
        "    filepath = \"visontmodel\"+str(folder)+\".h5\" \n",
        "    \n",
        "    checkpoint = ModelCheckpoint(filepath, monitor='val_accuracy', verbose=1, save_best_only=True, mode='min')\n",
        "    \n",
        "  \n",
        "    callbacks_list = [earlystopping, checkpoint]\n",
        "    \n",
        "    EPOCHS = 100\n",
        "\n",
        "    history = model.fit(train_gen,\n",
        "          steps_per_epoch = STEP_SIZE_TRAIN,\n",
        "          validation_data = val_gen,\n",
        "          validation_steps = STEP_SIZE_VAL,\n",
        "          epochs = EPOCHS,\n",
        "          callbacks = callbacks_list)\n",
        "    \n",
        "        \n",
        "    test_batch_x, test_batch_y = test_gen.next()\n",
        "    pred_batch = model.predict(test_batch_x)\n",
        "    \n",
        "    test_labels = np.argmax(test_batch_y, axis=1)\n",
        "    test_pred = np.argmax(pred_batch, axis=1)\n",
        "    \n",
        "    test_acc = sum(test_labels == test_pred) / len(test_labels)\n",
        "    print('Accuracy str(folder): %.3f' % test_acc) \n",
        "     \n",
        "    Y_pred = model.predict(test_gen, test_gen.samples // BATCH_SIZE+1)\n",
        "    y_pred = np.argmax(Y_pred, axis=1)\n",
        "\n",
        "    from sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n",
        "    print(precision_score(test_gen.classes, y_pred , average=\"macro\"))\n",
        "    print(recall_score(test_gen.classes, y_pred , average=\"macro\"))\n",
        "    print(f1_score(test_gen.classes, y_pred , average=\"macro\"))\n",
        "    \n",
        "    from sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\n",
        "    Y_pred = model.predict(test_gen, test_gen.samples // BATCH_SIZE+1)\n",
        "    y_pred = np.argmax(Y_pred, axis=1)\n",
        "    \n",
        "    \n",
        "    from sklearn.metrics import classification_report, confusion_matrix\n",
        "    # code found here: https://stackoverflow.com/questions/66386561/keras-classification-report-accuracy-is-different-between-model-predict-accurac?noredirect=1&lq=1\n",
        "    \n",
        "    print(confusion_matrix(test_gen.classes, y_pred))\n",
        "    \n",
        "    print('Classification Report')\n",
        "    \n",
        "    target_names = ['asian_pied', 'bali_myna', 'bluethroat', 'bm_leafbird', 'bn_oriole', 'bw_leafbird', 'bw_myna', 'c_laughingthrush', 'c_whiteeye', 'cc_thrush', 'chestnut_munia', \n",
        "                    'common_myna', 'fischers_lovebird', 'gg_leafbird', 'grey_parrot', 'grosbeak', 'hill_myna', 'hwamei',\n",
        "                    'javan_sparrow', 'jg_magpie', 'leiothrix', 'lt_shrike', 'oh_thrush', 'om_robin', 'r_laughingthrush', \n",
        "                    'red_billedstarling', 'red_whiskered', 'rubythroat', 'sb_munia', 'sh_bulbul', 'silver_eared',\n",
        "                    'su_laughingthrush', 'swinhoes', 'wr_munia', 'ws_shama', 'z_dove', 'z_finch']\n",
        "    \n",
        "    print(classification_report(test_gen.classes, y_pred, target_names=target_names))\n",
        "    \n",
        "    cm = confusion_matrix(test_gen.classes, y_pred)\n",
        "    \n",
        "    import itertools\n",
        "    \n",
        "    def plot_confusion_matrix(cm, classes, normalize=True, title='Confusion matrix', cmap=plt.cm.Oranges):#was Blues before, refer here for help: https://stackoverflow.com/questions/57043260/how-change-the-color-of-boxes-in-confusion-matrix-using-sklearn\n",
        "    \n",
        "        \"\"\"\n",
        "    \n",
        "        This function prints and plots the confusion matrix.\n",
        "    \n",
        "        Normalization can be applied by setting `normalize=True`.\n",
        "    \n",
        "        \"\"\"\n",
        "    \n",
        "        plt.figure(figsize=(20,20))\n",
        "    \n",
        "    \n",
        "    \n",
        "        plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    \n",
        "        plt.title(title)\n",
        "    \n",
        "        plt.colorbar()\n",
        "    \n",
        "    \n",
        "    \n",
        "        tick_marks = np.arange(len(classes))\n",
        "    \n",
        "        plt.xticks(tick_marks, classes, rotation=90)\n",
        "    \n",
        "        plt.yticks(tick_marks, classes)\n",
        "    \n",
        "    \n",
        "    \n",
        "        if normalize:\n",
        "    \n",
        "            cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "    \n",
        "            cm = np.around(cm, decimals=2)\n",
        "    \n",
        "            cm[np.isnan(cm)] = 0.0\n",
        "    \n",
        "            print(\"Normalized confusion matrix\")\n",
        "    \n",
        "        else:\n",
        "    \n",
        "            print('Confusion matrix, without normalization')\n",
        "    \n",
        "        thresh = cm.max() / 2.\n",
        "    \n",
        "        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "    \n",
        "            plt.text(j, i, cm[i, j],\n",
        "    \n",
        "                     horizontalalignment=\"center\",\n",
        "    \n",
        "                     color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "    \n",
        "        plt.tight_layout()\n",
        "    \n",
        "        plt.ylabel('True label')\n",
        "    \n",
        "        plt.xlabel('Predicted label')\n",
        "        \n",
        "    plot_confusion_matrix(cm, target_names, title='Confusion Matrix')\n",
        "    \n",
        "# This plots a confusion matrix for each fold"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}