notebooks/submission_tutorial/2_behind_the_scenes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# How we evaluate the competition entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### This notebook will show how we calculate the competition scores on our website\n",
    "\n",
    "Here, we illustrate in detail how we use the submission file to compute the scores of the competition. We do this by taking one of the datasets that is not part of the competition (i.e. one of the \"pre-training\" datasets). All of the \"pre-training\" datasets also have a test set (i.e. the `live_test` set), and the responses of all neurons to these test set images are present. Thus, you can test your model by training on one or all of the \"pre-training\" recordings and see how the model performs on the test set.\n",
    "\n",
    "Here, we use one of the sets to show how we extract the responses from the test set into a `ground_truth` file. And how we use the submitted files and the ground truth file to calculate the scores.\n",
    "\n",
    "In detail, this notebooks includes these steps:\n",
    "- we first load a pretrained model\n",
    "- we select a pre-training dataset, which is not part of the competition, and treat the \"test\" set in it as if it was part of the competition track\n",
    "- this example then illustrates the complete process how the ground truth responses are extracted, and how the scores are getting calculated between ground truth and the submitted responses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from nnfabrik.builder import get_data, get_model, get_trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Get dataloader and model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#get dataloader\n",
    "basepath = \"../data/\"\n",
    "filenames = [os.path.join(basepath, file) for file in os.listdir(basepath) if \".zip\" in file ]\n",
    "\n",
    "dataset_fn = 'sensorium.datasets.static_loaders'\n",
    "dataset_config = {'paths': filenames,\n",
    "                 'normalize': True,\n",
    "                 'include_behavior': False,\n",
    "                 'include_eye_position': False,\n",
    "                 'batch_size': 128,\n",
    "                 'scale':.25,\n",
    "                 }\n",
    "\n",
    "dataloaders = get_data(dataset_fn, dataset_config)\n",
    "\n",
    "# get model\n",
    "model_fn = 'sensorium.models.stacked_core_full_gauss_readout'\n",
    "model_config = {'pad_input': False,\n",
    "  'layers': 4,\n",
    "  'input_kern': 9,\n",
    "  'gamma_input': 6.3831,\n",
    "  'gamma_readout': 0.0076,\n",
    "  'hidden_kern': 7,\n",
    "  'hidden_channels': 64,\n",
    "  'depth_separable': True,\n",
    "  'grid_mean_predictor': {'type': 'cortex',\n",
    "   'input_dimensions': 2,\n",
    "   'hidden_layers': 1,\n",
    "   'hidden_features': 30,\n",
    "   'final_tanh': True},\n",
    "  'init_sigma': 0.1,\n",
    "  'init_mu_range': 0.3,\n",
    "  'gauss_type': 'full',\n",
    "  'shifter': False,\n",
    "  'stack': -1,\n",
    "}\n",
    "\n",
    "model = get_model(model_fn=model_fn,\n",
    "                  model_config=model_config,\n",
    "                  dataloaders=dataloaders,\n",
    "                  seed=42,)\n",
    "\n",
    "# load model weights\n",
    "model.load_state_dict(torch.load(\"../model_tutorial/model_checkpoints/pretrained/generalization_model.pth\"));\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# How we calculate the competition scores behind the scenes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "- we are withholding the ground truth neuronal responses to the test set images in the actual competition\n",
    "- here we show \n",
    " - how we extract the ground truth responses from the demo dataset (where the test set responses are present)\n",
    " - how the metrics of the competition are calculated\n",
    "\n",
    "The following steps are necessary:\n",
    "1. pick a dataset\n",
    "2. generate a file that contains the ground truth responses to the test set\n",
    "3. generate a submission file that contains the predictions\n",
    "4. Calculate the performance metrics based on these 2 files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# !! Important !!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Our grund truth file is storing **standardized responses**, meaning the responses of each neuron normalized by its own STD. Our dataloader is automatically normalizing the images and responses, and we encourage you to use our DataLoader and our submission API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# import the API from the competition repo\n",
    "from sensorium.utility import submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 1) Example Dataset:'21067-10-18' from the pre-training recordings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "filename = ['../data/static21067-10-18-GrayImageNet-94c6ff995dac583098847cfecd43e7b6.zip',]\n",
    "\n",
    "dataset_name = \"21067-10-18\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 2) Generate Ground Truth File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission file saved for tier: test. Saved in: ./ground_truth_files/ground_truth_file_test.csv\n"
     ]
    }
   ],
   "source": [
    "# we load the dataset which contains the held-out \"test\" responses, and save them in the .csv format\n",
    "# for the demo dataset that we provide here, these \"test\" responses are present\n",
    "\n",
    "submission.generate_ground_truth_file(filename=filename,\n",
    "                                      path='./ground_truth_files/',\n",
    "                                      tier=\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "##### Inspect the Ground Truth File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trial_indices</th>\n",
       "      <th>image_ids</th>\n",
       "      <th>responses</th>\n",
       "      <th>neuron_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>126</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.6678206920623779, 0.5115134119987488, 0.546...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>297</td>\n",
       "      <td>2214</td>\n",
       "      <td>[1.1717146635055542, 8.103653748037232e-09, 2....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>597</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.46323058009147644, 3.492245426173213e-08, 0...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>852</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.27945539355278015, 1.862973597610562e-08, 1...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>908</td>\n",
       "      <td>2214</td>\n",
       "      <td>[5.324940488549146e-08, 0.9885492324829102, 4....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>2752</td>\n",
       "      <td>3487</td>\n",
       "      <td>[4.292914468351228e-09, 4.884256998849423e-09,...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>3039</td>\n",
       "      <td>3487</td>\n",
       "      <td>[5.395119906381751e-09, 9.930156252835332e-09,...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>4312</td>\n",
       "      <td>3487</td>\n",
       "      <td>[8.72513794547558e-09, 0.5518566966056824, 4.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>4380</td>\n",
       "      <td>3487</td>\n",
       "      <td>[3.2182767562005665e-09, 1.3249121799674413e-0...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>4571</td>\n",
       "      <td>3487</td>\n",
       "      <td>[3.7211631553901725e-09, 1.0701143970948124e-0...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>998 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     trial_indices  image_ids  \\\n",
       "0              126       2214   \n",
       "1              297       2214   \n",
       "2              597       2214   \n",
       "3              852       2214   \n",
       "4              908       2214   \n",
       "..             ...        ...   \n",
       "993           2752       3487   \n",
       "994           3039       3487   \n",
       "995           4312       3487   \n",
       "996           4380       3487   \n",
       "997           4571       3487   \n",
       "\n",
       "                                             responses  \\\n",
       "0    [0.6678206920623779, 0.5115134119987488, 0.546...   \n",
       "1    [1.1717146635055542, 8.103653748037232e-09, 2....   \n",
       "2    [0.46323058009147644, 3.492245426173213e-08, 0...   \n",
       "3    [0.27945539355278015, 1.862973597610562e-08, 1...   \n",
       "4    [5.324940488549146e-08, 0.9885492324829102, 4....   \n",
       "..                                                 ...   \n",
       "993  [4.292914468351228e-09, 4.884256998849423e-09,...   \n",
       "994  [5.395119906381751e-09, 9.930156252835332e-09,...   \n",
       "995  [8.72513794547558e-09, 0.5518566966056824, 4.4...   \n",
       "996  [3.2182767562005665e-09, 1.3249121799674413e-0...   \n",
       "997  [3.7211631553901725e-09, 1.0701143970948124e-0...   \n",
       "\n",
       "                                            neuron_ids  \n",
       "0    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "1    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "2    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "3    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "4    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "..                                                 ...  \n",
       "993  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "994  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "995  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "996  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "997  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "\n",
       "[998 rows x 4 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('./ground_truth_files/ground_truth_file_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 3) Generate Submission file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission file saved for tier: live_test. Saved in: ./submission_files/submission_file_live_test.csv\n"
     ]
    }
   ],
   "source": [
    "# generate the submission file\n",
    "submission.generate_submission_file(trained_model=model, \n",
    "                                    dataloaders=dataloaders,\n",
    "                                    data_key=dataset_name,\n",
    "                                    path='./submission_files/',\n",
    "                                    device=\"cuda\",\n",
    "                                    tier=\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "##### Inspect content of submission file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trial_indices</th>\n",
       "      <th>image_ids</th>\n",
       "      <th>prediction</th>\n",
       "      <th>neuron_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>126</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>297</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>597</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>852</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>908</td>\n",
       "      <td>2214</td>\n",
       "      <td>[0.24674898386001587, 0.23912948369979858, 0.4...</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>2752</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>3039</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>4312</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>4380</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>4571</td>\n",
       "      <td>3487</td>\n",
       "      <td>[0.040638625621795654, 0.11597681045532227, 0....</td>\n",
       "      <td>[1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>998 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     trial_indices  image_ids  \\\n",
       "0              126       2214   \n",
       "1              297       2214   \n",
       "2              597       2214   \n",
       "3              852       2214   \n",
       "4              908       2214   \n",
       "..             ...        ...   \n",
       "993           2752       3487   \n",
       "994           3039       3487   \n",
       "995           4312       3487   \n",
       "996           4380       3487   \n",
       "997           4571       3487   \n",
       "\n",
       "                                            prediction  \\\n",
       "0    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "1    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "2    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "3    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "4    [0.24674898386001587, 0.23912948369979858, 0.4...   \n",
       "..                                                 ...   \n",
       "993  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "994  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "995  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "996  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "997  [0.040638625621795654, 0.11597681045532227, 0....   \n",
       "\n",
       "                                            neuron_ids  \n",
       "0    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "1    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "2    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "3    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "4    [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "..                                                 ...  \n",
       "993  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "994  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "995  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "996  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "997  [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 1...  \n",
       "\n",
       "[998 rows x 4 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_csv('./submission_files/submission_file_live_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 4) Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This is what is happening in the backend of our competition website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sensorium import evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# specify submission and ground truth file\n",
    "ground_truth_file = './ground_truth_files/ground_truth_file_test.csv'\n",
    "submission_file = './submission_files/submission_file_live_test.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sensorium import evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# specify submission and ground truth file\n",
    "ground_truth_file = './ground_truth_files/ground_truth_file_test.csv'\n",
    "submission_file = './submission_files/submission_file_live_test.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "out = evaluate(submission_file, ground_truth_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for the SOTA model:\n",
      "Single Trial Correlation: 0.286\n",
      "Correlation to Average: 0.542\n",
      "FEVE: 0.452\n"
     ]
    }
   ],
   "source": [
    "print(\"Results for the SOTA model:\")\n",
    "for metric, value in out.items():\n",
    "    print(f\"{metric}: {np.round(value, 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### These scores are calcualted in the backend of our website\n",
    "- we have two test sets, so these scores will be computed for our **live** test set, and our **final** test set\n",
    "- the **live** scores will get published on the live leaderboard\n",
    "- the **final** scores will be hidden, and we will release them to the public on Oct 22, after checking the scores carefully\n",
    "- the **final** scores will then determine the winner of the competition"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}