notebooks/01_datacreation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Music Classification Mini-Project\n",
    "## Author: Collin Guidry\n",
    "\n",
    "## Step 1: Data Creation Pipeline\n",
    "\n",
    "### Purpose:\n",
    "This notebook is designed collect, format, split, and label MIDI files for both model training and testing purposes.\n",
    "Portions of this notebook can be easily duplicated to create an inference pipeline.\n",
    "\n",
    "### Available Parameters:\n",
    "- Input and output paths for train and test data\n",
    "- chunk size (in seconds)\n",
    "\n",
    "### Processing Steps:\n",
    "\n",
    "    1. Load Midi files for model training\n",
    "        - These are located in separate folders, partitioned by the composer's name\n",
    "        - Save each parent folder name as the label for each respective midi file\n",
    "\n",
    "    2. Load Midi files for model inference/testing\n",
    "        - Midi files used for model testing are unlabeled.\n",
    "        - They should be processed independently from training data, but in the same fashion\n",
    "        - As the labels are unknown, this will later be known as \"df_final_test\"\n",
    "\n",
    "    3. Processs loaded Midi files as follows (Train and Test):\n",
    "        - Extraxt metadata from filepath\n",
    "        - Split midi file into 30-second chunks\n",
    "            - Trim each chunk as precicely as possible\n",
    "            - \"Stretch\" chunks that fall just short of 30 seconds \n",
    "        - Write out chunks to intermediate folders, with an iter_id for each chunk\n",
    "        - Note: Modifying the chunk size will not overwrite existing files\n",
    "\n",
    "    4. Load 30-second chunks into DataFrame (Train and Test)\n",
    "        - Read each chunk from disk\n",
    "        - Store each midi file object in a Pandas DataFrame column\n",
    "        - Format available medatada such as composer, composition name, chunk_iter as columns\n",
    "        - For training data, create train/test split column\n",
    "            - Note: Does not allow same composition to appear in both splits\n",
    "\n",
    "    5. Write out DataFrames as .pkl to be further processed\n",
    "        - train_processed.pkl\n",
    "        - test_processed.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pretty_midi\n",
    "import os\n",
    "import numpy as np\n",
    "import copy\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import GroupShuffleSplit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper functions\n",
    "\n",
    "def trim_midi_file(file:pretty_midi.pretty_midi.PrettyMIDI , start:int, end:int):\n",
    "    midi_file = copy.deepcopy(file)\n",
    "    start_time = float(start)\n",
    "    end_time = float(end)\n",
    "    new_start_time = 0.0\n",
    "    new_end_time = end_time - start_time  # Calculate the duration of the cropped portion\n",
    "    midi_file.adjust_times([start_time, end_time], [new_start_time, new_end_time])\n",
    "    return midi_file\n",
    "\n",
    "\n",
    "def create_chunk_interval_times(length, chunk_size):\n",
    "    chunks = []\n",
    "    for i in range(0, length, chunk_size):\n",
    "        start = i\n",
    "        end = min(i + chunk_size, length)\n",
    "        chunks.append([start, end])\n",
    "    return chunks\n",
    "\n",
    "\n",
    "def stretch_midi_file_to_desired_length(file:pretty_midi.pretty_midi.PrettyMIDI, length:int):\n",
    "    midi_file = copy.deepcopy(file)\n",
    "    start_time = 0.0\n",
    "    end_time = midi_file.get_end_time()\n",
    "    new_start_time = 0.0\n",
    "    new_end_time = length\n",
    "    midi_file.adjust_times([start_time, end_time], [new_start_time, new_end_time])\n",
    "    return midi_file\n",
    "\n",
    "\n",
    "def process_and_save_midi_splits(midi_file:pretty_midi.pretty_midi.PrettyMIDI, \n",
    "                                 file_name: str,\n",
    "                                 output_dir: str,\n",
    "                                 chunk_size_seconds:int ):\n",
    "    \n",
    "    total_duration = int(midi_file.get_end_time())\n",
    "    intervals = create_chunk_interval_times(total_duration, chunk_size_seconds)[:-1]\n",
    "\n",
    "    for i, interval in enumerate(intervals):\n",
    "        \n",
    "        trimmed_file = trim_midi_file(file=midi_file, start=interval[0], end=interval[1])\n",
    "\n",
    "        trimmed_stretched_file = stretch_midi_file_to_desired_length(file=trimmed_file, length=chunk_size_seconds)\n",
    "        \n",
    "        output_filename = os.path.join(output_dir, f'{file_name}_chunksize_{chunk_size_seconds}_iter_{i}_.mid')\n",
    "\n",
    "        trimmed_stretched_file.write(output_filename)\n",
    "\n",
    "    return None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dictionary (midi_dict) with full midi files and composer labels (training data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\cguid\\anaconda3\\envs\\CG_DS_Env_1\\lib\\site-packages\\pretty_midi\\pretty_midi.py:101: RuntimeWarning: Tempo, Key or Time signature change events found on non-zero tracks.  This is not a valid type 0 or type 1 MIDI file.  Tempo, Key or Time Signature may be wrong.\n",
      "  RuntimeWarning)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "midi_folder_train = f\"../data/raw/training\"\n",
    "composer_names = ['Bach','Beethoven','Brahms','Schubert']\n",
    "\n",
    "midi_dict_train = {}\n",
    "\n",
    "# Import files from each composer folder\n",
    "# Create a dictionary midi files, indexed by new filename to write out\n",
    "# The new filename contains the composer and will later be parsed apart\n",
    "\n",
    "for composer in composer_names:\n",
    "    midi_folder = midi_folder_train+\"/\"+composer\n",
    "\n",
    "    for filename in os.listdir(midi_folder):\n",
    "        if filename.endswith('.mid'):\n",
    "            \n",
    "            filename_split = filename.split('_')\n",
    "            composition = filename_split[0]\n",
    "            catalog_name = filename_split[1]\n",
    "            id = filename_split[2]\n",
    "            rest = ''.join(filename_split[3:]).split('.')[0]\n",
    "            \n",
    "            filename_new = f\"{composer}_{composition}_{id}_{catalog_name}_{rest}\"\n",
    "\n",
    "            midi_file = pretty_midi.PrettyMIDI(os.path.join(midi_folder, filename))\n",
    "\n",
    "            midi_dict_train[filename_new] = {}\n",
    "            midi_dict_train[filename_new]['MidiFile'] = midi_file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "## params\n",
    "output_dir_train = '../data/intermediate/midi_chunks_train/'\n",
    "chunk_size_seconds = 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split into chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "######\n",
    "# Iterate through the dictionary of files to generate sub-files which are 30-second chunks\n",
    "# Write these files out with the new file names \n",
    "\n",
    "for out_file_name in midi_dict_train:\n",
    "\n",
    "    midi_file = midi_dict_train[out_file_name]['MidiFile']\n",
    "\n",
    "    process_and_save_midi_splits(midi_file = midi_file, \n",
    "                                    file_name = out_file_name,\n",
    "                                    output_dir = output_dir_train,\n",
    "                                    chunk_size_seconds = chunk_size_seconds )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load into DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load training chunks into a new dictionary & DataFrame\n",
    "\n",
    "midi_dict_train = {}\n",
    "\n",
    "for filename in os.listdir(output_dir_train):\n",
    "    if filename.endswith('.mid'):\n",
    "\n",
    "        midi_file = pretty_midi.PrettyMIDI(os.path.join(output_dir_train, filename))\n",
    "        midi_dict_train[filename] = midi_file\n",
    "\n",
    "\n",
    "df = pd.DataFrame()\n",
    "df['track_path'] = midi_dict_train.keys()\n",
    "df[['composer', 'track_name', 'id', 'catalog_name','path_other','chunk_size_seconds', 'iter']] = df['track_path'].str.split('_', expand=True)[[0,1,2,3,4,6,8]]\n",
    "df['midi'] = [*midi_dict_train.values()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>track_path</th>\n",
       "      <th>composer</th>\n",
       "      <th>track_name</th>\n",
       "      <th>id</th>\n",
       "      <th>catalog_name</th>\n",
       "      <th>path_other</th>\n",
       "      <th>chunk_size_seconds</th>\n",
       "      <th>iter</th>\n",
       "      <th>midi</th>\n",
       "      <th>split</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...</td>\n",
       "      <td>Bach</td>\n",
       "      <td>Cello Suite 3</td>\n",
       "      <td>2217</td>\n",
       "      <td>BWV1009</td>\n",
       "      <td>cs3-1pre</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...</td>\n",
       "      <td>Bach</td>\n",
       "      <td>Cello Suite 3</td>\n",
       "      <td>2217</td>\n",
       "      <td>BWV1009</td>\n",
       "      <td>cs3-1pre</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...</td>\n",
       "      <td>Bach</td>\n",
       "      <td>Cello Suite 3</td>\n",
       "      <td>2217</td>\n",
       "      <td>BWV1009</td>\n",
       "      <td>cs3-1pre</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...</td>\n",
       "      <td>Bach</td>\n",
       "      <td>Cello Suite 3</td>\n",
       "      <td>2217</td>\n",
       "      <td>BWV1009</td>\n",
       "      <td>cs3-1pre</td>\n",
       "      <td>30</td>\n",
       "      <td>3</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...</td>\n",
       "      <td>Bach</td>\n",
       "      <td>Cello Suite 3</td>\n",
       "      <td>2217</td>\n",
       "      <td>BWV1009</td>\n",
       "      <td>cs3-1pre</td>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2313</th>\n",
       "      <td>Schubert_String Quintet in C major_1742_OP163_...</td>\n",
       "      <td>Schubert</td>\n",
       "      <td>String Quintet in C major</td>\n",
       "      <td>1742</td>\n",
       "      <td>OP163</td>\n",
       "      <td>sb163m2</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2314</th>\n",
       "      <td>Schubert_String Quintet in C major_1742_OP163_...</td>\n",
       "      <td>Schubert</td>\n",
       "      <td>String Quintet in C major</td>\n",
       "      <td>1742</td>\n",
       "      <td>OP163</td>\n",
       "      <td>sb163m2</td>\n",
       "      <td>30</td>\n",
       "      <td>6</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2315</th>\n",
       "      <td>Schubert_String Quintet in C major_1742_OP163_...</td>\n",
       "      <td>Schubert</td>\n",
       "      <td>String Quintet in C major</td>\n",
       "      <td>1742</td>\n",
       "      <td>OP163</td>\n",
       "      <td>sb163m2</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2316</th>\n",
       "      <td>Schubert_String Quintet in C major_1742_OP163_...</td>\n",
       "      <td>Schubert</td>\n",
       "      <td>String Quintet in C major</td>\n",
       "      <td>1742</td>\n",
       "      <td>OP163</td>\n",
       "      <td>sb163m2</td>\n",
       "      <td>30</td>\n",
       "      <td>8</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2317</th>\n",
       "      <td>Schubert_String Quintet in C major_1742_OP163_...</td>\n",
       "      <td>Schubert</td>\n",
       "      <td>String Quintet in C major</td>\n",
       "      <td>1742</td>\n",
       "      <td>OP163</td>\n",
       "      <td>sb163m2</td>\n",
       "      <td>30</td>\n",
       "      <td>9</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "      <td>train</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2318 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             track_path  composer  \\\n",
       "0     Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...      Bach   \n",
       "1     Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...      Bach   \n",
       "2     Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...      Bach   \n",
       "3     Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...      Bach   \n",
       "4     Bach_Cello Suite 3_2217_BWV1009_cs3-1pre_chunk...      Bach   \n",
       "...                                                 ...       ...   \n",
       "2313  Schubert_String Quintet in C major_1742_OP163_...  Schubert   \n",
       "2314  Schubert_String Quintet in C major_1742_OP163_...  Schubert   \n",
       "2315  Schubert_String Quintet in C major_1742_OP163_...  Schubert   \n",
       "2316  Schubert_String Quintet in C major_1742_OP163_...  Schubert   \n",
       "2317  Schubert_String Quintet in C major_1742_OP163_...  Schubert   \n",
       "\n",
       "                     track_name    id catalog_name path_other  \\\n",
       "0                 Cello Suite 3  2217      BWV1009   cs3-1pre   \n",
       "1                 Cello Suite 3  2217      BWV1009   cs3-1pre   \n",
       "2                 Cello Suite 3  2217      BWV1009   cs3-1pre   \n",
       "3                 Cello Suite 3  2217      BWV1009   cs3-1pre   \n",
       "4                 Cello Suite 3  2217      BWV1009   cs3-1pre   \n",
       "...                         ...   ...          ...        ...   \n",
       "2313  String Quintet in C major  1742        OP163    sb163m2   \n",
       "2314  String Quintet in C major  1742        OP163    sb163m2   \n",
       "2315  String Quintet in C major  1742        OP163    sb163m2   \n",
       "2316  String Quintet in C major  1742        OP163    sb163m2   \n",
       "2317  String Quintet in C major  1742        OP163    sb163m2   \n",
       "\n",
       "     chunk_size_seconds iter  \\\n",
       "0                    30    0   \n",
       "1                    30    1   \n",
       "2                    30    2   \n",
       "3                    30    3   \n",
       "4                    30    4   \n",
       "...                 ...  ...   \n",
       "2313                 30    5   \n",
       "2314                 30    6   \n",
       "2315                 30    7   \n",
       "2316                 30    8   \n",
       "2317                 30    9   \n",
       "\n",
       "                                                   midi  split  \n",
       "0     <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "1     <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "2     <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "3     <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "4     <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "...                                                 ...    ...  \n",
       "2313  <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "2314  <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "2315  <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "2316  <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "2317  <pretty_midi.pretty_midi.PrettyMIDI object at ...  train  \n",
       "\n",
       "[2318 rows x 10 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create train / test split before saving the dataset. \n",
    "# Ensure the same song is not seen in both splits (data leakage)\n",
    "\n",
    "splitter = GroupShuffleSplit(n_splits=1, test_size=0.3, random_state=42)\n",
    "\n",
    "train_indices, val_indices = next(splitter.split(df, groups=df['track_name']))\n",
    "\n",
    "df['split'] = 'train'\n",
    "df.loc[val_indices, 'split'] = 'test'\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "train    0.726057\n",
       "test     0.273943\n",
       "Name: split, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train/test split - sanity check\n",
    "df['split'].value_counts()/df['split'].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle('../data/processed/train_processed.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test/Inference/Holdout Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Load test data into dictionary\n",
    "midi_folder_test = f\"../data/raw/holdout\"\n",
    "\n",
    "midi_dict_test = {}\n",
    "for filename in os.listdir(midi_folder_test):\n",
    "    if filename.endswith('.mid'):\n",
    "\n",
    "        midi_file = pretty_midi.PrettyMIDI(os.path.join(midi_folder_test, filename))\n",
    "\n",
    "        filename_new = filename[:-4]\n",
    "        midi_dict_test[filename_new] = {}\n",
    "        midi_dict_test[filename_new]['MidiFile'] = midi_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split into chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "###### Split test data from dictionary into chunks\n",
    "output_dir_test = '../data/intermediate/midi_chunks_test/'\n",
    "chunk_size_seconds = 30\n",
    "\n",
    "for out_file_name in midi_dict_test:\n",
    "\n",
    "    midi_file = midi_dict_test[out_file_name]['MidiFile']\n",
    "\n",
    "    process_and_save_midi_splits(midi_file = midi_file, \n",
    "                                    file_name = out_file_name,\n",
    "                                    output_dir = output_dir_test,\n",
    "                                    chunk_size_seconds = chunk_size_seconds )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load test chunks into a new dictionary & dataFrame\n",
    "\n",
    "midi_dict_test = {}\n",
    "\n",
    "for filename in os.listdir(output_dir_test):\n",
    "    if filename.endswith('.mid'):\n",
    "\n",
    "        midi_file = pretty_midi.PrettyMIDI(os.path.join(output_dir_test, filename))\n",
    "        midi_dict_test[filename] = midi_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load into DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>track_path</th>\n",
       "      <th>track_path_original</th>\n",
       "      <th>prob</th>\n",
       "      <th>chunk_size_seconds</th>\n",
       "      <th>iter</th>\n",
       "      <th>pred_given</th>\n",
       "      <th>midi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.002716920481628_adj_chunksize_30_iter_0_.mid</td>\n",
       "      <td>0.002716920481628_adj.mid</td>\n",
       "      <td>0.002717</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002716920481628_adj_chunksize_30_iter_10_.mid</td>\n",
       "      <td>0.002716920481628_adj.mid</td>\n",
       "      <td>0.002717</td>\n",
       "      <td>30</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002716920481628_adj_chunksize_30_iter_11_.mid</td>\n",
       "      <td>0.002716920481628_adj.mid</td>\n",
       "      <td>0.002717</td>\n",
       "      <td>30</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.002716920481628_adj_chunksize_30_iter_12_.mid</td>\n",
       "      <td>0.002716920481628_adj.mid</td>\n",
       "      <td>0.002717</td>\n",
       "      <td>30</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.002716920481628_adj_chunksize_30_iter_13_.mid</td>\n",
       "      <td>0.002716920481628_adj.mid</td>\n",
       "      <td>0.002717</td>\n",
       "      <td>30</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>0.981087291054314_adj_chunksize_30_iter_5_.mid</td>\n",
       "      <td>0.981087291054314_adj.mid</td>\n",
       "      <td>0.981087</td>\n",
       "      <td>30</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>0.981087291054314_adj_chunksize_30_iter_6_.mid</td>\n",
       "      <td>0.981087291054314_adj.mid</td>\n",
       "      <td>0.981087</td>\n",
       "      <td>30</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>0.981087291054314_adj_chunksize_30_iter_7_.mid</td>\n",
       "      <td>0.981087291054314_adj.mid</td>\n",
       "      <td>0.981087</td>\n",
       "      <td>30</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>0.981087291054314_adj_chunksize_30_iter_8_.mid</td>\n",
       "      <td>0.981087291054314_adj.mid</td>\n",
       "      <td>0.981087</td>\n",
       "      <td>30</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>0.981087291054314_adj_chunksize_30_iter_9_.mid</td>\n",
       "      <td>0.981087291054314_adj.mid</td>\n",
       "      <td>0.981087</td>\n",
       "      <td>30</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;pretty_midi.pretty_midi.PrettyMIDI object at ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>305 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          track_path  \\\n",
       "0     0.002716920481628_adj_chunksize_30_iter_0_.mid   \n",
       "1    0.002716920481628_adj_chunksize_30_iter_10_.mid   \n",
       "2    0.002716920481628_adj_chunksize_30_iter_11_.mid   \n",
       "3    0.002716920481628_adj_chunksize_30_iter_12_.mid   \n",
       "4    0.002716920481628_adj_chunksize_30_iter_13_.mid   \n",
       "..                                               ...   \n",
       "300   0.981087291054314_adj_chunksize_30_iter_5_.mid   \n",
       "301   0.981087291054314_adj_chunksize_30_iter_6_.mid   \n",
       "302   0.981087291054314_adj_chunksize_30_iter_7_.mid   \n",
       "303   0.981087291054314_adj_chunksize_30_iter_8_.mid   \n",
       "304   0.981087291054314_adj_chunksize_30_iter_9_.mid   \n",
       "\n",
       "           track_path_original      prob chunk_size_seconds iter  pred_given  \\\n",
       "0    0.002716920481628_adj.mid  0.002717                 30    0           0   \n",
       "1    0.002716920481628_adj.mid  0.002717                 30   10           0   \n",
       "2    0.002716920481628_adj.mid  0.002717                 30   11           0   \n",
       "3    0.002716920481628_adj.mid  0.002717                 30   12           0   \n",
       "4    0.002716920481628_adj.mid  0.002717                 30   13           0   \n",
       "..                         ...       ...                ...  ...         ...   \n",
       "300  0.981087291054314_adj.mid  0.981087                 30    5           1   \n",
       "301  0.981087291054314_adj.mid  0.981087                 30    6           1   \n",
       "302  0.981087291054314_adj.mid  0.981087                 30    7           1   \n",
       "303  0.981087291054314_adj.mid  0.981087                 30    8           1   \n",
       "304  0.981087291054314_adj.mid  0.981087                 30    9           1   \n",
       "\n",
       "                                                  midi  \n",
       "0    <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "1    <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "2    <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "3    <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "4    <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "..                                                 ...  \n",
       "300  <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "301  <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "302  <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "303  <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "304  <pretty_midi.pretty_midi.PrettyMIDI object at ...  \n",
       "\n",
       "[305 rows x 7 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.DataFrame()\n",
    "df_test['track_path'] = midi_dict_test.keys()\n",
    "df_test['track_path_original'] = df_test['track_path'].str.split('_').apply(lambda lst: '_'.join(lst[0:2])+'.mid')\n",
    "df_test[['prob','chunk_size_seconds','iter']] = df_test['track_path'].str.split('_', expand=True)[[0,3,5]]\n",
    "df_test['prob'] = df_test['prob'].astype(float)\n",
    "df_test['pred_given'] = df_test['prob'].apply(lambda p: 1 if p > .5 else 0)\n",
    "df_test['midi'] = [*midi_dict_test.values()]\n",
    "df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test.to_pickle('../data/processed/test_processed.pkl')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "4ef2a6079f6d387da3ea4ab411a6e80313bea7bf94e551c237eefc44274569bb"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}