Speaker2Vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DRAFT VERSION.\n",
    "\n",
    "In this notebook, we are trying to reproduce [the paper](http://scuba.usc.edu/pdf/jati2017_Speaker2Vec.pdf).\n",
    "\n",
    "For review of the paper, you can look [here](https://hedonistrh.github.io/2018-07-09-Literature-Review-for-Speaker-Change-Detection/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction\n",
    "\n",
    "_\"We have adopted 40 dimensional high definition MFCC features extracted from 40 mel-spaced filters over a 25ms hamming window with a shift of 10ms using Kaldi toolkit\"_ However, I will use [Librosa](https://librosa.github.io).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data\n",
    "\n",
    "_\"We have used d = 100 frames (1s) for all training scenarios. This makes the size of input and output layers of the DNN models to be 4000.\"_\n",
    "\n",
    "_\"40 mel-spaced filters over a 25ms hamming window with a shift of 10ms\"_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir audio_files # this folder store audio_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use some data from Youtube. To download these files, [youtube-dl](https://rg3.github.io/youtube-dl/index.html) is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://askubuntu.com/questions/564567/how-to-download-playlist-from-youtube-dl\n",
    "# https://www.slashgeek.net/2016/06/24/5-youtube-dl-tips-might-not-know/\n",
    "!youtube-dl --extract-audio --audio-format m4a -o \"./audio_files/%(title)s.%(ext)s\" https://www.youtube.com/playlist?list="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import librosa\n",
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import sys\n",
    "\n",
    "def wav_to_matrix(filename, hop, win_len): # hop and win_len in milisecond \n",
    "    audio, sr = librosa.load(filename)\n",
    "    # https://github.com/librosa/librosa/issues/584\n",
    "    mfccs = librosa.feature.mfcc(audio, sr, n_mfcc=40, hop_length=int(float(hop/1000)*sr), n_fft=int(float(win_len/1000)*sr))\n",
    "    # line_mfccs = np.ravel(mfccs, order='F')\n",
    "    return mfccs\n",
    "\n",
    "def create_data_for_unsupervised(root_dir, hop, win_len):\n",
    "    all_wav_paths = glob.glob(os.path.join(root_dir, '*m4a'))\n",
    "    matrix_of_all_wav = []\n",
    "    print (all_wav_paths)\n",
    "\n",
    "    # All midi have to be in same shape. (?)\n",
    "    for single_wav_path in all_wav_paths:\n",
    "        print (single_wav_path)\n",
    "        matrix_of_single_wav = wav_to_matrix(single_wav_path, hop, win_len)\n",
    "        array_of_single_wav = np.ravel(matrix_of_single_wav)\n",
    "        \n",
    "        if (matrix_of_single_wav is not None):\n",
    "            print (matrix_of_single_wav.shape)\n",
    "            matrix_of_all_wav.extend(array_of_single_wav)\n",
    "            print (single_wav_path + \" is done.\")\n",
    "            \n",
    "    wav_array = np.asarray(matrix_of_all_wav)\n",
    "    wav_array = np.reshape(matrix_of_all_wav, (40,-1))\n",
    "    input_array = []\n",
    "    output_array = []\n",
    "\n",
    "    print (wav_array.shape)\n",
    "    \n",
    "    start_point = 0\n",
    "    \n",
    "    while (start_point+200 < wav_array.shape[1]):\n",
    "        single_input = wav_array[0:40,start_point:start_point+100]\n",
    "        single_output = wav_array[:40, start_point+100:start_point+200]\n",
    "        input_array.append(single_input)\n",
    "        output_array.append(single_output)\n",
    "        start_point += 100\n",
    "        \n",
    "        \n",
    "    input_array = np.asarray(input_array)\n",
    "    input_array = input_array.reshape((len(input_array), np.prod(input_array.shape[1:])))  \n",
    "    print(input_array.shape)\n",
    "\n",
    "    output_array = np.asarray(output_array)\n",
    "    output_array = output_array.reshape((len(output_array), np.prod(output_array.shape[1:])))  \n",
    "    print(output_array.shape)\n",
    "    \n",
    "    return (input_array, output_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_array, output_array = create_data_for_unsupervised('./audio_files/', 10, 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's focus on AMI Corpus**\n",
    "\n",
    "In the [dev.mdtm](https://github.com/pyannote/pyannote-db-odessa-ami/tree/master/AMI/data/speaker_diarization), we can see the:\n",
    "    - filename, offset, duration, \"speaker NA unknown\", id of speaker\n",
    "\n",
    "We should focus on this. We can download the corresponding .wav files easily. \n",
    "\n",
    "Firsly, I will create database for speaker change detection. It will be not-optimized until the end of test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir txt_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "def create_ground_truth(main_set):\n",
    "    # start_time_array = []\n",
    "    end_time_array = []\n",
    "    output_frame_array = []\n",
    "    with open(main_set) as f:\n",
    "        content = f.readlines()\n",
    "    # you may also want to remove whitespace characters like `\\n` at the end of each line\n",
    "    content = [x.strip() for x in content] \n",
    "    tmp_filename = content[0].split(' ')[0]\n",
    "    # sys.exit(\"WAIT, TEST :)\")\n",
    "\n",
    "    for single_line in content:\n",
    "        filename = single_line.split(' ')[0]\n",
    "        # print (filename)\n",
    "        if (filename != tmp_filename):\n",
    "            # start_time_array = np.asarray(start_time_array)\n",
    "            # np.savetxt(fname=tmp_filename + \"_start_time.txt\", X=start_time_array, delimiter=' ', fmt='%1.3f')\n",
    "            # end_time_array = np.asarray(end_time_array)\n",
    "            np.savetxt(fname=tmp_filename + \"_end_time.txt\", X=end_time_array, delimiter=' ', fmt='%1.3f')\n",
    "            # start_time_array = []\n",
    "            end_time_array = []\n",
    "            \n",
    "        tmp_filename = single_line.split(' ')[0]\n",
    "        offset = float(single_line.split(' ')[2])\n",
    "        duration = float(single_line.split(' ')[3])\n",
    "        end_time = offset+duration\n",
    "        # start_time_array.append(offset)\n",
    "        end_time_array.append(end_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd txt_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/pyannote/pyannote-db-odessa-ami/master/AMI/data/speaker_diarization/dev.mdtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_ground_truth('dev.mdtm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Architectures\n",
    "\n",
    "- They use 2 different autoencoder. \n",
    "    - Smaller is 4000 → 2000 → 40 → 2000 → 4000\n",
    "    - Larger is 4000 → 6000 → 2000 → 40 → 2000 → 6000 → 4000\n",
    "![Image of Autoencoder](https://docs.google.com/uc?id=1epse9ba1fRTdmyN3pF0XoECEHcaiLsa3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import layers\n",
    "from keras import models\n",
    "import keras\n",
    "from keras.models import Model\n",
    "import tensorflow as tf\n",
    "from keras.layers.advanced_activations import *\n",
    "\n",
    "\n",
    "input_frame = keras.Input(shape=(4000,), name='main_input')\n",
    "encoded_1 = layers.Dense(4000, activation='relu', name='encoded_1')(input_frame) # not quite sure about the activation\n",
    "encoded_2 = layers.Dense(2000, activation='relu', name='encoded_2')(encoded_1)\n",
    "\n",
    "embedding = layers.Dense(40, activation='relu', name='embedding')(encoded_2)\n",
    "\n",
    "decoded_1 = layers.Dense(2000, activation='relu', name='decoded_1')(embedding)\n",
    "decoded_2 = layers.Dense(4000, activation='linear', name='decoded_2')(decoded_1)\n",
    "\n",
    "autoencoder = Model(input_frame, decoded_2)\n",
    "autoencoder.compile(optimizer='adadelta', loss='mean_squared_logarithmic_error')\n",
    "\n",
    "autoencoder.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "autoencoder.fit(input_array, output_array,\n",
    "               epochs=2,\n",
    "               batch_size=32,\n",
    "               shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's try to detect speaker change points. We will use encoder part of autoencoder.**\n",
    "- Firstly, extract the embedding,\n",
    "- Compare neighboorhood embeddings via different parameters,\n",
    "- If result of comparision exceed determined threshold, it represent the speaker change point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_system = Model(input_frame, embedding) # create system to extraxt embed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have embedding system. We should feed this sytem via input frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def KL(P,Q):\n",
    "    \"\"\" Epsilon is used here to avoid conditional code for\n",
    "    checking that neither P nor Q is equal to 0. \"\"\"\n",
    "    epsilon = 0.00001\n",
    "\n",
    "     # You may want to instead make copies to avoid changing the np arrays.\n",
    "    P = P+epsilon\n",
    "    Q = Q+epsilon\n",
    "\n",
    "    divergence = np.sum(P*np.log(P/Q))\n",
    "    return divergence\n",
    "\n",
    "def create_prediction(wav_file, hop, win_len, threshold, embedding_system):\n",
    "    prediction_array = []\n",
    "    audio, sr = librosa.load(wav_file)\n",
    "    # at that point, we should not use overlapping segment\n",
    "    # https://github.com/librosa/librosa/issues/584\n",
    "    mfccs = librosa.feature.mfcc(audio, sr, n_mfcc=40, hop_length=int(hop/1000*sr), n_fft=int(win_len/1000*sr))\n",
    "    # now we have mfcc of all audio file.\n",
    "    offset=0\n",
    "    while (offset+200 < mfccs.shape[1]):\n",
    "    # line_mfccs = np.ravel(mfccs, order='F')\n",
    "        first_frame = np.ravel(mfccs[0:40,offset:offset+100])\n",
    "        first_frame = np.expand_dims(first_frame, axis=0)\n",
    "        first_embed = embedding_system.predict(first_frame)\n",
    "        \n",
    "        second_frame = np.ravel(mfccs[0:40,offset+100:offset+200])\n",
    "        second_frame = np.expand_dims(second_frame, axis=0)\n",
    "        second_embed = embedding_system.predict(second_frame)\n",
    "        \n",
    "        if (KL(first_embed, second_embed) > threshold):\n",
    "            ms_version = float(win_len + ((offset+100) * hop)) # milisecond version to represent end point of first embed\n",
    "            prediction_array.append(ms_version/1000)\n",
    "        offset += 100\n",
    "    prediction_array = np.asarray(prediction_array)\n",
    "    print (\"cx\")\n",
    "    np.savetxt(fname=wav_file + \"_prediction.txt\", X=prediction_array, \n",
    "               delimiter=' ', fmt='%1.3f')\n",
    "\n",
    "    return prediction_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now, implement transfer learning part. We will do it because of unsupervised domain adaptation.**\n",
    "- Find the speaker change points by trained DNN,\n",
    "- Get all possible speaker homogeneous regions,\n",
    "- Retrain the same DNN again on these homogeneous segments of speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "change_points = create_prediction('test.wav', hop=10, win_len=25, threshold=0.6, embedding_system=embedding_system)\n",
    "\n",
    "# This is for unsupervised adaptation.\n",
    "\n",
    "## Very bad according to optimization view. !!!\n",
    "def creata_data_for_unsuper_adaptation(root_dir, hop, win_len):\n",
    "    all_wav_paths = glob.glob(os.path.join(root_dir, '*wav'))\n",
    "    matrix_of_all_wav = []\n",
    "    \n",
    "    input_array = []\n",
    "    output_array = []\n",
    "    \n",
    "    for single_wav_path in all_wav_paths:\n",
    "        prediction_array = create_prediction(wav_file, hop, win_len, threshold, embedding_system)\n",
    "        matrix_of_single_wav = wav_to_matrix(single_wav_path, hop, win_len)\n",
    "            \n",
    "        start_point = 0\n",
    "        while (start_point+200 < matrix_of_single_wav.shape[1]):\n",
    "            change = 0 # value to check any value in prediction array is in range of frame\n",
    "            for single_change in prediction_array:\n",
    "                if (single_change in range(float(win_len + ((offset) * hop)), \n",
    "                                               float(win_len + ((offset+200) * hop)))):\n",
    "                    change = 1\n",
    "            if (change == 0):\n",
    "                single_input = wav_array[0:40,start_point:start_point+100]\n",
    "                single_output = wav_array[:40, start_point+100:start_point+200]\n",
    "                input_array.append(single_input)\n",
    "                output_array.append(single_output)\n",
    "            start_point += 100\n",
    "\n",
    "    input_array = np.asarray(input_array)\n",
    "    input_array = input_array.reshape((len(input_array), np.prod(input_array.shape[1:])))\n",
    "    \n",
    "    output_array = np.asarray(output_array)\n",
    "    output_array = output_array.reshape((len(output_array)*output_array.shape[1], output_array.shape[2]*output_array.shape[3] ))\n",
    "\n",
    "    print(output_array.shape)\n",
    "    \n",
    "    return (input_array, output_array)\n",
    "\n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}