2.3 Combining Language-Vision Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d9bfcced",
   "metadata": {
    "id": "fLkH5evlYL8E"
   },
   "source": [
    "## Overview\n",
    "\n",
    "In this jupyter notebook, we showcase a possible product of our sub-task in building language-vision model.\n",
    "- First, we create a mock data from visual model, consists of label names, x-y coordinates of the grabbing point, in the real-time scene. We also import our trained language models, its dictionary and tokenizer. \n",
    "- Then, we build a function that takes textual description, the model, and data from visual models. Through the function `max_prob`, we outputs the coordinates ranked and accompanied by their predicted probability. This is only one of the simplest way of combining the results of our language and vision model. In further research, we could further explore other possible combining methods. We send this output to the central control system where the robot arm automatically grab the piece.\n",
    "- In the last part, we simulate the real-time word-by-word input (incremental feeding), which our group would actually receive from ASR system."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee391bd",
   "metadata": {},
   "source": [
    "## Create mock coordinate data from visual model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c3235666",
   "metadata": {},
   "outputs": [],
   "source": [
    "mock_visual = [('blue v', {'x': 50, 'y': 63}),\n",
    " ('wooden v', {'x': 73, 'y': 56}),\n",
    " ('green w', {'x': 32, 'y': 98}),\n",
    " ('green t', {'x': 129, 'y': 69}),\n",
    " ('purple n', {'x': 58, 'y': 139}),\n",
    " ('wooden z', {'x': 40, 'y': 105}),\n",
    " ('blue z', {'x': 137, 'y': 163}),\n",
    " ('wooden w', {'x': 57, 'y': 24}),\n",
    "# ('blue i', {'x': 74, 'y': 133}),\n",
    "# ('yellow t', {'x': 48, 'y': 26}),\n",
    "# ('wooden i', {'x': 145, 'y': 29}),\n",
    "# ('yellow u', {'x': 67, 'y': 27})\n",
    "              ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c75b8da",
   "metadata": {},
   "source": [
    "# Import tokenizer and language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5af4b9a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import json\n",
    "from keras_preprocessing.text import tokenizer_from_json\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import load_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "15617a09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras_preprocessing.text.Tokenizer at 0x24b4d50e0a0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import tokenizer\n",
    "with open('tokenizer.json') as f:\n",
    "    data = json.load(f)\n",
    "    tokenizer = tokenizer_from_json(data)\n",
    "    \n",
    "tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f2a52d70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get label dictionary\n",
    "# Read data from file:\n",
    "all_label_dict = json.load( open( \"all_label_dict.json\" ) )\n",
    "all_label = list(all_label_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a42057fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 287, 100)          20000     \n",
      "_________________________________________________________________\n",
      "spatial_dropout1d_1 (Spatial (None, 287, 100)          0         \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, 256)               234496    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 23)                5911      \n",
      "=================================================================\n",
      "Total params: 260,407\n",
      "Trainable params: 260,407\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "#load model\n",
    "model = load_model(\"blstm_aug.h5\")\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bc70a1cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(None, 287, 100), (None, 287, 100), (None, 256), (None, 23)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_shape = []\n",
    "for l in model.layers:\n",
    "    model_shape.append(l.output_shape)\n",
    "model_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066ad7fc",
   "metadata": {},
   "source": [
    "### Test the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "e1ef574d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def text2pred(text, model):\n",
    "    seq = tokenizer.texts_to_sequences(text)\n",
    "    padded = pad_sequences(seq, maxlen=MAX_SEQUENCE_LENGTH)    \n",
    "    return model.predict(padded)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "ed60bc8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "green w\n"
     ]
    }
   ],
   "source": [
    "# check if model is predicting\n",
    "MAX_SEQUENCE_LENGTH = model_shape[1][1] # shape[1] from embedding_layer of the model\n",
    "MAX_NB_WORDS = 200\n",
    "\n",
    "# test with new data\n",
    "new_text = ['we are looking for the dark wooden piece that shape like l']\n",
    "new_text = ['yellow u']\n",
    "new_text = ['Wshaped green Center Above']\n",
    "pred = text2pred(new_text,model)\n",
    "\n",
    "print(np.argmax(pred))\n",
    "print(all_label[np.argmax(pred)]) #the list of all label here MUST match the list of label that the model is trained on as well"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "888aa7ad",
   "metadata": {},
   "source": [
    "# Predicting coordinates\n",
    "\n",
    "We input textual description, model, and visual coordinates to the function `max_prob`. The function outputs probabilities of each label, then filters those only detected in the visual scene. Then, it returns pairs of `n` most probable coordinates of the description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "56586802",
   "metadata": {},
   "outputs": [],
   "source": [
    "# with neural model\n",
    "def max_prob(text, model, visual_dict, n):\n",
    "    \n",
    "    visual_list = [i for (i,j) in visual_dict]\n",
    "\n",
    "    #get label idx from the list\n",
    "    idx_list = [all_label_dict.get(k) for k in all_label if k in visual_list]\n",
    "    \n",
    "    # make prediction\n",
    "    pred = text2pred(text,model)\n",
    "\n",
    "    # map proba with labels\n",
    "    pred_list = list(pred.reshape(-1))\n",
    "    proba_dict = {i:pred_list[idx] for idx,i in enumerate(all_label)}\n",
    "    \n",
    "    # get only prediction of the visible piece\n",
    "    pick_from_result = {d:proba_dict.get(d) for d in visual_list}\n",
    "    max_prob = sorted(pick_from_result.items(), key=lambda x:x[1], reverse=True)[:n]\n",
    "    \n",
    "    res_dict = {}\n",
    "    \n",
    "    for i in max_prob:\n",
    "        for j in visual_dict:\n",
    "            if i[0] == j[0]:\n",
    "                res_dict.update({i[1]:j[1]})\n",
    "            \n",
    "    return res_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "51d7cebd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample texts\n",
    "text = 'the green one'\n",
    "text = 'give me the straight piece please'\n",
    "text = 'the yellowish one'\n",
    "text = 'give me the pink piece'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "51d59c07",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0.62383866: {'x': 32, 'y': 98},\n",
       " 0.58254784: {'x': 129, 'y': 69},\n",
       " 0.45683467: {'x': 73, 'y': 56}}"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_prob(text, model, mock_visual,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adbb065d",
   "metadata": {
    "id": "29e30e02"
   },
   "source": [
    "# Incrememtal feeding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "51b870f4",
   "metadata": {
    "id": "b93011e2"
   },
   "outputs": [],
   "source": [
    "def feed_one(text,k):\n",
    "    \"\"\"\n",
    "    input: one input string\n",
    "    output: a list, consist of strings where 1 word is added at a time\n",
    "    \"\"\"\n",
    "    feed_one = []\n",
    "    tokenized = text.split()\n",
    "    for i, t in enumerate(tokenized):\n",
    "        if i+1 >= k:\n",
    "            feed_one.append(' '.join(tokenized[:i+1]))\n",
    "    return feed_one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "926a52cd",
   "metadata": {
    "id": "269437b1",
    "outputId": "2094d913-10d4-4eee-a471-53e42282505a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['top center Lshaped',\n",
       " 'top center Lshaped stone',\n",
       " 'top center Lshaped stone blue']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test 'feed_one' function\n",
    "feed_one(\"top center Lshaped stone blue\", 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "c9051b43",
   "metadata": {
    "id": "0c58cab8"
   },
   "outputs": [],
   "source": [
    "def increm_feed_predict(model, text, top_proba, threshold):\n",
    "    \"\"\"\n",
    "    input: text string\n",
    "    output: list of dictionary of top_proba\n",
    "\n",
    "    *the function also print the input, its probs, the prediction\n",
    "    \n",
    "    \"\"\"\n",
    "    list_of_proba = []\n",
    "    \n",
    "    for t in feed_one(text,3):\n",
    "        x = list()\n",
    "        x.append(t)\n",
    "        print(t)\n",
    "        \n",
    "        print (max_prob([t], model, mock_visual,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "91112360",
   "metadata": {
    "id": "e2b6b0c8",
    "outputId": "46c622e3-6b79-4740-9301-091c37305e9d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I want the\n",
      "{0.99552584: {'x': 137, 'y': 163}, 0.3412792: {'x': 57, 'y': 24}, 0.2778092: {'x': 58, 'y': 139}}\n",
      "I want the pink\n",
      "{0.99927044: {'x': 137, 'y': 163}, 0.99729013: {'x': 73, 'y': 56}, 0.9928199: {'x': 57, 'y': 24}}\n",
      "I want the pink one\n",
      "{0.99952453: {'x': 137, 'y': 163}, 0.99868923: {'x': 73, 'y': 56}, 0.9970174: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh\n",
      "{0.99952453: {'x': 137, 'y': 163}, 0.99868923: {'x': 73, 'y': 56}, 0.9970174: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh no\n",
      "{0.99952453: {'x': 137, 'y': 163}, 0.99868923: {'x': 73, 'y': 56}, 0.9970174: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh no I\n",
      "{0.9999925: {'x': 137, 'y': 163}, 0.99315476: {'x': 73, 'y': 56}, 0.9867716: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh no I actually\n",
      "{0.9999925: {'x': 137, 'y': 163}, 0.99315476: {'x': 73, 'y': 56}, 0.9867716: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh no I actually want\n",
      "{0.9999925: {'x': 137, 'y': 163}, 0.99315476: {'x': 73, 'y': 56}, 0.9867716: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh no I actually want the\n",
      "{0.9999846: {'x': 137, 'y': 163}, 0.9958007: {'x': 73, 'y': 56}, 0.99271566: {'x': 57, 'y': 24}}\n",
      "I want the pink one oh no I actually want the purple\n",
      "{0.9998718: {'x': 50, 'y': 63}, 0.9993148: {'x': 57, 'y': 24}, 0.9977622: {'x': 137, 'y': 163}}\n",
      "I want the pink one oh no I actually want the purple one\n",
      "{0.99955815: {'x': 50, 'y': 63}, 0.9991479: {'x': 57, 'y': 24}, 0.99683833: {'x': 73, 'y': 56}}\n"
     ]
    }
   ],
   "source": [
    "# use random made-up sentence to get prediction and prob\n",
    "# comment out when not need\n",
    "test_text = \"looking for brown piece with weird Lshape with elogated part jutted out from the longer side\"\n",
    "test_text = \"give me the yellow one there on your right?\"\n",
    "test_text = \"I want that something weirdly angular shape that's in pink\"\n",
    "test_text = 'the n as a piece consisting of two rectangles'\n",
    "test_text = 'the green t shape on the top left corner'\n",
    "test_text = 'I want the pink one oh no I actually want the purple one'\n",
    "\n",
    "increm_feed_predict(model,test_text,5,0.1) # input = text, max no. of proba to output, threshold of proba"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "558b91e1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}