notebooks/qa-gen-query.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Synthetic Query Generation: Zero-Shot and Few-Shot\n",
    "\n",
    "This notebook illustrates the utilization of the Llama2-Chat prompt templates to create synthetic context-query data. It utilizes the [Llama-2-13b-chat-hf](https://huggingface.co/meta-llama), integrating seamlessly with the Hugging Face library for this purpose.\n",
    "\n",
    "[Two prompting techniques](https://blog.reachsumit.com/posts/2023/03/llm-for-text-ranking/) are demonstrated:\n",
    "1) Basic zero-shot query generation - referred to as vanilla\n",
    "2) Few-shot with Guided by Bad Questions (GBQ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# User Inputs and Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from types import SimpleNamespace\n",
    "\n",
    "# Specify paths to data, prompt templates, llama model, etc.\n",
    "paths = {'base_dir': Path.cwd().parents[0],\n",
    "         'prompt_vanilla': 'notebooks/question-answering-prompts/vanilla.txt',\n",
    "         'prompt_gbq': 'notebooks/question-answering-prompts/gbq.txt',\n",
    "         'squad_data': 'data/squad_v2',\n",
    "         'model': '/nvme4tb/Projects/llama2_models/Llama-2-13b-chat-hf',\n",
    "         }\n",
    "\n",
    "# Number of context samples for experimentation\n",
    "NUM_SAMPLES = 3\n",
    "\n",
    "# Convert from dictionary to SimpleNamespace\n",
    "paths = SimpleNamespace(**paths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries and packages\n",
    "import pandas as pd\n",
    "from time import time\n",
    "from IPython.display import clear_output\n",
    "import re\n",
    "import torch\n",
    "from transformers import LlamaTokenizer, LlamaForCausalLM\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load Data\n",
    "The [Stanford Question Answering Dataset squad_v2](https://huggingface.co/datasets/squad_v2) dataset was downloaded from Hugging Face and stored locally. The Stanford Question Answering Dataset (SQuAD) is a collection of question-answer pairs derived from Wikipedia articles. In SQuAD, the correct answers of questions can be any sequence of tokens in the given text. Because the questions and answers are produced by humans through crowdsourcing, it is more diverse than some other question-answering datasets. \n",
    "\n",
    "If internet connection is available you can alternatively download the dataset as shown:\n",
    "```python\n",
    "df = load_dataset('squad_v2')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df.shape: (19029, 5)\n",
      "Columns: ['id', 'title', 'context', 'question', 'answers']\n",
      "Number of Words in Context\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count    19029.000000\n",
       "mean       116.600137\n",
       "std         49.666777\n",
       "min         20.000000\n",
       "25%         87.000000\n",
       "50%        107.000000\n",
       "75%        139.000000\n",
       "max        653.000000\n",
       "Name: num_words_context, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Load squad_v2 data locally from disk\n",
    "df = load_dataset(str(paths.base_dir / paths.squad_data),\n",
    "                  split='train').to_pandas()\n",
    "\n",
    "# Remove redundant context\n",
    "df = df.drop_duplicates(subset=['context', 'title']).reset_index(drop=True)\n",
    "print(f'df.shape: {df.shape}')\n",
    "print(f'Columns: {df.columns.tolist()}')\n",
    "\n",
    "# Approximate the # of words in context\n",
    "df['num_words_context'] = df.context.apply(lambda x: len(x.split()))\n",
    "print('Number of Words in Context')\n",
    "display(df.num_words_context.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example # 1\n",
      "Context: The Oklahoma City Police Department, has a uniformed force of 1,169 officers and 300+ civilian employees. The Department has a central police station and five substations covering 2,500 police reporting districts that average 1/4 square mile in size.\n",
      "Squad Query: How many substations does Oklahoma city have?\n",
      "\n",
      "Example # 2\n",
      "Context: The U.S. Federal Reserve and central banks around the world have taken steps to expand money supplies to avoid the risk of a deflationary spiral, in which lower wages and higher unemployment lead to a self-reinforcing decline in global consumption. In addition, governments have enacted large fiscal stimulus packages, by borrowing and spending to offset the reduction in private sector demand caused by the crisis. The U.S. Federal Reserve's new and expanded liquidity facilities were intended to enable the central bank to fulfill its traditional lender-of-last-resort role during the crisis while mitigating stigma, broadening the set of institutions with access to liquidity, and increasing the flexibility with which institutions could tap such liquidity.\n",
      "Squad Query: What have central banks around the world done to avoid the risk of a deflationary spiral?\n",
      "\n",
      "Example # 3\n",
      "Context: The two finalists were Kris Allen and Adam Lambert, both of whom had previously landed in the bottom three at the top five. Allen won the contest in the most controversial voting result since season two. It was claimed, later retracted, that 38 million of the 100 million votes cast on the night came from Allen's home state of Arkansas alone, and that AT&T employees unfairly influenced the votes by giving lessons on power-texting at viewing parties in Arkansas.\n",
      "Squad Query: Who were the final two contestants on season eight of American Idol?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Randomly select 50 contexts\n",
    "df = df.sample(n=NUM_SAMPLES, random_state=42)[['id', 'context', 'question']]\n",
    "\n",
    "# View a few context and questions from original dataset\n",
    "for ii in range(NUM_SAMPLES):\n",
    "    print(f'Example # {ii + 1}')\n",
    "    print(f'Context: {df.iloc[ii].context}')\n",
    "    print(f'Squad Query: {df.iloc[ii].question}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prompt Templates\n",
    "\n",
    "Two different prompt templates will be demonstrated in this notebook:\n",
    "1) Basic vanilla zero-shot query generation.\n",
    "2) [Few-shot with Guided by Bad Questions (GBQ)](https://blog.reachsumit.com/posts/2023/03/llm-for-text-ranking/): illustrated below image and detailed in [InPars paper](https://arxiv.org/abs/2301.01820).\n",
    "<p align=\"center\"> \n",
    "    <img src=\"https://raw.githubusercontent.com/mddunlap924/LLM-Prompting/main/imgs/inpars-gbq.png\"\n",
    "    style=\"width:756;height:512px;\">\n",
    "    <br>\n",
    "    Left: Vanilla template, Right: GBQ prompts <a href=\"https://blog.gopenai.com/enrich-llms-with-retrieval-augmented-generation-rag-17b82a96b6f0\">[Source]</a>.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt Template: vanilla.txt\n",
      "<s>[INST] <<SYS>>\n",
      "You are a question generating assistant. \n",
      "Given a document, please generate a simple and short question based on the information provided.\n",
      "The question can be a maximum of 10 words long.\n",
      "Return only the question in the JSON format shown in the examples.\n",
      "<</SYS>>\n",
      "\n",
      "\"DOCUMENT\": The Oklahoma City Police Department, has a uniformed force of 1,169 officers and 300+ civilian employees. The Department has a central police station and five substations covering 2,500 police reporting districts that average 1/4 square mile in size.\n",
      "{\"QUESTION\": Your question here.}\n",
      "[/INST]\n",
      "\n",
      "Prompt Template: gbq.txt\n",
      "<s>[INST] <<SYS>>\n",
      "You are a question generating assistant. Given a document and a bad question; please generate a better more detailed question based on the information provided. \n",
      "Below are three examples of bad questions and good questions with sample relevant documents for each question.\n",
      "Return only the question in the JSON format shown in the examples.\n",
      "\n",
      "Example # 1\n",
      "DOCUMENT: Guam lies between 13.2°N and 13.7°N and between 144.6°E and 145.0°E, and has an area of 212 square miles (549 km2), making it the 32nd largest island of the United States. It is the southernmost and largest island in the Mariana island chain and is also the largest island in Micronesia. This island chain was created by the colliding Pacific and Philippine Sea tectonic plates. Guam is the closest land mass to the Mariana Trench, a deep subduction zone, that lies beside the island chain to the east. Challenger Deep, the deepest surveyed point in the Oceans, is southwest of Guam at 35,797 feet (10,911 meters) deep. The highest point in Guam is Mount Lamlam at an elevation of 1,334 feet (407 meters).\n",
      "BAD QUESTION: How many square miles is Guam?\n",
      "{QUESTION: The island chain that Guam belongs to was created by the collision of what two tectonic plates?}\n",
      "\n",
      "Example # 2\n",
      "DOCUMENT: A TiVo service update in July 2008 allowed the system to search and play YouTube videos. In January 2009, YouTube launched \"YouTube for TV\", a version of the website tailored for set-top boxes and other TV-based media devices with web browsers, initially allowing its videos to be viewed on the PlayStation 3 and Wii video game consoles. In June 2009, YouTube XL was introduced, which has a simplified interface designed for viewing on a standard television screen. YouTube is also available as an app on Xbox Live. On November 15, 2012, Google launched an official app for the Wii, allowing users to watch YouTube videos from the Wii channel. An app is also available for Wii U and Nintendo 3DS, and videos can be viewed on the Wii U Internet Browser using HTML5. Google made YouTube available on the Roku player on December 17, 2013 and in October 2014, the Sony PlayStation 4.\n",
      "BAD QUESTION: What service was able to search and play youtube videos as of 2008?\n",
      "{QUESTION: In early 2009 what service did YouTube launch that played videos on PlayStation?}\n",
      "\n",
      "Example # 3\n",
      "DOCUMENT: Eisenhower retired to the place where he and Mamie had spent much of their post-war time, a working farm adjacent to the battlefield at Gettysburg, Pennsylvania, only 70 miles from his ancestral home in Elizabethville, Dauphin County, Pennsylvania. In 1967 the Eisenhowers donated the farm to the National Park Service. In retirement, the former president did not completely retreat from political life; he spoke at the 1964 Republican National Convention and appeared with Barry Goldwater in a Republican campaign commercial from Gettysburg. However, his endorsement came somewhat reluctantly because Goldwater had attacked the former president as \"a dime-store New Dealer\".\n",
      "BAD QUESTION: What town did Eisenhower retire to after his presidency?\n",
      "{QUESTION: After Eisenhower finished his presidency where did he and his wife retire?}\n",
      "<</SYS>>\n",
      "\n",
      "Generate a good question only for Example 4 and respond using the JSON format!\n",
      "Example 4:\n",
      "DOCUMENT: The Oklahoma City Police Department, has a uniformed force of 1,169 officers and 300+ civilian employees. The Department has a central police station and five substations covering 2,500 police reporting districts that average 1/4 square mile in size.\n",
      "{\"QUESTION\": Your question here.}\n",
      "[/INST]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load the each prompt template and insert context\n",
    "for template_name in [paths.prompt_vanilla, paths.prompt_gbq]:\n",
    "    # Name of prompt template\n",
    "    print(f'Prompt Template: {(paths.base_dir / template_name).name}')\n",
    "    \n",
    "    # Load the prompt template \n",
    "    prompt_template = open(paths.base_dir / template_name, 'r').read()\n",
    "    \n",
    "    # Insert the context into the prompt template\n",
    "    prompts = [prompt_template.replace('[CONTEXT]', i) for i in df.context.tolist()]\n",
    "\n",
    "    # Example prompt for the first instance of data\n",
    "    print(f'{prompts[0]}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate Synthetic Queries\n",
    "\n",
    "This section will generate queries for each of the two prompts and then compare them to the original Squad query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a6c3cc867174eb69e0d8e7fa73f309e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sun Nov  5 08:05:36 2023       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |\n",
      "| 31%   43C    P2   103W / 350W |  14875MiB / 24576MiB |      9%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "|    0   N/A  N/A     57005      C   ...GenQuery/.venv/bin/python    14872MiB |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "# Load the tokenizer and model\n",
    "tokenizer = LlamaTokenizer.from_pretrained(paths.model)\n",
    "model = LlamaForCausalLM.from_pretrained(paths.model,\n",
    "                                         load_in_8bit=True,\n",
    "                                         device_map='cuda:0',\n",
    "                                         torch_dtype=torch.float32)\n",
    "\n",
    "# View GPU vRAM\n",
    "!nvidia-smi\n",
    "\n",
    "# Notice: Llama-2 13B with 8bit quantization is ~14.8GB of vRAM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Time to Generate 6 Queries: 0.9 mins.\n",
      "Avg. Amount of Seconds Per Sample: 8.9\n",
      "[INST] <<SYS>>\n",
      "You are a question generating assistant. Given a document and a bad question; please generate a better more detailed question based on the information provided. \n",
      "Below are three examples of bad questions and good questions with sample relevant documents for each question.\n",
      "Return only the question in the JSON format shown in the examples.\n",
      "\n",
      "Example # 1\n",
      "DOCUMENT: Guam lies between 13.2°N and 13.7°N and between 144.6°E and 145.0°E, and has an area of 212 square miles (549 km2), making it the 32nd largest island of the United States. It is the southernmost and largest island in the Mariana island chain and is also the largest island in Micronesia. This island chain was created by the colliding Pacific and Philippine Sea tectonic plates. Guam is the closest land mass to the Mariana Trench, a deep subduction zone, that lies beside the island chain to the east. Challenger Deep, the deepest surveyed point in the Oceans, is southwest of Guam at 35,797 feet (10,911 meters) deep. The highest point in Guam is Mount Lamlam at an elevation of 1,334 feet (407 meters).\n",
      "BAD QUESTION: How many square miles is Guam?\n",
      "{QUESTION: The island chain that Guam belongs to was created by the collision of what two tectonic plates?}\n",
      "\n",
      "Example # 2\n",
      "DOCUMENT: A TiVo service update in July 2008 allowed the system to search and play YouTube videos. In January 2009, YouTube launched \"YouTube for TV\", a version of the website tailored for set-top boxes and other TV-based media devices with web browsers, initially allowing its videos to be viewed on the PlayStation 3 and Wii video game consoles. In June 2009, YouTube XL was introduced, which has a simplified interface designed for viewing on a standard television screen. YouTube is also available as an app on Xbox Live. On November 15, 2012, Google launched an official app for the Wii, allowing users to watch YouTube videos from the Wii channel. An app is also available for Wii U and Nintendo 3DS, and videos can be viewed on the Wii U Internet Browser using HTML5. Google made YouTube available on the Roku player on December 17, 2013 and in October 2014, the Sony PlayStation 4.\n",
      "BAD QUESTION: What service was able to search and play youtube videos as of 2008?\n",
      "{QUESTION: In early 2009 what service did YouTube launch that played videos on PlayStation?}\n",
      "\n",
      "Example # 3\n",
      "DOCUMENT: Eisenhower retired to the place where he and Mamie had spent much of their post-war time, a working farm adjacent to the battlefield at Gettysburg, Pennsylvania, only 70 miles from his ancestral home in Elizabethville, Dauphin County, Pennsylvania. In 1967 the Eisenhowers donated the farm to the National Park Service. In retirement, the former president did not completely retreat from political life; he spoke at the 1964 Republican National Convention and appeared with Barry Goldwater in a Republican campaign commercial from Gettysburg. However, his endorsement came somewhat reluctantly because Goldwater had attacked the former president as \"a dime-store New Dealer\".\n",
      "BAD QUESTION: What town did Eisenhower retire to after his presidency?\n",
      "{QUESTION: After Eisenhower finished his presidency where did he and his wife retire?}\n",
      "<</SYS>>\n",
      "\n",
      "Generate a good question only for Example 4 and respond using the JSON format!\n",
      "Example 4:\n",
      "DOCUMENT: The Oklahoma City Police Department, has a uniformed force of 1,169 officers and 300+ civilian employees. The Department has a central police station and five substations covering 2,500 police reporting districts that average 1/4 square mile in size.\n",
      "{\"QUESTION\": Your question here.}\n",
      "[/INST]  Sure! Here's a better, more detailed question based on the information provided in Example 4:\n",
      "\n",
      "{\n",
      "\"Question\": \"What is the total size of the Oklahoma City Police Department's police reporting districts, on average?\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Start time\n",
    "st = time()\n",
    "\n",
    "# Loop over each prompt template\n",
    "counter = 0\n",
    "prompt_names = []\n",
    "for template_name in [paths.prompt_vanilla, paths.prompt_gbq]:\n",
    "    # Name of prompt template\n",
    "    prompt_name = (paths.base_dir / template_name).name.split('.txt')[0]\n",
    "        \n",
    "    # Load the prompt template \n",
    "    prompt_template = open(paths.base_dir / template_name, 'r').read()\n",
    "    \n",
    "    # Insert the context into the prompt template\n",
    "    prompts = [prompt_template.replace('[CONTEXT]', i) for i in df.context.tolist()]\n",
    "    \n",
    "    # Loop over each prompt\n",
    "    llama_questions = []\n",
    "    for prompt in prompts:\n",
    "        # Tokenize the prompt\n",
    "        batch = tokenizer(prompt, return_tensors='pt')\n",
    "        \n",
    "        # Generate the response from Llama2\n",
    "        response = model.generate(batch[\"input_ids\"].cuda(),\n",
    "                                  do_sample=True,\n",
    "                                  top_k=50,\n",
    "                                  top_p=0.9,\n",
    "                                  temperature=0.75)\n",
    "        # Decode the response\n",
    "        decode_response = tokenizer.decode(response[0], skip_special_tokens=True)\n",
    "        llama_questions.append(decode_response)\n",
    "        clear_output()\n",
    "        counter += 1\n",
    "        \n",
    "    # Store llama queries in the dataframe\n",
    "    df[f'prompt_{prompt_name}'] = llama_questions\n",
    "    prompt_names.append(f'prompt_{prompt_name}')\n",
    "\n",
    "# Total time to generate the queries\n",
    "total_secs = time() - st\n",
    "secs_per_sample = (total_secs / counter)\n",
    "print(f'Total Time to Generate {counter} Queries: {(total_secs / 60):.1f} mins.')\n",
    "print(f'Avg. Amount of Seconds Per Sample: {secs_per_sample:.1f}')\n",
    "\n",
    "# Print an example of a returned llama response\n",
    "print(df['prompt_gbq'].iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 7)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>prompt_vanilla</th>\n",
       "      <th>prompt_gbq</th>\n",
       "      <th>prompt_vanilla_cleaned</th>\n",
       "      <th>prompt_gbq_cleaned</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2141</th>\n",
       "      <td>56df5cdd96943c1400a5d438</td>\n",
       "      <td>The Oklahoma City Police Department, has a uni...</td>\n",
       "      <td>How many substations does Oklahoma city have?</td>\n",
       "      <td>[INST] &lt;&lt;SYS&gt;&gt;\\nYou are a question generating ...</td>\n",
       "      <td>[INST] &lt;&lt;SYS&gt;&gt;\\nYou are a question generating ...</td>\n",
       "      <td>How many civilian employees work for the oklah...</td>\n",
       "      <td>What is the total size of the oklahoma city po...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18339</th>\n",
       "      <td>5733823bd058e614000b5c03</td>\n",
       "      <td>The U.S. Federal Reserve and central banks aro...</td>\n",
       "      <td>What have central banks around the world done ...</td>\n",
       "      <td>[INST] &lt;&lt;SYS&gt;&gt;\\nYou are a question generating ...</td>\n",
       "      <td>[INST] &lt;&lt;SYS&gt;&gt;\\nYou are a question generating ...</td>\n",
       "      <td>What steps have central banks taken to avoid a...</td>\n",
       "      <td>What actions have central banks and government...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>980</th>\n",
       "      <td>56d37ac659d6e414001464d5</td>\n",
       "      <td>The two finalists were Kris Allen and Adam Lam...</td>\n",
       "      <td>Who were the final two contestants on season e...</td>\n",
       "      <td>[INST] &lt;&lt;SYS&gt;&gt;\\nYou are a question generating ...</td>\n",
       "      <td>[INST] &lt;&lt;SYS&gt;&gt;\\nYou are a question generating ...</td>\n",
       "      <td>Did kris allen's home state advantage influenc...</td>\n",
       "      <td>What were the controversial voting results tha...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             id  \\\n",
       "2141   56df5cdd96943c1400a5d438   \n",
       "18339  5733823bd058e614000b5c03   \n",
       "980    56d37ac659d6e414001464d5   \n",
       "\n",
       "                                                 context  \\\n",
       "2141   The Oklahoma City Police Department, has a uni...   \n",
       "18339  The U.S. Federal Reserve and central banks aro...   \n",
       "980    The two finalists were Kris Allen and Adam Lam...   \n",
       "\n",
       "                                                question  \\\n",
       "2141       How many substations does Oklahoma city have?   \n",
       "18339  What have central banks around the world done ...   \n",
       "980    Who were the final two contestants on season e...   \n",
       "\n",
       "                                          prompt_vanilla  \\\n",
       "2141   [INST] <<SYS>>\\nYou are a question generating ...   \n",
       "18339  [INST] <<SYS>>\\nYou are a question generating ...   \n",
       "980    [INST] <<SYS>>\\nYou are a question generating ...   \n",
       "\n",
       "                                              prompt_gbq  \\\n",
       "2141   [INST] <<SYS>>\\nYou are a question generating ...   \n",
       "18339  [INST] <<SYS>>\\nYou are a question generating ...   \n",
       "980    [INST] <<SYS>>\\nYou are a question generating ...   \n",
       "\n",
       "                                  prompt_vanilla_cleaned  \\\n",
       "2141   How many civilian employees work for the oklah...   \n",
       "18339  What steps have central banks taken to avoid a...   \n",
       "980    Did kris allen's home state advantage influenc...   \n",
       "\n",
       "                                      prompt_gbq_cleaned  \n",
       "2141   What is the total size of the oklahoma city po...  \n",
       "18339  What actions have central banks and government...  \n",
       "980    What were the controversial voting results tha...  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Clean up the llama response to parse only the returned question\n",
    "def parse_response(text: str):\n",
    "    \n",
    "    # Extract Llama response\n",
    "    text = text.split('[/INST]')[-1].strip(\"</s>\").strip()\n",
    "\n",
    "    # Remove only the question\n",
    "    if 'question' in text.lower():\n",
    "        text = text.lower().split('question')[-1].split('?')[0].strip() + '?'\n",
    "    elif '?' in text:\n",
    "        text = text.split('?')[0].split('\\n')[-1] + '?'\n",
    "    else:\n",
    "        text = 'NAN'\n",
    "    text = re.sub('[\":]', '', text)\n",
    "    text = text.strip()\n",
    "    text = text.capitalize()\n",
    "    return text\n",
    "\n",
    "# Parse each llama response\n",
    "for prompt_name in prompt_names:\n",
    "    df[f'{prompt_name}_cleaned'] = df[f'{prompt_name}'].apply(lambda x: parse_response(text=x))\n",
    "\n",
    "# Display dataframe\n",
    "print(df.shape)\n",
    "display(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example # 1\n",
      "Context: The Oklahoma City Police Department, has a uniformed force of 1,169 officers and 300+ civilian employees. The Department has a central police station and five substations covering 2,500 police reporting districts that average 1/4 square mile in size.\n",
      "Original Squad Query: How many substations does Oklahoma city have?\n",
      "Llama-2 Vanilla Query: How many civilian employees work for the oklahoma city police department?\n",
      "Llama-2 GBQ Query: What is the total size of the oklahoma city police department's police reporting districts, on average?\n",
      "\n",
      "Example # 2\n",
      "Context: The U.S. Federal Reserve and central banks around the world have taken steps to expand money supplies to avoid the risk of a deflationary spiral, in which lower wages and higher unemployment lead to a self-reinforcing decline in global consumption. In addition, governments have enacted large fiscal stimulus packages, by borrowing and spending to offset the reduction in private sector demand caused by the crisis. The U.S. Federal Reserve's new and expanded liquidity facilities were intended to enable the central bank to fulfill its traditional lender-of-last-resort role during the crisis while mitigating stigma, broadening the set of institutions with access to liquidity, and increasing the flexibility with which institutions could tap such liquidity.\n",
      "Original Squad Query: What have central banks around the world done to avoid the risk of a deflationary spiral?\n",
      "Llama-2 Vanilla Query: What steps have central banks taken to avoid a deflationary spiral during the crisis?\n",
      "Llama-2 GBQ Query: What actions have central banks and governments taken to mitigate the risk of a deflationary spiral and stimulate economic growth during the current financial crisis?\n",
      "\n",
      "Example # 3\n",
      "Context: The two finalists were Kris Allen and Adam Lambert, both of whom had previously landed in the bottom three at the top five. Allen won the contest in the most controversial voting result since season two. It was claimed, later retracted, that 38 million of the 100 million votes cast on the night came from Allen's home state of Arkansas alone, and that AT&T employees unfairly influenced the votes by giving lessons on power-texting at viewing parties in Arkansas.\n",
      "Original Squad Query: Who were the final two contestants on season eight of American Idol?\n",
      "Llama-2 Vanilla Query: Did kris allen's home state advantage influence his victory on american idol season 8?\n",
      "Llama-2 GBQ Query: What were the controversial voting results that led to kris allen winning the season 8 finale of american idol, and what were the allegations made regarding the influence of arkansas voters and at&t employees?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# View the examples\n",
    "for ii in range(NUM_SAMPLES):\n",
    "    print(f'Example # {ii + 1}')\n",
    "    print(f'Context: {df.iloc[ii].context}')\n",
    "    print(f'Original Squad Query: {df.iloc[ii].question}')\n",
    "    print(f'Llama-2 Vanilla Query: {df.iloc[ii].prompt_vanilla_cleaned}')\n",
    "    print(f'Llama-2 GBQ Query: {df.iloc[ii].prompt_gbq_cleaned}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Takeaways:\n",
    "- Varying prompts (Vanilla vs. GBQ) generates diverse queries, showcasing the flexibility of LLM prompts.\n",
    "- Further experimentation with prompts can refine query outcomes as needed."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}