14_llamaindex_eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3a43a4b1-b424-41d1-8d39-153cc37f3a08",
   "metadata": {},
   "source": [
    "# Evaluating llamaindex ingestion pipelines\n",
    "\n",
    "This week we are focusing on building and evaluating indexes.\n",
    "\n",
    "There are many ways to chunk and index documents and some are going to perform better than others.\n",
    "Every dataset is different, so you need to find out for your dataset which approach (set of hyperparameters) works best.\n",
    "\n",
    "This notebook demonstrates two ways you can evaluate the quality of the answers, so you can choose the set of hyperparameters that give the best results.\n",
    "\n",
    "- The first evaluation approach doesn't require correct answers, just questions.\n",
    "- The second evaluation approach requires questions and correct answers.\n",
    "\n",
    "Next week we will talk about [DSPy](https://github.com/stanfordnlp/dspy), a framework for optimizing the queries sent to the index.\n",
    "\n",
    "The week after that we will talk about [Optuna](https://optuna.org/), a library to make finding the best hyperparameters easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:20.014116631Z",
     "start_time": "2024-06-02T06:51:20.013126125Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%load_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4f82c275efc6aa93",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:27.966680878Z",
     "start_time": "2024-06-02T06:51:20.554743009Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import chromadb\n",
    "\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, set_global_handler\n",
    "from llama_index.core.ingestion import IngestionPipeline\n",
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "from llama_index.vector_stores.chroma import ChromaVectorStore\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "\n",
    "import nest_asyncio\n",
    "import pandas as pd\n",
    "import phoenix as px\n",
    "from phoenix.evals import (\n",
    "    HUMAN_VS_AI_PROMPT_RAILS_MAP,\n",
    "    HUMAN_VS_AI_PROMPT_TEMPLATE,\n",
    "    HallucinationEvaluator,\n",
    "    OpenAIModel,\n",
    "    QAEvaluator,\n",
    "    RelevanceEvaluator,\n",
    "    run_evals,\n",
    "    llm_classify,\n",
    ")\n",
    "from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents\n",
    "from phoenix.trace import DocumentEvaluations, SpanEvaluations\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ab8255e29444203f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:28.392069312Z",
     "start_time": "2024-06-02T06:51:27.969236719Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# configure\n",
    "filename = 'sleeping_gods.md'\n",
    "qa_filename = 'sleeping_gods-selected.csv'\n",
    "\n",
    "sample_size = 20\n",
    "\n",
    "pd.set_option('display.max_colwidth', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6af2e69948b718d7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:30.267713496Z",
     "start_time": "2024-06-02T06:51:29.580538598Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🌍 To view the Phoenix app in your browser, visit http://localhost:6006/\n",
      "📖 For more information on how to use Phoenix, check out https://docs.arize.com/phoenix\n"
     ]
    }
   ],
   "source": [
    "# setup\n",
    "nest_asyncio.apply()  # needed for concurrent evals in notebook environments\n",
    "\n",
    "# launch the phoenix \n",
    "px.delete_all(prompt_before_delete=False)\n",
    "px_session = px.launch_app()\n",
    "\n",
    "# integrate phoenix into llamaindex\n",
    "set_global_handler(\"arize_phoenix\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4507f257-7671-4fdc-a069-187289b61f2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>manual quote</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://boardgamegeek.com/thread/2873881/i-have-some-questions-after-first-session</td>\n",
       "      <td>If your character has two equipped items, and you wish to use both during a challenge, would you play a command token on each equipped card?</td>\n",
       "      <td>Depends on what you mean. Some ability cards have the ability to discard themselves (when equipped). This will explicitly state so on the card itself. Also: you never put command on ability cards (that command just goes back to the supply). Weapon (not sure whether you meant this instead by 'item') cannot be discarded or re-equipped, and cannot have command on them.</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://boardgamegeek.com/thread/2574093/ship-action-clarification</td>\n",
       "      <td>Before the ship action I have one command left in the pool. I place the worker on the bridge to draw 2 command and place all used command back in the pool. Is this action draw 2 then return all or is it a simultaneous action?</td>\n",
       "      <td>On page 10 of the rulebook it states \"Ship action effects can be applied in any order\". So you would be able to return all the used command to the pool then take 2.</td>\n",
       "      <td>\"Ship action effects can be applied in any order\".</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://boardgamegeek.com/thread/3085176/is-everyone-playing-this-way-with-the-challenges-f</td>\n",
       "      <td>Which crew board abilities can help in challenges?</td>\n",
       "      <td>Only +1 to Fate (Mac) and redraw Fate if 1 (Kasumi).</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                           url  \\\n",
       "0           https://boardgamegeek.com/thread/2873881/i-have-some-questions-after-first-session   \n",
       "1                           https://boardgamegeek.com/thread/2574093/ship-action-clarification   \n",
       "2  https://boardgamegeek.com/thread/3085176/is-everyone-playing-this-way-with-the-challenges-f   \n",
       "\n",
       "                                                                                                                                                                                                                            question  \\\n",
       "0                                                                                       If your character has two equipped items, and you wish to use both during a challenge, would you play a command token on each equipped card?   \n",
       "1  Before the ship action I have one command left in the pool. I place the worker on the bridge to draw 2 command and place all used command back in the pool. Is this action draw 2 then return all or is it a simultaneous action?   \n",
       "2                                                                                                                                                                                 Which crew board abilities can help in challenges?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                             answer  \\\n",
       "0  Depends on what you mean. Some ability cards have the ability to discard themselves (when equipped). This will explicitly state so on the card itself. Also: you never put command on ability cards (that command just goes back to the supply). Weapon (not sure whether you meant this instead by 'item') cannot be discarded or re-equipped, and cannot have command on them.   \n",
       "1                                                                                                                                                                                                              On page 10 of the rulebook it states \"Ship action effects can be applied in any order\". So you would be able to return all the used command to the pool then take 2.   \n",
       "2                                                                                                                                                                                                                                                                                                                              Only +1 to Fate (Mac) and redraw Fate if 1 (Kasumi).   \n",
       "\n",
       "                                         manual quote  \n",
       "0                                                      \n",
       "1  \"Ship action effects can be applied in any order\".  \n",
       "2                                                      "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read question-answers\n",
    "qa_df = pd.read_csv(f'data/{qa_filename}', na_filter=False)\n",
    "print(len(qa_df))\n",
    "qa_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0a4e00f0-db84-46c5-8878-b3247a197b66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20 20\n"
     ]
    }
   ],
   "source": [
    "sample_df = qa_df.sample(n=sample_size, random_state=42)\n",
    "questions = sample_df['question'].tolist()\n",
    "answers = sample_df['answer'].tolist()\n",
    "print(len(questions), len(answers))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa15844d-4cc3-4fbd-9c97-2352caed3a25",
   "metadata": {},
   "source": [
    "## Create an index\n",
    "\n",
    "Creating an index involves a sequence of steps (a pipeline). Each step is configured using hyperparameters:\n",
    "- split each document into chunks\n",
    "- add metadata - e.g., document title, summary of previous and next chunks, pointer to parent chunk\n",
    "- add an embedding (vector) - decide whether you want the embedding to include chunk metadata or just the text\n",
    "- index the chunk - choose a vector store and index the embeddings, keywords, or both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7188773706598f27",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:38.604999793Z",
     "start_time": "2024-06-02T06:51:34.534023729Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "97\n"
     ]
    }
   ],
   "source": [
    "# configure hyperparameters\n",
    "chunk_size = 1024\n",
    "chunk_overlap = 200\n",
    "top_k = 2\n",
    "embed_model = OpenAIEmbedding()\n",
    "\n",
    "# load documents\n",
    "documents = SimpleDirectoryReader(None, [\n",
    "    f'data/{filename}',\n",
    "]).load_data()\n",
    "print(len(documents))\n",
    "\n",
    "# create vector store (delete if exists)\n",
    "chroma_client = chromadb.EphemeralClient()\n",
    "if any(coll.name == 'test' for coll in chroma_client.list_collections()):\n",
    "    chroma_client.delete_collection('test')\n",
    "chroma_collection = chroma_client.create_collection('test')\n",
    "vector_store = ChromaVectorStore(chroma_collection=chroma_collection)\n",
    "\n",
    "# create a simple ingestion pipeline: chunk the documents, create embeddings, and add to the vector store\n",
    "pipeline = IngestionPipeline(\n",
    "    transformations=[\n",
    "        SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap),\n",
    "        embed_model,\n",
    "    ],\n",
    "    vector_store=vector_store,\n",
    ")\n",
    "\n",
    "# run pipeline to populate the vector store\n",
    "pipeline.run(documents=documents)\n",
    "\n",
    "# create an index from the vector store\n",
    "index = VectorStoreIndex.from_vector_store(\n",
    "    vector_store,\n",
    "    embed_model=embed_model,\n",
    ")\n",
    "\n",
    "# create a query engine from the index\n",
    "query_engine = index.as_query_engine(similarity_top_k=top_k)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88f49a31-7079-43f0-a395-82884763f8a0",
   "metadata": {},
   "source": [
    "### Test the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9d5056f1cdaa9510",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:39.802412275Z",
     "start_time": "2024-06-02T06:51:38.607042780Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "QUESTION Can you change the number of players mid campaign? If so, how?\n",
      "RESPONSE You can change the number of players mid-campaign by following these steps:\n",
      "\n",
      "To add a player:\n",
      "1. After the current turn ends, split up crew boards and assign them to each player as evenly as possible (except Captain Odessa, who is always controlled by the active player).\n",
      "\n",
      "To remove a player:\n",
      "1. After the current turn ends, reassign crew members to players as evenly as possible (except Captain Odessa, who is always controlled by the active player).\n",
      "2. The player that is leaving must discard 1 ability card and all of their command but 1. The active player decides how to distribute the remaining cards and command token to the other players.\n"
     ]
    }
   ],
   "source": [
    "# test the index by issuing a question\n",
    "question = questions[0]\n",
    "response = query_engine.query(question)\n",
    "print('QUESTION', question)\n",
    "print('RESPONSE', response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ef997d55278a031d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:39.856952432Z",
     "start_time": "2024-06-02T06:51:39.800673831Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://localhost:6006/\n"
     ]
    }
   ],
   "source": [
    "# view the session logs\n",
    "print(px_session.url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "052494f5-ef59-49ae-b07b-cefbc9e65e37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🌍 To view the Phoenix app in your browser, visit http://localhost:6006/\n",
      "📖 For more information on how to use Phoenix, check out https://docs.arize.com/phoenix\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR [asyncio] Task was destroyed but it is pending!\n",
      "task: <Task pending name='Task-5' coro=<BulkInserter._bulk_insert() running at /home/dallan/dallanq/PathwayInternshipPresentations/.venv/lib/python3.11/site-packages/phoenix/db/bulk_inserter.py:103> wait_for=<Future pending cb=[Task.__wakeup()]>>\n"
     ]
    }
   ],
   "source": [
    "# reset the session\n",
    "px.close_app()\n",
    "px.delete_all(prompt_before_delete=False)\n",
    "px_session = px.launch_app()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4226352c-894a-4e14-b1f0-a6d3b7a4c6bb",
   "metadata": {},
   "source": [
    "## Ask sample questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "679eb22d-40c3-48be-9196-8e0a68249f97",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████| 20/20 [00:27<00:00,  1.38s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "ai_answers = []\n",
    "for question in tqdm(questions):\n",
    "    ai_answers.append(query_engine.query(question))\n",
    "print(len(ai_answers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6d18f02a-04fe-47c2-973a-34b3afcb389d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question Can you change the number of players mid campaign? If so, how?\n",
      "Answer You play the same 9 characters no matter the player count, so it's pretty easy to divvy them up however you like.\n",
      "AI Answer You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.\n"
     ]
    }
   ],
   "source": [
    "print('Question', questions[0])\n",
    "print('Answer', answers[0])\n",
    "print('AI Answer', ai_answers[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3018e895-30aa-4878-b2c2-b9edc2ea621a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>correct_answer</th>\n",
       "      <th>ai_generated_answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Can you change the number of players mid campaign? If so, how?</td>\n",
       "      <td>You play the same 9 characters no matter the player count, so it's pretty easy to divvy them up however you like.</td>\n",
       "      <td>You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?</td>\n",
       "      <td>Fortunately for you, it's neither of those!  You can remove any 3 fatigue and damage tokens you'd like, from any characters and in any combination.</td>\n",
       "      <td>The \"Flapjacks\" card's ability to remove 3 x health tokens and 3 x fatigue tokens can be split to heal one character while removing fatigue from another. The card's effect is not limited to only one crew member and can be divided among different crew members as needed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Do ongoing cards leave the play area when the event card deck runs out?</td>\n",
       "      <td>It depends on the card. Some of them will direct you to remove it when the deck runs out. Some won't. The cards will indicate when. So.. sorta yes.</td>\n",
       "      <td>Ongoing cards do not leave the play area when the event card deck runs out.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                         question  \\\n",
       "0                                                                                                                                                                                                                  Can you change the number of players mid campaign? If so, how?   \n",
       "1  I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?   \n",
       "2                                                                                                                                                                                                         Do ongoing cards leave the play area when the event card deck runs out?   \n",
       "\n",
       "                                                                                                                                        correct_answer  \\\n",
       "0                                    You play the same 9 characters no matter the player count, so it's pretty easy to divvy them up however you like.   \n",
       "1  Fortunately for you, it's neither of those!  You can remove any 3 fatigue and damage tokens you'd like, from any characters and in any combination.   \n",
       "2  It depends on the card. Some of them will direct you to remove it when the deck runs out. Some won't. The cards will indicate when. So.. sorta yes.   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ai_generated_answer  \n",
       "0  You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.  \n",
       "1                                                                                                                                                                                                                                            The \"Flapjacks\" card's ability to remove 3 x health tokens and 3 x fatigue tokens can be split to heal one character while removing fatigue from another. The card's effect is not limited to only one crew member and can be divided among different crew members as needed.  \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                              Ongoing cards do not leave the play area when the event card deck runs out.  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df = pd.DataFrame({\n",
    "    'question': questions,\n",
    "    'correct_answer': answers,\n",
    "    'ai_generated_answer': ai_answers,\n",
    "})\n",
    "print(len(results_df))\n",
    "results_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9018fca-857c-4e55-874d-30761bc67cc0",
   "metadata": {},
   "source": [
    "## Ask the LLM to evaluate answers without comparing to human answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0ccc8e3f-78e5-4082-9ca3-7fe273f7c7f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n",
      "40\n"
     ]
    }
   ],
   "source": [
    "# get the query responses and the documents that were retrieved for each query into separate dataframes\n",
    "queries_df = get_qa_with_reference(px.Client())\n",
    "print(len(queries_df))\n",
    "retrieved_documents_df = get_retrieved_documents(px.Client())\n",
    "print(len(retrieved_documents_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8ca42442-cc8c-43e3-b69a-fa9f51470eb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>reference</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>context.span_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22b77a335508da37</th>\n",
       "      <td>Can you change the number of players mid campaign? If so, how?</td>\n",
       "      <td>You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.</td>\n",
       "      <td>Saving the Game (page 31)\\n\\nAt the start of any player's turn, you can decide to stop playing and save your game. You only need to do this if you want to pack up the game. This is simply a way to keep track of your progress so you can continue another time. A campaign takes around 10-20 hours to complete, so if you have a dedicated space to keep the game set up, you do not need to save the game.\\n\\nTo save the game and pack up mid-campaign, follow these steps:\\n\\n1. On the next available line on your journey sheet, write down your ship's location, the ship's current damage, and the last ship action that was used. Also write down which ability cards each player has in their hand, and how many unspent command tokens they have.\\n\\n2. Place each crew board and all associated damage, status tokens, ability cards, level cards, and equipped weapons in a separate crew plastic bag. Place these in storage box 1 (see pg. 2).\\n\\n3. Place all of your resources, coins, adventure cards, ability cards in hand, unused command tokens you own, and quest cards in the campaign box. When placing adventure cards in this box, put them in their own bag, and put any that have command tokens on them face down in the pile. Place the current event deck in this box as well.\\n\\n4. Place the ability card draw deck in a plastic bag. Place the ability card discard pile face up in the bag (keep the draw deck face down). Indicate the bottom of this pile by placing a synergy token at the bottom.\\n\\n5. Clean up the rest of the game components.\\n\\nAdding &amp; Removing Players (page 33)\\n\\nYou can add or remove players at the end of any turn.\\n\\nFollow these steps to add a player:\\n\\n1. After the current turn ends, split up crew boards and assign them to each player as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\nFollow these steps to remove a player:\\n\\n1. After the current turn ends, reassign crew members to players as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\n2. The player that is leaving must discard 1 ability card and all of their command but 1. The active player decides how to distribute the remaining cards and command token to the other players.\\n\\nNOTE: Remember to turn the ship board to the correct side depending on the number of players in the game.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94db120a4a2c3441</th>\n",
       "      <td>I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?</td>\n",
       "      <td>The \"Flapjacks\" card's ability to remove 3 x health tokens and 3 x fatigue tokens can be split to heal one character while removing fatigue from another. The card's effect is not limited to only one crew member and can be divided among different crew members as needed.</td>\n",
       "      <td>Fatigue (page 8)\\n\\nWhen crew members participate in challenges, they gain fatigue, represented by a fatigue token. Each crew member can hold up to 2 fatigue tokens.\\n\\nEach fatigue token is double-sided. If a crew member has only 1 token, place the blank side face up. If a crew member has a 2nd fatigue token, it should have the \"-1 damage\" side face up. This causes the crew member to deal -1 damage in combat.\\n\\nA crew member with 2 fatigue tokens cannot participate in challenges (pg. 19), but can continue to participate in combat (pg. 21).\\n\\nYou can remove fatigue mainly by cooking recipes or performing a port action.\\n\\nHealth (page 8)\\n\\nHealth is the amount of damage a crew member can take and still function normally. If a crew member has damage equal to their health (known as having 0 health), they can no longer attack, participate in challenges, or activate any of their crew board abilities or ability cards until they regain at least 1 health. Thematically, at 0 health they are nearly unconscious, able to speak and move, but badly hurt.\\n\\nIf all crew members have 0 health, you are defeated (see \"Defeat\" on pg. 28).\\n\\nA crew member's damage cannot exceed their health—any excess must be placed on another crew member.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28db97856036139c</th>\n",
       "      <td>Do ongoing cards leave the play area when the event card deck runs out?</td>\n",
       "      <td>Ongoing cards do not leave the play area when the event card deck runs out.</td>\n",
       "      <td>2. Event (page 12)\\n\\nDraw the top card of the event deck and read it aloud. Apply the effect or complete the challenge.\\n\\n- Some event cards have multiple choices. In this case, choose one and apply the effect.\\n\\n- If you draw an event card with the word \"ongoing,\" place it face up near the ship board. Apply the effects until you're able to discard the card. (The way to discard the card is described on each card.)\\n\\n- You cannot ignore an event card. If it only has 1 challenge listed, you must complete it.\\n\\nOver the course of each campaign, you will go through the event deck 3 times. Each time you reach the end of the event deck, follow these instructions:\\n\\nTurn Overview (page 10)\\n\\nThis is a summary of the steps you take on your turn. The steps are explained in detail on pgs. 10-16.\\n\\nStarting with the first player, players take turns in clockwise order. Follow these steps on your turn:\\n\\n1. Ship Action: Choose a ship action. Move the ship action figure to one of the five ship rooms and apply the effect. You must move the ship action figure to a new room each turn (you cannot repeat the same room two turns in a row). (See pg. 10-11.)\\n\\n2. Event: Draw an event card and read the effect. Events may present a choice, a challenge, consequence, or other effect. (For challenges, see pg. 19.) Resolve the event card before continuing your turn. (For event card rules, see pg. 12.)\\n\\n3. Two Actions: Perform two actions. Available actions are listed on the bottom of the ship board. You may perform the same action twice in one turn. These actions are explained starting on pg. 13.\\n\\n4. End of Turn: After you have performed your two actions, pass the captain token to the player on the left. That player now starts their turn.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                           input  \\\n",
       "context.span_id                                                                                                                                                                                                                                                                                    \n",
       "22b77a335508da37                                                                                                                                                                                                                  Can you change the number of players mid campaign? If so, how?   \n",
       "94db120a4a2c3441  I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?   \n",
       "28db97856036139c                                                                                                                                                                                                         Do ongoing cards leave the play area when the event card deck runs out?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   output  \\\n",
       "context.span_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             \n",
       "22b77a335508da37  You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.   \n",
       "94db120a4a2c3441                                                                                                                                                                                                                                            The \"Flapjacks\" card's ability to remove 3 x health tokens and 3 x fatigue tokens can be split to heal one character while removing fatigue from another. The card's effect is not limited to only one crew member and can be divided among different crew members as needed.   \n",
       "28db97856036139c                                                                                                                                                                                                                                                                                                                                                                                                                                              Ongoing cards do not leave the play area when the event card deck runs out.   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               reference  \n",
       "context.span_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "22b77a335508da37  Saving the Game (page 31)\\n\\nAt the start of any player's turn, you can decide to stop playing and save your game. You only need to do this if you want to pack up the game. This is simply a way to keep track of your progress so you can continue another time. A campaign takes around 10-20 hours to complete, so if you have a dedicated space to keep the game set up, you do not need to save the game.\\n\\nTo save the game and pack up mid-campaign, follow these steps:\\n\\n1. On the next available line on your journey sheet, write down your ship's location, the ship's current damage, and the last ship action that was used. Also write down which ability cards each player has in their hand, and how many unspent command tokens they have.\\n\\n2. Place each crew board and all associated damage, status tokens, ability cards, level cards, and equipped weapons in a separate crew plastic bag. Place these in storage box 1 (see pg. 2).\\n\\n3. Place all of your resources, coins, adventure cards, ability cards in hand, unused command tokens you own, and quest cards in the campaign box. When placing adventure cards in this box, put them in their own bag, and put any that have command tokens on them face down in the pile. Place the current event deck in this box as well.\\n\\n4. Place the ability card draw deck in a plastic bag. Place the ability card discard pile face up in the bag (keep the draw deck face down). Indicate the bottom of this pile by placing a synergy token at the bottom.\\n\\n5. Clean up the rest of the game components.\\n\\nAdding & Removing Players (page 33)\\n\\nYou can add or remove players at the end of any turn.\\n\\nFollow these steps to add a player:\\n\\n1. After the current turn ends, split up crew boards and assign them to each player as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\nFollow these steps to remove a player:\\n\\n1. After the current turn ends, reassign crew members to players as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\n2. The player that is leaving must discard 1 ability card and all of their command but 1. The active player decides how to distribute the remaining cards and command token to the other players.\\n\\nNOTE: Remember to turn the ship board to the correct side depending on the number of players in the game.  \n",
       "94db120a4a2c3441                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Fatigue (page 8)\\n\\nWhen crew members participate in challenges, they gain fatigue, represented by a fatigue token. Each crew member can hold up to 2 fatigue tokens.\\n\\nEach fatigue token is double-sided. If a crew member has only 1 token, place the blank side face up. If a crew member has a 2nd fatigue token, it should have the \"-1 damage\" side face up. This causes the crew member to deal -1 damage in combat.\\n\\nA crew member with 2 fatigue tokens cannot participate in challenges (pg. 19), but can continue to participate in combat (pg. 21).\\n\\nYou can remove fatigue mainly by cooking recipes or performing a port action.\\n\\nHealth (page 8)\\n\\nHealth is the amount of damage a crew member can take and still function normally. If a crew member has damage equal to their health (known as having 0 health), they can no longer attack, participate in challenges, or activate any of their crew board abilities or ability cards until they regain at least 1 health. Thematically, at 0 health they are nearly unconscious, able to speak and move, but badly hurt.\\n\\nIf all crew members have 0 health, you are defeated (see \"Defeat\" on pg. 28).\\n\\nA crew member's damage cannot exceed their health—any excess must be placed on another crew member.  \n",
       "28db97856036139c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              2. Event (page 12)\\n\\nDraw the top card of the event deck and read it aloud. Apply the effect or complete the challenge.\\n\\n- Some event cards have multiple choices. In this case, choose one and apply the effect.\\n\\n- If you draw an event card with the word \"ongoing,\" place it face up near the ship board. Apply the effects until you're able to discard the card. (The way to discard the card is described on each card.)\\n\\n- You cannot ignore an event card. If it only has 1 challenge listed, you must complete it.\\n\\nOver the course of each campaign, you will go through the event deck 3 times. Each time you reach the end of the event deck, follow these instructions:\\n\\nTurn Overview (page 10)\\n\\nThis is a summary of the steps you take on your turn. The steps are explained in detail on pgs. 10-16.\\n\\nStarting with the first player, players take turns in clockwise order. Follow these steps on your turn:\\n\\n1. Ship Action: Choose a ship action. Move the ship action figure to one of the five ship rooms and apply the effect. You must move the ship action figure to a new room each turn (you cannot repeat the same room two turns in a row). (See pg. 10-11.)\\n\\n2. Event: Draw an event card and read the effect. Events may present a choice, a challenge, consequence, or other effect. (For challenges, see pg. 19.) Resolve the event card before continuing your turn. (For event card rules, see pg. 12.)\\n\\n3. Two Actions: Perform two actions. Available actions are listed on the bottom of the ship board. You may perform the same action twice in one turn. These actions are explained starting on pg. 13.\\n\\n4. End of Turn: After you have performed your two actions, pass the captain token to the player on the left. That player now starts their turn.  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "queries_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f564b38b-7051-486e-8606-324c5d0b1b92",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>context.trace_id</th>\n",
       "      <th>input</th>\n",
       "      <th>reference</th>\n",
       "      <th>document_score</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>context.span_id</th>\n",
       "      <th>document_position</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">66d33090fbf01637</th>\n",
       "      <th>0</th>\n",
       "      <td>281ea0c5f735d5e4693d18c020120a73</td>\n",
       "      <td>Can you change the number of players mid campaign? If so, how?</td>\n",
       "      <td>Saving the Game (page 31)\\n\\nAt the start of any player's turn, you can decide to stop playing and save your game. You only need to do this if you want to pack up the game. This is simply a way to keep track of your progress so you can continue another time. A campaign takes around 10-20 hours to complete, so if you have a dedicated space to keep the game set up, you do not need to save the game.\\n\\nTo save the game and pack up mid-campaign, follow these steps:\\n\\n1. On the next available line on your journey sheet, write down your ship's location, the ship's current damage, and the last ship action that was used. Also write down which ability cards each player has in their hand, and how many unspent command tokens they have.\\n\\n2. Place each crew board and all associated damage, status tokens, ability cards, level cards, and equipped weapons in a separate crew plastic bag. Place these in storage box 1 (see pg. 2).\\n\\n3. Place all of your resources, coins, adventure cards, ability cards in hand, unused command tokens you own, and quest cards in the campaign box. When placing adventure cards in this box, put them in their own bag, and put any that have command tokens on them face down in the pile. Place the current event deck in this box as well.\\n\\n4. Place the ability card draw deck in a plastic bag. Place the ability card discard pile face up in the bag (keep the draw deck face down). Indicate the bottom of this pile by placing a synergy token at the bottom.\\n\\n5. Clean up the rest of the game components.</td>\n",
       "      <td>0.672417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>281ea0c5f735d5e4693d18c020120a73</td>\n",
       "      <td>Can you change the number of players mid campaign? If so, how?</td>\n",
       "      <td>Adding &amp; Removing Players (page 33)\\n\\nYou can add or remove players at the end of any turn.\\n\\nFollow these steps to add a player:\\n\\n1. After the current turn ends, split up crew boards and assign them to each player as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\nFollow these steps to remove a player:\\n\\n1. After the current turn ends, reassign crew members to players as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\n2. The player that is leaving must discard 1 ability card and all of their command but 1. The active player decides how to distribute the remaining cards and command token to the other players.\\n\\nNOTE: Remember to turn the ship board to the correct side depending on the number of players in the game.</td>\n",
       "      <td>0.671891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9a7ce8a10f0d702c</th>\n",
       "      <th>0</th>\n",
       "      <td>8811debf97a654626f96d3a75909da9c</td>\n",
       "      <td>I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?</td>\n",
       "      <td>Fatigue (page 8)\\n\\nWhen crew members participate in challenges, they gain fatigue, represented by a fatigue token. Each crew member can hold up to 2 fatigue tokens.\\n\\nEach fatigue token is double-sided. If a crew member has only 1 token, place the blank side face up. If a crew member has a 2nd fatigue token, it should have the \"-1 damage\" side face up. This causes the crew member to deal -1 damage in combat.\\n\\nA crew member with 2 fatigue tokens cannot participate in challenges (pg. 19), but can continue to participate in combat (pg. 21).\\n\\nYou can remove fatigue mainly by cooking recipes or performing a port action.</td>\n",
       "      <td>0.742488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    context.trace_id  \\\n",
       "context.span_id  document_position                                     \n",
       "66d33090fbf01637 0                  281ea0c5f735d5e4693d18c020120a73   \n",
       "                 1                  281ea0c5f735d5e4693d18c020120a73   \n",
       "9a7ce8a10f0d702c 0                  8811debf97a654626f96d3a75909da9c   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                             input  \\\n",
       "context.span_id  document_position                                                                                                                                                                                                                                                                                   \n",
       "66d33090fbf01637 0                                                                                                                                                                                                                                  Can you change the number of players mid campaign? If so, how?   \n",
       "                 1                                                                                                                                                                                                                                  Can you change the number of players mid campaign? If so, how?   \n",
       "9a7ce8a10f0d702c 0                  I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       reference  \\\n",
       "context.span_id  document_position                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 \n",
       "66d33090fbf01637 0                  Saving the Game (page 31)\\n\\nAt the start of any player's turn, you can decide to stop playing and save your game. You only need to do this if you want to pack up the game. This is simply a way to keep track of your progress so you can continue another time. A campaign takes around 10-20 hours to complete, so if you have a dedicated space to keep the game set up, you do not need to save the game.\\n\\nTo save the game and pack up mid-campaign, follow these steps:\\n\\n1. On the next available line on your journey sheet, write down your ship's location, the ship's current damage, and the last ship action that was used. Also write down which ability cards each player has in their hand, and how many unspent command tokens they have.\\n\\n2. Place each crew board and all associated damage, status tokens, ability cards, level cards, and equipped weapons in a separate crew plastic bag. Place these in storage box 1 (see pg. 2).\\n\\n3. Place all of your resources, coins, adventure cards, ability cards in hand, unused command tokens you own, and quest cards in the campaign box. When placing adventure cards in this box, put them in their own bag, and put any that have command tokens on them face down in the pile. Place the current event deck in this box as well.\\n\\n4. Place the ability card draw deck in a plastic bag. Place the ability card discard pile face up in the bag (keep the draw deck face down). Indicate the bottom of this pile by placing a synergy token at the bottom.\\n\\n5. Clean up the rest of the game components.   \n",
       "                 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Adding & Removing Players (page 33)\\n\\nYou can add or remove players at the end of any turn.\\n\\nFollow these steps to add a player:\\n\\n1. After the current turn ends, split up crew boards and assign them to each player as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\nFollow these steps to remove a player:\\n\\n1. After the current turn ends, reassign crew members to players as evenly as possible (except Captain Odessa, who is always controlled by the active player).\\n\\n2. The player that is leaving must discard 1 ability card and all of their command but 1. The active player decides how to distribute the remaining cards and command token to the other players.\\n\\nNOTE: Remember to turn the ship board to the correct side depending on the number of players in the game.   \n",
       "9a7ce8a10f0d702c 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Fatigue (page 8)\\n\\nWhen crew members participate in challenges, they gain fatigue, represented by a fatigue token. Each crew member can hold up to 2 fatigue tokens.\\n\\nEach fatigue token is double-sided. If a crew member has only 1 token, place the blank side face up. If a crew member has a 2nd fatigue token, it should have the \"-1 damage\" side face up. This causes the crew member to deal -1 damage in combat.\\n\\nA crew member with 2 fatigue tokens cannot participate in challenges (pg. 19), but can continue to participate in combat (pg. 21).\\n\\nYou can remove fatigue mainly by cooking recipes or performing a port action.   \n",
       "\n",
       "                                    document_score  \n",
       "context.span_id  document_position                  \n",
       "66d33090fbf01637 0                        0.672417  \n",
       "                 1                        0.671891  \n",
       "9a7ce8a10f0d702c 0                        0.742488  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieved_documents_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "04e1e224-c5a6-4f49-b673-a50f14290f64",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6baf698932174790b4f32271aa18dfac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "run_evals |          | 0/40 (0.0%) | ⏳ 00:00<? | ?it/s"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e5914d251234237ba43bcedf083ca91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "run_evals |          | 0/40 (0.0%) | ⏳ 00:00<? | ?it/s"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_model = OpenAIModel(\n",
    "    model=\"gpt-4-turbo-preview\",\n",
    "    temperature=0.0,\n",
    ")\n",
    "hallucination_evaluator = HallucinationEvaluator(eval_model)\n",
    "qa_correctness_evaluator = QAEvaluator(eval_model)\n",
    "relevance_evaluator = RelevanceEvaluator(eval_model)\n",
    "\n",
    "hallucination_eval_df, qa_correctness_eval_df = run_evals(\n",
    "    dataframe=queries_df,\n",
    "    evaluators=[hallucination_evaluator, qa_correctness_evaluator],\n",
    "    provide_explanation=True,\n",
    ")\n",
    "relevance_eval_df = run_evals(\n",
    "    dataframe=retrieved_documents_df,\n",
    "    evaluators=[relevance_evaluator],\n",
    "    provide_explanation=True,\n",
    ")[0]\n",
    "\n",
    "px.Client().log_evaluations(\n",
    "    SpanEvaluations(eval_name=\"Hallucination\", dataframe=hallucination_eval_df),\n",
    "    SpanEvaluations(eval_name=\"QA Correctness\", dataframe=qa_correctness_eval_df),\n",
    "    DocumentEvaluations(eval_name=\"Relevance\", dataframe=relevance_eval_df),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4a3832c-d444-4554-a16f-b911159560d6",
   "metadata": {},
   "source": [
    "### Review evaluations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ce7e9f9b-f27b-4e2a-8ba6-b903a59f2dcb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://localhost:6006/\n"
     ]
    }
   ],
   "source": [
    "print(px_session.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f1b3327-f0f9-4281-ac18-e8e530cb2dca",
   "metadata": {},
   "source": [
    "## Ask an LLM to evaluate answers by comparing to human answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "117a0731-90d8-4cad-848d-7b471fea7add",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "You are comparing a human ground truth answer from an expert to an answer from an AI model.\n",
      "Your goal is to determine if the AI answer correctly matches, in substance, the human answer.\n",
      "    [BEGIN DATA]\n",
      "    ************\n",
      "    [Question]: {question}\n",
      "    ************\n",
      "    [Human Ground Truth Answer]: {correct_answer}\n",
      "    ************\n",
      "    [AI Answer]: {ai_generated_answer}\n",
      "    ************\n",
      "    [END DATA]\n",
      "Compare the AI answer to the human ground truth answer, if the AI correctly answers the question,\n",
      "then the AI answer is \"correct\". If the AI answer is longer but contains the main idea of the\n",
      "Human answer please answer \"correct\". If the AI answer divergences or does not contain the main\n",
      "idea of the human answer, please answer \"incorrect\".\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(HUMAN_VS_AI_PROMPT_TEMPLATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d881d06e-35ca-4de9-b203-480fab3b5406",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be2aff1af2ba4b91a5348e6422322a1e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "llm_classify |          | 0/20 (0.0%) | ⏳ 00:00<? | ?it/s"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The rails is used to hold the output to specific values based on the template\n",
    "# It will remove text such as \",,,\" or \"...\"\n",
    "# Will ensure the binary value expected from the template is returned\n",
    "rails = list(HUMAN_VS_AI_PROMPT_RAILS_MAP.values())\n",
    "\n",
    "eval_df = llm_classify(\n",
    "    dataframe=results_df,\n",
    "    template=HUMAN_VS_AI_PROMPT_TEMPLATE,\n",
    "    model=eval_model,\n",
    "    rails=rails,\n",
    "    verbose=False,\n",
    "    provide_explanation=True,\n",
    "    concurrency=50,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "cc467f58-2eed-4bd9-8d6f-41703dbba76b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>explanation</th>\n",
       "      <th>exceptions</th>\n",
       "      <th>execution_status</th>\n",
       "      <th>execution_seconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>correct</td>\n",
       "      <td>The AI answer provides a detailed process for changing the number of players mid-campaign, including steps for adding or removing players, reassigning crew members, and adjusting game components accordingly. This suggests that changing the number of players is indeed possible, aligning with the human ground truth answer's implication that player count can be adjusted by divvying up the characters. Although the AI's response is more detailed and introduces specific steps not mentioned in the human answer, it fundamentally agrees with the human answer's core assertion that the number of players can be changed mid-campaign. Therefore, the AI answer is relevant as it expands on the human answer by providing a how-to guide that matches the substance of the human answer, which is about the flexibility in player count adjustment.</td>\n",
       "      <td>[]</td>\n",
       "      <td>COMPLETED</td>\n",
       "      <td>8.372679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>correct</td>\n",
       "      <td>To determine if the AI answer is 'relevant' or 'irrelevant', we compare the substance of the AI's answer to the human ground truth answer. The key points to compare include: 1) Whether the AI understands that the \"Flapjacks\" card's ability can be applied in a flexible manner, not limited to a single crew member. 2) If the AI recognizes that the healing and fatigue removal can be distributed among different crew members as needed. The human answer indicates that the ability allows for any combination of removing fatigue and damage tokens from any characters. The AI's answer aligns with this by stating that the card's effect can be split to heal one character while removing fatigue from another, and it is not limited to only one crew member. This shows that the AI's response captures the main idea of the human answer, which is the flexibility in applying the card's effects across multiple crew members in any combination desired.</td>\n",
       "      <td>[]</td>\n",
       "      <td>COMPLETED</td>\n",
       "      <td>8.702796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>incorrect</td>\n",
       "      <td>To determine if the AI answer matches the human ground truth answer, we need to compare the substance of both answers. The human answer indicates that the outcome depends on the specific card, with some cards requiring removal when the event card deck runs out and others not. This suggests variability based on the card's instructions. The AI's answer, however, provides a definitive statement that ongoing cards do not leave the play area when the event card deck runs out, which contradicts the nuanced explanation provided by the human expert. The AI's answer fails to capture the conditional nature of the human answer, which is that the action to be taken depends on the instructions of the individual cards. Therefore, the AI's answer does not correctly match the substance of the human answer.</td>\n",
       "      <td>[]</td>\n",
       "      <td>COMPLETED</td>\n",
       "      <td>5.730891</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       label  \\\n",
       "0    correct   \n",
       "1    correct   \n",
       "2  incorrect   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    explanation  \\\n",
       "0                                                                                                            The AI answer provides a detailed process for changing the number of players mid-campaign, including steps for adding or removing players, reassigning crew members, and adjusting game components accordingly. This suggests that changing the number of players is indeed possible, aligning with the human ground truth answer's implication that player count can be adjusted by divvying up the characters. Although the AI's response is more detailed and introduces specific steps not mentioned in the human answer, it fundamentally agrees with the human answer's core assertion that the number of players can be changed mid-campaign. Therefore, the AI answer is relevant as it expands on the human answer by providing a how-to guide that matches the substance of the human answer, which is about the flexibility in player count adjustment.   \n",
       "1  To determine if the AI answer is 'relevant' or 'irrelevant', we compare the substance of the AI's answer to the human ground truth answer. The key points to compare include: 1) Whether the AI understands that the \"Flapjacks\" card's ability can be applied in a flexible manner, not limited to a single crew member. 2) If the AI recognizes that the healing and fatigue removal can be distributed among different crew members as needed. The human answer indicates that the ability allows for any combination of removing fatigue and damage tokens from any characters. The AI's answer aligns with this by stating that the card's effect can be split to heal one character while removing fatigue from another, and it is not limited to only one crew member. This shows that the AI's response captures the main idea of the human answer, which is the flexibility in applying the card's effects across multiple crew members in any combination desired.   \n",
       "2                                                                                                                                             To determine if the AI answer matches the human ground truth answer, we need to compare the substance of both answers. The human answer indicates that the outcome depends on the specific card, with some cards requiring removal when the event card deck runs out and others not. This suggests variability based on the card's instructions. The AI's answer, however, provides a definitive statement that ongoing cards do not leave the play area when the event card deck runs out, which contradicts the nuanced explanation provided by the human expert. The AI's answer fails to capture the conditional nature of the human answer, which is that the action to be taken depends on the instructions of the individual cards. Therefore, the AI's answer does not correctly match the substance of the human answer.   \n",
       "\n",
       "  exceptions execution_status  execution_seconds  \n",
       "0         []        COMPLETED           8.372679  \n",
       "1         []        COMPLETED           8.702796  \n",
       "2         []        COMPLETED           5.730891  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b0ce37fb-1cc3-4f6c-8c8a-3a4ad33250ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>correct_answer</th>\n",
       "      <th>ai_generated_answer</th>\n",
       "      <th>label</th>\n",
       "      <th>explanation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Can you change the number of players mid campaign? If so, how?</td>\n",
       "      <td>You play the same 9 characters no matter the player count, so it's pretty easy to divvy them up however you like.</td>\n",
       "      <td>You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.</td>\n",
       "      <td>correct</td>\n",
       "      <td>The AI answer provides a detailed process for changing the number of players mid-campaign, including steps for adding or removing players, reassigning crew members, and adjusting game components accordingly. This suggests that changing the number of players is indeed possible, aligning with the human ground truth answer's implication that player count can be adjusted by divvying up the characters. Although the AI's response is more detailed and introduces specific steps not mentioned in the human answer, it fundamentally agrees with the human answer's core assertion that the number of players can be changed mid-campaign. Therefore, the AI answer is relevant as it expands on the human answer by providing a how-to guide that matches the substance of the human answer, which is about the flexibility in player count adjustment.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?</td>\n",
       "      <td>Fortunately for you, it's neither of those!  You can remove any 3 fatigue and damage tokens you'd like, from any characters and in any combination.</td>\n",
       "      <td>The \"Flapjacks\" card's ability to remove 3 x health tokens and 3 x fatigue tokens can be split to heal one character while removing fatigue from another. The card's effect is not limited to only one crew member and can be divided among different crew members as needed.</td>\n",
       "      <td>correct</td>\n",
       "      <td>To determine if the AI answer is 'relevant' or 'irrelevant', we compare the substance of the AI's answer to the human ground truth answer. The key points to compare include: 1) Whether the AI understands that the \"Flapjacks\" card's ability can be applied in a flexible manner, not limited to a single crew member. 2) If the AI recognizes that the healing and fatigue removal can be distributed among different crew members as needed. The human answer indicates that the ability allows for any combination of removing fatigue and damage tokens from any characters. The AI's answer aligns with this by stating that the card's effect can be split to heal one character while removing fatigue from another, and it is not limited to only one crew member. This shows that the AI's response captures the main idea of the human answer, which is the flexibility in applying the card's effects across multiple crew members in any combination desired.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Do ongoing cards leave the play area when the event card deck runs out?</td>\n",
       "      <td>It depends on the card. Some of them will direct you to remove it when the deck runs out. Some won't. The cards will indicate when. So.. sorta yes.</td>\n",
       "      <td>Ongoing cards do not leave the play area when the event card deck runs out.</td>\n",
       "      <td>incorrect</td>\n",
       "      <td>To determine if the AI answer matches the human ground truth answer, we need to compare the substance of both answers. The human answer indicates that the outcome depends on the specific card, with some cards requiring removal when the event card deck runs out and others not. This suggests variability based on the card's instructions. The AI's answer, however, provides a definitive statement that ongoing cards do not leave the play area when the event card deck runs out, which contradicts the nuanced explanation provided by the human expert. The AI's answer fails to capture the conditional nature of the human answer, which is that the action to be taken depends on the instructions of the individual cards. Therefore, the AI's answer does not correctly match the substance of the human answer.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                         question  \\\n",
       "0                                                                                                                                                                                                                  Can you change the number of players mid campaign? If so, how?   \n",
       "1  I see that the \"Flapjacks\" card has an ability to remove 3 x health tokens and 3 x fatigue tokens from a crew member. My question is, can I split that to heal one character while removing fatigue from another? Or does the card collectively apply to only one crew member?   \n",
       "2                                                                                                                                                                                                         Do ongoing cards leave the play area when the event card deck runs out?   \n",
       "\n",
       "                                                                                                                                        correct_answer  \\\n",
       "0                                    You play the same 9 characters no matter the player count, so it's pretty easy to divvy them up however you like.   \n",
       "1  Fortunately for you, it's neither of those!  You can remove any 3 fatigue and damage tokens you'd like, from any characters and in any combination.   \n",
       "2  It depends on the card. Some of them will direct you to remove it when the deck runs out. Some won't. The cards will indicate when. So.. sorta yes.   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       ai_generated_answer  \\\n",
       "0  You can change the number of players mid-campaign by following these steps: After the current turn ends, you can add or remove players. To add a player, split up crew boards and assign them as evenly as possible. To remove a player, reassign crew members evenly, and the leaving player must discard 1 ability card and all but 1 command token. The active player decides how to distribute the remaining cards and command token. Remember to adjust the ship board based on the number of players in the game.   \n",
       "1                                                                                                                                                                                                                                            The \"Flapjacks\" card's ability to remove 3 x health tokens and 3 x fatigue tokens can be split to heal one character while removing fatigue from another. The card's effect is not limited to only one crew member and can be divided among different crew members as needed.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                              Ongoing cards do not leave the play area when the event card deck runs out.   \n",
       "\n",
       "       label  \\\n",
       "0    correct   \n",
       "1    correct   \n",
       "2  incorrect   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    explanation  \n",
       "0                                                                                                            The AI answer provides a detailed process for changing the number of players mid-campaign, including steps for adding or removing players, reassigning crew members, and adjusting game components accordingly. This suggests that changing the number of players is indeed possible, aligning with the human ground truth answer's implication that player count can be adjusted by divvying up the characters. Although the AI's response is more detailed and introduces specific steps not mentioned in the human answer, it fundamentally agrees with the human answer's core assertion that the number of players can be changed mid-campaign. Therefore, the AI answer is relevant as it expands on the human answer by providing a how-to guide that matches the substance of the human answer, which is about the flexibility in player count adjustment.  \n",
       "1  To determine if the AI answer is 'relevant' or 'irrelevant', we compare the substance of the AI's answer to the human ground truth answer. The key points to compare include: 1) Whether the AI understands that the \"Flapjacks\" card's ability can be applied in a flexible manner, not limited to a single crew member. 2) If the AI recognizes that the healing and fatigue removal can be distributed among different crew members as needed. The human answer indicates that the ability allows for any combination of removing fatigue and damage tokens from any characters. The AI's answer aligns with this by stating that the card's effect can be split to heal one character while removing fatigue from another, and it is not limited to only one crew member. This shows that the AI's response captures the main idea of the human answer, which is the flexibility in applying the card's effects across multiple crew members in any combination desired.  \n",
       "2                                                                                                                                             To determine if the AI answer matches the human ground truth answer, we need to compare the substance of both answers. The human answer indicates that the outcome depends on the specific card, with some cards requiring removal when the event card deck runs out and others not. This suggests variability based on the card's instructions. The AI's answer, however, provides a definitive statement that ongoing cards do not leave the play area when the event card deck runs out, which contradicts the nuanced explanation provided by the human expert. The AI's answer fails to capture the conditional nature of the human answer, which is that the action to be taken depends on the instructions of the individual cards. Therefore, the AI's answer does not correctly match the substance of the human answer.  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's view the data\n",
    "merged_df = pd.merge(results_df, eval_df, left_index=True, right_index=True)\n",
    "summary_df = merged_df[[\"question\", \"correct_answer\", \"ai_generated_answer\", \"label\", \"explanation\"]]\n",
    "summary_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "908efb76-9130-4994-bb80-ab1f29a85599",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "# print the total number of evaluations and the number of 'correct' labels\n",
    "print(len(eval_df))\n",
    "print(sum(eval_df['label'] == 'correct')) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c246a817-ca13-4e2a-a1ec-1181adf81d27",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}