tutorials/10_Full_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Research CoPilot\n",
    "## Multimodal RAG with Code Execution (RAG-CE)\n",
    "\n",
    "This notebook will demo the capabilities of the solution. In the below cells, we will ingest PDF documents that are located in the `cases` directory. Then we will generate questions based on **all** the information extracted from the 2 case documents. Then, with our multimodal Search that is supported by the Code Interpreter, we will generate potential answers to these questions and rate the ground truth vs the generated answer. \n",
    "\n",
    "This notebook is for demo purposes only, and therefore we will ignore the bias in the the below demo: by generating questions based on the extracted contents of the documents, this introduces a bias, since the questions are only limited to what the solution already has ingested. But again, this is for demo only. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest the Directory\n",
    "Import Python packages, and ingest the PDF directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "\n",
    "import sys\n",
    "sys.path.append('..\\\\code')\n",
    "\n",
    "\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "\n",
    "from IPython.display import display, Markdown, HTML\n",
    "from PIL import Image\n",
    "from doc_utils import *\n",
    "from processor import *\n",
    "\n",
    "\n",
    "def show_img(img_path, width = None):\n",
    "    if width is not None:\n",
    "        display(HTML(f'<img src=\"{img_path}\" width={width}>'))  \n",
    "    else:\n",
    "        display(Image.open(img_path))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_path = r\"sample_data\\minion-tech.pdf\"\n",
    "\n",
    "ingestion_params_dict = {\n",
    "        \"index_name\" : 'project_minions',\n",
    "        \"delete_existing_output_dir\": True,\n",
    "        \"processing_mode_pdf\" : 'hybrid',\n",
    "        \"doc_path\" : doc_path,\n",
    "        'num_threads': 1\n",
    "    }\n",
    "\n",
    "pdf1 = PdfProcessor(ingestion_params_dict)\n",
    "pdf1.ingest_document()\n",
    "\n",
    "\n",
    "doc_path = r\"sample_data\\wile_e_coyote.pdf\"\n",
    "\n",
    "ingestion_params_dict = {\n",
    "        \"index_name\" : 'project_minions',\n",
    "        \"delete_existing_output_dir\": False,\n",
    "        \"processing_mode_pdf\" : 'hybrid',\n",
    "        \"doc_path\" : doc_path,\n",
    "        'num_threads': 1\n",
    "    }\n",
    "\n",
    "pdf2 = PdfProcessor(ingestion_params_dict)\n",
    "pdf2.ingest_document()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Genereate questions based on the ingested output of the case documents\n",
    "\n",
    "The below uses only 5 prompts to generate questions / answers pairs that are considered \"ground truth\" answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "11.02.2024_21.17.35 :: \u001b[92mGeneral Q&A: \u001b[94m{\n",
      "    \"qna_pairs\": [\n",
      "        {\n",
      "            \"question\": \"What is the projected increase in annual revenue for Gru's Enterprises over the next three years according to the investment appeal document?\",\n",
      "            \"answer\": \"Gru's Enterprises projects a 25% increase in annual revenue over the next three years.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"What is the total valuation of Gru's Enterprises as calculated in Wile E. Coyote's investment proposal?\",\n",
      "            \"answer\": \"The total valuation of Gru's Enterprises is calculated to be $5 million.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"What are the main components of the manufacturing process at Gru's Enterprises as detailed in their business analysis document?\",\n",
      "            \"answer\": \"The main components of the manufacturing process at Gru's Enterprises are Design and Prototyping, Material Sourcing, Assembly, and Testing.\"\n",
      "        }\n",
      "    ]\n",
      "}\u001b[0m\n",
      "\n",
      "11.02.2024_21.17.57 :: \u001b[92mSpecialized Q&A: \u001b[94m{\n",
      "    \"qna_pairs\": [\n",
      "        {\n",
      "            \"question\": \"What investment amount is Wile E. Coyote proposing for a 40% stake in Gru's Enterprises according to his investment proposal?\",\n",
      "            \"answer\": \"Wile E. Coyote is proposing an investment of $2 million for a 40% stake in Gru's Enterprises.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"What is the projected Return on Investment (ROI) percentage after 3 years for Gru's Enterprises as per Wile E. Coyote's financial projections?\",\n",
      "            \"answer\": \"The projected Return on Investment (ROI) percentage after 3 years for Gru's Enterprises is 35%.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"According to Gru's Enterprises business analysis document, what are the primary revenue streams for the company?\",\n",
      "            \"answer\": \"The primary revenue streams for Gru's Enterprises are direct product sales, licensing deals, and merchandising.\"\n",
      "        }\n",
      "    ]\n",
      "}\u001b[0m\n",
      "\n",
      "11.02.2024_21.18.10 :: \u001b[92mNumerical Q&A: \u001b[94m{\n",
      "    \"qna_pairs\": [\n",
      "        {\n",
      "            \"question\": \"What is the projected net income for Gru's Enterprises for the fiscal year 2023 according to the profit and loss statement?\",\n",
      "            \"answer\": \"The projected net income for Gru's Enterprises for the fiscal year 2023 is $80,000.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"How much is ACME Enterprises proposing to invest in Gru's Enterprises for a 40% stake according to Wile E. Coyote's investment proposal?\",\n",
      "            \"answer\": \"ACME Enterprises is proposing to invest $2 million in Gru's Enterprises for a 40% stake.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"What is the amount of long-term debt listed on Gru's Enterprises' balance sheet as of the fiscal year 2023?\",\n",
      "            \"answer\": \"The amount of long-term debt listed on Gru's Enterprises' balance sheet as of the fiscal year 2023 is $1,700,000.\"\n",
      "        }\n",
      "    ]\n",
      "}\u001b[0m\n",
      "\n",
      "11.02.2024_21.18.27 :: \u001b[92mTables Q&A: \u001b[94m{\n",
      "    \"qna_pairs\": [\n",
      "        {\n",
      "            \"question\": \"What is the average effectiveness rating of the weapons in Coyote's arsenal according to the hierarchy table in Wile E. Coyote's document?\",\n",
      "            \"answer\": \"The average effectiveness rating of the weapons in Coyote's arsenal cannot be precisely calculated as the ratings are qualitative (High, Medium, Low, Very Low).\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"What is the total number of attempts made with explosives and mechanical traps combined as per Coyote's past failed weapons categories table?\",\n",
      "            \"answer\": \"The total number of attempts made with explosives and mechanical traps combined is 35.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"Based on the financial projection table in Wile E. Coyote's proposal, what is the total projected revenue for Gru's Enterprises over the three years from 2024 to 2026?\",\n",
      "            \"answer\": \"The total projected revenue for Gru's Enterprises over the three years from 2024 to 2026 is $19,500,000.\"\n",
      "        }\n",
      "    ]\n",
      "}\u001b[0m\n",
      "\n",
      "11.02.2024_21.18.43 :: \u001b[92mImages Q&A: \u001b[94m{\n",
      "    \"qna_pairs\": [\n",
      "        {\n",
      "            \"question\": \"What is the name of the head of operations at Gru's Enterprises as shown in their organizational chart?\",\n",
      "            \"answer\": \"The name of the head of operations at Gru's Enterprises is Kevin.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"In the R&D expenditure bar chart from Gru's Enterprises, what is the R&D spending for the year 2022?\",\n",
      "            \"answer\": \"The R&D spending for the year 2022 is $580,000.\"\n",
      "        },\n",
      "        {\n",
      "            \"question\": \"What are the three products listed in the Best-selling Products Analysis table from Gru's Enterprises and their respective gross margins?\",\n",
      "            \"answer\": \"The three products listed are the Freeze Ray with a 70% gross margin, Rocket Boots with a 60% gross margin, and Bubble Gun with a 66.7% gross margin.\"\n",
      "        }\n",
      "    ]\n",
      "}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "path_1 = r\"..\\project_minions\\minion-tech.pdf\\minion-tech.txt\"\n",
    "path_2 = r\"..\\project_minions\\wile_e_coyote.pdf\\wile_e_coyote.txt\"\n",
    "\n",
    "text_template = \"\"\"\n",
    "Gru's Document: Business Analysis Document for Minion Tech\n",
    "## START OF GRU'S DOCUMENT\n",
    "{gru_doc}\n",
    "## END OF GRU'S DOCUMENT\n",
    "\n",
    "\n",
    "\n",
    "Wile E. Coyote's Document: Investment Proposal for Gru's Enterprises\n",
    "## START OF WILE E. COYOTE'S DOCUMENT\n",
    "{wile_doc}\n",
    "## END OF WILE E. COYOTE'S DOCUMENT\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "text = text_template.format(gru_doc = read_asset_file(path_1)[0] , wile_doc = read_asset_file(path_2)[0])\n",
    "\n",
    "past_questions = []\n",
    "\n",
    "qna_pairs = recover_json(ask_LLM_with_JSON(general_prompt_template.format(context=text, past_questions=past_questions), temperature = 0.5))\n",
    "past_questions.extend(qna_pairs['qna_pairs'])\n",
    "logc(\"General Q&A\", json.dumps(qna_pairs, indent=4))\n",
    "\n",
    "qna_pairs = recover_json(ask_LLM_with_JSON(specialized_prompt_template.format(context=text, past_questions=past_questions), temperature = 0.5))\n",
    "past_questions.extend(qna_pairs['qna_pairs'])\n",
    "logc(\"Specialized Q&A\", json.dumps(qna_pairs, indent=4))\n",
    "\n",
    "qna_pairs = recover_json(ask_LLM_with_JSON(numerical_prompt_template.format(context=text, past_questions=past_questions), temperature = 0.5))\n",
    "past_questions.extend(qna_pairs['qna_pairs'])\n",
    "logc(\"Numerical Q&A\", json.dumps(qna_pairs, indent=4))\n",
    "\n",
    "qna_pairs = recover_json(ask_LLM_with_JSON(table_prompt_template.format(context=text, past_questions=past_questions), temperature = 0.5))\n",
    "past_questions.extend(qna_pairs['qna_pairs'])\n",
    "logc(\"Tables Q&A\", json.dumps(qna_pairs, indent=4))\n",
    "    \n",
    "qna_pairs = recover_json(ask_LLM_with_JSON(image_prompt_template.format(context=text, past_questions=past_questions), temperature = 0.5))\n",
    "past_questions.extend(qna_pairs['qna_pairs'])\n",
    "logc(\"Images Q&A\", json.dumps(qna_pairs, indent=4))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate answers\n",
    "\n",
    "With our multimodal Search that is potentially supported by the Code Interpreter, we will generate answers to the questions and rate the ground truth vs the generated answer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "11.02.2024_21.22.54 :: \u001b[92mRating for question 0 is --> 0: \u001b[94m\n",
      "Question: What is the projected increase in annual revenue for Gru's Enterprises over the next three years according to the investment appeal document?\n",
      "\u001b[96mGround truth answer: Gru's Enterprises projects a 25% increase in annual revenue over the next three years.\n",
      "\u001b[35mGenerated answer: The projected increase in annual revenue for Gru's Enterprises over the next three years is calculated based on the provided revenue figures for each year. The computation is as follows:\n",
      "\n",
      "- The projected revenue for Year 1 (2024) is $5,500,000.\n",
      "- The projected revenue for Year 2 (2025) is $6,500,000.\n",
      "- The projected revenue for Year 3 (2026) is $7,500,000.\n",
      "\n",
      "To calculate the increase from Year 1 to Year 2, we subtract the Year 1 revenue from the Year 2 revenue:\n",
      "$6,500,000 (Year 2) - $5,500,000 (Year 1) = $1,000,000\n",
      "\n",
      "Similarly, to calculate the increase from Year 2 to Year 3, we subtract the Year 2 revenue from the Year 3 revenue:\n",
      "$7,500,000 (Year 3) - $6,500,000 (Year 2) = $1,000,000\n",
      "\n",
      "Adding these two increases together gives us the total projected increase in annual revenue over the three years:\n",
      "$1,000,000 (Year 1 to Year 2 increase) + $1,000,000 (Year 2 to Year 3 increase) = $2,000,000\n",
      "\n",
      "Therefore, the total projected increase in annual revenue over the next three years is $2,000,000. This answer was derived from the document 'wile_e_coyote.pdf', page 4.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\tables\\\\page_4_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"table\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.25.35 :: \u001b[92mRating for question 1 is --> 10: \u001b[94m\n",
      "Question: What is the total valuation of Gru's Enterprises as calculated in Wile E. Coyote's investment proposal?\n",
      "\u001b[96mGround truth answer: The total valuation of Gru's Enterprises is calculated to be $5 million.\n",
      "\u001b[35mGenerated answer: The total valuation of Gru's Enterprises, as calculated in Wile E. Coyote's investment proposal, is $5,000,000. This valuation is based on the financials provided in the proposal, which include the company's assets, revenue streams, and market potential. The valuation is mentioned in the context of a proposal by Wile E. Coyote to acquire a 40% stake in Gru's Enterprises, with the intention of infusing $2 million of capital for expansion and product development. The valuation is a crucial piece of information as it forms the basis for the proposed investment amount and the anticipated revenue growth over the next three years.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\text\\\\page_4.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.25.53 :: \u001b[92mRating for question 2 is --> 8: \u001b[94m\n",
      "Question: What are the main components of the manufacturing process at Gru's Enterprises as detailed in their business analysis document?\n",
      "\u001b[96mGround truth answer: The main components of the manufacturing process at Gru's Enterprises are Design and Prototyping, Material Sourcing, Assembly, and Testing.\n",
      "\u001b[35mGenerated answer: The main components of the manufacturing process at Gru's Enterprises, as detailed in their business analysis document, include the following stages: 1. Design and Prototyping, led by Dr. Nefario and his team, which involves brainstorming, sketching, and creating prototypes. 2. Material Sourcing, where high-quality, durable materials are sourced to ensure the products are safe and long-lasting. 3. Assembly, which is handled by Minions with precision and care, adhering to strict quality control standards. 4. Testing, where each product undergoes rigorous testing to ensure it meets safety and performance criteria. Additionally, the document mentions that the supply chain is managed by a robust logistics team, supervised by Minion Kevin, ensuring timely delivery of materials and finished products.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\text\\\\page_10.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 10,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.30.02 :: \u001b[92mRating for question 3 is --> 10: \u001b[94m\n",
      "Question: What investment amount is Wile E. Coyote proposing for a 40% stake in Gru's Enterprises according to his investment proposal?\n",
      "\u001b[96mGround truth answer: Wile E. Coyote is proposing an investment of $2 million for a 40% stake in Gru's Enterprises.\n",
      "\u001b[35mGenerated answer: Wile E. Coyote is proposing to invest $2,000,000 for a 40% stake in Gru's Enterprises. This proposal is based on the valuation of Gru's Enterprises at $5,000,000. The investment amount is calculated by taking 40% of the total company valuation, which is the stake that Wile E. Coyote seeks to acquire. The calculation is as follows: Investment Amount = Company Valuation * Stake Percentage, which results in Investment Amount = $5,000,000 * 40% = $2,000,000. This calculation is derived from the financial data provided in the 'wile_e_coyote.pdf' document, specifically from page 4.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\text\\\\page_4.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"text\"\n",
      "    },\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\tables\\\\page_4_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"table\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.33.11 :: \u001b[92mRating for question 4 is --> 10: \u001b[94m\n",
      "Question: What is the projected Return on Investment (ROI) percentage after 3 years for Gru's Enterprises as per Wile E. Coyote's financial projections?\n",
      "\u001b[96mGround truth answer: The projected Return on Investment (ROI) percentage after 3 years for Gru's Enterprises is 35%.\n",
      "\u001b[35mGenerated answer: The projected Return on Investment (ROI) percentage after 3 years for Gru's Enterprises, as per Wile E. Coyote's financial projections, is 35%. This information is found in the financial projections table on page 4 of the 'wile_e_coyote.pdf' document. The table lists the projected revenue, profit, and ROI for the years 2024, 2025, and 2026. For the year 2026, which marks the end of the 3-year period post-investment, the ROI is given as 35%. This percentage represents the expected profit from the investment relative to the cost of the investment after three years.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\tables\\\\page_4_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"table\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.33.26 :: \u001b[92mRating for question 5 is --> 0: \u001b[94m\n",
      "Question: According to Gru's Enterprises business analysis document, what are the primary revenue streams for the company?\n",
      "\u001b[96mGround truth answer: The primary revenue streams for Gru's Enterprises are direct product sales, licensing deals, and merchandising.\n",
      "\u001b[35mGenerated answer: The primary revenue streams for Gru's Enterprises are derived from the design and manufacture of unique, cartoonish \"evil\" weapons. These products cater to a niche market of enthusiasts who are interested in fantasy weaponry with a humorous twist. The company's product range includes items such as freeze rays and rocket boots, which combine innovation with practical jokes to offer a unique experience in the realm of fantasy weaponry. This information was deduced from the 'minion-tech.pdf' document, which provides an overview of Gru's Business Model in the Executive Summary section on page 4.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\text\\\\page_4.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.35.55 :: \u001b[92mRating for question 6 is --> 10: \u001b[94m\n",
      "Question: What is the projected net income for Gru's Enterprises for the fiscal year 2023 according to the profit and loss statement?\n",
      "\u001b[96mGround truth answer: The projected net income for Gru's Enterprises for the fiscal year 2023 is $80,000.\n",
      "\u001b[35mGenerated answer: The projected net income for Gru's Enterprises for the fiscal year 2023 is $80,000. This figure is taken from the Profit and Loss Statement for the Year 2023, which details the company's financial performance. The net income is calculated after accounting for all revenues, cost of goods sold (COGS), operating expenses, other expenses, and taxes. The calculation provided in the document is as follows:\n",
      "\n",
      "- Total Revenue: $4,200,000\n",
      "- Total COGS: $2,200,000\n",
      "- Gross Profit: $2,000,000\n",
      "- Total Operating Expenses: $1,800,000\n",
      "- Operating Income: $200,000\n",
      "- Total Other Expenses: $100,000\n",
      "- Net Income Before Taxes: $100,000\n",
      "- Taxes (20%): $20,000\n",
      "- Net Income: $80,000\n",
      "\n",
      "This answer was derived from document 'minion-tech.pdf', page 17.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\text\\\\page_17.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 17,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.38.51 :: \u001b[92mRating for question 7 is --> 10: \u001b[94m\n",
      "Question: How much is ACME Enterprises proposing to invest in Gru's Enterprises for a 40% stake according to Wile E. Coyote's investment proposal?\n",
      "\u001b[96mGround truth answer: ACME Enterprises is proposing to invest $2 million in Gru's Enterprises for a 40% stake.\n",
      "\u001b[35mGenerated answer: ACME Enterprises, as proposed by Wile E. Coyote, is planning to invest $2,000,000 for a 40% stake in Gru's Enterprises. This investment figure is based on the valuation of Gru's Enterprises at $5 million. The investment proposal is detailed on page 4 of the 'wile_e_coyote.pdf' document, where it states that the proposed investment amount by ACME Enterprises is to acquire a 40% stake in Gru's Enterprises. The financial projections and analysis section of the same page provides further insight into the expected financial outcomes of this investment, including the projected revenue, profit, and ROI for the years 2024, 2025, and 2026.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\tables\\\\page_4_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"table\"\n",
      "    },\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\text\\\\page_4.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.39.20 :: \u001b[92mRating for question 8 is --> 10: \u001b[94m\n",
      "Question: What is the amount of long-term debt listed on Gru's Enterprises' balance sheet as of the fiscal year 2023?\n",
      "\u001b[96mGround truth answer: The amount of long-term debt listed on Gru's Enterprises' balance sheet as of the fiscal year 2023 is $1,700,000.\n",
      "\u001b[35mGenerated answer: The amount of long-term debt listed on Gru's Enterprises' balance sheet for the fiscal year 2023 is $1,700,000. This information is found in the balance sheet details provided in the financial statements section of the 'minion-tech.pdf' document. The balance sheet shows the long-term debt under the 'Liabilities and Equity' section, which is part of the company's total liabilities.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\tables\\\\page_15_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 15,\n",
      "        \"type\": \"table\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.41.13 :: \u001b[92mRating for question 9 is --> 0: \u001b[94m\n",
      "Question: What is the average effectiveness rating of the weapons in Coyote's arsenal according to the hierarchy table in Wile E. Coyote's document?\n",
      "\u001b[96mGround truth answer: The average effectiveness rating of the weapons in Coyote's arsenal cannot be precisely calculated as the ratings are qualitative (High, Medium, Low, Very Low).\n",
      "\u001b[35mGenerated answer: The average effectiveness rating of the weapons in Coyote's arsenal, according to the hierarchy table in Wile E. Coyote's document, is 2.0. To calculate this, we first assign numerical values to the qualitative effectiveness ratings: 'Very Low' as 0, 'Low' as 1, 'Medium' as 2, and 'High' as 3. Then, we list the effectiveness ratings for each weapon: Giant Catapult (High = 3), Rocket Skates (Medium = 2), ACME Birdseed Trap (Low = 1), Disguise Kit (Low = 1), and ACME Portable Hole (High = 3). Adding these values together gives us a total of 10. Since there are 5 weapons listed, we divide the total by 5 to get the average effectiveness rating: 10 / 5 = 2.0. This calculation is based on the information provided on page 8 of the 'wile_e_coyote.pdf' document.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\tables\\\\page_8_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 8,\n",
      "        \"type\": \"table\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.45.03 :: \u001b[92mRating for question 10 is --> 10: \u001b[94m\n",
      "Question: What is the total number of attempts made with explosives and mechanical traps combined as per Coyote's past failed weapons categories table?\n",
      "\u001b[96mGround truth answer: The total number of attempts made with explosives and mechanical traps combined is 35.\n",
      "\u001b[35mGenerated answer: The total number of attempts made with explosives and mechanical traps combined, according to Coyote's past failed weapons categories table, is 35. This is calculated by adding the number of attempts made with explosives, which is 15, to the number of attempts made with mechanical traps, which is 20. The data is derived from the document 'wile_e_coyote.pdf', page 8.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\text\\\\page_8.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 8,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.49.02 :: \u001b[92mRating for question 11 is --> 10: \u001b[94m\n",
      "Question: Based on the financial projection table in Wile E. Coyote's proposal, what is the total projected revenue for Gru's Enterprises over the three years from 2024 to 2026?\n",
      "\u001b[96mGround truth answer: The total projected revenue for Gru's Enterprises over the three years from 2024 to 2026 is $19,500,000.\n",
      "\u001b[35mGenerated answer: The total projected revenue for Gru's Enterprises over the three years from 2024 to 2026 is $19,500,000. This total is the sum of the projected revenues for each of the three years. Here's the breakdown:\n",
      "\n",
      "- For the year 2024, the projected revenue is $5,500,000.\n",
      "- For the year 2025, the projected revenue is $6,500,000.\n",
      "- For the year 2026, the projected revenue is $7,500,000.\n",
      "\n",
      "To arrive at the total, we simply add these amounts together: $5,500,000 + $6,500,000 + $7,500,000 = $19,500,000. This calculation is based on the financial projection table provided in Wile E. Coyote's proposal, which is found on page 4 of the 'wile_e_coyote.pdf' document.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\tables\\\\page_4_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"table\"\n",
      "    },\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\text\\\\page_4.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\wile_e_coyote.pdf\\\\wile_e_coyote.pdf\",\n",
      "        \"pdf_document\": \"wile_e_coyote.pdf\",\n",
      "        \"page\": 4,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.49.21 :: \u001b[92mRating for question 12 is --> 10: \u001b[94m\n",
      "Question: What is the name of the head of operations at Gru's Enterprises as shown in their organizational chart?\n",
      "\u001b[96mGround truth answer: The name of the head of operations at Gru's Enterprises is Kevin.\n",
      "\u001b[35mGenerated answer: The Head of Operations at Gru's Enterprises is Kevin. This information is found in the organizational chart of the company, which outlines the hierarchy and structure. According to the chart, Felonius Gru is the CEO, and directly under him are three key positions: Dr. Nefario as the Lead Scientist, Kevin as the Head of Operations, and Stuart as the Chief Financial Officer. Kevin manages the Manufacturing Team, which consists of 75 Minions.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\images\\\\page_5_image_1.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 5,\n",
      "        \"type\": \"image\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.52.49 :: \u001b[92mRating for question 13 is --> 10: \u001b[94m\n",
      "Question: In the R&D expenditure bar chart from Gru's Enterprises, what is the R&D spending for the year 2022?\n",
      "\u001b[96mGround truth answer: The R&D spending for the year 2022 is $580,000.\n",
      "\u001b[35mGenerated answer: The R&D spending for Gru's Enterprises in the year 2022 is $580,000. This figure is obtained from the bar chart titled 'R&D Expenditure (USD)' which is part of the 'minion-tech.pdf' document. The chart provides a visual representation of the annual research and development expenses for the years 2021, 2022, and 2023, with the specific expenditure for 2022 being marked at $580,000. The bar chart complements the textual data found on the previous page of the document, which confirms the R&D expenditure for the year 2023 as $600,000, thereby supporting the accuracy of the visual data presented for 2022.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\images\\\\page_18_image_1.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 18,\n",
      "        \"type\": \"image\"\n",
      "    },\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\text\\\\page_18.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 18,\n",
      "        \"type\": \"text\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n",
      "\n",
      "11.02.2024_21.53.09 :: \u001b[92mRating for question 14 is --> 10: \u001b[94m\n",
      "Question: What are the three products listed in the Best-selling Products Analysis table from Gru's Enterprises and their respective gross margins?\n",
      "\u001b[96mGround truth answer: The three products listed are the Freeze Ray with a 70% gross margin, Rocket Boots with a 60% gross margin, and Bubble Gun with a 66.7% gross margin.\n",
      "\u001b[35mGenerated answer: The three products listed in the Best-selling Products Analysis table from Gru's Enterprises and their respective gross margins are as follows:\n",
      "\n",
      "| Product     | Gross Margin (%) |\n",
      "|-------------|------------------|\n",
      "| Freeze Ray  | 70%              |\n",
      "| Rocket Boots| 60%              |\n",
      "| Bubble Gun  | 66.7%            |\n",
      "\n",
      "These products and their gross margins are detailed in the table provided on page 19 of the 'minion-tech.pdf' document. The gross margin is a financial metric that indicates the percentage of sales revenue that exceeds the cost of goods sold (COGS) and is a measure of production efficiency and profitability before other operating expenses are taken into account.\n",
      "\u001b[91mReferences: [\n",
      "    {\n",
      "        \"asset\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\tables\\\\page_19_table_0.txt\",\n",
      "        \"document_path\": \"..\\\\project_minions\\\\minion-tech.pdf\\\\minion-tech.pdf\",\n",
      "        \"pdf_document\": \"minion-tech.pdf\",\n",
      "        \"page\": 19,\n",
      "        \"type\": \"table\"\n",
      "    }\n",
      "]\n",
      "\n",
      "\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "for index, qna_pair in enumerate(past_questions):\n",
    "    gen_answer, references, _, _ = search(\n",
    "        qna_pair['question'], \n",
    "        top=3, \n",
    "        computation_approach = \"Taskweaver\",  ## other options are \"NoComputationTextOnly\", \"Taskweaver\", \"AssistantsAPI\", or \"LocalPythonExec\"\n",
    "        computation_decision = \"LLM\", \n",
    "        index_name = index_name,\n",
    "        vision_support = False,  \n",
    "        count = False, \n",
    "        verbose = False)\n",
    "\n",
    "    rating = recover_json(ask_LLM_with_JSON(rate_answers_prompt_template.format(question=qna_pair['question'], ground_truth_answer=qna_pair['answer'], generated_answer=gen_answer)))\n",
    "    rating = int(rating['rating'])\n",
    "    past_questions[index]['rating'] = rating\n",
    "    logc(\"Rating for question \" + str(index) + \" is --> \" + str(rating), \n",
    "         f\"\\nQuestion: {qna_pair['question']}\\n{bc.OKCYAN}Ground truth answer: {qna_pair['answer']}\\n{bc.MAGENTA}Generated answer: {gen_answer}\\n{bc.LIGHT_RED}References: {json.dumps(references, indent=4)}\\n\\n\\n\")\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mmdoc",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}