Cortex_LLM/Cortex_RAG_Eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RAG Evaluation with Snowflake Cortex\n",
    "\n",
    "This notebook demonstrates how to use Snowflake Cortex to generate synthetic questions and answers for evaluating a RAG application.\n",
    "\n",
    "This notebooks is based on the [Hugging Face RAG evaluation notebook](https://huggingface.co/learn/cookbook/rag_evaluation), but using Cortex to generate the synthetic questions and answers and performing the operations in Snowpark (and not on the local machine).\n",
    "\n",
    "There are four main steps in this notebook:\n",
    "-   **Step 1**: Generating Synthetic Questions and Answers\n",
    "-   **Step 2**: Evaluating Synthetic Questions for Groundness, Relevance, and Context\n",
    "-   **Step 3**: Getting Predictions for Synthetic Questions\n",
    "-   **Step 4**: Evaluating Performance on Synthetic Questions\n",
    "\n",
    "The notebook uses `mixtral-8x7b` for generarting questions/answers and `mistral-large` for the final evaluation on the synthetic questions.\n",
    "\n",
    "There are many ways to evaluate a RAG model, and this notebook is just one example. The evaluation metrics used in this notebook are not exhaustive, and you may want to consider additional metrics when evaluating your RAG model. This notebook was put together for one project, so plenty of room for improvement and customization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# Snowpark for Python\n",
    "from snowflake.snowpark.session import Session\n",
    "from snowflake.snowpark.types import Variant\n",
    "from snowflake.snowpark.version import VERSION\n",
    "\n",
    "# Snowpark ML\n",
    "# Misc\n",
    "import pandas as pd\n",
    "import json\n",
    "import logging \n",
    "logger = logging.getLogger(\"snowflake.snowpark.session\")\n",
    "logger.setLevel(logging.ERROR)\n",
    "\n",
    "from snowflake import connector\n",
    "from snowflake.ml.utils import connection_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "with open('../../creds.json') as f:\n",
    "    data = json.load(f)\n",
    "    USERNAME = data['user']\n",
    "    PASSWORD = data['password']\n",
    "    SF_ACCOUNT = data['account']\n",
    "    SF_WH = data['warehouse']\n",
    "\n",
    "CONNECTION_PARAMETERS = {\n",
    "   \"account\": SF_ACCOUNT,\n",
    "   \"user\": USERNAME,\n",
    "   \"password\": PASSWORD,\n",
    "}\n",
    "\n",
    "session = Session.builder.configs(CONNECTION_PARAMETERS).create()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User                        : RSHAH\n",
      "Role                        : \"RAJIV\"\n",
      "Database                    : \"RAJIV\"\n",
      "Schema                      : \"DOCAI\"\n",
      "Warehouse                   : \"RAJIV\"\n",
      "Snowflake version           : 8.18.0\n",
      "Snowpark for Python version : 1.11.1\n",
      "Snowflake ML version        : 1.5.0\n"
     ]
    }
   ],
   "source": [
    "snowflake_environment = session.sql('select current_user(), current_version()').collect()\n",
    "snowpark_version = VERSION\n",
    "\n",
    "from snowflake.ml import version\n",
    "mlversion = version.VERSION\n",
    "\n",
    "\n",
    "# Current Environment Details\n",
    "print('User                        : {}'.format(snowflake_environment[0][0]))\n",
    "print('Role                        : {}'.format(session.get_current_role()))\n",
    "print('Database                    : {}'.format(session.get_current_database()))\n",
    "print('Schema                      : {}'.format(session.get_current_schema()))\n",
    "print('Warehouse                   : {}'.format(session.get_current_warehouse()))\n",
    "print('Snowflake version           : {}'.format(snowflake_environment[0][1]))\n",
    "print('Snowpark for Python version : {}.{}.{}'.format(snowpark_version[0],snowpark_version[1],snowpark_version[2]))\n",
    "print('Snowflake ML version        : {}.{}.{}'.format(mlversion[0],mlversion[2],mlversion[4]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Generating Synthetic Questions and Answers\n",
    "My starting point is the [SEC RAG example](https://medium.com/snowflake/retrieval-augmented-generation-rag-application-using-snowflake-cortex-and-streamlit-9cb261e81c2e) where the data has already been chunked. I then filter down to just the Rivian entries. \n",
    "\n",
    "Feel free to modify this, the goal is a representative set of topics for the synthetic questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"SEC_DOCUMENT_ID\"          |\"DOCUMENT_TYPE\"   |\"COMPANY_NAME\"                |\"SIC_CODE_CATEGORY\"      |\"SIC_CODE_DESCRIPTION\"                 |\"COUNTRY\"  |\"PERIOD_END_DATE\"  |\"CONTENT_CHUNK\"                                     |\"START_INDEX\"  |\"EMBEDDING\"                                         |\"DOCUMENT_INDEX_ROWNUM\"  |\"ROWNUM\"  |\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |rivn-20221231 UNITED STATES SECURITIES AND EXCH...  |4              |[0.0134749636054039, -0.01612430065870285, -0.0...  |1                        |3635739   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.001 par value per share RIVN The Nasdaq Stock ...  |963            |[-0.013030052185058594, -0.014440304599702358, ...  |2                        |3635740   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.    Yes  ☒    No ☐  Indicate by check mark whe...  |1712           |[-0.006698981858789921, -0.03849697485566139, -...  |3                        |3635741   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. See the definitions of “large accelerated fil...  |2256           |[-0.04192263260483742, -0.0486314631998539, -0....  |4                        |3635742   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.S.C. 7262(b)) by the registered public account...  |3076           |[-0.015177706256508827, -0.03369320556521416, -...  |5                        |3635743   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. In the case of 5% or greater stockholders, we...  |3911           |[0.004648115485906601, -0.05180753022432327, -0...  |6                        |3635744   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.  As of February 14, 2023,  919,265,013  share...  |4511           |[0.010190531611442566, -0.026317495852708817, -...  |7                        |3635745   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Mine Safety Disclosures  49 Part II 50 Item 5...  |5300           |[0.0006388800102286041, -0.024540066719055176, ...  |8                        |3635746   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Security Ownership of Certain Beneficial Owne...  |6074           |[-0.0035113098565489054, -0.015815898776054382,...  |9                        |3635747   |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. All statements other than statements of histo...  |6851           |[0.011535296216607094, -0.040209297090768814, -...  |10                       |3635748   |\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from snowflake.snowpark.functions import col\n",
    "import snowflake.snowpark.functions as f\n",
    "from snowflake.snowpark.functions import col, split, lit, regexp_extract\n",
    "from snowflake.cortex import Complete\n",
    "\n",
    "\n",
    "article_df = session.table(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K\")\n",
    "filtered_df = article_df.filter(col(\"company_name\").like(\"RIVIAN%\"))\n",
    "\n",
    "# Now you can perform further operations on filtered_df, like displaying the data\n",
    "filtered_df.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "497"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Count the number of starting rows\n",
    "filtered_df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will only want to select a subset of the chunks to build your generated examples. The first time you run this, start small, but your final evaluation should probably be at least 50 questions up to hundreds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "40"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shuffled_df = filtered_df.sample(n=40)\n",
    "shuffled_df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build the Q&A pairs for the RAG evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "QA_generation_prompt = \"\"\"\n",
    "Your task is to write a factoid question and an answer given a context.\n",
    "Your factoid question should be answerable with a specific, concise piece of factual information from the context.\n",
    "Your factoid question should be formulated in the same style as questions users could ask in a search engine.\n",
    "This means that your factoid question MUST NOT mention something like \"according to the passage\" or \"context\".\n",
    "\n",
    "Provide your answer as follows:\n",
    "\n",
    "Output:::\n",
    "Factoid question: (your factoid question)\n",
    "Answer: (your answer to the factoid question)\n",
    "\n",
    "Now here is the context.\n",
    "\n",
    "Context: \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Complete() is experimental since 1.0.12. Do not use it in production. \n"
     ]
    }
   ],
   "source": [
    "outdf = shuffled_df.withColumn(\n",
    "    \"GenQA\",\n",
    "    Complete(\n",
    "        model='mixtral-8x7b',prompt = f.concat(\n",
    "            f.lit(QA_generation_prompt),\n",
    "            f.col(\"CONTENT_CHUNK\"),\n",
    "            f.lit(\"\\n Output:::\")\n",
    "            ))\n",
    ")\n",
    "outputs = outdf.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' Factoid question: How many shares of common stock were issued and outstanding by Rivian Automotive, Inc. as of December 31, 2022?\\nAnswer: 926 shares'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs.GENQA[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"SEC_DOCUMENT_ID\"          |\"DOCUMENT_TYPE\"   |\"COMPANY_NAME\"                |\"SIC_CODE_CATEGORY\"      |\"SIC_CODE_DESCRIPTION\"                 |\"COUNTRY\"  |\"PERIOD_END_DATE\"  |\"CONTENT_CHUNK\"                                     |\"START_INDEX\"  |\"EMBEDDING\"                                         |\"DOCUMENT_INDEX_ROWNUM\"  |\"ROWNUM\"  |\"GENQA\"                                             |\"SPLIT_COL\"                                         |\"GENQ\"                                              |\"GENA\"                                              |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.  Holders As of February 14, 2023, there were ...  |262748         |[0.00538471108302474, -0.05729801580309868, -0....  |317                      |3636055   | Factoid question: Has the company ever declare...  |[                                                   |\" Factoid question: Has the company ever declar...  |\"Answer: No, the company has never declared or ...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: No, the company has never declared or p...  |  \" Factoid question: Has the company ever decl...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: No, the company has never declared o...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. By Order dated February 16, 2023  48 RIVIAN A...  |257392         |[0.02331852912902832, -0.05648023635149002, -0....  |310                      |3636048   | Factoid question: Has the State of Georgia's p...  |[                                                   |\" Factoid question: Has the State of Georgia's ...  |\"Answer: No, the petition was denied by Order d...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: No, the petition was denied by Order da...  |  \" Factoid question: Has the State of Georgia'...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: No, the petition was denied by Order...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. In 2021, changes to the SAFE Vehicles rule we...  |133902         |[-0.014899153262376785, -0.031731441617012024, ...  |164                      |3635902   | Factoid question: When were fuel economy stand...  |[                                                   |\" Factoid question: When were fuel economy stan...  |\"Answer: Fuel economy standards were last made ...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Fuel economy standards were last made m...  |  \" Factoid question: When were fuel economy st...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Fuel economy standards were last mad...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Our processes and controls to identify, measu...  |218016         |[-0.0010109544964507222, -0.02101697027683258, ...  |267                      |3636005   | Factoid question: What factors could impact a ...  |[                                                   |\" Factoid question: What factors could impact a...  |\"Answer: ESG metrics measurement methodologies,...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: ESG metrics measurement methodologies, ...  |  \" Factoid question: What factors could impact...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: ESG metrics measurement methodologie...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.S.C. 7262(b)) by the registered public account...  |3076           |[-0.015177706256508827, -0.03369320556521416, -...  |5                        |3635743   | Factoid question: What was the approximate mar...  |[                                                   |\" Factoid question: What was the approximate ma...  |\"Answer: $12.3 billion\"                             |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: $12.3 billion                               |  \" Factoid question: What was the approximate ...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: $12.3 billion\"                           |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. The R1T, R1S, and EDV are fully compliant wit...  |31140          |[0.004507666919380426, -0.03386825695633888, -0...  |41                       |3635779   | Factoid question: Is the R1T, R1S, and EDV com...  |[                                                   |\" Factoid question: Is the R1T, R1S, and EDV co...  |\"Answer: Yes, the R1T, R1S, and EDV are fully c...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Yes, the R1T, R1S, and EDV are fully co...  |  \" Factoid question: Is the R1T, R1S, and EDV ...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Yes, the R1T, R1S, and EDV are fully...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.  As of February 14, 2023,  919,265,013  share...  |4511           |[0.010190531611442566, -0.026317495852708817, -...  |7                        |3635745   | Factoid question: How many shares of Class A c...  |[                                                   |\" Factoid question: How many shares of Class A ...  |\"Answer: 919,265,013 shares of Class A common s...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: 919,265,013 shares of Class A common st...  |  \" Factoid question: How many shares of Class ...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: 919,265,013 shares of Class A common...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. 52 RIVIAN AUTOMOTIVE, INC. • Ability to Conve...  |271026         |[-0.005223366431891918, -0.02997530624270439, -...  |327                      |3636065   | Factoid question: How does Rivian Automotive, ...  |[                                                   |\" Factoid question: How does Rivian Automotive,...  |\"Answer: By creating a higher-margin, recurring...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: By creating a higher-margin, recurring ...  |  \" Factoid question: How does Rivian Automotiv...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: By creating a higher-margin, recurri...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. These estimates require the use of judgments ...  |326180         |[-0.0019456164445728064, -0.011028078384697437,...  |392                      |3636130   | Factoid question: What are some examples of it...  |[                                                   |\" Factoid question: What are some examples of i...  |\"Answer: Inventory valuation, property, plant, ...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Inventory valuation, property, plant, a...  |  \" Factoid question: What are some examples of...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Inventory valuation, property, plant...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. All statements other than statements of histo...  |6851           |[0.011535296216607094, -0.040209297090768814, -...  |10                       |3635748   | Factoid question: What types of statements in ...  |[                                                   |\" Factoid question: What types of statements in...  |\"Answer: The forward-looking statements in this...  |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The forward-looking statements in this ...  |  \" Factoid question: What types of statements ...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The forward-looking statements in th...  |                                                    |                                                    |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "outdf = outdf.withColumn(\"split_col\", split(col(\"GENQA\"), lit('\\n')))\n",
    "\n",
    "# Create new columns 'GENQ' and 'GENA' by accessing elements of the array\n",
    "outdf = outdf.withColumn(\"GENQ\", col(\"split_col\")[0])\n",
    "outdf = outdf.withColumn(\"GENA\", col(\"split_col\")[1])\n",
    "\n",
    "outdf.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "#save our work periodically\n",
    "table_name = \"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\"  # Specify your new table name here\n",
    "outdf.write.mode(\"overwrite\").save_as_table(table_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "outdf = session.table(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Evaluating Synthetic Questions for Groundness, Relevance, and Context\n",
    "The questions generated by the previous agent can have many flaws: we should do a quality check before validating these questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Groundness Criteria"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "question_groundedness_critique_prompt = \"\"\"\n",
    "You will be given a context and a question.\n",
    "Your task is to provide a 'total rating' scoring how well one can answer the given question unambiguously with the given context.\n",
    "Give your answer on a scale of 1 to 5, where 1 means that the question is not answerable at all given the context, and 5 means that the question is clearly and unambiguously answerable with the context.\n",
    "\n",
    "Provide your answer as follows:\n",
    "\n",
    "Answer:::\n",
    "Evaluation: (your rationale for the rating, as a text)\n",
    "Total rating: (your rating, as a integer between 1 and 5)\n",
    "\n",
    "You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.\n",
    "\n",
    "Now here are the question and context.\n",
    "\n",
    "Question: \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "outdf = outdf.withColumn(\n",
    "    \"Eval_Groundness\",\n",
    "    Complete(\n",
    "        model='mixtral-8x7b',prompt = f.concat(\n",
    "            f.lit(question_groundedness_critique_prompt),\n",
    "            f.col(\"GENQ\"),\n",
    "            f.lit(\"\\n Context: \"),\n",
    "            f.col(\"CONTENT_CHUNK\"),\n",
    "            f.lit(\"\\n Answer:::\")\n",
    "            ))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' Evaluation: The context provides a detailed explanation of the potential consequences if the business is unable to adequately service their vehicles. It mentions adverse publicity, regulatory action, safety recall, fines, loss of permits and licenses, and negative impact on the business, prospects, financial condition, results of operations, and cash flows. Therefore, the question is clearly and unambiguously answerable with the context.\\n\\nTotal rating: 5'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## you can always pull this into pandas to see the results\n",
    "outputs = outdf.to_pandas()\n",
    "outputs.EVAL_GROUNDNESS[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split out the score,which will later filter on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"SEC_DOCUMENT_ID\"          |\"DOCUMENT_TYPE\"   |\"COMPANY_NAME\"                |\"SIC_CODE_CATEGORY\"      |\"SIC_CODE_DESCRIPTION\"                 |\"COUNTRY\"  |\"PERIOD_END_DATE\"  |\"CONTENT_CHUNK\"                                     |\"START_INDEX\"  |\"EMBEDDING\"                                         |\"DOCUMENT_INDEX_ROWNUM\"  |\"ROWNUM\"  |\"GENQA\"                                             |\"SPLIT_COL\"                                         |\"GENQ\"                                              |\"GENA\"                                              |\"EVAL_GROUNDNESS\"                                   |\"SCORE_EG\"  |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |rivn-20221231 UNITED STATES SECURITIES AND EXCH...  |4              |[0.0134749636054039, -0.01612430065870285, -0.0...  |1                        |3635739   | Factoid question: What is the trading symbol f...  |[                                                   |\" Factoid question: What is the trading symbol ...  |\"Answer: The trading symbol for Rivian Automoti...  | Evaluation: The context provides the trading s...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The trading symbol for Rivian Automotiv...  |  \" Factoid question: What is the trading symbo...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The trading symbol for Rivian Automo...  |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We do not have any control over these analyst...  |229817         |[0.014223161153495312, -0.02481968328356743, -0...  |279                      |3636017   | Factoid question: Why might demand for Rivian ...  |[                                                   |\" Factoid question: Why might demand for Rivian...  |\"Answer: Demand for Rivian Automotive's Class A...  | Evaluation: The context provides several reaso...  |4           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Demand for Rivian Automotive's Class A ...  |  \" Factoid question: Why might demand for Rivi...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Demand for Rivian Automotive's Class...  |                                                    |                                                    |Total rating: 4                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |The context provides a clear and unambiguous an...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Our vehicles use lithium-ion battery cells, w...  |114360         |[0.019371476024389267, -0.005720599088817835, -...  |141                      |3635879   | Factoid question: What type of battery cells a...  |[                                                   |\" Factoid question: What type of battery cells ...  |\"Answer: Our vehicles use lithium-ion battery c...  | Answer: The type of battery cells used in the ...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Our vehicles use lithium-ion battery ce...  |  \" Factoid question: What type of battery cell...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Our vehicles use lithium-ion battery...  |                                                    |                                                    |Evaluation: The context provides a clear and di...  |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.  NET LOSS PER SHARE The Company's basic net l...  |389484         |[-0.01139928586781025, -0.029863249510526657, -...  |473                      |3636211   | Factoid question: How are net loss and common ...  |[                                                   |\" Factoid question: How are net loss and common...  |\"Answer: The Company's basic net loss per share...  | Evaluation: The context provides information a...  |1           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The Company's basic net loss per share ...  |  \" Factoid question: How are net loss and comm...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The Company's basic net loss per sha...  |                                                    |                                                    |Total rating: 1                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Answer:::                                           |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Evaluation: The context provides information ab...  |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 1                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Such damage or injury could also lead to adve...  |116092         |[-0.001115779741667211, -0.01557418704032898, -...  |143                      |3635881   | Factoid question: What could be the potential ...  |[                                                   |\" Factoid question: What could be the potential...  |\"Answer: The business, prospects, financial con...  | Evaluation: The context provides a detailed ex...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The business, prospects, financial cond...  |  \" Factoid question: What could be the potenti...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The business, prospects, financial c...  |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We use our Investor Relations website as a me...  |55419          |[0.007679999805986881, -0.015060197561979294, -...  |70                       |3635808   | Factoid question: What is one way the company ...  |[                                                   |\" Factoid question: What is one way the company...  |\"Answer: The company uses its Investor Relation...  | Answer: The company discloses material informa...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The company uses its Investor Relations...  |  \" Factoid question: What is one way the compa...  |                                                    |                                                    |Evaluation: The context directly answers the qu...  |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The company uses its Investor Relati...  |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. The grant-date fair value of stock options gr...  |376076         |[-0.0316321924328804, -0.018513798713684082, -0...  |456                      |3636194   | Factoid question: How are the grant-date fair ...  |[                                                   |\" Factoid question: How are the grant-date fair...  |\"Answer: The grant-date fair value of RSUs gran...  | Answer: The grant-date fair value of RSUs gran...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The grant-date fair value of RSUs grant...  |  \" Factoid question: How are the grant-date fa...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The grant-date fair value of RSUs gr...  |                                                    |                                                    |Evaluation: The context clearly states that the...  |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Any reduction, elimination, or discriminatory...  |127973         |[0.006324884947389364, -0.027103247120976448, -...  |157                      |3635895   | Factoid question: Why are certain hybrids and ...  |[                                                   |\" Factoid question: Why are certain hybrids and...  |\"Answer: This is to support transportation infr...  | Evaluation: The context explains that many sta...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: This is to support transportation infra...  |  \" Factoid question: Why are certain hybrids a...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: This is to support transportation in...  |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Confidence: 90%                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.1** Certification of Chief Executive Officer p...  |408362         |[-0.01204388216137886, -0.008523665368556976, -...  |494                      |3636232   | Factoid question: What type of document is the...  |[                                                   |\" Factoid question: What type of document is th...  |\"Answer: The Inline XBRL Instance Document is a...  | Evaluation: The context provides information a...  |3           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The Inline XBRL Instance Document is an...  |  \" Factoid question: What type of document is ...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The Inline XBRL Instance Document is...  |                                                    |                                                    |Total rating: 3                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Explanation: While the context does provide som...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We market our ability to provide our customer...  |110525         |[-0.010174417868256569, -0.024404698982834816, ...  |136                      |3635874   | Factoid question: What are some of Rivian's ch...  |[                                                   |\" Factoid question: What are some of Rivian's c...  |\"Answer: Rivian offers comprehensive charging s...  | Evaluation: The context clearly states that Ri...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Rivian offers comprehensive charging so...  |  \" Factoid question: What are some of Rivian's...  |                                                    |                                                    |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Rivian offers comprehensive charging...  |                                                    |                                                    |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Extracting the score by taking the rightmost character of the last part of split_rating\n",
    "outdf = outdf.withColumn(\"score_EG\", regexp_extract(col(\"EVAL_GROUNDNESS\"), r\"Total rating:\\s*(\\d)\", 1))\n",
    "\n",
    "outdf.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relevance Criteria"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "question_relevance_critique_prompt = \"\"\"\n",
    "You will be given a question.\n",
    "Your task is to provide a 'total rating' representing how useful this question can be to analysts.\n",
    "Give your answer on a scale of 1 to 5, where 1 means that the question is not useful at all, and 5 means that the question is extremely useful.\n",
    "\n",
    "Provide your answer as follows:\n",
    "\n",
    "Answer:::\n",
    "Evaluation: (your rationale for the rating, as a text)\n",
    "Total rating: (your rating, as an integer between 1 and 5)\n",
    "\n",
    "You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.\n",
    "\n",
    "Now here is the question.\n",
    "\n",
    "Question: \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"SEC_DOCUMENT_ID\"          |\"DOCUMENT_TYPE\"   |\"COMPANY_NAME\"                |\"SIC_CODE_CATEGORY\"      |\"SIC_CODE_DESCRIPTION\"                 |\"COUNTRY\"  |\"PERIOD_END_DATE\"  |\"CONTENT_CHUNK\"                                     |\"START_INDEX\"  |\"EMBEDDING\"                                         |\"DOCUMENT_INDEX_ROWNUM\"  |\"ROWNUM\"  |\"GENQA\"                                             |\"SPLIT_COL\"                                         |\"GENQ\"                                              |\"GENA\"                                              |\"EVAL_GROUNDNESS\"                                   |\"SCORE_EG\"  |\"EVAL_RELEVANCE\"                                    |\"SCORE_ER\"  |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |rivn-20221231 UNITED STATES SECURITIES AND EXCH...  |4              |[0.0134749636054039, -0.01612430065870285, -0.0...  |1                        |3635739   | Factoid question: What is the trading symbol f...  |[                                                   |\" Factoid question: What is the trading symbol ...  |\"Answer: The trading symbol for Rivian Automoti...  | Evaluation: The context provides the trading s...  |5           | Evaluation: This question is seeking specific ...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The trading symbol for Rivian Automotiv...  |  \" Factoid question: What is the trading symbo...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The trading symbol for Rivian Automo...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |Note: The trading symbol for Rivian Automotive ...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We do not have any control over these analyst...  |229817         |[0.014223161153495312, -0.02481968328356743, -0...  |279                      |3636017   | Factoid question: Why might demand for Rivian ...  |[                                                   |\" Factoid question: Why might demand for Rivian...  |\"Answer: Demand for Rivian Automotive's Class A...  | Evaluation: The context provides several reaso...  |4           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Demand for Rivian Automotive's Class A ...  |  \" Factoid question: Why might demand for Rivi...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Demand for Rivian Automotive's Class...  |                                                    |                                                    |Total rating: 4                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |The context provides several clear and unambigu...  |            |Confidence: 90%                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Our vehicles use lithium-ion battery cells, w...  |114360         |[0.019371476024389267, -0.005720599088817835, -...  |141                      |3635879   | Factoid question: What type of battery cells a...  |[                                                   |\" Factoid question: What type of battery cells ...  |\"Answer: Our vehicles use lithium-ion battery c...  | Answer: The type of battery cells used in the ...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Our vehicles use lithium-ion battery ce...  |  \" Factoid question: What type of battery cell...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Our vehicles use lithium-ion battery...  |                                                    |                                                    |Evaluation: The context provides a clear and di...  |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 5                                     |            |The question is open-ended and encourages a det...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.  NET LOSS PER SHARE The Company's basic net l...  |389484         |[-0.01139928586781025, -0.029863249510526657, -...  |473                      |3636211   | Factoid question: How are net loss and common ...  |[                                                   |\" Factoid question: How are net loss and common...  |\"Answer: The Company's basic net loss per share...  | Evaluation: The context provides information a...  |1           | Evaluation: This question is useful for analys...  |4           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The Company's basic net loss per share ...  |  \" Factoid question: How are net loss and comm...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The Company's basic net loss per sha...  |                                                    |                                                    |Total rating: 1                                     |            |Total rating: 4                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Answer:::                                           |            |Explanation: While the question is useful, it c...  |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Evaluation: The context provides information ab...  |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 1                                     |            |                                                    |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Such damage or injury could also lead to adve...  |116092         |[-0.001115779741667211, -0.01557418704032898, -...  |143                      |3635881   | Factoid question: What could be the potential ...  |[                                                   |\" Factoid question: What could be the potential...  |\"Answer: The business, prospects, financial con...  | Evaluation: The context provides a detailed ex...  |5           | Evaluation: This question is open-ended and re...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The business, prospects, financial cond...  |  \" Factoid question: What could be the potenti...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The business, prospects, financial c...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |The question is useful in highlighting the pote...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We use our Investor Relations website as a me...  |55419          |[0.007679999805986881, -0.015060197561979294, -...  |70                       |3635808   | Factoid question: What is one way the company ...  |[                                                   |\" Factoid question: What is one way the company...  |\"Answer: The company uses its Investor Relation...  | Answer: The company discloses material informa...  |5           | Evaluation: This question is seeking informati...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The company uses its Investor Relations...  |  \" Factoid question: What is one way the compa...  |                                                    |                                                    |Evaluation: The context directly answers the qu...  |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The company uses its Investor Relati...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |This question is extremely useful for analysts ...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. The grant-date fair value of stock options gr...  |376076         |[-0.0316321924328804, -0.018513798713684082, -0...  |456                      |3636194   | Factoid question: How are the grant-date fair ...  |[                                                   |\" Factoid question: How are the grant-date fair...  |\"Answer: The grant-date fair value of RSUs gran...  | Answer: The grant-date fair value of RSUs gran...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The grant-date fair value of RSUs grant...  |  \" Factoid question: How are the grant-date fa...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The grant-date fair value of RSUs gr...  |                                                    |                                                    |Evaluation: The context clearly states that the...  |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 5                                     |            |Note: The rating of 5 is given because the ques...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Any reduction, elimination, or discriminatory...  |127973         |[0.006324884947389364, -0.027103247120976448, -...  |157                      |3635895   | Factoid question: Why are certain hybrids and ...  |[                                                   |\" Factoid question: Why are certain hybrids and...  |\"Answer: This is to support transportation infr...  | Evaluation: The context explains that many sta...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: This is to support transportation infra...  |  \" Factoid question: Why are certain hybrids a...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: This is to support transportation in...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Confidence: 90%                                     |            |Confidence: 90%                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.1** Certification of Chief Executive Officer p...  |408362         |[-0.01204388216137886, -0.008523665368556976, -...  |494                      |3636232   | Factoid question: What type of document is the...  |[                                                   |\" Factoid question: What type of document is th...  |\"Answer: The Inline XBRL Instance Document is a...  | Evaluation: The context provides information a...  |3           | Evaluation: This question is seeking to unders...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The Inline XBRL Instance Document is an...  |  \" Factoid question: What type of document is ...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The Inline XBRL Instance Document is...  |                                                    |                                                    |Total rating: 3                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Explanation: While the context does provide som...  |            |Reference(s):                                       |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |- Inline XBRL (eXtensible Business Reporting La...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We market our ability to provide our customer...  |110525         |[-0.010174417868256569, -0.024404698982834816, ...  |136                      |3635874   | Factoid question: What are some of Rivian's ch...  |[                                                   |\" Factoid question: What are some of Rivian's c...  |\"Answer: Rivian offers comprehensive charging s...  | Evaluation: The context clearly states that Ri...  |5           | Evaluation: This question is useful for analys...  |4           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Rivian offers comprehensive charging so...  |  \" Factoid question: What are some of Rivian's...  |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Rivian offers comprehensive charging...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 4                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |Note: While this question is useful, it is limi...  |            |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "outdf = outdf.withColumn(\n",
    "    \"EVAL_RELEVANCE\",\n",
    "    Complete(\n",
    "        model='mixtral-8x7b',prompt = f.concat(\n",
    "            f.lit(question_relevance_critique_prompt),\n",
    "            f.col(\"GENQ\"),\n",
    "            f.lit(\"\\n Answer:::\")\n",
    "            ))\n",
    ")\n",
    "# Extracting the score by taking the rightmost character of the last part of split_rating\n",
    "outdf = outdf.withColumn(\"score_ER\", regexp_extract(col(\"EVAL_RELEVANCE\"), r\"Total rating:\\s*(\\d)\", 1))\n",
    "\n",
    "outdf.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context Criteria"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "question_standalone_critique_prompt = \"\"\"\n",
    "You will be given a question.\n",
    "Your task is to provide a 'total rating' representing how context-independant this question is.\n",
    "Give your answer on a scale of 1 to 5, where 1 means that the question depends on additional information to be understood, and 5 means that the question makes sense by itself.\n",
    "For instance, if the question refers to a particular setting, like 'in the context' or 'in the document', the rating must be 1.\n",
    "The questions can contain obscure technical nouns or acronyms like Gradio, Hub, Hugging Face or Space and still be a 5: it must simply be clear to an operator with access to documentation what the question is about.\n",
    "\n",
    "For instance, \"What is the name of the checkpoint from which the ViT model is imported?\" should receive a 1, since there is an implicit mention of a context, thus the question is not independant from the context.\n",
    "\n",
    "Provide your answer as follows:\n",
    "\n",
    "Answer:::\n",
    "Evaluation: (your rationale for the rating, as a text)\n",
    "Total rating: (your rating, as an integer between 1 and 5)\n",
    "\n",
    "You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.\n",
    "\n",
    "Now here is the question.\n",
    "\n",
    "Question: \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"SEC_DOCUMENT_ID\"          |\"DOCUMENT_TYPE\"   |\"COMPANY_NAME\"                |\"SIC_CODE_CATEGORY\"      |\"SIC_CODE_DESCRIPTION\"                 |\"COUNTRY\"  |\"PERIOD_END_DATE\"  |\"CONTENT_CHUNK\"                                     |\"START_INDEX\"  |\"EMBEDDING\"                                         |\"DOCUMENT_INDEX_ROWNUM\"  |\"ROWNUM\"  |\"GENQA\"                                             |\"SPLIT_COL\"                                         |\"GENQ\"                                              |\"GENA\"                                              |\"EVAL_GROUNDNESS\"                                   |\"SCORE_EG\"  |\"EVAL_RELEVANCE\"                                    |\"SCORE_ER\"  |\"EVAL_CONTEXT\"                                      |\"SCORE_EC\"  |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |rivn-20221231 UNITED STATES SECURITIES AND EXCH...  |4              |[0.0134749636054039, -0.01612430065870285, -0.0...  |1                        |3635739   | Factoid question: What is the trading symbol f...  |[                                                   |\" Factoid question: What is the trading symbol ...  |\"Answer: The trading symbol for Rivian Automoti...  | Evaluation: The context provides the trading s...  |5           | Evaluation: This question is seeking specific ...  |5           | Evaluation: This question is seeking specific ...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The trading symbol for Rivian Automotiv...  |  \" Factoid question: What is the trading symbo...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The trading symbol for Rivian Automo...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |Note: The trading symbol for Rivian Automotive ...  |            |Note: The trading symbol for Rivian Automotive ...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We do not have any control over these analyst...  |229817         |[0.014223161153495312, -0.02481968328356743, -0...  |279                      |3636017   | Factoid question: Why might demand for Rivian ...  |[                                                   |\" Factoid question: Why might demand for Rivian...  |\"Answer: Demand for Rivian Automotive's Class A...  | Evaluation: The context provides several reaso...  |5           | Evaluation: This question is useful for analys...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Demand for Rivian Automotive's Class A ...  |  \" Factoid question: Why might demand for Rivi...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Demand for Rivian Automotive's Class...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |The context clearly and unambiguously answers t...  |            |Confidence: 90%                                     |            |Confidence: 90%                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Our vehicles use lithium-ion battery cells, w...  |114360         |[0.019371476024389267, -0.005720599088817835, -...  |141                      |3635879   | Factoid question: What type of battery cells a...  |[                                                   |\" Factoid question: What type of battery cells ...  |\"Answer: Our vehicles use lithium-ion battery c...  | Answer: The type of battery cells used in the ...  |5           | Evaluation: This question is useful for analys...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Our vehicles use lithium-ion battery ce...  |  \" Factoid question: What type of battery cell...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Our vehicles use lithium-ion battery...  |                                                    |                                                    |Evaluation: The context provides a clear and di...  |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 5                                     |            |The question is open-ended and encourages a det...  |            |The question is open-ended and encourages a det...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.  NET LOSS PER SHARE The Company's basic net l...  |389484         |[-0.01139928586781025, -0.029863249510526657, -...  |473                      |3636211   | Factoid question: How are net loss and common ...  |[                                                   |\" Factoid question: How are net loss and common...  |\"Answer: The Company's basic net loss per share...  | Evaluation: The context provides information a...  |1           | Evaluation: This question is useful for analys...  |4           | Evaluation: This question is useful for analys...  |4           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The Company's basic net loss per share ...  |  \" Factoid question: How are net loss and comm...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The Company's basic net loss per sha...  |                                                    |                                                    |Total rating: 1                                     |            |Total rating: 4                                     |            |Total rating: 4                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Answer:::                                           |            |Explanation: While the question is useful, it c...  |            |Explanation: While the question is useful, it c...  |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Evaluation: The context provides information ab...  |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 1                                     |            |                                                    |            |                                                    |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Such damage or injury could also lead to adve...  |116092         |[-0.001115779741667211, -0.01557418704032898, -...  |143                      |3635881   | Factoid question: What could be the potential ...  |[                                                   |\" Factoid question: What could be the potential...  |\"Answer: The business, prospects, financial con...  | Evaluation: The context provides a detailed ex...  |5           | Evaluation: This question is open-ended and re...  |5           | Evaluation: This question is open-ended and re...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The business, prospects, financial cond...  |  \" Factoid question: What could be the potenti...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The business, prospects, financial c...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |The question is useful in highlighting the pote...  |            |The question is useful in highlighting the pote...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We use our Investor Relations website as a me...  |55419          |[0.007679999805986881, -0.015060197561979294, -...  |70                       |3635808   | Factoid question: What is one way the company ...  |[                                                   |\" Factoid question: What is one way the company...  |\"Answer: The company uses its Investor Relation...  | Answer: The company discloses material informa...  |5           | Evaluation: This question is seeking informati...  |5           | Evaluation: This question is seeking informati...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The company uses its Investor Relations...  |  \" Factoid question: What is one way the compa...  |                                                    |                                                    |Evaluation: The context directly answers the qu...  |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The company uses its Investor Relati...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |This question is extremely useful for analysts ...  |            |This question is extremely useful for analysts ...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. The grant-date fair value of stock options gr...  |376076         |[-0.0316321924328804, -0.018513798713684082, -0...  |456                      |3636194   | Factoid question: How are the grant-date fair ...  |[                                                   |\" Factoid question: How are the grant-date fair...  |\"Answer: The grant-date fair value of RSUs gran...  | Answer: The grant-date fair value of RSUs gran...  |5           | Evaluation: This question is useful for analys...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The grant-date fair value of RSUs grant...  |  \" Factoid question: How are the grant-date fa...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The grant-date fair value of RSUs gr...  |                                                    |                                                    |Evaluation: The context clearly states that the...  |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Total rating: 5                                     |            |Note: The rating of 5 is given because the ques...  |            |Note: The rating of 5 is given because the ques...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. Any reduction, elimination, or discriminatory...  |127973         |[0.006324884947389364, -0.027103247120976448, -...  |157                      |3635895   | Factoid question: Why are certain hybrids and ...  |[                                                   |\" Factoid question: Why are certain hybrids and...  |\"Answer: This is to support transportation infr...  | Evaluation: The context explains that many sta...  |5           | Evaluation: This question is useful for analys...  |5           | Evaluation: This question is useful for analys...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: This is to support transportation infra...  |  \" Factoid question: Why are certain hybrids a...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: This is to support transportation in...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Confidence: 90%                                     |            |Confidence: 90%                                     |            |Confidence: 90%                                     |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |.1** Certification of Chief Executive Officer p...  |408362         |[-0.01204388216137886, -0.008523665368556976, -...  |494                      |3636232   | Factoid question: What type of document is the...  |[                                                   |\" Factoid question: What type of document is th...  |\"Answer: The Inline XBRL Instance Document is a...  | Evaluation: The context provides information a...  |3           | Evaluation: This question is seeking to unders...  |5           | Evaluation: This question is seeking to unders...  |5           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: The Inline XBRL Instance Document is an...  |  \" Factoid question: What type of document is ...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: The Inline XBRL Instance Document is...  |                                                    |                                                    |Total rating: 3                                     |            |Total rating: 5                                     |            |Total rating: 5                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |Explanation: While the context does provide som...  |            |Reference(s):                                       |            |Reference(s):                                       |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |- Inline XBRL (eXtensible Business Reporting La...  |            |- Inline XBRL (eXtensible Business Reporting La...  |            |\n",
      "|0001874178-23-000009_10-K  |10-K Filing Text  |RIVIAN AUTOMOTIVE, INC. / DE  |Office of Manufacturing  |MOTOR VEHICLES & PASSENGER CAR BODIES  |US         |2022-12-31         |. We market our ability to provide our customer...  |110525         |[-0.010174417868256569, -0.024404698982834816, ...  |136                      |3635874   | Factoid question: What are some of Rivian's ch...  |[                                                   |\" Factoid question: What are some of Rivian's c...  |\"Answer: Rivian offers comprehensive charging s...  | Evaluation: The context clearly states that Ri...  |5           | Evaluation: This question is useful for analys...  |4           | Evaluation: This question is useful for analys...  |4           |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |Answer: Rivian offers comprehensive charging so...  |  \" Factoid question: What are some of Rivian's...  |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |  \"Answer: Rivian offers comprehensive charging...  |                                                    |                                                    |Total rating: 5                                     |            |Total rating: 4                                     |            |Total rating: 4                                     |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |]                                                   |                                                    |                                                    |                                                    |            |                                                    |            |                                                    |            |\n",
      "|                           |                  |                              |                         |                                       |           |                   |                                                    |               |                                                    |                         |          |                                                    |                                                    |                                                    |                                                    |                                                    |            |Note: While this question is useful, it is limi...  |            |Note: While this question is useful, it is limi...  |            |\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "outdf = outdf.withColumn(\n",
    "    \"EVAL_CONTEXT\",\n",
    "    Complete(\n",
    "        model='mixtral-8x7b',prompt = f.concat(\n",
    "            f.lit(question_relevance_critique_prompt),\n",
    "            f.col(\"GENQ\"),\n",
    "            f.lit(\"\\n Answer:::\")\n",
    "            ))\n",
    ")\n",
    "\n",
    "# Extracting the score by taking the rightmost character of the last part of split_rating\n",
    "outdf = outdf.withColumn(\"score_EC\", regexp_extract(col(\"EVAL_CONTEXT\"), r\"Total rating:\\s*(\\d)\", 1))\n",
    "\n",
    "# Show results to verify\n",
    "outdf.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "outdf.write.saveAsTable(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA_All\", mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's keep only the questions that have scored well, by filtering on the score. This part may error on you if the scoring wasn't done correctly. I need to rework this scoring to be more robust. Sometimes the models don't output a number and this affects the following filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28\n"
     ]
    }
   ],
   "source": [
    "# Assuming 'outdf' is your DataFrame\n",
    "filtered_df = outdf.filter(\n",
    "    col(\"SCORE_EC\").isin([\"4\", \"5\"]) &\n",
    "    col(\"SCORE_ER\").isin([\"4\", \"5\"]) &\n",
    "    col(\"SCORE_EG\").isin([\"4\", \"5\"])\n",
    ")\n",
    "\n",
    "# To count the number of rows in the filtered DataFrame\n",
    "row_count = filtered_df.count()\n",
    "print(row_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "selected_df = filtered_df.select(\"CONTENT_CHUNK\", \"GENQ\", \"GENA\")\n",
    "# Now, to save this DataFrame as a new table in Snowflake\n",
    "selected_df.write.mode(\"overwrite\").saveAsTable(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Get Predictions for Synthetic Questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "from snowflake.snowpark.functions import col\n",
    "import snowflake.snowpark.functions as f\n",
    "from snowflake.snowpark.functions import col, split, lit, regexp_extract\n",
    "from snowflake.cortex import Complete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get predictions for the snythetic questions we have already created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CONTENT_CHUNK</th>\n",
       "      <th>GENQ</th>\n",
       "      <th>GENA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rivn-20221231 UNITED STATES SECURITIES AND EXC...</td>\n",
       "      <td>\" Factoid question: What is the trading symbol...</td>\n",
       "      <td>\"Answer: The trading symbol for Rivian Automot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>. We do not have any control over these analys...</td>\n",
       "      <td>\" Factoid question: Why might demand for Rivia...</td>\n",
       "      <td>\"Answer: Demand for Rivian Automotive's Class ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>. Our vehicles use lithium-ion battery cells, ...</td>\n",
       "      <td>\" Factoid question: What type of battery cells...</td>\n",
       "      <td>\"Answer: Our vehicles use lithium-ion battery ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>. Such damage or injury could also lead to adv...</td>\n",
       "      <td>\" Factoid question: What could be the potentia...</td>\n",
       "      <td>\"Answer: The business, prospects, financial co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>. We use our Investor Relations website as a m...</td>\n",
       "      <td>\" Factoid question: What is one way the compan...</td>\n",
       "      <td>\"Answer: The company uses its Investor Relatio...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       CONTENT_CHUNK  \\\n",
       "0  rivn-20221231 UNITED STATES SECURITIES AND EXC...   \n",
       "1  . We do not have any control over these analys...   \n",
       "2  . Our vehicles use lithium-ion battery cells, ...   \n",
       "3  . Such damage or injury could also lead to adv...   \n",
       "4  . We use our Investor Relations website as a m...   \n",
       "\n",
       "                                                GENQ  \\\n",
       "0  \" Factoid question: What is the trading symbol...   \n",
       "1  \" Factoid question: Why might demand for Rivia...   \n",
       "2  \" Factoid question: What type of battery cells...   \n",
       "3  \" Factoid question: What could be the potentia...   \n",
       "4  \" Factoid question: What is one way the compan...   \n",
       "\n",
       "                                                GENA  \n",
       "0  \"Answer: The trading symbol for Rivian Automot...  \n",
       "1  \"Answer: Demand for Rivian Automotive's Class ...  \n",
       "2  \"Answer: Our vehicles use lithium-ion battery ...  \n",
       "3  \"Answer: The business, prospects, financial co...  \n",
       "4  \"Answer: The company uses its Investor Relatio...  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_qa = session.table(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\")\n",
    "df_gen_qa = gen_qa.toPandas()\n",
    "df_gen_qa.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "def get_response(session, question):\n",
    "    # Prepare the SQL command\n",
    "    sql_command = f\"\"\"\n",
    "    WITH question AS (\n",
    "        SELECT '{question.replace(\"'\", \"''\")}' AS question,\n",
    "               snowflake.cortex.embed_text('e5-base-v2', question) AS embedding\n",
    "    ),\n",
    "    context AS (\n",
    "        SELECT content_chunk AS context,\n",
    "               sec_document_id,\n",
    "               start_index AS document_index\n",
    "        FROM RAJIV.PUBLIC.CONTENT_CHUNKS_10K AS content,\n",
    "             question\n",
    "        WHERE company_name = 'RIVIAN AUTOMOTIVE, INC. / DE'\n",
    "        ORDER BY vector_cosine_distance(content.embedding, question.embedding) DESC\n",
    "        LIMIT 1\n",
    "    ),\n",
    "    response AS (\n",
    "        SELECT snowflake.cortex.complete('mistral-7b',\n",
    "                                         'Use only the context provided to answer the question.' ||\n",
    "                                         '### ' || 'Context: ' || context.context ||\n",
    "                                         '### ' || 'Question: ' || question.question ||\n",
    "                                         'Answer: ') AS response\n",
    "        FROM context, question\n",
    "    )\n",
    "    SELECT response FROM response\n",
    "    \"\"\"\n",
    "\n",
    "    #print(\"Executing SQL command:\", sql_command)  # Log the SQL command\n",
    "\n",
    "    # Execute the SQL command\n",
    "    result = session.sql(sql_command).collect()\n",
    "\n",
    "    if result:\n",
    "        return result[0][0]\n",
    "    else:\n",
    "        return \"No suitable answer found.\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                        CONTENT_CHUNK  \\\n",
      "0   rivn-20221231 UNITED STATES SECURITIES AND EXC...   \n",
      "1   . We do not have any control over these analys...   \n",
      "2   . Our vehicles use lithium-ion battery cells, ...   \n",
      "3   . Such damage or injury could also lead to adv...   \n",
      "4   . We use our Investor Relations website as a m...   \n",
      "5   . The grant-date fair value of stock options g...   \n",
      "6   . Any reduction, elimination, or discriminator...   \n",
      "7   . We market our ability to provide our custome...   \n",
      "8   . We primarily rely on Amazon Web Services in ...   \n",
      "9   . In addition, our independent registered publ...   \n",
      "10  . Errors in the design, implementation or exec...   \n",
      "11  . The Company is subject to taxation and files...   \n",
      "12  . Item 9B. Other Information None. Item 9C. Di...   \n",
      "13  . For example, we will not be able to utilize ...   \n",
      "14  . As our vehicles are produced, we will need t...   \n",
      "15  . These services will generate long-term brand...   \n",
      "16  . In 2021, changes to the SAFE Vehicles rule w...   \n",
      "17  .   Our executive officers, directors, and pri...   \n",
      "18  . Environmental, health and safety laws, and r...   \n",
      "19  . The Court denied the petition by Order dated...   \n",
      "20  . In designing and evaluating the disclosure c...   \n",
      "21  . We have incurred net losses since our incept...   \n",
      "22  . Threats to networks and information technolo...   \n",
      "23  . We are subject to anti-corruption, anti-brib...   \n",
      "24  . Stock options granted under the 2015 Stock P...   \n",
      "25  .0   % $ 1,250   11.3   % Total long-term debt...   \n",
      "26  . Therefore, the automated and manual process ...   \n",
      "27  . We have limited experience in the actual pro...   \n",
      "\n",
      "                                                 GENQ  \\\n",
      "0   \" Factoid question: What is the trading symbol...   \n",
      "1   \" Factoid question: Why might demand for Rivia...   \n",
      "2   \" Factoid question: What type of battery cells...   \n",
      "3   \" Factoid question: What could be the potentia...   \n",
      "4   \" Factoid question: What is one way the compan...   \n",
      "5   \" Factoid question: How are the grant-date fai...   \n",
      "6   \" Factoid question: Why are certain hybrids an...   \n",
      "7   \" Factoid question: What are some of Rivian's ...   \n",
      "8   \" Factoid question: Who provides cloud computi...   \n",
      "9   \" Factoid question: What was the opinion of th...   \n",
      "10  \" Factoid question: What are some potential ri...   \n",
      "11  \" Factoid question: When did the 2021 Incentiv...   \n",
      "12  \" Factoid question: When will the information ...   \n",
      "13  \" Factoid question: What could affect a manufa...   \n",
      "14  \" Factoid question: How can inadequate warrant...   \n",
      "15  \" Factoid question: What connection do Rivian'...   \n",
      "16  \" Factoid question: When were fuel economy sta...   \n",
      "17  \" Factoid question: Who holds voting power rep...   \n",
      "18  \" Factoid question: What are the potential con...   \n",
      "19  \" Factoid question: When did five individuals ...   \n",
      "20  \" Factoid question: Were the Company's disclos...   \n",
      "21  \" Factoid question: How much was the net loss ...   \n",
      "22  \" Factoid question: Who are some of the entiti...   \n",
      "23  \" Factoid question: What is the U.S. law that ...   \n",
      "24  \" Factoid question: When did stock options and...   \n",
      "25  \" Factoid question: When did the Company pay o...   \n",
      "26  \" Factoid question: Did KPMG LLP express an ad...   \n",
      "27  \" Factoid question: What are some challenges R...   \n",
      "\n",
      "                                                 GENA  \\\n",
      "0   \"Answer: The trading symbol for Rivian Automot...   \n",
      "1   \"Answer: Demand for Rivian Automotive's Class ...   \n",
      "2   \"Answer: Our vehicles use lithium-ion battery ...   \n",
      "3   \"Answer: The business, prospects, financial co...   \n",
      "4   \"Answer: The company uses its Investor Relatio...   \n",
      "5   \"Answer: The grant-date fair value of RSUs gra...   \n",
      "6   \"Answer: This is to support transportation inf...   \n",
      "7   \"Answer: Rivian offers comprehensive charging ...   \n",
      "8   \"Answer: Amazon Web Services primarily provide...   \n",
      "9   \"Answer: The independent registered public acc...   \n",
      "10  \"Answer: Errors in design, implementation, or ...   \n",
      "11  \"Answer: The 2021 Incentive Award Plan became ...   \n",
      "12  \"Answer: The information will be included in t...   \n",
      "13  \"Answer: Regulatory limitations could affect a...   \n",
      "14  \"Answer: Inadequate warranty reserves can mate...   \n",
      "15  \"Answer: Rivian's vehicles use the Combined Ch...   \n",
      "16  \"Answer: Fuel economy standards were last made...   \n",
      "17  \"Answer: Our executive officers, directors, an...   \n",
      "18  \"Answer: Failures to comply with environmental...   \n",
      "19                     \"Answer: On October 21, 2022.\"   \n",
      "20  \"Answer: No, the Company's disclosure controls...   \n",
      "21  \"Answer: The net loss for the company in 2022 ...   \n",
      "22  \"Answer: Traditional computer \\\"hackers,\\\" mal...   \n",
      "23  \"Answer: The U.S. Foreign Corrupt Practices Ac...   \n",
      "24  \"Answer: Stock options and RSUs granted under ...   \n",
      "25  \"Answer: The Company paid all outstanding amou...   \n",
      "26  \"Answer: Yes, according to the context, KPMG L...   \n",
      "27  \"Answer: Challenges include charging station p...   \n",
      "\n",
      "                                             RESPONSE  \n",
      "0                                                RIVN  \n",
      "1    Demand for Rivian Automotive's Class A common...  \n",
      "2    The vehicles use lithium-ion battery cells. I...  \n",
      "3    The potential consequences for the business i...  \n",
      "4    The company discloses material information to...  \n",
      "5    The grant-date fair value of Restricted Stock...  \n",
      "6    Certain hybrids and EVs are subject to additi...  \n",
      "7    Rivian offers comprehensive charging solution...  \n",
      "8    Amazon Web Services (AWS) provides cloud comp...  \n",
      "9    The independent registered public accounting ...  \n",
      "10   Advanced driver assistance technologies come ...  \n",
      "11   The context does not provide information on w...  \n",
      "12   The information about Rivian Automotive Inc.'...  \n",
      "13   A manufacturer's ability to sell vehicles dir...  \n",
      "14   Inadequate warranty reserves can have a mater...  \n",
      "15   Rivian's vehicles can be charged using variou...  \n",
      "16   The fuel economy standards were made more str...  \n",
      "17   The executive officers, directors, and stockh...  \n",
      "18   Non-compliance with environmental, health and...  \n",
      "19   Five individuals filed a Petition for Writ of...  \n",
      "20   No, the Company's disclosure controls and pro...  \n",
      "21   The net loss for the company in 2022 was $6.8...  \n",
      "22   The entities that pose threats to information...  \n",
      "23   The U.S. law that prohibits offering, promisi...  \n",
      "24   Stock options and RSUs granted under the 2015...  \n",
      "25   The Company paid all outstanding amounts rela...  \n",
      "26   Yes, according to the context provided, KPMG ...  \n",
      "27   Rivian may face several challenges in providi...  \n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Assume 'session' is your Snowflake session object and 'df_gen_qa' is your pandas DataFrame\n",
    "df_gen_qa['RESPONSE'] = df_gen_qa['GENQ'].apply(lambda question: get_response(session, question))\n",
    "\n",
    "# Now df_gen_qa has an additional column 'Response' with the answers\n",
    "print(df_gen_qa)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CONTENT_CHUNK</th>\n",
       "      <th>GENQ</th>\n",
       "      <th>GENA</th>\n",
       "      <th>RESPONSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>rivn-20221231 UNITED STATES SECURITIES AND EXC...</td>\n",
       "      <td>\" Factoid question: What is the trading symbol...</td>\n",
       "      <td>\"Answer: The trading symbol for Rivian Automot...</td>\n",
       "      <td>RIVN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>. We do not have any control over these analys...</td>\n",
       "      <td>\" Factoid question: Why might demand for Rivia...</td>\n",
       "      <td>\"Answer: Demand for Rivian Automotive's Class ...</td>\n",
       "      <td>Demand for Rivian Automotive's Class A common...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>. Our vehicles use lithium-ion battery cells, ...</td>\n",
       "      <td>\" Factoid question: What type of battery cells...</td>\n",
       "      <td>\"Answer: Our vehicles use lithium-ion battery ...</td>\n",
       "      <td>The vehicles use lithium-ion battery cells. I...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>. Such damage or injury could also lead to adv...</td>\n",
       "      <td>\" Factoid question: What could be the potentia...</td>\n",
       "      <td>\"Answer: The business, prospects, financial co...</td>\n",
       "      <td>The potential consequences for the business i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>. We use our Investor Relations website as a m...</td>\n",
       "      <td>\" Factoid question: What is one way the compan...</td>\n",
       "      <td>\"Answer: The company uses its Investor Relatio...</td>\n",
       "      <td>The company discloses material information to...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       CONTENT_CHUNK  \\\n",
       "0  rivn-20221231 UNITED STATES SECURITIES AND EXC...   \n",
       "1  . We do not have any control over these analys...   \n",
       "2  . Our vehicles use lithium-ion battery cells, ...   \n",
       "3  . Such damage or injury could also lead to adv...   \n",
       "4  . We use our Investor Relations website as a m...   \n",
       "\n",
       "                                                GENQ  \\\n",
       "0  \" Factoid question: What is the trading symbol...   \n",
       "1  \" Factoid question: Why might demand for Rivia...   \n",
       "2  \" Factoid question: What type of battery cells...   \n",
       "3  \" Factoid question: What could be the potentia...   \n",
       "4  \" Factoid question: What is one way the compan...   \n",
       "\n",
       "                                                GENA  \\\n",
       "0  \"Answer: The trading symbol for Rivian Automot...   \n",
       "1  \"Answer: Demand for Rivian Automotive's Class ...   \n",
       "2  \"Answer: Our vehicles use lithium-ion battery ...   \n",
       "3  \"Answer: The business, prospects, financial co...   \n",
       "4  \"Answer: The company uses its Investor Relatio...   \n",
       "\n",
       "                                            RESPONSE  \n",
       "0                                               RIVN  \n",
       "1   Demand for Rivian Automotive's Class A common...  \n",
       "2   The vehicles use lithium-ion battery cells. I...  \n",
       "3   The potential consequences for the business i...  \n",
       "4   The company discloses material information to...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gen_qa.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "snowpark_df = session.createDataFrame(df_gen_qa)\n",
    "snowpark_df.write.saveAsTable(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\", mode=\"overwrite\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluate Performance on Synthetic Questions\n",
    "\n",
    "Let's now start to analyze the results. In this notebooks, I use one criteria, but three are many more criteria you can use to evaluate the performance of the RAG model. For example, [Ragas](https://docs.ragas.io/en/stable/getstarted/evaluation.html#get-started-evaluation) does provides several metrics to evaluate various aspects of your RAG systems:\n",
    "\n",
    "Retriever: Offers context_precision and context_recall that measure the performance of your retrieval system.\n",
    "\n",
    "Generator (LLM): Provides faithfulness that measures hallucinations and answer_relevancy that measures how relevant the answers are to the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"GENQ\"                                              |\"GENA\"                                              |\"RESPONSE\"                                          |\n",
      "----------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\" Factoid question: What is the trading symbol ...  |\"Answer: The trading symbol for Rivian Automoti...  | RIVN                                               |\n",
      "|\" Factoid question: Why might demand for Rivian...  |\"Answer: Demand for Rivian Automotive's Class A...  | Demand for Rivian Automotive's Class A common ...  |\n",
      "|\" Factoid question: What type of battery cells ...  |\"Answer: Our vehicles use lithium-ion battery c...  | The vehicles use lithium-ion battery cells. If...  |\n",
      "|\" Factoid question: What could be the potential...  |\"Answer: The business, prospects, financial con...  | The potential consequences for the business if...  |\n",
      "|\" Factoid question: What is one way the company...  |\"Answer: The company uses its Investor Relation...  | The company discloses material information to ...  |\n",
      "|\" Factoid question: How are the grant-date fair...  |\"Answer: The grant-date fair value of RSUs gran...  | The grant-date fair value of Restricted Stock ...  |\n",
      "|\" Factoid question: Why are certain hybrids and...  |\"Answer: This is to support transportation infr...  | Certain hybrids and EVs are subject to additio...  |\n",
      "|\" Factoid question: What are some of Rivian's c...  |\"Answer: Rivian offers comprehensive charging s...  | Rivian offers comprehensive charging solutions...  |\n",
      "|\" Factoid question: Who provides cloud computin...  |\"Answer: Amazon Web Services primarily provides...  | Amazon Web Services (AWS) provides cloud compu...  |\n",
      "|\" Factoid question: What was the opinion of the...  |\"Answer: The independent registered public acco...  | The independent registered public accounting f...  |\n",
      "----------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "snowpark_df = session.table(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\").select(\"GENQ\", \"GENA\", \"RESPONSE\")\n",
    "snowpark_df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "EVALUATION_PROMPT = \"\"\"###Task Description:\n",
    "An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n",
    "1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n",
    "2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n",
    "3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n",
    "4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n",
    "\n",
    "###The instruction to evaluate:\n",
    "{instruction}\n",
    "\n",
    "###Response to evaluate:\n",
    "{response}\n",
    "\n",
    "###Reference Answer (Score 5):\n",
    "{reference_answer}\n",
    "\n",
    "###Score Rubrics:\n",
    "[Is the response correct, accurate, and factual based on the reference answer?]\n",
    "Score 1: The response is completely incorrect, inaccurate, and/or not factual.\n",
    "Score 2: The response is mostly incorrect, inaccurate, and/or not factual.\n",
    "Score 3: The response is somewhat correct, accurate, and/or factual.\n",
    "Score 4: The response is mostly correct, accurate, and factual.\n",
    "Score 5: The response is completely correct, accurate, and factual.\n",
    "\n",
    "Answer:::\n",
    "Feedback: (your feedback for the rating, as a text)\n",
    "Total score: (your score, as an integer between 1 and 5)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "part0 = \"\"\"###Task Description:\n",
    "An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n",
    "1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n",
    "2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n",
    "3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n",
    "4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n",
    "\n",
    "###The instruction to evaluate: \"\"\"\n",
    "part1 = \"\"\"###Response to evaluate: \"\"\"\n",
    "\n",
    "part2 = \"\"\"###Reference Answer (Score 5): \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "part3 = \"\"\"###Score Rubrics:\n",
    "[Is the response correct, accurate, and factual based on the reference answer?]\n",
    "Score 1: The response is completely incorrect, inaccurate, and/or not factual.\n",
    "Score 2: The response is mostly incorrect, inaccurate, and/or not factual.\n",
    "Score 3: The response is somewhat correct, accurate, and/or factual.\n",
    "Score 4: The response is mostly correct, accurate, and factual.\n",
    "Score 5: The response is completely correct, accurate, and factual.\n",
    "\n",
    "Answer:::\n",
    "Feedback: (your feedback for the rating, as a text)\n",
    "Total score: (an integer between 1 and 5)\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\"GENQ\"                                              |\"GENA\"                                              |\"RESPONSE\"                                          |\"EVAL_CRITIQUE\"                                     |\"SCORE_EVAL\"  |\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "|\" Factoid question: What is the trading symbol ...  |\"Answer: The trading symbol for Rivian Automoti...  | RIVN                                               |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: Why might demand for Rivian...  |\"Answer: Demand for Rivian Automotive's Class A...  | Demand for Rivian Automotive's Class A common ...  |                                                    |4             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is mostly correct, accur...  |              |\n",
      "|                                                    |                                                    |                                                    |[RESULT] Total score: 4                             |              |\n",
      "|\" Factoid question: What type of battery cells ...  |\"Answer: Our vehicles use lithium-ion battery c...  | The vehicles use lithium-ion battery cells. If...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: What could be the potential...  |\"Answer: The business, prospects, financial con...  | The potential consequences for the business if...  |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is mostly correct, accur...  |              |\n",
      "|                                                    |                                                    |                                                    |[RESULT] 4                                          |              |\n",
      "|\" Factoid question: What is one way the company...  |\"Answer: The company uses its Investor Relation...  | The company discloses material information to ...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: How are the grant-date fair...  |\"Answer: The grant-date fair value of RSUs gran...  | The grant-date fair value of Restricted Stock ...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: Why are certain hybrids and...  |\"Answer: This is to support transportation infr...  | Certain hybrids and EVs are subject to additio...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: What are some of Rivian's c...  |\"Answer: Rivian offers comprehensive charging s...  | Rivian offers comprehensive charging solutions...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: Who provides cloud computin...  |\"Answer: Amazon Web Services primarily provides...  | Amazon Web Services (AWS) provides cloud compu...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "|\" Factoid question: What was the opinion of the...  |\"Answer: The independent registered public acco...  | The independent registered public accounting f...  |                                                    |5             |\n",
      "|                                                    |                                                    |                                                    |                                                    |              |\n",
      "|                                                    |                                                    |                                                    |Feedback: The response is completely correct, a...  |              |\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from snowflake.snowpark.functions import col, regexp_extract, coalesce, lit\n",
    "\n",
    "snowpark_df = snowpark_df.withColumn(\n",
    "    \"EVAL_CRITIQUE\",\n",
    "    Complete(\n",
    "        model='mistral-large',prompt = f.concat(\n",
    "            f.lit(part0),\n",
    "            f.col(\"GENQ\"),\n",
    "            f.lit(part1),\n",
    "            f.col(\"RESPONSE\"),\n",
    "            f.lit(part2),\n",
    "            f.col(\"GENA\"),\n",
    "            f.lit(part3)\n",
    "            ))\n",
    ")\n",
    "\n",
    "# Extracting the score by taking the rightmost character of the last part of split_rating\n",
    "#snowpark_df = snowpark_df.withColumn(\"SCORE_EVAL\", regexp_extract(col(\"EVAL_CRITIQUE\"), r\"Total score:\\s*(\\d)\", 1).cast(\"int\"))\n",
    "snowpark_df = snowpark_df.withColumn(\"SCORE_EVAL\",\n",
    "    coalesce(regexp_extract(col(\"EVAL_CRITIQUE\"), r\"Total score:\\s*(\\d)\", 1), lit(\"0\")))\n",
    "\n",
    "snowpark_df.write.mode(\"overwrite\").save_as_table(\"RAJIV.PUBLIC.CONTENT_CHUNKS_10K_GQA\")\n",
    "# Show results to verify\n",
    "snowpark_df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pulling into pandas for the final calculations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "output = snowpark_df.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "output['SCORE_EVAL'] = pd.to_numeric(output['SCORE_EVAL'], errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    17.000000\n",
      "mean      4.823529\n",
      "std       0.727607\n",
      "min       2.000000\n",
      "25%       5.000000\n",
      "50%       5.000000\n",
      "75%       5.000000\n",
      "max       5.000000\n",
      "Name: SCORE_EVAL, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "summary_stats = output['SCORE_EVAL'].describe()\n",
    "print(summary_stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAHHCAYAAABKudlQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1qElEQVR4nO3de1xUdeL/8fcIOKACCorKCmopmqLkLWs1lQ1TNFO7eMkL0n3XS6aWy7pqpIZamm26Wm2JXdQy08r1mnnJylLM7LbeMsHEawaCOSKc3x/9mG8TV0fgzKHX8/GYx3Y+8zln3jO1+e5zzpmxGYZhCAAAwKKqmB0AAADgalBmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmgKvUqFEjjRgxwuwYld7TTz+ta665Rl5eXrr++uvNjgPAg1BmgN9ITk6WzWbT7t27C32+W7duioyMvOrXWbt2rZ544omrPs4fxcaNG/X444+rU6dOWrx4sZ566qli57///vvq2rWrQkJCVK1aNV1zzTUaMGCA1q9fX2BuZmamEhMTFRUVpRo1asjPz0+RkZGaOHGijh8/XmD+mjVr1LNnTwUHB8vX11cRERGaMGGCzp49W2DuiBEjZLPZnA+73a6IiAhNmTJFFy9eLDD/t3N//3j44YdL/Xlt3bq12GMtX75ce/bskc1m0z//+c8ij3Pw4EHZbDaNGzfOZfzxxx+XzWbTwIEDC93vhx9+kM1m0zPPPFPqzMDV8DY7AGB1+/fvV5UqV/bfBWvXrtWCBQsoNKX04YcfqkqVKnr55ZdVtWrVYuc+88wzeuyxx9S1a1clJCSoWrVqOnTokD744AMtX75cPXv2dM79/vvvFRMTo9TUVN1999168MEHVbVqVe3bt08vv/yyVq1apQMHDjjnT5gwQXPmzFFUVJQmTpyooKAg7dmzR/Pnz9fy5cu1efNmNWvWzCWP3W7Xf/7zH0lSRkaG3n33XU2bNk2HDx/WG2+8USB/9+7dNXz48ALjERERV/SZSdKYMWPUoUOHAuM33XSTGjZsqObNm2vZsmWaPn16ofsvXbpUkjR06FDnmGEYWrZsmRo1aqT3339f58+fl7+//xVnA8qUAcBp8eLFhiRj165dhT7ftWtXo2XLllf9OiNHjjTM/L9fVlaWaa/tjvj4eKN69eolzsvJyTECAgKM7t27F/r8yZMnXeZGRUUZ1apVMz766KMCczMyMox//OMfzu2lS5cakoyBAwcaly9fdpn72WefGdWqVTNatWpl5OTkOMfj4uIK5M7LyzNuvPFGw2azGSdOnHB5TpIxcuTIEt9nSbZs2WJIMlasWFHsvGnTphmSjE8//bTQ55s1a2Y0b97cZezDDz80JBkffvih4ePjYyQnJxfY78iRI4Yk4+mnn3b/TQBXgNNMwFX6/TUzOTk5SkxMVNOmTeXr66vg4GB17txZmzZtkvTrqYcFCxZIcj2tkC87O1vjx49XWFiY7Ha7mjVrpmeeeUbG737g/pdfftGYMWNUu3Zt+fv76/bbb9ePP/4om83msuLzxBNPyGaz6dtvv9U999yjWrVqqXPnzpKkffv2acSIEbrmmmvk6+urevXq6d577y1wyiT/GAcOHNDQoUMVGBioOnXqaPLkyTIMQ2lpaerbt68CAgJUr149zZkzp1Sf3eXLlzVt2jRde+21stvtatSokf7xj3/I4XA459hsNi1evFjZ2dnOzyo5ObnQ4505c0aZmZnq1KlToc+HhIQ4/3rlypX68ssvNWnSJOfn8VsBAQGaMWOGczsxMVG1atXSiy++KC8vL5e5N9xwgyZOnKivvvpKb7/9drHv2WazqXPnzjIMQ99//32xc8vbkCFDJP3fCsxvpaSkaP/+/c45+d544w21aNFC0dHRiomJKXR1CaholBmgEBkZGTpz5kyBR05OTon7PvHEE0pMTFR0dLTmz5+vSZMmKTw8XHv27JEkPfTQQ+revbsk6bXXXnM+pF+X8G+//XY9++yz6tmzp+bOnatmzZrpscceK3DdwogRI/T888+rV69emjVrlvz8/NS7d+8ic9199926cOGCnnrqKT3wwAOSpE2bNun7779XfHy8nn/+eQ0aNEjLly9Xr169CpQnSRo4cKDy8vI0c+ZMdezYUdOnT9e8efPUvXt3/elPf9KsWbPUpEkTTZgwQdu3by/xs7r//vs1ZcoUtW3bVs8++6y6du2qpKQkDRo0yDnntdde08033yy73e78rLp06VLo8UJCQuTn56f3339fP/30U7Gv/d5770mShg0bVmLOgwcPav/+/c7CVpj8U0Nr1qwp8Xg//PCDJKlWrVoFnrt48WKh/+xdunSpxOP+3vnz5ws9Vv7f28aNG+vPf/6z3nrrLeXm5rrsm19w7rnnHueYw+HQypUrNXjwYEnS4MGD9eGHH+rEiRNXnA0oU2YuCwGeJv80U3GP359matiwoREXF+fcjoqKMnr37l3s6xR1mmn16tWGJGP69Oku43fddZdhs9mMQ4cOGYZhGCkpKYYkY+zYsS7zRowYYUgypk6d6hybOnWqIckYPHhwgde7cOFCgbFly5YZkozt27cXOMaDDz7oHLt8+bLRoEEDw2azGTNnznSOnzt3zvDz83P5TAqzd+9eQ5Jx//33u4xPmDDBeRojX2Gna4oyZcoUQ5JRvXp1IzY21pgxY4aRkpJSYF6bNm2MwMDAUh0z/+/Ls88+W+y8gIAAo23btgVynz592jh9+rRx6NAh45lnnjFsNpsRGRlp5OXluexf3D93y5YtK1VWw/i/00xFPdLT051zFyxYYEgyNmzY4BzLzc01/vSnPxk33XSTy3HffvttQ5Jx8OBBwzAMIzMz0/D19S3wuXCaCRWNlRmgEAsWLNCmTZsKPFq3bl3ivjVr1tQ333yjgwcPXvHrrl27Vl5eXhozZozL+Pjx42UYhtatWydJzrty/va3v7nMGz16dJHHLuxuGD8/P+df568I3HjjjZLkXEn6rfvvv9/5115eXmrfvr0Mw9B9993nHK9Zs6aaNWtW4imUtWvXSlKBFafx48dLkv773/8Wu39REhMTtXTpUrVp00YbNmzQpEmT1K5dO7Vt21bfffedc15mZmapL1w9f/68JJU439/fX5mZmS5j2dnZqlOnjurUqeNcterUqZPeffddl9OL+fr27VvoP3vR0dGlyvpbU6ZMKfRYQUFBzjkDBw6Uj4+Py6mmbdu26ccffyz0FFP79u3VpEkT5/vt3bs3p5pgOu5mAgpxww03qH379gXGa9WqpTNnzhS775NPPqm+ffsqIiJCkZGR6tmzp4YNG1aqInT06FGFhoYW+EPzuuuucz6f/79VqlRR48aNXebl/yFTmN/PlaSffvpJiYmJWr58uU6dOuXyXEZGRoH54eHhLtuBgYHy9fVV7dq1C4wXdqvyb+W/h99nrlevnmrWrOl8r+4YPHiwBg8erMzMTH322WdKTk7W0qVL1adPH3399dfy9fVVQEBAqa9Zyf/7kV9qinL+/HmX63IkydfXV++//74k6dixY5o9e7ZOnTrlUiR/q0GDBoqJiSlVrpK0atWqxGMFBwerR48eWrVqlRYtWiRfX18tXbpU3t7eGjBggHPezz//rLVr12rUqFE6dOiQc7xTp05auXKlDhw44NYdV0BZYGUGKGNdunTR4cOH9corrygyMlL/+c9/1LZtW+ftuWYp7A/PAQMG6KWXXtLDDz+sd955Rxs3bnSu+uTl5RWY//sLX4sak1ToNTeFKWx1oqwEBASoe/fueuONNxQXF6fDhw/rs88+kyQ1b95cGRkZSktLK/E4+WVy3759Rc45evSoMjMz1aJFC5dxLy8vxcTEKCYmRiNGjNDmzZt14sQJPfTQQ1fxzsrW0KFDlZmZqTVr1ujSpUtauXKlbr31VtWpU8c5Z8WKFXI4HJozZ46aNm3qfOSvrLE6AzNRZoByEBQUpPj4eC1btkxpaWlq3bq1yx1GRf0B3rBhQx0/frzACsD//vc/5/P5/5uXl6cjR464zPvtfzGX5Ny5c9q8ebP+/ve/KzExUf3791f37t11zTXXlPoYVyP/Pfz+dNzJkyf1888/O99rWclfaUtPT5ck9enTR5L0+uuvl7hvRESEIiIitHr16iJXZ1599VVJ0m233VbsserXr69HH31U77//vnbu3Fnq/OXp9ttvl7+/v5YuXap169bp3LlzhZ5iioyM1IoVKwo8YmJiCr0jCqgolBmgjP3+9EqNGjXUpEkTl9uNq1evLunXpfvf6tWrl3JzczV//nyX8WeffVY2m02xsbGSpB49ekiS/v3vf7vMe/7550udM39F5fcrKPPmzSv1Ma5Gr169Cn29uXPnSlKxd2YV5cKFC/r0008LfS7/eqP8L7W766671KpVK82YMaPQfc6fP69JkyY5t6dMmaJz587p4YcfLnDnT0pKimbNmqXIyEjdeeedJeYcPXq0qlWrppkzZ5b6vZUnPz8/9e/fX2vXrtXChQtVvXp19e3b1/l8Wlqatm/frgEDBuiuu+4q8IiPj9ehQ4ecq15AReOaGaCMtWjRQt26dVO7du0UFBSk3bt36+2339aoUaOcc9q1ayfp129o7dGjh7y8vDRo0CD16dNH0dHRmjRpkn744QdFRUVp48aNevfddzV27Fhde+21zv3vvPNOzZs3T2fPntWNN96obdu2Ob+ttjSnbgICAtSlSxfNnj1bOTk5+tOf/qSNGzcWWO0pL1FRUYqLi9OLL76on3/+WV27dtXnn3+uJUuWqF+/fm5d8HrhwgX9+c9/1o033qiePXsqLCxMP//8s1avXq2PPvpI/fr1U5s2bSRJPj4+eueddxQTE6MuXbpowIAB6tSpk3x8fPTNN99o6dKlqlWrlvO7ZoYMGaJdu3bpueee07fffqshQ4aoVq1a2rNnj1555RUFBwfr7bfflo+PT4k5g4ODFR8fr3//+9/67rvvnKexJOnAgQOFrhbVrVvXeUt/aX300UeF/mxC69atC1zDNXToUL366qvasGGDhgwZ4izc0q+3aRv//2sDCtOrVy95e3vrjTfeUMeOHZ3jmzdvLvT1+/XrVyY/CwI4mXovFeBh3PkG4N/fmj19+nTjhhtuMGrWrGn4+fkZzZs3N2bMmGFcunTJOefy5cvG6NGjjTp16hg2m83lNu3z588bjz76qBEaGmr4+PgYTZs2NZ5++ukCt/FmZ2cbI0eONIKCgowaNWoY/fr1M/bv329IcrlVOv+26tOnTxd4P8eOHTP69+9v1KxZ0wgMDDTuvvtu4/jx40Xe3v37YxR1y3Rpvyk5JyfHSExMNBo3bmz4+PgYYWFhRkJCgnHx4sVSvU5hx3vppZeMfv36GQ0bNjTsdrtRrVo1o02bNsbTTz9tOByOAvucO3fOmDJlitGqVSujWrVqhq+vrxEZGWkkJCS43MKcb/Xq1Ub37t2NWrVqGXa73WjSpIkxfvz4Qj/f4nIfPnzY8PLycvlnR8XcTt21a9cS33++km7N/u3f23yXL1826tevb0gy1q5d6/Jcq1atjPDw8GJfs1u3bkZISIiRk5PjvDW7qMdrr71W6vcClIbNMEp5lR4Aj7d37161adNGr7/+eoFrHgCgsuKaGcCifvnllwJj8+bNU5UqVYr8hlwAqIy4ZgawqNmzZyslJUXR0dHy9vbWunXrtG7dOj344IMKCwszOx7KwS+//FLo9//8VlBQUIm/LA5UNpxmAixq06ZNSkxM1LfffqusrCyFh4dr2LBhmjRpkry9+e+Uyig5OVnx8fHFztmyZYu6detWMYEAD0GZAQCLSE9P1zfffFPsnHbt2hX6A5ZAZUaZAQAAlsYFwAAAwNIq/Yn1vLw8HT9+XP7+/uX6GzAAAKDsGIah8+fPKzQ0VFWqFL/2UunLzPHjx7mzAwAAi0pLS1ODBg2KnVPpy4y/v7+kXz+MgIAAk9MAAIDSyMzMVFhYmPPP8eJU+jKTf2opICCAMgMAgMWU5hIRLgAGAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWZmqZ2b59u/r06aPQ0FDZbDatXr26wJzvvvtOt99+uwIDA1W9enV16NBBqampFR8WAAB4JFPLTHZ2tqKiorRgwYJCnz98+LA6d+6s5s2ba+vWrdq3b58mT54sX1/fCk4KAAA8lc0wDMPsENKvPyS1atUq9evXzzk2aNAg+fj46LXXXnP7uJmZmQoMDFRGRgY/NAkAgEVcyZ/fHnvNTF5env773/8qIiJCPXr0UEhIiDp27FjoqSgAAPDH5W12gKKcOnVKWVlZmjlzpqZPn65Zs2Zp/fr1uuOOO7RlyxZ17dq10P0cDoccDodzOzMzs6IiAwA8SGpqqs6cOWN2jEqvdu3aCg8PNzWDx5aZvLw8SVLfvn316KOPSpKuv/56ffLJJ1q0aFGRZSYpKUmJiYkVlhMA4HlSU1PVrPl1uvjLBbOjVHq+ftW0/3/fmVpoPLbM1K5dW97e3mrRooXL+HXXXacdO3YUuV9CQoLGjRvn3M7MzFRYWFi55QQAeJ4zZ87o4i8XFHzbePkE82dAeck5m6aza+bozJkzlJnCVK1aVR06dND+/ftdxg8cOKCGDRsWuZ/dbpfdbi/veAAAC/AJDpO9XhOzY6CcmVpmsrKydOjQIef2kSNHtHfvXgUFBSk8PFyPPfaYBg4cqC5duig6Olrr16/X+++/r61bt5oXGgAAeBRTy8zu3bsVHR3t3M4/PRQXF6fk5GT1799fixYtUlJSksaMGaNmzZpp5cqV6ty5s1mRAQCAhzG1zHTr1k0lfc3Nvffeq3vvvbeCEgEAAKvx2O+ZAQAAKA3KDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDTKDAAAsDRTy8z27dvVp08fhYaGymazafXq1UXOffjhh2Wz2TRv3rwKywcAADyfqWUmOztbUVFRWrBgQbHzVq1apZ07dyo0NLSCkgEAAKvwNvPFY2NjFRsbW+ycH3/8UaNHj9aGDRvUu3fvCkoGAACswqOvmcnLy9OwYcP02GOPqWXLlmbHAQAAHsjUlZmSzJo1S97e3hozZkyp93E4HHI4HM7tzMzM8ogGAAA8hMeuzKSkpOi5555TcnKybDZbqfdLSkpSYGCg8xEWFlaOKQEAgNk8tsx89NFHOnXqlMLDw+Xt7S1vb28dPXpU48ePV6NGjYrcLyEhQRkZGc5HWlpaxYUGAAAVzmNPMw0bNkwxMTEuYz169NCwYcMUHx9f5H52u112u7284wEAAA9hapnJysrSoUOHnNtHjhzR3r17FRQUpPDwcAUHB7vM9/HxUb169dSsWbOKjgoAADyUqWVm9+7dio6Odm6PGzdOkhQXF6fk5GSTUgEAACsxtcx069ZNhmGUev4PP/xQfmEAAIAleewFwAAAAKVBmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZGmQEAAJZmapnZvn27+vTpo9DQUNlsNq1evdr5XE5OjiZOnKhWrVqpevXqCg0N1fDhw3X8+HHzAgMAAI9japnJzs5WVFSUFixYUOC5CxcuaM+ePZo8ebL27Nmjd955R/v379ftt99uQlIAAOCpvM188djYWMXGxhb6XGBgoDZt2uQyNn/+fN1www1KTU1VeHh4RUQEAAAezlLXzGRkZMhms6lmzZpmRwEAAB7C1JWZK3Hx4kVNnDhRgwcPVkBAQJHzHA6HHA6HczszM7Mi4gEAAJNYYmUmJydHAwYMkGEYWrhwYbFzk5KSFBgY6HyEhYVVUEoAAGAGjy8z+UXm6NGj2rRpU7GrMpKUkJCgjIwM5yMtLa2CkgIAADN49Gmm/CJz8OBBbdmyRcHBwSXuY7fbZbfbKyAdAADwBKaWmaysLB06dMi5feTIEe3du1dBQUGqX7++7rrrLu3Zs0dr1qxRbm6uTpw4IUkKCgpS1apVzYoNAAA8iKllZvfu3YqOjnZujxs3TpIUFxenJ554Qu+9954k6frrr3fZb8uWLerWrVtFxQQAAB7M1DLTrVs3GYZR5PPFPQcAACBZ4AJgAACA4lBmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApVFmAACApZlaZrZv364+ffooNDRUNptNq1evdnneMAxNmTJF9evXl5+fn2JiYnTw4EFzwgIAAI9kapnJzs5WVFSUFixYUOjzs2fP1r/+9S8tWrRIn332mapXr64ePXro4sWLFZwUAAB4Km8zXzw2NlaxsbGFPmcYhubNm6d//vOf6tu3ryTp1VdfVd26dbV69WoNGjSoIqMCAAAP5bHXzBw5ckQnTpxQTEyMcywwMFAdO3bUp59+amIyAADgSUxdmSnOiRMnJEl169Z1Ga9bt67zucI4HA45HA7ndmZmZvkEBAAAHsFjV2bclZSUpMDAQOcjLCzM7EgAAKAceWyZqVevniTp5MmTLuMnT550PleYhIQEZWRkOB9paWnlmhMAAJjLrTLz/fffl3WOAho3bqx69epp8+bNzrHMzEx99tlnuummm4rcz263KyAgwOUBAAAqL7fKTJMmTRQdHa3XX3/9qm6TzsrK0t69e7V3715Jv170u3fvXqWmpspms2ns2LGaPn263nvvPX311VcaPny4QkND1a9fP7dfEwAAVC5ulZk9e/aodevWGjdunOrVq6eHHnpIn3/++RUfZ/fu3WrTpo3atGkjSRo3bpzatGmjKVOmSJIef/xxjR49Wg8++KA6dOigrKwsrV+/Xr6+vu7EBgAAlZBbZeb666/Xc889p+PHj+uVV15Renq6OnfurMjISM2dO1enT58u1XG6desmwzAKPJKTkyVJNptNTz75pE6cOKGLFy/qgw8+UEREhDuRAQBAJXVVFwB7e3vrjjvu0IoVKzRr1iwdOnRIEyZMUFhYmIYPH6709PSyygkAAFCoqyozu3fv1t/+9jfVr19fc+fO1YQJE3T48GFt2rRJx48fd35zLwAAQHlx60vz5s6dq8WLF2v//v3q1auXXn31VfXq1UtVqvzajRo3bqzk5GQ1atSoLLMCAAAU4FaZWbhwoe69916NGDFC9evXL3ROSEiIXn755asKBwAAUBK3yszBgwdLnFO1alXFxcW5c3gAAIBSc+uamcWLF2vFihUFxlesWKElS5ZcdSgAAIDScqvMJCUlqXbt2gXGQ0JC9NRTT111KAAAgNJyq8ykpqaqcePGBcYbNmyo1NTUqw4FAABQWm6VmZCQEO3bt6/A+Jdffqng4OCrDgUAAFBabpWZwYMHa8yYMdqyZYtyc3OVm5urDz/8UI888ogGDRpU1hkBAACK5NbdTNOmTdMPP/ygW265Rd7evx4iLy9Pw4cP55oZAABQodwqM1WrVtWbb76padOm6csvv5Sfn59atWqlhg0blnU+AACAYrlVZvJFRETww48AAMBUbpWZ3NxcJScna/PmzTp16pTy8vJcnv/www/LJBwAAEBJ3CozjzzyiJKTk9W7d29FRkbKZrOVdS4AAIBScavMLF++XG+99ZZ69epV1nkAAACuiFu3ZletWlVNmjQp6ywAAABXzK0yM378eD333HMyDKOs8wAAAFwRt04z7dixQ1u2bNG6devUsmVL+fj4uDz/zjvvlEk4AACAkrhVZmrWrKn+/fuXdRYAAIAr5laZWbx4cVnnAAAAcItb18xI0uXLl/XBBx/ohRde0Pnz5yVJx48fV1ZWVpmFAwAAKIlbKzNHjx5Vz549lZqaKofDoe7du8vf31+zZs2Sw+HQokWLyjonAABAodxamXnkkUfUvn17nTt3Tn5+fs7x/v37a/PmzWUWDgAAoCRurcx89NFH+uSTT1S1alWX8UaNGunHH38sk2AAAACl4dbKTF5ennJzcwuMHzt2TP7+/lcdCgAAoLTcKjO33nqr5s2b59y22WzKysrS1KlT+YkDAABQodw6zTRnzhz16NFDLVq00MWLF3XPPffo4MGDql27tpYtW1bWGQEAAIrkVplp0KCBvvzySy1fvlz79u1TVlaW7rvvPg0ZMsTlgmAAAIDy5laZkSRvb28NHTq0LLMAAABcMbfKzKuvvlrs88OHD3crDAAAwJVyq8w88sgjLts5OTm6cOGCqlatqmrVqlFmAABAhXHrbqZz5865PLKysrR//3517tyZC4ABAECFcvu3mX6vadOmmjlzZoFVm6uRm5uryZMnq3HjxvLz89O1116radOmyTCMMnsNAABgbW5fAFzowby9dfz48TI73qxZs7Rw4UItWbJELVu21O7duxUfH6/AwECNGTOmzF4HAABYl1tl5r333nPZNgxD6enpmj9/vjp16lQmwSTpk08+Ud++fdW7d29Jv/5cwrJly/T555+X2WsAAABrc6vM9OvXz2XbZrOpTp06+stf/qI5c+aURS5J0p///Ge9+OKLOnDggCIiIvTll19qx44dmjt3bpm9BgAAsDa3ykxeXl5Z5yjU3//+d2VmZqp58+by8vJSbm6uZsyYoSFDhhS5j8PhkMPhcG5nZmZWRFQAAGCSMrsAuDy89dZbeuONN7R06VLt2bNHS5Ys0TPPPKMlS5YUuU9SUpICAwOdj7CwsApMDAAAKppbKzPjxo0r9dyrOSX02GOP6e9//7sGDRokSWrVqpWOHj2qpKQkxcXFFbpPQkKCS77MzEwKDQAAlZhbZeaLL77QF198oZycHDVr1kySdODAAXl5ealt27bOeTab7arCXbhwQVWquC4eeXl5FXuay263y263X9XrAgAA63CrzPTp00f+/v5asmSJatWqJenXL9KLj4/XzTffrPHjx5dJuD59+mjGjBkKDw9Xy5Yt9cUXX2ju3Lm69957y+T4AADA+twqM3PmzNHGjRudRUaSatWqpenTp+vWW28tszLz/PPPa/Lkyfrb3/6mU6dOKTQ0VA899JCmTJlSJscHAADW51aZyczM1OnTpwuMnz59WufPn7/qUPn8/f01b948zZs3r8yOCQAAKhe37mbq37+/4uPj9c477+jYsWM6duyYVq5cqfvuu0933HFHWWcEAAAoklsrM4sWLdKECRN0zz33KCcn59cDeXvrvvvu09NPP12mAQEAAIrjVpmpVq2a/v3vf+vpp5/W4cOHJUnXXnutqlevXqbhAAAASnJVX5qXnp6u9PR0NW3aVNWrV+fXrAEAQIVzq8ycPXtWt9xyiyIiItSrVy+lp6dLku67774yu5MJAACgNNwqM48++qh8fHyUmpqqatWqOccHDhyo9evXl1k4AACAkrh1zczGjRu1YcMGNWjQwGW8adOmOnr0aJkEAwAAKA23Vmays7NdVmTy/fTTT/yUAAAAqFBulZmbb75Zr776qnPbZrMpLy9Ps2fPVnR0dJmFAwAAKIlbp5lmz56tW265Rbt379alS5f0+OOP65tvvtFPP/2kjz/+uKwzAgAAFMmtlZnIyEgdOHBAnTt3Vt++fZWdna077rhDX3zxha699tqyzggAAFCkK16ZycnJUc+ePbVo0SJNmjSpPDIBAACU2hWvzPj4+Gjfvn3lkQUAAOCKuXWaaejQoXr55ZfLOgsAAMAVc+sC4MuXL+uVV17RBx98oHbt2hX4Taa5c+eWSTgAAICSXFGZ+f7779WoUSN9/fXXatu2rSTpwIEDLnNsNlvZpQMAACjBFZWZpk2bKj09XVu2bJH0688X/Otf/1LdunXLJRwAAEBJruiamd//Kva6deuUnZ1dpoEAAACuhFsXAOf7fbkBAACoaFdUZmw2W4FrYrhGBgAAmOmKrpkxDEMjRoxw/pjkxYsX9fDDDxe4m+mdd94pu4QAAADFuKIyExcX57I9dOjQMg0DAABwpa6ozCxevLi8cgAAALjlqi4ABgAAMBtlBgAAWBplBgAAWBplBgAAWBplBgAAWBplBgAAWBplBgAAWBplBgAAWBplBgAAWBplBgAAWJrHl5kff/xRQ4cOVXBwsPz8/NSqVSvt3r3b7FgAAMBDXNFvM1W0c+fOqVOnToqOjta6detUp04dHTx4ULVq1TI7GgAA8BAeXWZmzZqlsLAwlx+4bNy4sYmJAACAp/Ho00zvvfee2rdvr7vvvlshISFq06aNXnrpJbNjAQAAD+LRZeb777/XwoUL1bRpU23YsEF//etfNWbMGC1ZsqTIfRwOhzIzM10eAACg8vLo00x5eXlq3769nnrqKUlSmzZt9PXXX2vRokWKi4srdJ+kpCQlJiZWZEwAAGAij16ZqV+/vlq0aOEydt111yk1NbXIfRISEpSRkeF8pKWllXdMAABgIo9emenUqZP279/vMnbgwAE1bNiwyH3sdrvsdnt5RwMAAB7Co1dmHn30Ue3cuVNPPfWUDh06pKVLl+rFF1/UyJEjzY4GAAA8hEeXmQ4dOmjVqlVatmyZIiMjNW3aNM2bN09DhgwxOxoAAPAQHn2aSZJuu+023XbbbWbHAAAAHsqjV2YAAABKQpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWRpkBAACWZqkyM3PmTNlsNo0dO9bsKAAAwENYpszs2rVLL7zwglq3bm12FAAA4EEsUWaysrI0ZMgQvfTSS6pVq5bZcQAAgAexRJkZOXKkevfurZiYGLOjAAAAD+NtdoCSLF++XHv27NGuXbtKNd/hcMjhcDi3MzMzyysaAADwAB69MpOWlqZHHnlEb7zxhnx9fUu1T1JSkgIDA52PsLCwck4JAADM5NFlJiUlRadOnVLbtm3l7e0tb29vbdu2Tf/617/k7e2t3NzcAvskJCQoIyPD+UhLSzMhOQAAqCgefZrplltu0VdffeUyFh8fr+bNm2vixIny8vIqsI/dbpfdbq+oiAAAwGQeXWb8/f0VGRnpMla9enUFBwcXGAcAAH9MHn2aCQAAoCQevTJTmK1bt5odAQAAeBBWZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKVRZgAAgKV5fJlJSkpShw4d5O/vr5CQEPXr10/79+83OxYAAPAQHl9mtm3bppEjR2rnzp3atGmTcnJydOuttyo7O9vsaAAAwAN4mx2gJOvXr3fZTk5OVkhIiFJSUtSlSxeTUgEAAE/h8Sszv5eRkSFJCgoKMjkJAADwBB6/MvNbeXl5Gjt2rDp16qTIyMhC5zgcDjkcDud2ZmZmRcUDAAAmsNTKzMiRI/X1119r+fLlRc5JSkpSYGCg8xEWFlaBCQEAQEWzTJkZNWqU1qxZoy1btqhBgwZFzktISFBGRobzkZaWVoEpAQBARfP400yGYWj06NFatWqVtm7dqsaNGxc73263y263V1A6AABgNo8vMyNHjtTSpUv17rvvyt/fXydOnJAkBQYGys/Pz+R0AADAbB5/mmnhwoXKyMhQt27dVL9+fefjzTffNDsaAADwAB6/MmMYhtkRAACAB/P4lRkAAIDiUGYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAIClUWYAAICleZsdACiN1NRUnTlzxuwYlV7t2rUVHh5udgwAuCKUGXi81NRUNWt+nS7+csHsKJWer1817f/fdxQaAJZCmYHHO3PmjC7+ckHBt42XT3CY2XEqrZyzaTq7Zo7OnDlDmQFgKZQZWIZPcJjs9ZqYHQMA4GG4ABgAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFgaZQYAAFiaJcrMggUL1KhRI/n6+qpjx476/PPPzY4EAAA8hMeXmTfffFPjxo3T1KlTtWfPHkVFRalHjx46deqU2dEAAIAH8PgyM3fuXD3wwAOKj49XixYttGjRIlWrVk2vvPKK2dEAAIAH8Ogyc+nSJaWkpCgmJsY5VqVKFcXExOjTTz81MRkAAPAU3mYHKM6ZM2eUm5urunXruozXrVtX//vf/wrdx+FwyOFwOLczMjIkSZmZmeWS8cSJEzpx4kS5HBu/2r9/vyTJceKQ8i5dNDlN5ZXz0zFJUkpKirKyskxOU7lVqVJFeXl5Zseo1Pj3RsXI//dGVlZWmf85m388wzBKnOvRZcYdSUlJSkxMLDAeFhZmQhqUpXMb5psd4Q/hwQcfNDsCUGb490bF6Nq1a7kd+/z58woMDCx2jkeXmdq1a8vLy0snT550GT958qTq1atX6D4JCQkaN26cczsvL08//fSTgoODZbPZyjWvFWRmZiosLExpaWkKCAgwO06lxedcMficKwafc8Xgc3ZlGIbOnz+v0NDQEud6dJmpWrWq2rVrp82bN6tfv36Sfi0nmzdv1qhRowrdx263y263u4zVrFmznJNaT0BAAP9nqQB8zhWDz7li8DlXDD7n/1PSikw+jy4zkjRu3DjFxcWpffv2uuGGGzRv3jxlZ2crPj7e7GgAAMADeHyZGThwoE6fPq0pU6boxIkTuv7667V+/foCFwUDAIA/Jo8vM5I0atSoIk8r4crY7XZNnTq1wKk4lC0+54rB51wx+JwrBp+z+2xGae55AgAA8FAe/aV5AAAAJaHMAAAAS6PMAAAAS6PMAAAAS6PM/EFs375dffr0UWhoqGw2m1avXm12pEonKSlJHTp0kL+/v0JCQtSvXz/n78OgbC1cuFCtW7d2frnYTTfdpHXr1pkdq9KbOXOmbDabxo4da3aUSuWJJ56QzWZzeTRv3tzsWJZCmfmDyM7OVlRUlBYsWGB2lEpr27ZtGjlypHbu3KlNmzYpJydHt956q7Kzs82OVuk0aNBAM2fOVEpKinbv3q2//OUv6tu3r7755huzo1Vau3bt0gsvvKDWrVubHaVSatmypdLT052PHTt2mB3JUizxPTO4erGxsYqNjTU7RqW2fv16l+3k5GSFhIQoJSVFXbp0MSlV5dSnTx+X7RkzZmjhwoXauXOnWrZsaVKqyisrK0tDhgzRSy+9pOnTp5sdp1Ly9vYu8jcHUTJWZoBykpGRIUkKCgoyOUnllpubq+XLlys7O1s33XST2XEqpZEjR6p3796KiYkxO0qldfDgQYWGhuqaa67RkCFDlJqaanYkS2FlBigHeXl5Gjt2rDp16qTIyEiz41RKX331lW666SZdvHhRNWrU0KpVq9SiRQuzY1U6y5cv1549e7Rr1y6zo1RaHTt2VHJyspo1a6b09HQlJibq5ptv1tdffy1/f3+z41kCZQYoByNHjtTXX3/Nee9y1KxZM+3du1cZGRl6++23FRcXp23btlFoylBaWpoeeeQRbdq0Sb6+vmbHqbR+ewlA69at1bFjRzVs2FBvvfWW7rvvPhOTWQdlBihjo0aN0po1a7R9+3Y1aNDA7DiVVtWqVdWkSRNJUrt27bRr1y4999xzeuGFF0xOVnmkpKTo1KlTatu2rXMsNzdX27dv1/z58+VwOOTl5WViwsqpZs2aioiI0KFDh8yOYhmUGaCMGIah0aNHa9WqVdq6dasaN25sdqQ/lLy8PDkcDrNjVCq33HKLvvrqK5ex+Ph4NW/eXBMnTqTIlJOsrCwdPnxYw4YNMzuKZVBm/iCysrJcWv6RI0e0d+9eBQUFKTw83MRklcfIkSO1dOlSvfvuu/L399eJEyckSYGBgfLz8zM5XeWSkJCg2NhYhYeH6/z581q6dKm2bt2qDRs2mB2tUvH39y9wzVf16tUVHBzMtWBlaMKECerTp48aNmyo48ePa+rUqfLy8tLgwYPNjmYZlJk/iN27dys6Otq5PW7cOElSXFyckpOTTUpVuSxcuFCS1K1bN5fxxYsXa8SIERUfqBI7deqUhg8frvT0dAUGBqp169basGGDunfvbnY04IodO3ZMgwcP1tmzZ1WnTh117txZO3fuVJ06dcyOZhk2wzAMs0MAAAC4i++ZAQAAlkaZAQAAlkaZAQAAlkaZAQAAlkaZAQAAlkaZAQAAlkaZAQAAlkaZAQAAlkaZAWCK06dP669//avCw8Nlt9tVr1499ejRQx9//LHZ0QBYDD9nAMAUd955py5duqQlS5bommuu0cmTJ7V582adPXu2XF7v0qVLqlq1arkcG4C5WJkBUOF+/vlnffTRR5o1a5aio6PVsGFD3XDDDUpISNDtt9/unPPQQw+pbt268vX1VWRkpNasWeM8xsqVK9WyZUvZ7XY1atRIc+bMcXmNRo0aadq0aRo+fLgCAgL04IMPSpJ27Nihm2++WX5+fgoLC9OYMWOUnZ1dcW8eQJmjzACocDVq1FCNGjW0evVqORyOAs/n5eUpNjZWH3/8sV5//XV9++23mjlzpry8vCRJKSkpGjBggAYNGqSvvvpKTzzxhCZPnlzgR1OfeeYZRUVF6YsvvtDkyZN1+PBh9ezZU3feeaf27dunN998Uzt27NCoUaMq4m0DKCf80CQAU6xcuVIPPPCAfvnlF7Vt21Zdu3bVoEGD1Lp1a23cuFGxsbH67rvvFBERUWDfIUOG6PTp09q4caNz7PHHH9d///tfffPNN5J+XZlp06aNVq1a5Zxz//33y8vLSy+88IJzbMeOHeratauys7Pl6+tbju8YQHlhZQaAKe68804dP35c7733nnr27KmtW7eqbdu2Sk5O1t69e9WgQYNCi4wkfffdd+rUqZPLWKdOnXTw4EHl5uY6x9q3b+8y58svv1RycrJzZahGjRrq0aOH8vLydOTIkbJ/kwAqBBcAAzCNr6+vunfvru7du2vy5Mm6//77NXXqVE2YMKFMjl+9enWX7aysLD300EMaM2ZMgbnh4eFl8poAKh5lBoDHaNGihVavXq3WrVvr2LFjOnDgQKGrM9ddd12BW7g//vhjRUREOK+rKUzbtm317bffqkmTJmWeHYB5OM0EoMKdPXtWf/nLX/T6669r3759OnLkiFasWKHZs2erb9++6tq1q7p06aI777xTmzZt0pEjR7Ru3TqtX79ekjR+/Hht3rxZ06ZN04EDB7RkyRLNnz+/xBWdiRMn6pNPPtGoUaO0d+9eHTx4UO+++y4XAAMWx8oMgApXo0YNdezYUc8++6wOHz6snJwchYWF6YEHHtA//vEPSb9eIDxhwgQNHjxY2dnZatKkiWbOnCnp1xWWt956S1OmTNG0adNUv359PfnkkxoxYkSxr9u6dWtt27ZNkyZN0s033yzDMHTttddq4MCB5f2WAZQj7mYCAACWxmkmAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaZQZAABgaf8PBTVPuY70PfoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Plotting histogram\n",
    "plt.hist(output['SCORE_EVAL'], bins=range(1, 7), edgecolor='black', align='left')  # bins from 1 to 5\n",
    "plt.xlabel('Score')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Histogram of SCORE_EVAL')\n",
    "plt.xticks(range(1, 6))  # Setting x-ticks to show each score\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can calculate an accuracy here to use as a baseline. Next we can start improving parts of the model and verify the accuracy goes up. Start digging into why the model was inaccurate in some cases!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "metadata": {}
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9558823529411765\n"
     ]
    }
   ],
   "source": [
    "scores = (output[\"SCORE_EVAL\"] - 1) / 4\n",
    "print (scores.mean())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hf_public",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.-1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}