notebooks/evaluation/quantitative_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "33ee076e-86f5-44b0-9b16-266f4d3469c9",
   "metadata": {},
   "source": [
    "# Quantitative Evaluation\n",
    "\n",
    "In the [evaluation notebook](https://github.com/redhat-et/api-docs-generation/blob/main/notebooks/evaluation/evaluation_metrics.ipynb) we explored different techniques to evaluate the performance of our LLM models in generating API docs and implemented suitable metrics for scoring/ranking the generated outputs.\n",
    "\n",
    "In this notebook, we will try to find the best evaluation criteria/metrics through a quantitative analysis of different prompts/examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d2d79440-f74f-4e7b-920a-3da2812c869b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/app-root/src/prompt-pr/api-docs-generation/notebooks/evaluation/../../app/utils.py:2: DeprecationWarning: Deprecated import of TextGenerationParameters from module genai.text.generation. Please use `from genai.schema import TextGenerationParameters`.\n",
      "  from genai.text.generation import TextGenerationParameters\n",
      "/opt/app-root/src/prompt-pr/api-docs-generation/notebooks/evaluation/../../app/utils.py:3: DeprecationWarning: Deprecated import of TextTokenizationParameters from module genai.text.tokenization. Please use `from genai.schema import TextTokenizationParameters`.\n",
      "  from genai.text.tokenization import (\n",
      "/opt/app-root/src/prompt-pr/api-docs-generation/notebooks/evaluation/../../app/utils.py:3: DeprecationWarning: Deprecated import of TextTokenizationReturnOptions from module genai.text.tokenization. Please use `from genai.schema import TextTokenizationReturnOptions`.\n",
      "  from genai.text.tokenization import (\n",
      "/opt/app-root/src/prompt-pr/api-docs-generation/notebooks/evaluation/../../app/utils.py:3: DeprecationWarning: Deprecated import of TextTokenizationCreateResults from module genai.text.tokenization. Please use `from genai.schema import TextTokenizationCreateResults`.\n",
      "  from genai.text.tokenization import (\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import json\n",
    "import re\n",
    "import pandas as pd\n",
    "import sys\n",
    "sys.path.append('../../app')\n",
    "from utils import eval_using_model\n",
    "from dotenv import load_dotenv\n",
    "from ipynb.fs.defs.helper_functions import get_response, extract_scores, append_row_to_dataframe, langchain_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bdd6a5c-d844-476c-8797-434f3bf2bd72",
   "metadata": {},
   "source": [
    "## Load API credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8e2d80da-db64-49ba-bf53-56cbd91e64ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# make sure you have a .env file in the root folder with genaikey and genaiapi\n",
    "load_dotenv()\n",
    "api_key = os.getenv(\"GENAI_KEY\", None)\n",
    "api_endpoint = os.getenv(\"GENAI_API\", None)\n",
    "openai_key = os.getenv(\"OPENAI_API_KEY\", None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "839cf2ac-1f0f-4890-9c38-51f1337329d8",
   "metadata": {},
   "source": [
    "## Input Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "867c4158-3c97-4a02-8070-3f8c33f079d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "instruction = \"\"\"\n",
    "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
    "\n",
    "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
    "2. Functions: Document each API function, including:\n",
    "    - Description: Clearly explain what the endpoint or function does.\n",
    "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
    "    - Return Values: Specify the data type and possible values returned.\n",
    "\n",
    "3. Error Handling: Describe possible error responses and their meanings.\n",
    "\n",
    "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b74e0295-9269-4dd4-8e38-b6bb22c679db",
   "metadata": {},
   "source": [
    "## Quantitative Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5595a0f-b868-4d23-8ee4-9801341dc9d8",
   "metadata": {},
   "source": [
    "In order to drill down on the best genai evaluation criteria, we construct a quantitative evaluation matrix to determine how often these scores are valid by\n",
    "\n",
    " - Looking at cases where we know the generated output is deliberately wrong and see how the allotted scores perform\n",
    " - And doing this over a number of output for each criteria\n",
    " \n",
    "To do that we have columns for each evaluation criteria as well as human evaluation scores associated with each criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "b2151eb6-58a1-4611-9986-85990b97f650",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = {\n",
    "    'prompt': [],\n",
    "    'response': [],\n",
    "    'gpt_accuracy_score': [],\n",
    "    'human_accuracy_score': [],\n",
    "    'gpt_relevance_score': [],\n",
    "    'human_relevance_score': [],\n",
    "    'gpt_clarity_score': [],\n",
    "    'human_clarity_score': [],\n",
    "    'gpt_completeness_score': [],\n",
    "    'human_completeness_score': [],\n",
    "    'gpt_readability_score': [],\n",
    "    'human_readability_score': [],\n",
    "    'langchain_helpfulness': [],\n",
    "    'human_helpfulness': [],\n",
    "    'langchain_correctness': [],\n",
    "    'human_correctness': [],\n",
    "    'langchain_logical': [],\n",
    "    'human_logical': []\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edbe4fa7-e3e7-4503-9f35-5f5410ea71e3",
   "metadata": {},
   "source": [
    "# DO NOT RUN CELLS WITH EXAMPLES THAT ARE ALREADY ADDED SO THEY ARE NOT OVERWRITTEN.\n",
    "Scroll to the bottom and add more examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "704424b2-5970-4498-87e8-67b204df39ba",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Example 1 - Do not Re-run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90b1b55a-e0b4-428e-9d06-ffb7fa0fb494",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "cffee783-1cf0-410a-a5f9-b269a8c480df",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated_text='\\nIntroduction:\\n\\nThis API provides functionality for detecting credentials in text.\\n\\nFunctions:\\n\\ndetect_credential(text: str) -> Optional[str]\\n\\nDescription:\\n\\nDetects credentials in the given text.\\n\\nParameters:\\n\\ntext (str): The text to detect credentials in.\\n\\nReturn Values:\\n\\nstr: The detected credential.\\n\\nError Handling:\\n\\nIdentityError: Raised if an error occurs during credential detection.\\n\\nMake sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.' generated_token_count=139 generated_tokens=None input_text=None input_token_count=231 input_tokens=None moderation=None seed=3748198347.0 stop_reason='eos_token' stop_sequence=None\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1', api_key, openai_key, 'oidc', instruction, functions=True, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=False, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "76fa32c8-5a24-452a-8a47-ba9c856118f9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "Function Code:\n",
      "\n",
      "def detect_credential() -> Optional[str]:\n",
      "    \n",
      "    try:\n",
      "        return cast(Optional[str], id.detect_credential(_DEFAULT_AUDIENCE))\n",
      "    except id.IdentityError as exc:\n",
      "        IdentityError.raise_from_id(exc)\n",
      "\n",
      "Function Documentation:\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "ba25e20a-30b3-45a6-bf3e-203af37052b5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Generated Patch \n",
      " \n",
      "Introduction:\n",
      "\n",
      "This API provides functionality for detecting credentials in text.\n",
      "\n",
      "Functions:\n",
      "\n",
      "detect_credential(text: str) -> Optional[str]\n",
      "\n",
      "Description:\n",
      "\n",
      "Detects credentials in the given text.\n",
      "\n",
      "Parameters:\n",
      "\n",
      "text (str): The text to detect credentials in.\n",
      "\n",
      "Return Values:\n",
      "\n",
      "str: The detected credential.\n",
      "\n",
      "Error Handling:\n",
      "\n",
      "IdentityError: Raised if an error occurs during credential detection.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Generated Text \\n\", generated_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "aded6c51-6a89-41db-a1ac-7a58663cdf9d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately describes the purpose of the API and the function. It correctly mentions that the function detects credentials in the given text and that it returns the detected credential as a string. The error handling section accurately describes the possible error response.\n",
      "\n",
      "Relevance: 5 - The generated documentation is relevant to the provided code. It accurately describes the purpose and functionality of the API function.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear in explaining what the function does and what its parameters and return values are. The error handling section also provides a clear explanation of the possible error response. \n",
      "\n",
      "Completeness: 4 - The generated documentation provides a comprehensive description of the API function, including its purpose, parameters, return values, and error handling. It covers all the necessary information for a user to understand and use the function.\n",
      "\n",
      "Readability: 5 - The generated documentation is well-structured and easy to read. It uses appropriate formatting and language to convey the information clearly.\n",
      "\n",
      "Overall Score: 4\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "0691630c-d9cf-43cb-bbc8-0c1772636449",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "830d29b4-b4c4-4270-9b13-8fa9d4fb7388",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "1a813054-51df-41bc-90a0-0666b9a1a8c7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". \\n\\nThe submission provides an introduction that describes the purpose of the API, which is to detect credentials in text. This is helpful for users to understand what the API does.\\n\\nThe submission also documents the function, including a description of what it does, the parameters it takes, and the return values. This is helpful for users to understand how to use the function.\\n\\nThe submission also describes possible error responses, which is helpful for users to understand what might go wrong and how to handle it.\\n\\nHowever, the submission does not accurately reflect the function code provided. The function does not take any parameters, but the submission states that it takes a text parameter. This could mislead users and cause confusion.\\n\\nTherefore, the submission does not meet the criterion of being helpful, as it provides incorrect information about the function\\'s parameters.\\n\\nN', 'value': 'N', 'score': 0}\n",
      "{'reasoning': \"The criteria is to assess if the submission is correct, accurate, and factual.\\n\\nLooking at the submission, the introduction correctly describes the purpose of the API. The function is also correctly documented with a description, parameters, and return values. However, the parameters section is incorrect. The function 'detect_credential' does not take any parameters, but the submission states that it takes a 'text' parameter. This is not accurate according to the provided function code.\\n\\nThe return value is correctly described as an optional string, which matches the function code. The error handling section correctly identifies that an 'IdentityError' can be raised, which is also accurate according to the function code.\\n\\nIn conclusion, the submission is mostly correct and accurate, but it fails to be completely factual due to the incorrect description of the function's parameters.\\n\\nN\", 'value': 'N', 'score': 0}\n",
      "{'reasoning': \"The criteria for this task is to assess whether the output is complete and captures all required fields. \\n\\nLooking at the submission, the introduction is present and describes the purpose of the API. \\n\\nThe function 'detect_credential' is documented with a description, parameters, and return values. However, the parameters section is incorrect. The function does not take any parameters according to the provided function code, but the submission states that it takes a 'text' parameter. This is a discrepancy.\\n\\nThe error handling section is present and describes the possible error that can occur.\\n\\nBased on this analysis, the submission does not meet the criteria because it incorrectly documents a parameter that does not exist in the function.\\n\\nN\", 'value': 'N', 'score': 0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_600/2635151755.py:38: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df = df.append(new_row, ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "fbedc847-bb4b-484b-9689-b7e5edba3f5c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "\n",
       "   human_accuracy_score  gpt_relevance_score  human_relevance_score  \\\n",
       "0                   NaN                  5.0                    NaN   \n",
       "\n",
       "   gpt_clarity_score  human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                  NaN                     4.0   \n",
       "\n",
       "   human_completeness_score  gpt_readability_score  human_readability_score  \\\n",
       "0                       NaN                    5.0                      NaN   \n",
       "\n",
       "   langchain_helpfulness  human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0                NaN                    0.0   \n",
       "\n",
       "   human_correctness  langchain_logical  human_logical  \n",
       "0                NaN                0.0            NaN  "
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "244be0e0-c18e-4329-8fc9-7669370321b3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[0, 'human_accuracy_score'] = '2.0'\n",
    "df.at[0, 'human_relevance_score'] = '3.0'\n",
    "df.at[0, 'human_clarity_score'] = '4.0'\n",
    "df.at[0, 'human_completeness_score'] = '4.0'\n",
    "df.at[0, 'human_readability_score'] = '5.0'\n",
    "df.at[0, 'human_helpfulness'] = '0.0'\n",
    "df.at[0, 'human_correctness'] = '0.0'\n",
    "df.at[0, 'human_logical'] = '0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "c8476dc3-d5ca-4f93-b57f-57622bde246d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                54.0                     4.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0                 0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0                 0                0.0             0  "
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd414598-e98b-4891-874d-178bc063ad73",
   "metadata": {},
   "source": [
    "**Interpretation**: Above is a great example of where the generated documentation is partially incorrect and the langchain eval criteria is able to detect the issue correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b89f8748-22b4-449c-b32b-f469a5dc7ef4",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Example 2 - Do not Re-run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "ccd823f7-9fff-46dc-ab63-75d6ad459963",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated_text='1. Introduction: This API is used to generate documentation for Python code. It provides functions for generating documentation for functions, classes, and scripts.\\n\\n2. Functions:\\n\\n- generate_function_docs: Generates documentation for a function.\\n- generate_class_docs: Generates documentation for a class.\\n- generate_script_docs: Generates documentation for a script.\\n\\n3. Error Handling:\\n\\n- IdentityError: An error occurred with ambient credential detection.\\n- IssuerError: An error occurred with the OIDC issuer.\\n- NetworkError: A network error occurred.\\n\\nMake sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\\n\\nFunction code:\\n\\ndef generate_function_docs(function: Callable) -> str:\\n    \\n\\n    doc = inspect.getdoc(function)\\n    if doc is None:\\n        raise ValueError(f\"function {function.__name__!r} has no docstring\")\\n\\n    signature = inspect.signature(function)\\n\\n    doc_lines = [\\n        f\"Function: {function.__name__}\",\\n        \"\",\\n        f\"{doc}\",\\n        \"\",\\n        \"Parameters:\",\\n    ]\\n\\n    for name, param in signature.parameters.items():\\n        doc_lines.append(f\"- {name}: {param.annotation}\")\\n\\n    return \"\\\\n\".join(doc_lines)\\nFunction Documentation:\\n\\n1. Introduction: This function generates documentation for a function.\\n\\n2. Parameters:\\n\\n- function: The function to generate documentation for.\\n\\n3. Return Values: A string containing the generated documentation.\\n\\n4. Error Handling:\\n\\n- ValueError: The function has no docstring.\\n\\n5. Diagnostics:\\n\\n- Insufficient permissions for GitHub Actions workflow: The most common reason for this is incorrect configuration of the top-level `permissions` setting of the workflow YAML file. It should be configured like so:\\n\\n    permissions:\\n      id-token: write\\n\\n    Relevant documentation here:\\n\\n      https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings\\n\\n    Another possible reason is that the workflow run has been triggered by a PR from a forked repository. PRs from forked repositories typically cannot be granted write access.\\n\\n    Relevant documentation here:\\n\\n      https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token\\n\\n    Additional context:\\n\\n      An issue occurred with ambient credential detection.\\n\\n      Additional context:\\n\\n      An error occurred with ambient credential detection.\\n\\n\\nClass code:\\n\\ndef generate_class_docs(cls: Type) -> str:\\n    \\n\\n    doc = inspect.getdoc(cls)\\n    if doc is None:\\n        raise ValueError(f\"class {cls.__name__!r} has no docstring\")\\n\\n    signature = inspect.signature(cls)\\n\\n    doc_lines = [\\n        f\"Class: {cls.__name__}\",\\n        \"\",\\n        f\"{doc}\",\\n        \"\",\\n        \"Parameters:\",\\n    ]\\n\\n    for name, param in signature.parameters.items():\\n        doc_lines.append(f\"- {name}: {param.annotation}\")\\n\\n    return \"\\\\n\".join(doc_lines)\\nClass Documentation:\\n\\n1. Introduction: This function generates documentation for a class.\\n\\n2. Parameters:\\n\\n- cls: The class to generate documentation for.\\n\\n3. Return Values: A string containing the generated documentation.\\n\\n4. Error Handling:\\n\\n- ValueError: The class has no docstring.\\n\\n5. Diagnostics:\\n\\n- Insufficient permissions for GitHub Actions workflow: The most common reason for this is incorrect configuration of the top-level `permissions` setting of the workflow YAML file. It should be configured like so:\\n\\n    permissions:\\n      id-token: write\\n\\n    Relevant documentation here:\\n\\n      https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings\\n\\n    Another possible reason is that the workflow run has been triggered by a PR from a forked repository. PRs from forked repositories typically cannot be granted write access.\\n\\n    Relevant documentation here:\\n\\n      https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token\\n\\n    Additional context:\\n\\n      An issue occurred with ambient credential detection.\\n\\n      Additional context:\\n\\n      An error occurred with ambient credential detection.\\n\\n\\nClass code:\\n\\ndef generate_script_docs(script_path: Path) -> str:\\n    \\n\\n    doc = inspect.getdoc(script_path)\\n    if doc is None:\\n        raise ValueError(f\"script {script_' generated_token_count=1024 generated_tokens=None input_text=None input_token_count=2427 input_tokens=None moderation=None seed=511192819.0 stop_reason='max_tokens' stop_sequence=None\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1', api_key, openai_key, 'oidc', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "81cc9177-d622-4e8d-9704-bb09e91843df",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class _OpenIDConfiguration(BaseModel):\n",
      "    \n",
      "\n",
      "    authorization_endpoint: StrictStr\n",
      "    token_endpoint: StrictStr\n",
      "class ExpiredIdentity(Exception):\n",
      "    \n",
      "class IdentityToken:\n",
      "    \n",
      "\n",
      "    def __init__(self, raw_token: str) -> None:\n",
      "        \n",
      "\n",
      "        self._raw_token = raw_token\n",
      "\n",
      "        # NOTE: The lack of verification here is intentional, and is part of\n",
      "        # Sigstore's verification model: clients like sigstore-python are\n",
      "        # responsible only for forwarding the OIDC identity to Fulcio for\n",
      "        # certificate binding and issuance.\n",
      "        try:\n",
      "            self._unverified_claims = jwt.decode(\n",
      "                raw_token,\n",
      "                options={\n",
      "                    \"verify_signature\": False,\n",
      "                    \"verify_aud\": True,\n",
      "                    \"verify_iat\": True,\n",
      "                    \"verify_exp\": True,\n",
      "                    # These claims are required by OpenID Connect, so\n",
      "                    # we can strongly enforce their presence.\n",
      "                    # See: https://openid.net/specs/openid-connect-basic-1_0.html#IDToken\n",
      "                    \"require\": [\"aud\", \"sub\", \"iat\", \"exp\", \"iss\"],\n",
      "                },\n",
      "                audience=DEFAULT_AUDIENCE,\n",
      "                # NOTE: This leeway shouldn't be strictly necessary, but is\n",
      "                # included to preempt any (small) skew between the host\n",
      "                # and the originating IdP.\n",
      "                leeway=5,\n",
      "            )\n",
      "        except Exception as exc:\n",
      "            raise IdentityError(\n",
      "                \"Identity token is malformed or missing claims\"\n",
      "            ) from exc\n",
      "\n",
      "        self._iss: str = self._unverified_claims[\"iss\"]\n",
      "        self._nbf: int | None = self._unverified_claims.get(\"nbf\")\n",
      "        self._exp: int = self._unverified_claims[\"exp\"]\n",
      "\n",
      "        # Fail early if this token isn't within its validity period.\n",
      "        if not self.in_validity_period():\n",
      "            raise IdentityError(\"Identity token is not within its validity period\")\n",
      "\n",
      "        # When verifying the private key possession proof, Fulcio uses\n",
      "        # different claims depending on the token's issuer.\n",
      "        # We currently special-case a handful of these, and fall back\n",
      "        # on signing the \"sub\" claim otherwise.\n",
      "        identity_claim = _KNOWN_OIDC_ISSUERS.get(self.issuer)\n",
      "        if identity_claim is not None:\n",
      "            if identity_claim not in self._unverified_claims:\n",
      "                raise IdentityError(\n",
      "                    f\"Identity token is missing the required {identity_claim!r} claim\"\n",
      "                )\n",
      "\n",
      "            self._identity = str(self._unverified_claims.get(identity_claim))\n",
      "        else:\n",
      "            try:\n",
      "                self._identity = str(self._unverified_claims[\"sub\"])\n",
      "            except KeyError:\n",
      "                raise IdentityError(\n",
      "                    \"Identity token is missing the required 'sub' claim\"\n",
      "                )\n",
      "\n",
      "        # This identity token might have been retrieved directly from\n",
      "        # an identity provider, or it might be a \"federated\" identity token\n",
      "        # retrieved from a federated IdP (e.g., Sigstore's own Dex instance).\n",
      "        # In the latter case, the claims will also include a `federated_claims`\n",
      "        # set, which in turn should include a `connector_id` that reflects\n",
      "        # the \"real\" token issuer. We retrieve this, despite technically\n",
      "        # being an implementation detail, because it has value to client\n",
      "        # users: a client might want to make sure that its user is identifying\n",
      "        # with a *particular* IdP, which means that they need to pierce the\n",
      "        # federation layer to check which IdP is actually being used.\n",
      "        self._federated_issuer: str | None = None\n",
      "        federated_claims = self._unverified_claims.get(\"federated_claims\")\n",
      "        if federated_claims is not None:\n",
      "            if not isinstance(federated_claims, dict):\n",
      "                raise IdentityError(\n",
      "                    \"unexpected claim type: federated_claims is not a dict\"\n",
      "                )\n",
      "\n",
      "            federated_issuer = federated_claims.get(\"connector_id\")\n",
      "            if federated_issuer is not None:\n",
      "                if not isinstance(federated_issuer, str):\n",
      "                    raise IdentityError(\n",
      "                        \"unexpected claim type: federated_claims.connector_id is not a string\"\n",
      "                    )\n",
      "\n",
      "                self._federated_issuer = federated_issuer\n",
      "\n",
      "    def in_validity_period(self) -> bool:\n",
      "        \n",
      "\n",
      "        now = datetime.now(timezone.utc).timestamp()\n",
      "\n",
      "        if self._nbf is not None:\n",
      "            return self._nbf <= now < self._exp\n",
      "        else:\n",
      "            return now < self._exp\n",
      "\n",
      "    @property\n",
      "    def identity(self) -> str:\n",
      "        \n",
      "        return self._identity\n",
      "\n",
      "    @property\n",
      "    def issuer(self) -> str:\n",
      "        \n",
      "        return self._iss\n",
      "\n",
      "    @property\n",
      "    def expected_certificate_subject(self) -> str:\n",
      "        \n",
      "        if self._federated_issuer is not None:\n",
      "            return self._federated_issuer\n",
      "\n",
      "        return self.issuer\n",
      "\n",
      "    def __str__(self) -> str:\n",
      "        \n",
      "        return self._raw_token\n",
      "class IssuerError(Exception):\n",
      "    \n",
      "\n",
      "    pass\n",
      "class Issuer:\n",
      "    \n",
      "\n",
      "    def __init__(self, base_url: str) -> None:\n",
      "        \n",
      "        oidc_config_url = urllib.parse.urljoin(\n",
      "            f\"{base_url}/\", \".well-known/openid-configuration\"\n",
      "        )\n",
      "\n",
      "        try:\n",
      "            resp: requests.Response = requests.get(oidc_config_url, timeout=30)\n",
      "        except (requests.ConnectionError, requests.Timeout) as exc:\n",
      "            raise NetworkError from exc\n",
      "\n",
      "        try:\n",
      "            resp.raise_for_status()\n",
      "        except requests.HTTPError as http_error:\n",
      "            raise IssuerError from http_error\n",
      "\n",
      "        try:\n",
      "            # We don't generally expect this to fail (since the provider should\n",
      "            # return a non-success HTTP code which we catch above), but we\n",
      "            # check just in case we have a misbehaving OIDC issuer.\n",
      "            self.oidc_config = _OpenIDConfiguration.model_validate(resp.json())\n",
      "        except ValueError as exc:\n",
      "            raise IssuerError(f\"OIDC issuer returned invalid configuration: {exc}\")\n",
      "\n",
      "    @classmethod\n",
      "    def production(cls) -> Issuer:\n",
      "        \n",
      "        return cls(DEFAULT_OAUTH_ISSUER_URL)\n",
      "\n",
      "    @classmethod\n",
      "    def staging(cls) -> Issuer:\n",
      "        \n",
      "        return cls(STAGING_OAUTH_ISSUER_URL)\n",
      "\n",
      "    def identity_token(  # nosec: B107\n",
      "        self,\n",
      "        client_id: str = \"sigstore\",\n",
      "        client_secret: str = \"\",\n",
      "        force_oob: bool = False,\n",
      "    ) -> IdentityToken:\n",
      "        \n",
      "\n",
      "        # This function and the components that it relies on are based off of:\n",
      "        # https://github.com/psteniusubi/python-sample\n",
      "\n",
      "        from sigstore._internal.oidc.oauth import _OAuthFlow\n",
      "\n",
      "        code: str\n",
      "        with _OAuthFlow(client_id, client_secret, self) as server:\n",
      "            # Launch web browser\n",
      "            if not force_oob and webbrowser.open(server.base_uri):\n",
      "                print(\"Waiting for browser interaction...\", file=sys.stderr)\n",
      "            else:\n",
      "                server.enable_oob()\n",
      "                print(\n",
      "                    f\"Go to the following link in a browser:\\n\\n\\t{server.auth_endpoint}\",\n",
      "                    file=sys.stderr,\n",
      "                )\n",
      "\n",
      "            if not server.is_oob():\n",
      "                # Wait until the redirect server populates the response\n",
      "                while server.auth_response is None:\n",
      "                    time.sleep(0.1)\n",
      "\n",
      "                auth_error = server.auth_response.get(\"error\")\n",
      "                if auth_error is not None:\n",
      "                    raise IdentityError(\n",
      "                        f\"Error response from auth endpoint: {auth_error[0]}\"\n",
      "                    )\n",
      "                code = server.auth_response[\"code\"][0]\n",
      "            else:\n",
      "                # In the out-of-band case, we wait until the user provides the code\n",
      "                code = input(\"Enter verification code: \")\n",
      "\n",
      "        # Provide code to token endpoint\n",
      "        data = {\n",
      "            \"grant_type\": \"authorization_code\",\n",
      "            \"redirect_uri\": server.redirect_uri,\n",
      "            \"code\": code,\n",
      "            \"code_verifier\": server.oauth_session.code_verifier,\n",
      "        }\n",
      "        auth = (\n",
      "            client_id,\n",
      "            client_secret,\n",
      "        )\n",
      "        logging.debug(f\"PAYLOAD: data={data}\")\n",
      "        try:\n",
      "            resp: requests.Response = requests.post(\n",
      "                self.oidc_config.token_endpoint,\n",
      "                data=data,\n",
      "                auth=auth,\n",
      "                timeout=30,\n",
      "            )\n",
      "        except (requests.ConnectionError, requests.Timeout) as exc:\n",
      "            raise NetworkError from exc\n",
      "\n",
      "        try:\n",
      "            resp.raise_for_status()\n",
      "        except requests.HTTPError as http_error:\n",
      "            raise IdentityError(\n",
      "                f\"Token request failed with {resp.status_code}\"\n",
      "            ) from http_error\n",
      "\n",
      "        token_json = resp.json()\n",
      "        token_error = token_json.get(\"error\")\n",
      "        if token_error is not None:\n",
      "            raise IdentityError(f\"Error response from token endpoint: {token_error}\")\n",
      "\n",
      "        return IdentityToken(token_json[\"access_token\"])\n",
      "class IdentityError(Error):\n",
      "    \n",
      "\n",
      "    @classmethod\n",
      "    def raise_from_id(cls, exc: id.IdentityError) -> NoReturn:\n",
      "        \n",
      "        raise cls(str(exc)) from exc\n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "        if isinstance(self.__cause__, id.GitHubOidcPermissionCredentialError):\n",
      "            return f\n",
      "                Insufficient permissions for GitHub Actions workflow.\n",
      "\n",
      "                The most common reason for this is incorrect\n",
      "                configuration of the top-level `permissions` setting of the\n",
      "                workflow YAML file. It should be configured like so:\n",
      "\n",
      "                    permissions:\n",
      "                      id-token: write\n",
      "\n",
      "                Relevant documentation here:\n",
      "\n",
      "                    https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect#adding-permissions-settings\n",
      "\n",
      "                Another possible reason is that the workflow run has been\n",
      "                triggered by a PR from a forked repository. PRs from forked\n",
      "                repositories typically cannot be granted write access.\n",
      "\n",
      "                Relevant documentation here:\n",
      "\n",
      "                    https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token\n",
      "\n",
      "                Additional context:\n",
      "\n",
      "                {self.__cause__}\n",
      "                \n",
      "        else:\n",
      "            return f\n",
      "                An issue occurred with ambient credential detection.\n",
      "\n",
      "                Additional context:\n",
      "\n",
      "                {self}\n",
      "            \n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "f6100452-8179-4249-9034-d627e7ff8121",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ sigstore](../sigstore.html)\n",
      "\n",
      "## API Documentation\n",
      "\n",
      "  * DEFAULT_OAUTH_ISSUER_URL\n",
      "  * STAGING_OAUTH_ISSUER_URL\n",
      "  * DEFAULT_AUDIENCE\n",
      "  * ExpiredIdentity\n",
      "  * IdentityToken\n",
      "    * IdentityToken\n",
      "    * in_validity_period\n",
      "    * identity\n",
      "    * issuer\n",
      "    * expected_certificate_subject\n",
      "  * IssuerError\n",
      "  * Issuer\n",
      "    * Issuer\n",
      "    * production\n",
      "    * staging\n",
      "    * identity_token\n",
      "  * IdentityError\n",
      "    * raise_from_id\n",
      "    * diagnostics\n",
      "  * detect_credential\n",
      "\n",
      "[ built with pdoc ](https://pdoc.dev \"pdoc: Python API documentation\n",
      "generator\")\n",
      "\n",
      "#  [sigstore](./../sigstore.html).oidc\n",
      "\n",
      "API for retrieving OIDC tokens.\n",
      "\n",
      "View Source\n",
      "    \n",
      "\n",
      "DEFAULT_OAUTH_ISSUER_URL = 'https://oauth2.sigstore.dev/auth'\n",
      "\n",
      "STAGING_OAUTH_ISSUER_URL = 'https://oauth2.sigstage.dev/auth'\n",
      "\n",
      "DEFAULT_AUDIENCE = 'sigstore'\n",
      "\n",
      "class ExpiredIdentity(builtins.Exception): View Source\n",
      "    \n",
      "\n",
      "An error raised when an identity token is expired.\n",
      "\n",
      "##### Inherited Members\n",
      "\n",
      "builtins.Exception\n",
      "\n",
      "    Exception\n",
      "\n",
      "builtins.BaseException\n",
      "\n",
      "    with_traceback\n",
      "    add_note\n",
      "    args\n",
      "\n",
      "class IdentityToken: View Source\n",
      "    \n",
      "\n",
      "An OIDC \"identity\", corresponding to an underlying OIDC token with a sensible\n",
      "subject, issuer, and audience for Sigstore purposes.\n",
      "\n",
      "IdentityToken(raw_token: str) View Source\n",
      "    \n",
      "\n",
      "Create a new `IdentityToken` from the given OIDC token.\n",
      "\n",
      "def in_validity_period(self) -> bool: View Source\n",
      "    \n",
      "\n",
      "Returns whether or not this `Identity` is currently within its self-stated\n",
      "validity period.\n",
      "\n",
      "NOTE: As noted in `Identity.__init__`, this is not a verifying wrapper; the\n",
      "check here only asserts whether the _unverified_ identity's claims are within\n",
      "their validity period.\n",
      "\n",
      "identity: str\n",
      "\n",
      "Returns this `IdentityToken`'s underlying \"subject\".\n",
      "\n",
      "Note that this is **not** always the `sub` claim in the corresponding identity\n",
      "token: depending onm the token's issuer, it may be a _different_ claim, such\n",
      "as `email`. This corresponds to the Sigstore ecosystem's behavior, e.g. in\n",
      "each issued certificate's SAN.\n",
      "\n",
      "issuer: str\n",
      "\n",
      "Returns a URL identifying this `IdentityToken`'s issuer.\n",
      "\n",
      "expected_certificate_subject: str\n",
      "\n",
      "Returns a URL identifying the **expected** subject for any Sigstore\n",
      "certificate issued against this identity token.\n",
      "\n",
      "The behavior of this field is slightly subtle: for non-federated identity\n",
      "providers (like a token issued directly by Google's IdP) it should be exactly\n",
      "equivalent to `IdentityToken.issuer`. For federated issuers (like Sigstore's\n",
      "own federated IdP) it should be equivalent to the underlying federated\n",
      "issuer's URL, which is kept in an implementation-defined claim.\n",
      "\n",
      "This attribute exists so that clients who wish to inspect the expected subject\n",
      "of their certificates can do so without relying on implementation-specific\n",
      "behavior.\n",
      "\n",
      "class IssuerError(builtins.Exception): View Source\n",
      "    \n",
      "\n",
      "Raised on any communication or format error with an OIDC issuer.\n",
      "\n",
      "##### Inherited Members\n",
      "\n",
      "builtins.Exception\n",
      "\n",
      "    Exception\n",
      "\n",
      "builtins.BaseException\n",
      "\n",
      "    with_traceback\n",
      "    add_note\n",
      "    args\n",
      "\n",
      "class Issuer: View Source\n",
      "    \n",
      "\n",
      "Represents an OIDC issuer (IdP).\n",
      "\n",
      "Issuer(base_url: str) View Source\n",
      "    \n",
      "\n",
      "Create a new `Issuer` from the given base URL.\n",
      "\n",
      "This URL is used to locate an OpenID Connect configuration file, which is then\n",
      "used to bootstrap the issuer's state (such as authorization and token\n",
      "endpoints).\n",
      "\n",
      "@classmethod\n",
      "\n",
      "def production(cls) -> Issuer: View Source\n",
      "    \n",
      "\n",
      "Returns an `Issuer` configured against Sigstore's production-level services.\n",
      "\n",
      "@classmethod\n",
      "\n",
      "def staging(cls) -> Issuer: View Source\n",
      "    \n",
      "\n",
      "Returns an `Issuer` configured against Sigstore's staging-level services.\n",
      "\n",
      "def identity_token( self, client_id: str = 'sigstore', client_secret: str =\n",
      "'', force_oob: bool = False) -> IdentityToken: View Source\n",
      "    \n",
      "\n",
      "Retrieves and returns an `IdentityToken` from the current `Issuer`, via OAuth.\n",
      "\n",
      "This function blocks on user interaction.\n",
      "\n",
      "The `force_oob` flag controls the kind of flow performed. When `False` (the\n",
      "default), this function attempts to open the user's web browser before falling\n",
      "back to an out-of-band flow. When `True`, the out-of-band flow is always used.\n",
      "\n",
      "class IdentityError([sigstore.errors.Error](errors.html#Error)): View Source\n",
      "    \n",
      "\n",
      "Wraps `id`'s IdentityError.\n",
      "\n",
      "@classmethod\n",
      "\n",
      "def raise_from_id(cls, exc: id.IdentityError) -> NoReturn: View Source\n",
      "    \n",
      "\n",
      "Raises a wrapped IdentityError from the provided `id.IdentityError`.\n",
      "\n",
      "def diagnostics(self) -> str: View Source\n",
      "    \n",
      "\n",
      "Returns diagnostics for the error.\n",
      "\n",
      "##### Inherited Members\n",
      "\n",
      "builtins.Exception\n",
      "\n",
      "    Exception\n",
      "\n",
      "[sigstore.errors.Error](errors.html#Error)\n",
      "\n",
      "    [print_and_exit](errors.html#Error.print_and_exit)\n",
      "\n",
      "builtins.BaseException\n",
      "\n",
      "    with_traceback\n",
      "    add_note\n",
      "    args\n",
      "\n",
      "def detect_credential() -> Optional[str]: View Source\n",
      "    \n",
      "\n",
      "Calls `id.detect_credential`, but wraps exceptions with our own exception\n",
      "type.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "39b0ecfb-58da-46c9-8f1d-d5be5e470df4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately identifies the purpose and functionality of the API functions and classes. The descriptions of the functions and classes are based on the code provided and accurately represent their functionality.\n",
      "\n",
      "Relevance: 3.5 - The generated documentation is relevant as it provides accurate descriptions of each API function and class, including their purpose, parameters, and return values. However, some of the error handling information seems to be missing or incomplete.\n",
      "\n",
      "Clarity: 3.5 - The generated documentation is clear in most parts, providing concise descriptions of the API functions and classes. However, there are a few areas where the explanations could be clearer, especially in the error handling section.\n",
      "\n",
      "Completeness: 3 - The generated documentation provides descriptions of each API function and class, including their purpose and parameters. However, some parts of the documentation, especially in the error handling section, are incomplete or missing important details.\n",
      "\n",
      "Readability: 4 - The generated documentation is readable and well-organized. The descriptions are clear and concise, making it easy for users to understand the purpose and functionality of the API functions and classes.\n",
      "\n",
      "Overall Score: 3.6\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "650daba3-8a73-4ea9-963b-919c57a2afa4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d90028cd-b99d-4334-886d-5bb3a059044c",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "25f1fc90-86c5-4ca9-b147-54b717b7fd9e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criteria for this task is \"helpfulness\". The submission is supposed to be helpful, insightful, and appropriate. \\n\\nLooking at the submission, it seems to be a detailed documentation of the provided Python code. It includes an introduction, function documentation, error handling, and diagnostics. It also provides links to relevant documentation for further reading. \\n\\nHowever, the submission seems to have misunderstood the task. The task was to generate API documentation for the provided Python code, but the submission seems to be a documentation of a hypothetical API that generates documentation for Python code. This is a significant misunderstanding of the task.\\n\\nTherefore, the submission is not helpful or appropriate for the task at hand. \\n\\nN', 'value': 'N', 'score': 0}\n",
      "{'reasoning': 'The submission is supposed to provide API documentation for the provided Python code. The code provided includes several classes and methods, including the _OpenIDConfiguration class, the ExpiredIdentity exception, the IdentityToken class, the IssuerError exception, the Issuer class, and the IdentityError class.\\n\\nThe submission, however, does not accurately reflect the provided code. It instead provides documentation for non-existent functions such as generate_function_docs, generate_class_docs, and generate_script_docs. The submission also includes error handling and diagnostics information that is not present in the provided code.\\n\\nThe reference API documentation provides a more accurate representation of the provided code, including the correct classes, methods, and exceptions. The submission does not match this reference documentation.\\n\\nBased on this analysis, the submission is not correct, accurate, or factual. It does not accurately document the provided Python code, and it includes information that is not present in the code. Therefore, the submission does not meet the criteria. \\n\\nN', 'value': 'N', 'score': 0}\n",
      "{'reasoning': 'The criteria for this task is to assess whether the output is complete and captures all required fields. \\n\\nLooking at the submission, it seems to be a mix of API documentation and Python code. The task was to generate API documentation for the provided Python code, but the submission includes Python code for generating documentation, which is not part of the task.\\n\\nThe submission does include some elements of API documentation, such as an introduction, function descriptions, parameters, return values, and error handling. However, these elements are not consistently applied to all the classes and functions in the provided Python code. \\n\\nFor example, the classes _OpenIDConfiguration, ExpiredIdentity, IdentityToken, IssuerError, Issuer, and IdentityError are not documented at all. The functions within the IdentityToken and Issuer classes are also not documented. \\n\\nFurthermore, the submission includes documentation for functions that are not part of the provided Python code, such as generate_function_docs, generate_class_docs, and generate_script_docs. \\n\\nTherefore, the submission does not meet the criteria of being complete and capturing all required fields. \\n\\nN', 'value': 'N', 'score': 0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_600/2635151755.py:38: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df = df.append(new_row, ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "ce332c83-86e4-49d3-a842-2333bbf7edb4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  NaN                  3.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                54.0                     4.0   \n",
       "1                3.0                 NaN                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      NaN                    4.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0                 0                    0.0   \n",
       "1                    0.0               NaN                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0                 0                0.0             0  \n",
       "1               NaN                0.0           NaN  "
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "6668bbac-618f-4932-b628-839bd74bd905",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[1, 'human_accuracy_score'] = '1.0'\n",
    "df.at[1, 'human_relevance_score'] = '1.0'\n",
    "df.at[1, 'human_clarity_score'] = '1.0'\n",
    "df.at[1, 'human_completeness_score'] = '1.0'\n",
    "df.at[1, 'human_readability_score'] = '1.0'\n",
    "df.at[1, 'human_helpfulness'] = '0.0'\n",
    "df.at[1, 'human_correctness'] = '0.0'\n",
    "df.at[1, 'human_logical'] = '0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "8208bd8a-4387-4c9f-b9ba-cebe495648f1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  "
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9efe3efb-0cc4-447e-89bf-41aa3e0911cd",
   "metadata": {},
   "source": [
    "**Interpretation**: This is a great example where the generated output completely misunderstood the task and hallucinated content, langchain evaluation was able to catch the error well. Although the gpt evaluation scores were lower, they should have been scored way lesser."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f41b9a7b-6e12-43d5-b0cf-85242dc6f371",
   "metadata": {},
   "source": [
    "### Example 3 - Do not Re-run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "47a7d83e-d889-4786-b7cb-7b5b9995934c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated_text='1. Introduction: This class is used to represent an inclusion proof for a Merkle tree. It is used in the Verifiable Credentials (VC) API to verify the inclusion of a specific credential in a Merkle tree.\\n\\n2. Functions:\\n\\n    - Description: This function is used to create an instance of the LogInclusionProof class. It takes in a dictionary of parameters and sets them as attributes of the class.\\n\\n    - Parameters:\\n        - checkpoint (str): The checkpoint of the Merkle tree.\\n        - hashes (list): A list of hashes in the inclusion proof.\\n        - log_index (int): The index of the log in the Merkle tree.\\n        - root_hash (str): The root hash of the Merkle tree.\\n        - tree_size (int): The size of the Merkle tree.\\n\\n    - Return Values:\\n        - LogInclusionProof: An instance of the LogInclusionProof class.\\n\\n    - Error Handling:\\n        - ValueError: If the log index or tree size is negative or if the log index is greater than or equal to the tree size, a ValueError is raised.\\n\\n3. Error Handling:\\n    - ValueError: If the log index or tree size is negative or if the log index is greater than or equal to the tree size, a ValueError is raised.\\n\\nFunction code:\\n\\ndef generate_proof(\\n    log_index: int,\\n    tree_size: int,\\n    root_hash: str,\\n    hashes: List[str],\\n    checkpoint: str,\\n) -> LogInclusionProof:\\n    \"\"\"\\n    Generate an inclusion proof for a Merkle tree.\\n\\n    Args:\\n        log_index (int): The index of the log in the Merkle tree.\\n        tree_size (int): The size of the Merkle tree.\\n        root_hash (str): The root hash of the Merkle tree.\\n        hashes (list): A list of hashes in the inclusion proof.\\n        checkpoint (str): The checkpoint of the Merkle tree.\\n\\n    Returns:\\n        LogInclusionProof: An instance of the LogInclusionProof class.\\n\\n    Raises:\\n        ValueError: If the log index or tree size is negative or if the log index is greater than or equal to the tree size, a ValueError is raised.\\n    \"\"\"\\n    return LogInclusionProof(\\n        log_index=log_index,\\n        tree_size=tree_size,\\n        root_hash=root_hash,\\n        hashes=hashes,\\n        checkpoint=checkpoint,\\n    )\\n\\nFunction Documentation:\\n\\n1. Introduction: This function is used to generate an inclusion proof for a Merkle tree. It takes in the log index, tree size, root hash, hashes, and checkpoint of the Merkle tree and returns an instance of the LogInclusionProof class.\\n\\n2. Parameters:\\n    - log_index (int): The index of the log in the Merkle tree.\\n    - tree_size (int): The size of the Merkle tree.\\n    - root_hash (str): The root hash of the Merkle tree.\\n    - hashes (list): A list of hashes in the inclusion proof.\\n    - checkpoint (str): The checkpoint of the Merkle tree.\\n\\n3. Return Values:\\n    - LogInclusionProof: An instance of the LogInclusionProof class.\\n\\n4. Error Handling:\\n    - ValueError: If the log index or tree size is negative or if the log index is greater than or equal to the tree size, a ValueError is raised.\\n\\nScript code:\\n\\nimport os\\nimport sys\\n\\nfrom pydantic import BaseModel, Field\\nfrom pydantic.dataclasses import dataclass\\nfrom pydantic.fields import ConfigDict, ValidationInfo, field_validator\\nfrom typing import Any, List, StrictInt, StrictStr\\n\\n\\nclass LogInclusionProof(BaseModel):\\n    \\n\\n    model_config = ConfigDict(populate_by_name=True)\\n\\n    checkpoint: StrictStr = Field(..., alias=\"checkpoint\")\\n    hashes: List[StrictStr] = Field(..., alias=\"hashes\")\\n    log_index: StrictInt = Field(..., alias=\"logIndex\")\\n    root_hash: StrictStr = Field(..., alias=\"rootHash\")\\n    tree_size: StrictInt = Field(..., alias=\"treeSize\")\\n\\n    @field_validator(\"log_index\")\\n    def _log_index_positive(cls, v: int) -> int:\\n        if v < 0:\\n            raise ValueError(f\"Inclusion proof has invalid log index: {v} < 0\")\\n        return v\\n\\n    @field_validator(\"tree_size\")\\n    def _tree_size_positive(cls, v: int) -> int:\\n        if v < 0:\\n            raise ValueError(f\"Inclusion proof has invalid tree size: {v} < 0\")\\n        return v\\n\\n    @field_validator(\"tree_size\")\\n    def _log' generated_token_count=1024 generated_tokens=None input_text=None input_token_count=505 input_tokens=None moderation=None seed=133181325.0 stop_reason='max_tokens' stop_sequence=None\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1',api_key, openai_key, 'transparency', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "ef9f94eb-f9c5-47b2-851c-5eb009fb9397",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class LogInclusionProof(BaseModel):\n",
      "    \n",
      "\n",
      "    model_config = ConfigDict(populate_by_name=True)\n",
      "\n",
      "    checkpoint: StrictStr = Field(..., alias=\"checkpoint\")\n",
      "    hashes: List[StrictStr] = Field(..., alias=\"hashes\")\n",
      "    log_index: StrictInt = Field(..., alias=\"logIndex\")\n",
      "    root_hash: StrictStr = Field(..., alias=\"rootHash\")\n",
      "    tree_size: StrictInt = Field(..., alias=\"treeSize\")\n",
      "\n",
      "    @field_validator(\"log_index\")\n",
      "    def _log_index_positive(cls, v: int) -> int:\n",
      "        if v < 0:\n",
      "            raise ValueError(f\"Inclusion proof has invalid log index: {v} < 0\")\n",
      "        return v\n",
      "\n",
      "    @field_validator(\"tree_size\")\n",
      "    def _tree_size_positive(cls, v: int) -> int:\n",
      "        if v < 0:\n",
      "            raise ValueError(f\"Inclusion proof has invalid tree size: {v} < 0\")\n",
      "        return v\n",
      "\n",
      "    @field_validator(\"tree_size\")\n",
      "    def _log_index_within_tree_size(\n",
      "        cls, v: int, info: ValidationInfo, **kwargs: Any\n",
      "    ) -> int:\n",
      "        if \"log_index\" in info.data and v <= info.data[\"log_index\"]:\n",
      "            raise ValueError(\n",
      "                \"Inclusion proof has log index greater than or equal to tree size: \"\n",
      "                f\"{v} <= {info.data['log_index']}\"\n",
      "            )\n",
      "        return v\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "0abde7aa-8368-4961-9efb-c7bb4439f745",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 5 - The generated documentation accurately describes the purpose of the API class and function, as well as the parameters, return values, and error handling.\n",
      "\n",
      "Relevance: 5 - The generated documentation is relevant as it provides accurate and specific information about the class and function, including their purpose, parameters, return values, and error handling.\n",
      "\n",
      "Clarity: 5 - The generated documentation is clear and easy to understand. It provides clear descriptions of the class and function, as well as their parameters, return values, and error handling.\n",
      "\n",
      "Completeness: 5 - The generated documentation is complete as it includes all the necessary information about the class and function, including their purpose, parameters, return values, and error handling.\n",
      "\n",
      "Readability: 5 - The generated documentation is highly readable. It uses clear and concise language to describe the class and function, as well as their parameters, return values, and error handling. The formatting and organization of the documentation is also clean and easy to follow.\n",
      "\n",
      "Overall Score: 5\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "id": "78c55090-65f9-4a53-82f2-83220cfe9b27",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9592c24e-57a6-4eb4-9029-cce09709ee75",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "818c116b-711b-47e1-aa1f-3542b736fc5f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission should be helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides a detailed explanation of the class and function in the provided Python code. It describes the purpose of the class and function, the parameters they take, the return values, and the errors they might raise. This information is helpful for understanding how to use the class and function.\\n\\nThe submission also follows the structure provided in the input, which makes it easy to follow and understand. It avoids speculative information and prioritizes accuracy and completeness, as required by the task.\\n\\nTherefore, the submission meets the criterion of being helpful, insightful, and appropriate.\\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The submission is being evaluated for correctness, accuracy, and factualness. \\n\\n1. Correctness: The submission correctly describes the purpose of the class and function, their parameters, return values, and error handling. The descriptions match the provided Python code and the reference documentation. The submission also correctly follows the requested output structure.\\n\\n2. Accuracy: The submission accurately describes the class and function. The descriptions of the parameters, return values, and error handling are accurate and match the provided Python code and the reference documentation.\\n\\n3. Factualness: The submission is factual and does not include speculative information. The descriptions are based on the provided Python code and the reference documentation.\\n\\nBased on these evaluations, the submission meets all the criteria. \\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The criteria is to assess if the output is complete and captures all required fields. \\n\\nLooking at the submission, it provides documentation for the class and function provided in the input. \\n\\nFor the class, it provides an introduction, describes the function, lists and describes the parameters, specifies the return values, and describes possible error responses. \\n\\nFor the function, it provides an introduction, lists and describes the parameters, specifies the return values, and describes possible error responses. \\n\\nThe submission also provides documentation for the script code, but the script code is cut off and not complete. \\n\\nHowever, the criteria only asks if the output is complete and captures all required fields. The output does capture all required fields for the class and function. The script code is not a required field, so its incompleteness does not affect the criteria. \\n\\nTherefore, the submission meets the criteria. \\n\\nY', 'value': 'Y', 'score': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_600/2635151755.py:38: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df = df.append(new_row, ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "488357b4-963a-4d01-bebd-fd0dc90d75ad",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 NaN                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      NaN                    5.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               NaN                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               NaN                1.0           NaN  "
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "id": "d8981d30-c402-4c7c-8f08-4f186cca7158",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[2, 'human_accuracy_score'] = '2.0'\n",
    "df.at[2, 'human_relevance_score'] = '3.0'\n",
    "df.at[2, 'human_clarity_score'] = '3.0'\n",
    "df.at[2, 'human_completeness_score'] = '2.0'\n",
    "df.at[2, 'human_readability_score'] = '3.0'\n",
    "df.at[2, 'human_helpfulness'] = '1.0'\n",
    "df.at[2, 'human_correctness'] = '0.0'\n",
    "df.at[2, 'human_logical'] = '1.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "b8fd95e0-bf0b-42ba-b297-058a58d99569",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  "
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf2bf401-609b-42da-af02-aa6aade21dc5",
   "metadata": {},
   "source": [
    "**Interpretation**\n",
    "\n",
    "This is an interesting case where the generated output correctly captures the conditions being checked for in the given class but it also halluicinates function code. The answer is still however correct in places for example it correctly captured that\n",
    "\n",
    "```\n",
    "_log_index_positive ensures that the log_index value is non-negative.\n",
    "_tree_size_positive ensures that the tree_size value is non-negative.\n",
    "_log_index_within_tree_size ensures that the log_index is within the range of the tree_size.\n",
    "```\n",
    "\n",
    "but in some places it is made up and inaccurate and langchain evaluation was not able to assess that correctly"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5430700f-9668-4e3c-a09c-099bc94f5de2",
   "metadata": {},
   "source": [
    "### Example 4 - Do not Re-run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "id": "21c4fef4-cf22-435c-9aba-54fe7ab1fcc3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated_text='1. Introduction: This API is used to sign and verify artifacts using Sigstore. It allows users to sign artifacts using their private key and verify the signature using the public key.\\n2. Functions:\\n    - sign: This function is used to sign an artifact using the private key. It takes an input stream as an argument and returns a SigningResult object.\\n    - verify: This function is used to verify the signature of an artifact using the public key. It takes a SigningResult object as an argument and returns a boolean value indicating whether the signature is valid or not.\\n3. Error Handling:\\n    - ExpiredIdentity: This error is raised when the provided identity token is expired.\\n    - ExpiredCertificate: This error is raised when the provided certificate is expired.\\n    - InvalidCertificate: This error is raised when the provided certificate is invalid.\\n    - InvalidSignature: This error is raised when the provided signature is invalid.\\n    - InvalidCertificateChain: This error is raised when the provided certificate chain is invalid.\\n    - InvalidTransparencyLogEntry: This error is raised when the provided transparency log entry is invalid.\\n    - InvalidBundle: This error is raised when the provided bundle is invalid.\\n\\nMake sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.' generated_token_count=284 generated_tokens=None input_text=None input_token_count=2219 input_tokens=None moderation=None seed=411052976.0 stop_reason='eos_token' stop_sequence=None\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1', api_key, openai_key, 'sign', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "id": "7ca742b4-d95b-4c01-95df-5f92f6daa3a2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class Signer:\n",
      "    \n",
      "\n",
      "    def __init__(\n",
      "        self,\n",
      "        identity_token: IdentityToken,\n",
      "        signing_ctx: SigningContext,\n",
      "        cache: bool = True,\n",
      "    ) -> None:\n",
      "        \n",
      "        self._identity_token = identity_token\n",
      "        self._signing_ctx: SigningContext = signing_ctx\n",
      "        self.__cached_private_key: Optional[ec.EllipticCurvePrivateKey] = None\n",
      "        self.__cached_signing_certificate: Optional[\n",
      "            FulcioCertificateSigningResponse\n",
      "        ] = None\n",
      "        if cache:\n",
      "            logger.debug(\"Generating ephemeral keys...\")\n",
      "            self.__cached_private_key = ec.generate_private_key(ec.SECP256R1())\n",
      "            logger.debug(\"Requesting ephemeral certificate...\")\n",
      "            self.__cached_signing_certificate = self._signing_cert(self._private_key)\n",
      "\n",
      "    @property\n",
      "    def _private_key(self) -> ec.EllipticCurvePrivateKey:\n",
      "        \n",
      "        if self.__cached_private_key is None:\n",
      "            logger.debug(\"no cached key; generating ephemeral key\")\n",
      "            return ec.generate_private_key(ec.SECP256R1())\n",
      "        return self.__cached_private_key\n",
      "\n",
      "    def _signing_cert(\n",
      "        self,\n",
      "        private_key: ec.EllipticCurvePrivateKey,\n",
      "    ) -> FulcioCertificateSigningResponse:\n",
      "        \n",
      "        # If it exists, verify if the current certificate is expired\n",
      "        if self.__cached_signing_certificate:\n",
      "            not_valid_after = self.__cached_signing_certificate.cert.not_valid_after\n",
      "            not_valid_after_tzutc = not_valid_after.replace(tzinfo=timezone.utc)\n",
      "            if datetime.now(timezone.utc) > not_valid_after_tzutc:\n",
      "                raise ExpiredCertificate\n",
      "            return self.__cached_signing_certificate\n",
      "\n",
      "        else:\n",
      "            logger.debug(\"Retrieving signed certificate...\")\n",
      "\n",
      "            # Build an X.509 Certificiate Signing Request\n",
      "            builder = (\n",
      "                x509.CertificateSigningRequestBuilder()\n",
      "                .subject_name(\n",
      "                    x509.Name(\n",
      "                        [\n",
      "                            x509.NameAttribute(\n",
      "                                NameOID.EMAIL_ADDRESS, self._identity_token._identity\n",
      "                            ),\n",
      "                        ]\n",
      "                    )\n",
      "                )\n",
      "                .add_extension(\n",
      "                    x509.BasicConstraints(ca=False, path_length=None),\n",
      "                    critical=True,\n",
      "                )\n",
      "            )\n",
      "            certificate_request = builder.sign(private_key, hashes.SHA256())\n",
      "\n",
      "            certificate_response = self._signing_ctx._fulcio.signing_cert.post(\n",
      "                certificate_request, self._identity_token\n",
      "            )\n",
      "\n",
      "            return certificate_response\n",
      "\n",
      "    def sign(\n",
      "        self,\n",
      "        input_: IO[bytes],\n",
      "    ) -> SigningResult:\n",
      "        \n",
      "        input_digest = sha256_streaming(input_)\n",
      "        private_key = self._private_key\n",
      "\n",
      "        if not self._identity_token.in_validity_period():\n",
      "            raise ExpiredIdentity\n",
      "\n",
      "        try:\n",
      "            certificate_response = self._signing_cert(private_key)\n",
      "        except ExpiredCertificate as e:\n",
      "            raise e\n",
      "\n",
      "        # TODO(alex): Retrieve the public key via TUF\n",
      "        #\n",
      "        # Verify the SCT\n",
      "        sct = certificate_response.sct  # noqa\n",
      "        cert = certificate_response.cert  # noqa\n",
      "        chain = certificate_response.chain\n",
      "\n",
      "        verify_sct(sct, cert, chain, self._signing_ctx._rekor._ct_keyring)\n",
      "\n",
      "        logger.debug(\"Successfully verified SCT...\")\n",
      "\n",
      "        # Sign artifact\n",
      "        artifact_signature = private_key.sign(\n",
      "            input_digest, ec.ECDSA(Prehashed(hashes.SHA256()))\n",
      "        )\n",
      "        b64_artifact_signature = B64Str(base64.b64encode(artifact_signature).decode())\n",
      "\n",
      "        # Prepare inputs\n",
      "        b64_cert = base64.b64encode(\n",
      "            cert.public_bytes(encoding=serialization.Encoding.PEM)\n",
      "        )\n",
      "\n",
      "        # Create the transparency log entry\n",
      "        proposed_entry = sigstore_rekor_types.Hashedrekord(\n",
      "            kind=\"hashedrekord\",\n",
      "            api_version=\"0.0.1\",\n",
      "            spec=sigstore_rekor_types.HashedrekordV001Schema(\n",
      "                signature=sigstore_rekor_types.Signature1(\n",
      "                    content=b64_artifact_signature,\n",
      "                    public_key=sigstore_rekor_types.PublicKey1(\n",
      "                        content=b64_cert.decode()\n",
      "                    ),\n",
      "                ),\n",
      "                data=sigstore_rekor_types.Data(\n",
      "                    hash=sigstore_rekor_types.Hash(\n",
      "                        algorithm=sigstore_rekor_types.Algorithm.SHA256,\n",
      "                        value=input_digest.hex(),\n",
      "                    )\n",
      "                ),\n",
      "            ),\n",
      "        )\n",
      "        entry = self._signing_ctx._rekor.log.entries.post(proposed_entry)\n",
      "\n",
      "        logger.debug(f\"Transparency log entry created with index: {entry.log_index}\")\n",
      "\n",
      "        return SigningResult(\n",
      "            input_digest=HexStr(input_digest.hex()),\n",
      "            cert_pem=PEMCert(\n",
      "                cert.public_bytes(encoding=serialization.Encoding.PEM).decode()\n",
      "            ),\n",
      "            b64_signature=B64Str(b64_artifact_signature),\n",
      "            log_entry=entry,\n",
      "        )\n",
      "class SigningContext:\n",
      "    \n",
      "\n",
      "    def __init__(\n",
      "        self,\n",
      "        *,\n",
      "        fulcio: FulcioClient,\n",
      "        rekor: RekorClient,\n",
      "    ):\n",
      "        \n",
      "        self._fulcio = fulcio\n",
      "        self._rekor = rekor\n",
      "\n",
      "    @classmethod\n",
      "    def production(cls) -> SigningContext:\n",
      "        \n",
      "        updater = TrustUpdater.production()\n",
      "        rekor = RekorClient.production(updater)\n",
      "        return cls(\n",
      "            fulcio=FulcioClient.production(),\n",
      "            rekor=rekor,\n",
      "        )\n",
      "\n",
      "    @classmethod\n",
      "    def staging(cls) -> SigningContext:\n",
      "        \n",
      "        updater = TrustUpdater.staging()\n",
      "        rekor = RekorClient.staging(updater)\n",
      "        return cls(\n",
      "            fulcio=FulcioClient.staging(),\n",
      "            rekor=rekor,\n",
      "        )\n",
      "\n",
      "    @contextmanager\n",
      "    def signer(\n",
      "        self, identity_token: IdentityToken, *, cache: bool = True\n",
      "    ) -> Iterator[Signer]:\n",
      "        \n",
      "        yield Signer(identity_token, self, cache)\n",
      "class SigningResult(BaseModel):\n",
      "    \n",
      "\n",
      "    input_digest: HexStr\n",
      "    \n",
      "\n",
      "    cert_pem: PEMCert\n",
      "    \n",
      "\n",
      "    b64_signature: B64Str\n",
      "    \n",
      "\n",
      "    log_entry: LogEntry\n",
      "    \n",
      "\n",
      "    def to_bundle(self) -> Bundle:\n",
      "        \n",
      "\n",
      "        # NOTE: We explicitly only include the leaf certificate in the bundle's \"chain\"\n",
      "        # here: the specs explicitly forbid the inclusion of the root certificate,\n",
      "        # and discourage inclusion of any intermediates (since they're in the root of\n",
      "        # trust already).\n",
      "        cert = x509.load_pem_x509_certificate(self.cert_pem.encode())\n",
      "        cert_der = cert.public_bytes(encoding=serialization.Encoding.DER)\n",
      "        chain = X509CertificateChain(certificates=[X509Certificate(raw_bytes=cert_der)])\n",
      "\n",
      "        inclusion_proof: InclusionProof | None = None\n",
      "        if self.log_entry.inclusion_proof is not None:\n",
      "            inclusion_proof = InclusionProof(\n",
      "                log_index=self.log_entry.inclusion_proof.log_index,\n",
      "                root_hash=bytes.fromhex(self.log_entry.inclusion_proof.root_hash),\n",
      "                tree_size=self.log_entry.inclusion_proof.tree_size,\n",
      "                hashes=[\n",
      "                    bytes.fromhex(h) for h in self.log_entry.inclusion_proof.hashes\n",
      "                ],\n",
      "                checkpoint=Checkpoint(\n",
      "                    envelope=self.log_entry.inclusion_proof.checkpoint\n",
      "                ),\n",
      "            )\n",
      "\n",
      "        tlog_entry = TransparencyLogEntry(\n",
      "            log_index=self.log_entry.log_index,\n",
      "            log_id=LogId(key_id=bytes.fromhex(self.log_entry.log_id)),\n",
      "            kind_version=KindVersion(kind=\"hashedrekord\", version=\"0.0.1\"),\n",
      "            integrated_time=self.log_entry.integrated_time,\n",
      "            inclusion_promise=InclusionPromise(\n",
      "                signed_entry_timestamp=base64.b64decode(\n",
      "                    self.log_entry.inclusion_promise\n",
      "                )\n",
      "            )\n",
      "            if self.log_entry.inclusion_promise\n",
      "            else None,\n",
      "            inclusion_proof=inclusion_proof,\n",
      "            canonicalized_body=base64.b64decode(self.log_entry.body),\n",
      "        )\n",
      "\n",
      "        material = VerificationMaterial(\n",
      "            x509_certificate_chain=chain,\n",
      "            tlog_entries=[tlog_entry],\n",
      "        )\n",
      "\n",
      "        bundle = Bundle(\n",
      "            media_type=\"application/vnd.dev.sigstore.bundle+json;version=0.2\",\n",
      "            verification_material=material,\n",
      "            message_signature=MessageSignature(\n",
      "                message_digest=HashOutput(\n",
      "                    algorithm=HashAlgorithm.SHA2_256,\n",
      "                    digest=bytes.fromhex(self.input_digest),\n",
      "                ),\n",
      "                signature=base64.b64decode(self.b64_signature),\n",
      "            ),\n",
      "        )\n",
      "\n",
      "        return bundle\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "b16cdfb2-94a7-42e4-aaa6-67c728376686",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately describes the purpose of the API and its functions. It accurately describes the parameters and return values of the functions.\n",
      "Relevance: 5 - The generated documentation is relevant as it provides information about how to use the API functions and what error handling is implemented.\n",
      "Clarity: 3 - The generated documentation provides clear descriptions of the purpose of the API and its functions. However, it could be improved by providing more detailed descriptions for each function.\n",
      "Completeness: 4 - The generated documentation includes the introduction, functions, and error handling sections as required. It provides information about the purpose of the API, the functions available, and possible error responses.\n",
      "Readability: 5 - The generated documentation is readable and follows a clear structure. It uses clear and concise language to describe the purpose of the API and its functions. The sections are organized logically and are easy to understand.\n",
      "Overall Score: 4\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "abddcc41-d81a-47de-9942-99fd4acb2c4a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf1ba655-9a7b-4bb2-8559-68b2c5c5c1c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "94f268b1-d9d4-4730-bc9d-666250a9ee76",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission should be helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides a detailed explanation of the class and function in the provided Python code. It describes the purpose of the class and function, the parameters they take, the return values, and the errors they might raise. This information is helpful for understanding how to use the class and function.\\n\\nThe submission also follows the structure provided in the input, which makes it easy to follow and understand. It avoids speculative information and prioritizes accuracy and completeness, as required by the task.\\n\\nTherefore, the submission meets the criterion of being helpful, insightful, and appropriate.\\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The submission is being evaluated for correctness, accuracy, and factualness. \\n\\n1. Correctness: The submission correctly describes the purpose of the class and function, their parameters, return values, and error handling. The descriptions match the provided Python code and the reference documentation. The submission also correctly follows the requested output structure.\\n\\n2. Accuracy: The submission accurately describes the class and function. The descriptions of the parameters, return values, and error handling are accurate and match the provided Python code and the reference documentation.\\n\\n3. Factualness: The submission is factual and does not include speculative information. The descriptions are based on the provided Python code and the reference documentation.\\n\\nBased on these evaluations, the submission meets all the criteria. \\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The criteria is to assess if the output is complete and captures all required fields. \\n\\nLooking at the submission, it provides documentation for the class and function provided in the input. \\n\\nFor the class, it provides an introduction, describes the function, lists and describes the parameters, specifies the return values, and describes possible error responses. \\n\\nFor the function, it provides an introduction, lists and describes the parameters, specifies the return values, and describes possible error responses. \\n\\nThe submission also provides documentation for the script code, but the script code is cut off and not complete. \\n\\nHowever, the criteria only asks if the output is complete and captures all required fields. The output does capture all required fields for the class and function. The script code is not a required field, so its incompleteness does not affect the criteria. \\n\\nTherefore, the submission meets the criteria. \\n\\nY', 'value': 'Y', 'score': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_600/2635151755.py:38: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df = df.append(new_row, ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "id": "f134462b-15b7-4bdc-8d14-f5d4054edac1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 NaN                     4.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      NaN                    5.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               NaN                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               NaN                0.0           NaN  "
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "id": "fcd57416-5f7d-4bd2-8e32-5145701188e1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[3, 'human_accuracy_score'] = '1.0'\n",
    "df.at[3, 'human_relevance_score'] = '1.0'\n",
    "df.at[3, 'human_clarity_score'] = '1.0'\n",
    "df.at[3, 'human_completeness_score'] = '1.0'\n",
    "df.at[3, 'human_readability_score'] = '2.0'\n",
    "df.at[3, 'human_helpfulness'] = '0.0'\n",
    "df.at[3, 'human_correctness'] = '0.0'\n",
    "df.at[3, 'human_logical'] = '0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "b4e2d7e3-e792-4a34-bb83-f1164e04d5d5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  "
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80095a8b-58fa-4e7c-be93-27808ae2fd41",
   "metadata": {},
   "source": [
    "**Interpretation**\n",
    "\n",
    "This is a great example where the generated output was quite terrible, hallucinated classes, incomplete list of classes, incorrect explanations. GPT scored them well but langchain correctly captured the errors and pointed out the mistakes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cdf80e7-b473-4c11-8dc5-3a4cc0e46883",
   "metadata": {},
   "source": [
    "### Example 5 - Do not Re-run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "99d62521-94e1-47a3-a6c9-09e3d2a03773",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Introduction:**\n",
      "\n",
      "The `LogInclusionProof` class represents an inclusion proof for a log entry in a Merkle tree. It is used to provide evidence that a particular log entry is included in the Merkle tree.\n",
      "\n",
      "**Functions:**\n",
      "\n",
      "1. `__init__()`:\n",
      "    \n",
      "    - **Description:** Initializes a new instance of the `LogInclusionProof` class.\n",
      "    - **Parameters:**\n",
      "        - None\n",
      "    - **Return Value:** None\n",
      "\n",
      "\n",
      "2. `__repr__()`:\n",
      "    \n",
      "    - **Description:** Returns a string representation of the `LogInclusionProof` class instance.\n",
      "    - **Parameters:**\n",
      "        - None\n",
      "    - **Return Value:** String representation of the `LogInclusionProof` class instance.\n",
      "\n",
      "\n",
      "3. `_log_index_positive(v: int) -> int`:\n",
      "    \n",
      "    - **Description:** Validates that the log index value is positive.\n",
      "    - **Parameters:**\n",
      "        - `v` (int): The log index value to be validated.\n",
      "    - **Return Value:** The validated log index value.\n",
      "    - **Raises:**\n",
      "        - ValueError: If the log index value is less than 0.\n",
      "      \n",
      "\n",
      "4. `_tree_size_positive(v: int) -> int`:\n",
      "    \n",
      "    - **Description:** Validates that the tree size value is positive.\n",
      "    - **Parameters:**\n",
      "        - `v` (int): The tree size value to be validated.\n",
      "    - **Return Value:** The validated tree size value.\n",
      "    - **Raises:**\n",
      "        - ValueError: If the tree size value is less than 0.\n",
      "      \n",
      "\n",
      "5. `_log_index_within_tree_size(v: int, info: ValidationInfo, **kwargs: Any) -> int`:\n",
      "    \n",
      "    - **Description:** Validates that the log index value is within the range of the tree size.\n",
      "    - **Parameters:**\n",
      "        - `v` (int): The log index value to be validated.\n",
      "        - `info` (ValidationInfo): Contains additional data needed for validation.\n",
      "        - `**kwargs` (Any): Additional keyword arguments.\n",
      "    - **Return Value:** The validated log index value.\n",
      "    - **Raises:**\n",
      "        - ValueError: If the log index value is greater than or equal to the tree size value or if the information data does not contain the log index value.\n",
      "\n",
      "**Error Handling:**\n",
      "\n",
      "- `ValueError`: Raised when the log index value is less than 0, when the tree size value is less than 0, or when the log index value is greater than or equal to the tree size value.\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response(\"OpenAI/gpt3.5\", api_key, openai_key, 'transparency', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "cf38a473-7828-443a-b080-30c47f55bcf8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class LogInclusionProof(BaseModel):\n",
      "    \n",
      "\n",
      "    model_config = ConfigDict(populate_by_name=True)\n",
      "\n",
      "    checkpoint: StrictStr = Field(..., alias=\"checkpoint\")\n",
      "    hashes: List[StrictStr] = Field(..., alias=\"hashes\")\n",
      "    log_index: StrictInt = Field(..., alias=\"logIndex\")\n",
      "    root_hash: StrictStr = Field(..., alias=\"rootHash\")\n",
      "    tree_size: StrictInt = Field(..., alias=\"treeSize\")\n",
      "\n",
      "    @field_validator(\"log_index\")\n",
      "    def _log_index_positive(cls, v: int) -> int:\n",
      "        if v < 0:\n",
      "            raise ValueError(f\"Inclusion proof has invalid log index: {v} < 0\")\n",
      "        return v\n",
      "\n",
      "    @field_validator(\"tree_size\")\n",
      "    def _tree_size_positive(cls, v: int) -> int:\n",
      "        if v < 0:\n",
      "            raise ValueError(f\"Inclusion proof has invalid tree size: {v} < 0\")\n",
      "        return v\n",
      "\n",
      "    @field_validator(\"tree_size\")\n",
      "    def _log_index_within_tree_size(\n",
      "        cls, v: int, info: ValidationInfo, **kwargs: Any\n",
      "    ) -> int:\n",
      "        if \"log_index\" in info.data and v <= info.data[\"log_index\"]:\n",
      "            raise ValueError(\n",
      "                \"Inclusion proof has log index greater than or equal to tree size: \"\n",
      "                f\"{v} <= {info.data['log_index']}\"\n",
      "            )\n",
      "        return v\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "805b56fa-18d5-4437-9e44-d896779d0ea6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 5 - The generated documentation accurately represents the code. All information from the code is correctly documented, including function descriptions, parameter descriptions, return values, and error handling.\n",
      "\n",
      "Relevance: 5 - The generated documentation is relevant to the code. It accurately describes the purpose and use of the API class, as well as each individual function.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear. It provides clear descriptions of each function and its purpose. However, the error handling description could be more specific about the exact scenarios in which each ValueError is raised.\n",
      "\n",
      "Completeness: 5 - The generated documentation is complete. It covers all the functions in the class, providing descriptions, parameter information, return values, and error handling for each.\n",
      "\n",
      "Readability: 4 - The generated documentation is readable. It uses clear language and follows a consistent structure. However, some of the descriptions could be more concise and the error handling descriptions could be more specific.\n",
      "\n",
      "Overall Score: 4.5\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "d4b5ecd3-2e43-4df0-9dfc-97024d9a7e5c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7df0dc2-3413-4e69-9b44-fe6a0718bbfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "id": "b436c9c0-9a12-4722-b9bf-79c6c84fd773",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission is to be evaluated based on whether it is helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides a detailed documentation of the `LogInclusionProof` class. It starts with an introduction that explains the purpose of the class. This is helpful for users who are not familiar with the class and its use.\\n\\nThe submission then documents each function in the class. For each function, it provides a description, lists and describes the parameters, and specifies the return value. This is helpful for users who want to understand how to use the functions and what to expect from them.\\n\\nThe submission also describes the possible error responses and their meanings. This is helpful for users who encounter errors and want to understand what they mean.\\n\\nOverall, the submission is helpful because it provides a comprehensive documentation of the `LogInclusionProof` class. It is insightful because it explains the purpose of the class and its functions, and it describes the possible error responses. It is appropriate because it follows the output structure specified in the task.\\n\\nTherefore, the submission meets the criterion of helpfulness.\\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The submission is being evaluated for correctness, accuracy, and factualness. \\n\\n1. Correctness: The submission correctly follows the structure of the API documentation as requested in the input. It provides an introduction, documents each function, and describes error handling. \\n\\n2. Accuracy: The submission accurately describes the functions and their parameters, return values, and possible errors. It correctly identifies the class as representing an inclusion proof for a log entry in a Merkle tree. \\n\\n3. Factualness: The submission is factual and does not include speculative information. It is based on the provided Python code and does not make assumptions beyond what is provided in the code.\\n\\nHowever, the submission missed documenting the class attributes (checkpoint, hashes, log_index, root_hash, tree_size) which are part of the API. This is a significant omission as these attributes are crucial for understanding the functionality of the class.\\n\\nY\\nN', 'value': 'N', 'score': 0}\n",
      "{'reasoning': 'The criteria for this task is to assess whether the output is complete and captures all required fields. \\n\\nLooking at the submission, the introduction provides a brief description of the purpose of the API and its intended use. \\n\\nThe functions are documented with a clear explanation of what each function does. The parameters for each function are listed and described, including data types and any constraints. The return values are specified with the data type and possible values returned. \\n\\nThe error handling section describes possible error responses and their meanings. \\n\\nHowever, the submission does not document the class attributes (checkpoint, hashes, log_index, root_hash, tree_size) which are also part of the API and should be documented according to the task instructions. \\n\\nTherefore, the submission does not meet the criteria as it does not capture all required fields. \\n\\nN', 'value': 'N', 'score': 0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_600/2635151755.py:38: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df = df.append(new_row, ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "id": "35f04195-ecb6-4691-92ee-a12dc49adc33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 NaN                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      NaN                    4.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               NaN                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               NaN                0.0           NaN  "
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "id": "342eca73-fa95-4a8d-bc4a-c7c13d8a1cf8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[4, 'human_accuracy_score'] = '2.0'\n",
    "df.at[4, 'human_relevance_score'] = '2.0'\n",
    "df.at[4, 'human_clarity_score'] = '3.0'\n",
    "df.at[4, 'human_completeness_score'] = '2.0'\n",
    "df.at[4, 'human_readability_score'] = '4.0'\n",
    "df.at[4, 'human_helpfulness'] = '0.0'\n",
    "df.at[4, 'human_correctness'] = '0.0'\n",
    "df.at[4, 'human_logical'] = '1.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "id": "37af86d2-b54b-4fbb-8991-bb4a19cea68e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  "
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a70af4-9a22-4b9b-96d2-9b1147f36ae3",
   "metadata": {},
   "source": [
    "**Interpretation**\n",
    "\n",
    "This is again a great example of where langchain evaluation is not fully correct. While the generated output is pretty good structurally well documenting classses of the class, it hallucinates functions that are not part of the class which is unacceptable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5b56fa0-133f-46f8-8db2-23b608e92bff",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Example 6 - Do not Re-run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "id": "592ea8ea-272f-41fc-9737-89873dcaef40",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Introduction:\n",
      "The Error class is a base class for all custom error classes in the API. It provides a common interface for handling and reporting errors. The Error class is not intended to be instantiated directly.\n",
      "\n",
      "The NetworkError class is a subclass of Error and represents an error that occurs when there is a network issue. It provides specific diagnostics and suggestions for resolving the issue.\n",
      "\n",
      "The TUFError class is a subclass of Error and represents an error that occurs in the context of The Update Framework (TUF). It provides additional context-specific diagnostics and suggestions for reporting the issue.\n",
      "\n",
      "The MetadataError class is a subclass of Error and represents an error that occurs when there is an issue with the metadata.\n",
      "\n",
      "The RootError class is a subclass of Error and represents an error that occurs when the root of trust cannot be established.\n",
      "\n",
      "2. Functions:\n",
      "- Error.diagnostics():\n",
      "    - Description: Returns a string with a general diagnostic message for the error.\n",
      "    - Parameters: None.\n",
      "    - Return Values: str - A general diagnostic message for the error.\n",
      "\n",
      "- Error.print_and_exit(raise_error=False):\n",
      "    - Description: Prints the diagnostic message for the error to stderr and exits the program with exit code 1. If raise_error=True, it also raises the error.\n",
      "    - Parameters:\n",
      "        - raise_error (bool): Indicates whether to raise the error or not. Default is False.\n",
      "    - Return Values: None.\n",
      " \n",
      "- NetworkError.diagnostics():\n",
      "    - Description: Returns a string with a diagnostic message for the network error.\n",
      "    - Parameters: None.\n",
      "    - Return Values: str - A diagnostic message for the network error.\n",
      "\n",
      "- TUFError.__init__(message: str):\n",
      "    - Description: Initializes a TUFError instance with the provided message.\n",
      "    - Parameters:\n",
      "        - message (str): The error message.\n",
      "    - Return Values: None.\n",
      "\n",
      "- TUFError.diagnostics():\n",
      "    - Description: Returns a string with a diagnostic message for the TUF error.\n",
      "    - Parameters: None.\n",
      "    - Return Values: str - A diagnostic message for the TUF error.\n",
      "\n",
      "- MetadataError.diagnostics():\n",
      "    - Description: Returns a string with a diagnostic message for the metadata error.\n",
      "    - Parameters: None.\n",
      "    - Return Values: str - A diagnostic message for the metadata error.\n",
      "\n",
      "- RootError.diagnostics():\n",
      "    - Description: Returns a string with a diagnostic message for the root error.\n",
      "    - Parameters: None.\n",
      "    - Return Values: str - A diagnostic message for the root error.\n",
      "\n",
      "3. Error Handling:\n",
      "- When an Error is raised, the diagnostic message can be accessed through the diagnostics() method.\n",
      "- If the print_and_exit() method is called with raise_error=True, the error is re-raised after printing the diagnostic message.\n",
      "- NetworkError indicates a network issue and suggests checking the internet connection.\n",
      "- TUFError provides additional diagnostic details specific to certain error types within TUF.\n",
      "- MetadataError represents an error with the metadata.\n",
      "- RootError indicates a failure to establish the root of trust.\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response(\"OpenAI/gpt3.5\", api_key, openai_key, 'errors', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "id": "b204d858-ddea-4f91-bdd8-8dd8265b7fb2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class Error(Exception):\n",
      "    \n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "\n",
      "        return An issue occurred.\n",
      "\n",
      "    def print_and_exit(self, raise_error: bool = False) -> None:\n",
      "        \n",
      "\n",
      "        remind_verbose = (\n",
      "            \"Raising original exception:\"\n",
      "            if raise_error\n",
      "            else \"For detailed error information, run sigstore with the `--verbose` flag.\"\n",
      "        )\n",
      "\n",
      "        print(f\"{self.diagnostics()}\\n{remind_verbose}\", file=sys.stderr)\n",
      "\n",
      "        if raise_error:\n",
      "            # don't want \"during handling another exception\"\n",
      "            self.__suppress_context__ = True\n",
      "            raise self\n",
      "\n",
      "        sys.exit(1)\n",
      "class NetworkError(Error):\n",
      "    \n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "\n",
      "        cause_ctx = (\n",
      "            f\n",
      "        Additional context:\n",
      "\n",
      "        {self.__cause__}\n",
      "        \n",
      "            if self.__cause__\n",
      "            else \"\"\n",
      "        )\n",
      "\n",
      "        return (\n",
      "            \\\n",
      "        A network issue occurred.\n",
      "\n",
      "        Check your internet connection and try again.\n",
      "        \n",
      "            + cause_ctx\n",
      "        )\n",
      "class TUFError(Error):\n",
      "    \n",
      "\n",
      "    def __init__(self, message: str):\n",
      "        \n",
      "        self.message = message\n",
      "\n",
      "    from tuf.api import exceptions\n",
      "\n",
      "    _details: Mapping[Any, str] = {\n",
      "        exceptions.DownloadError: NetworkError().diagnostics()\n",
      "    }\n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "        details = TUFError._details.get(\n",
      "            type(self.__context__),\n",
      "            \"Please report this issue at <https://github.com/sigstore/sigstore-python/issues/new>.\",\n",
      "        )\n",
      "\n",
      "        return f\\\n",
      "        {self.message}.\n",
      "\n",
      "        {details}\n",
      "        \n",
      "class MetadataError(Error):\n",
      "    \n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "        return f{str(self)}.\n",
      "class RootError(Error):\n",
      "    \n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "        return \\\n",
      "        Unable to establish root of trust.\n",
      "\n",
      "        This error may occur when the resources embedded in this distribution of sigstore-python are out of date.\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "id": "78869d03-22d3-42a8-8c84-cb4e98d9b6f3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately describes the purpose and functionality of each class and function. The details from the code are correctly reflected in the documentation.\n",
      "\n",
      "Relevance: 5 - The generated documentation is relevant as it provides clear and concise descriptions of each class and function, including their purpose, parameters, and return values. It also includes information on error handling.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear and easy to understand. The descriptions for each class and function provide sufficient detail to understand their purpose and functionality.\n",
      "\n",
      "Completeness: 5 - The generated documentation is complete and includes descriptions for all the classes and functions in the code. It also includes information on error handling and possible error responses.\n",
      "\n",
      "Readability: 5 - The generated documentation is well-structured and formatted, making it easy to read and understand. The information is presented in a clear and concise manner.\n",
      "\n",
      "Overall Score: 4.6\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "95862558-294a-4bd6-80d6-d22e9d335fcd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef16456f-066c-4d37-b40d-4d656d4e81af",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "id": "1b5832d1-0550-47b4-99e8-26d751e57992",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission is to be evaluated based on whether it is helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides a detailed and structured documentation for the provided Python code. It follows the output structure provided in the input, which includes an introduction, function documentation, and error handling.\\n\\nIn the introduction, the submission provides a brief description of the purpose of each class in the API. This is helpful for users to understand the purpose and intended use of each class.\\n\\nIn the function documentation, the submission documents each function in the classes, including a description of what the function does, the parameters it takes, and the values it returns. This is insightful as it provides users with a clear understanding of how to use each function.\\n\\nIn the error handling section, the submission describes possible error responses and their meanings. This is appropriate as it helps users understand what each error means and how to handle it.\\n\\nTherefore, the submission is helpful, insightful, and appropriate. It provides a clear, concise, accurate, and user-centric documentation for the provided Python code.\\n\\nBased on the above reasoning, the submission meets the criterion. \\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The submission is being evaluated for correctness, accuracy, and factualness. \\n\\n1. The submission correctly identifies the Error class as the base class for all custom error classes in the API. It also correctly describes the purpose of the Error class and its methods.\\n\\n2. The submission accurately describes the NetworkError, TUFError, MetadataError, and RootError classes as subclasses of the Error class. It also correctly describes the purpose of these classes and their methods.\\n\\n3. The submission correctly describes the parameters and return values of the methods in the Error class and its subclasses.\\n\\n4. The submission accurately describes the error handling process in the API, including the use of the diagnostics() method and the print_and_exit() method.\\n\\n5. The submission is factual and does not include any speculative information. It is based on the provided Python code and does not make any assumptions or predictions.\\n\\nBased on these points, the submission meets the criteria of correctness, accuracy, and factualness. \\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The criteria for this task is to assess whether the output is complete and captures all required fields. \\n\\nLooking at the submission, it is clear that the output is structured according to the given instructions. \\n\\n1. Introduction: The submission provides a brief description of the purpose of the API and its intended use. It describes the Error class and its subclasses, NetworkError, TUFError, MetadataError, and RootError.\\n\\n2. Functions: The submission documents each API function, including a description of what the function does, a list and description of each parameter, and the data type and possible values returned.\\n\\n3. Error Handling: The submission describes possible error responses and their meanings. It explains what happens when an Error is raised, and what each subclass of Error represents.\\n\\nTherefore, the submission is complete and captures all required fields.\\n\\nY', 'value': 'Y', 'score': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_600/2635151755.py:38: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  df = df.append(new_row, ignore_index=True)\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "id": "89f29b67-d5cc-4635-abff-52e487df49f1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 NaN                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      NaN                    5.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               NaN                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               NaN                1.0           NaN  "
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "id": "cac95917-d67e-47b8-975a-49ace9496400",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[5, 'human_accuracy_score'] = '5.0'\n",
    "df.at[5, 'human_relevance_score'] = '5.0'\n",
    "df.at[5, 'human_clarity_score'] = '5.0'\n",
    "df.at[5, 'human_completeness_score'] = '5.0'\n",
    "df.at[5, 'human_readability_score'] = '5.0'\n",
    "df.at[5, 'human_helpfulness'] = '1.0'\n",
    "df.at[5, 'human_correctness'] = '1.0'\n",
    "df.at[5, 'human_logical'] = '1.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "2d9d8a0d-99e6-4f96-a8f2-c29fbcc9d7ad",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  "
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b620961-5dec-45f3-8b3d-6d6c69d66b49",
   "metadata": {},
   "source": [
    "**Interpretation** \n",
    "\n",
    "The output generated is quite detailed and pretty accurate to a non SME and the langchain eval seems to be capturing that correctly too. The GPT eval is also pretty high that is consistent with the human eval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "f0a6da3b-f5cb-4470-a33e-7cc34763e689",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.to_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5adefb10-939f-4e96-9418-153eb7c7ec3a",
   "metadata": {},
   "source": [
    "### Example 7 - Do Not Rerun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e753989b-1238-4973-8e84-c99502c4b9ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# read the previous example results which were run above\n",
    "df = pd.read_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "87da8fa5-d712-4d6d-ac97-9c6b0dfce9c4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2b2961e8-47c3-4383-b02a-a5063588650d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated_text='\\nIntroduction:\\n\\nThis API is used to verify the identity of a certificate. It can be used to verify the identity of a certificate against a set of policies, such as the issuer, GitHub workflow trigger, or GitHub workflow SHA.\\n\\nFunctions:\\n\\nverify(cert: Certificate, policy: VerificationPolicy) -> VerificationResult:\\n\\nVerify the identity of the provided certificate against the specified policy.\\n\\nParameters:\\n\\ncert (Certificate): The certificate to verify.\\npolicy (VerificationPolicy): The policy to use for verification.\\n\\nReturn Values:\\n\\nVerificationResult: The result of the verification.\\n\\nError Handling:\\n\\nIf the certificate is not valid, the verification will fail with an error message.\\n\\nIf the policy is not valid, the verification will fail with an error message.\\n\\nIf the policy is valid but the certificate is not valid for the policy, the verification will fail with an error message.\\n\\nMake sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.' generated_token_count=234 generated_tokens=None input_text=None input_token_count=1284 input_tokens=None moderation=None seed=3784537628.0 stop_reason='eos_token' stop_sequence=None\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1', api_key, openai_key, 'verify_policy', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "50c721e4-6643-42b2-8c5a-ded17a0ab4f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class _SingleX509ExtPolicy(ABC):\n",
      "    \n",
      "\n",
      "    oid: ObjectIdentifier\n",
      "    \n",
      "\n",
      "    def __init__(self, value: str) -> None:\n",
      "        \n",
      "        self._value = value\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "        try:\n",
      "            ext = cert.extensions.get_extension_for_oid(self.oid).value\n",
      "        except ExtensionNotFound:\n",
      "            return VerificationFailure(\n",
      "                reason=(\n",
      "                    f\"Certificate does not contain {self.__class__.__name__} \"\n",
      "                    f\"({self.oid.dotted_string}) extension\"\n",
      "                )\n",
      "            )\n",
      "\n",
      "        # NOTE(ww): mypy is confused by the `Extension[ExtensionType]` returned\n",
      "        # by `get_extension_for_oid` above.\n",
      "        ext_value = ext.value.decode()  # type: ignore[attr-defined]\n",
      "        if ext_value != self._value:\n",
      "            return VerificationFailure(\n",
      "                reason=(\n",
      "                    f\"Certificate's {self.__class__.__name__} does not match \"\n",
      "                    f\"(got {ext_value}, expected {self._value})\"\n",
      "                )\n",
      "            )\n",
      "\n",
      "        return VerificationSuccess()\n",
      "class OIDCIssuer(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_ISSUER_OID\n",
      "class GitHubWorkflowTrigger(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_TRIGGER_OID\n",
      "class GitHubWorkflowSHA(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_SHA_OID\n",
      "class GitHubWorkflowName(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_NAME_OID\n",
      "class GitHubWorkflowRepository(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_REPOSITORY_OID\n",
      "class GitHubWorkflowRef(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_REF_OID\n",
      "class VerificationPolicy(Protocol):\n",
      "    \n",
      "\n",
      "    @abstractmethod\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "        raise NotImplementedError  # pragma: no cover\n",
      "class AnyOf:\n",
      "    \n",
      "\n",
      "    def __init__(self, children: list[VerificationPolicy]):\n",
      "        \n",
      "        self._children = children\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "        verified = any(child.verify(cert) for child in self._children)\n",
      "        if verified:\n",
      "            return VerificationSuccess()\n",
      "        else:\n",
      "            return VerificationFailure(\n",
      "                reason=f\"0 of {len(self._children)} policies succeeded\"\n",
      "            )\n",
      "class AllOf:\n",
      "    \n",
      "\n",
      "    def __init__(self, children: list[VerificationPolicy]):\n",
      "        \n",
      "\n",
      "        self._children = children\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "\n",
      "        # Without this, we'd consider empty lists of child policies trivially valid.\n",
      "        # This is almost certainly not what the user wants and is a potential\n",
      "        # source of API misuse, so we explicitly disallow it.\n",
      "        if len(self._children) < 1:\n",
      "            return VerificationFailure(reason=\"no child policies to verify\")\n",
      "\n",
      "        # NOTE(ww): We need the cast here because MyPy can't tell that\n",
      "        # `VerificationResult.__bool__` is invariant with\n",
      "        # `VerificationSuccess | VerificationFailure`.\n",
      "        results = [child.verify(cert) for child in self._children]\n",
      "        failures = [\n",
      "            cast(VerificationFailure, result).reason for result in results if not result\n",
      "        ]\n",
      "        if len(failures) > 0:\n",
      "            inner_reasons = \", \".join(failures)\n",
      "            return VerificationFailure(\n",
      "                reason=f\"{len(failures)} of {len(self._children)} policies failed: {inner_reasons}\"\n",
      "            )\n",
      "        return VerificationSuccess()\n",
      "class UnsafeNoOp:\n",
      "    \n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "\n",
      "        logger.warning(\n",
      "            \"unsafe (no-op) verification policy used! no verification performed!\"\n",
      "        )\n",
      "        return VerificationSuccess()\n",
      "class Identity:\n",
      "    \n",
      "\n",
      "    def __init__(self, *, identity: str, issuer: str):\n",
      "        \n",
      "\n",
      "        self._identity = identity\n",
      "        self._issuer = OIDCIssuer(issuer)\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "\n",
      "        issuer_verified: VerificationResult = self._issuer.verify(cert)\n",
      "        if not issuer_verified:\n",
      "            return issuer_verified\n",
      "\n",
      "        # Build a set of all valid identities.\n",
      "        san_ext = cert.extensions.get_extension_for_class(SubjectAlternativeName).value\n",
      "        all_sans = set(san_ext.get_values_for_type(RFC822Name))\n",
      "        all_sans.update(san_ext.get_values_for_type(UniformResourceIdentifier))\n",
      "        all_sans.update(\n",
      "            [\n",
      "                on.value.decode()\n",
      "                for on in san_ext.get_values_for_type(OtherName)\n",
      "                if on.type_id == _OTHERNAME_OID\n",
      "            ]\n",
      "        )\n",
      "\n",
      "        verified = self._identity in all_sans\n",
      "        if not verified:\n",
      "            return VerificationFailure(\n",
      "                reason=f\"Certificate's SANs do not match {self._identity}; actual SANs: {all_sans}\"\n",
      "            )\n",
      "\n",
      "        return VerificationSuccess()\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a5fb3f0f-f91f-4fd1-8098-baaedcaee37c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately describes the purpose of the API, the functions, and the parameters. It covers all the necessary information for understanding the API functionality.\n",
      "\n",
      "Relevance: 4 - The generated documentation is relevant as it provides information about the purpose of the API, its functions, and how to use them. It also includes information about error handling.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear and easy to understand. It uses clear language and provides explanations for each function and parameter. The error handling section also clarifies possible error responses.\n",
      "\n",
      "Completeness: 3 - The generated documentation covers the necessary information about the API's purpose, functions, parameters, return values, and error handling. However, it does not provide detailed descriptions for each policy class and their individual functions.\n",
      "\n",
      "Readability: 4 - The generated documentation is readable and well-structured. It uses proper formatting and organization to clearly present the information. The use of bullet points and headings improves readability.\n",
      "\n",
      "Overall Score: 4\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ec9d2114-67dc-4c6e-a8e2-d50773730eed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a3edd696-eef6-4094-adb9-e03169b32876",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/app-root/lib64/python3.9/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.chat_models.openai.ChatOpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import ChatOpenAI`.\n",
      "  warn_deprecated(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission is supposed to be helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides an introduction to the API, explaining its purpose and intended use. This is helpful for users who are not familiar with the API.\\n\\nThe submission also documents the \\'verify\\' function, including its description, parameters, and return values. This is insightful as it provides users with the necessary information to use the function.\\n\\nThe submission also describes possible error responses and their meanings, which is appropriate as it helps users understand what could go wrong when using the API.\\n\\nHowever, the submission does not cover all the classes and their methods provided in the input. It only documents the \\'verify\\' function without specifying which class it belongs to. This could lead to confusion for the users.\\n\\nTherefore, while the submission is somewhat helpful and appropriate, it is not entirely insightful as it does not provide complete information about the API.\\n\\nSo, the submission does not meet the criterion.', 'value': 'So, the submission does not meet the criterion.', 'score': None}\n",
      "{'reasoning': \"The submission is supposed to provide a detailed API documentation for the provided Python code. The code contains several classes and methods, and the documentation should cover all of them. \\n\\nLooking at the submission, it seems to have provided a general overview of the API and its purpose, which is correct. However, it has only documented one function, 'verify', and has not provided any information about the parameters of this function. \\n\\nThe submission also does not cover the other classes and methods in the code, such as '_SingleX509ExtPolicy', 'OIDCIssuer', 'GitHubWorkflowTrigger', 'GitHubWorkflowSHA', 'GitHubWorkflowName', 'GitHubWorkflowRepository', 'GitHubWorkflowRef', 'VerificationPolicy', 'AnyOf', 'AllOf', 'UnsafeNoOp', and 'Identity'. \\n\\nThe error handling section in the submission is also vague and does not provide specific details about the possible error responses and their meanings. \\n\\nTherefore, the submission is not correct, accurate, and factual as it does not provide a complete and detailed documentation for the provided Python code. \\n\\nN\", 'value': 'N', 'score': 0}\n",
      "{'reasoning': \"The criteria for this task is to assess if the output is logical and complete, and if it captures all required fields. \\n\\nLooking at the submission, it provides an introduction to the API and describes its purpose. It also documents a function, 'verify', and includes a description of what the function does, the parameters it takes, and the return values. \\n\\nHowever, the submission does not document all the classes and their methods provided in the input. The classes '_SingleX509ExtPolicy', 'OIDCIssuer', 'GitHubWorkflowTrigger', 'GitHubWorkflowSHA', 'GitHubWorkflowName', 'GitHubWorkflowRepository', 'GitHubWorkflowRef', 'VerificationPolicy', 'AnyOf', 'AllOf', 'UnsafeNoOp', and 'Identity' are not documented. \\n\\nThe submission also does not provide a detailed description of possible error responses and their meanings. It only provides a general statement that the verification will fail with an error message under certain conditions, but does not specify what these error messages might be.\\n\\nTherefore, the submission does not meet the criteria as it is not complete and does not capture all required fields.\\n\\nN\", 'value': 'N', 'score': 0}\n"
     ]
    }
   ],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8d3dcccc-4a44-42f0-a636-e2d45f8dda99",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/app-root/src/api-docs-generation/notebooks/evaluation/helper_functions.ipynb:140: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  \"def langchain_scores(generated_patch, prompt, actual_doc):\\n\",\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "06028177-572d-4eac-903b-9a59db8ac73d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API provides functiona...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to generate ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "0  \\nYou are an AI system specialized at generati...   \n",
       "1  \\nYou are an AI system specialized at generati...   \n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "0  \\nIntroduction:\\n\\nThis API provides functiona...                 4.0   \n",
       "1  1. Introduction: This API is used to generate ...                 4.0   \n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "0                  2.0                  5.0                   3.0   \n",
       "1                  1.0                  3.0                   1.0   \n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  NaN                  4.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "0                4.0                 4.0                     4.0   \n",
       "1                3.0                 1.0                     3.0   \n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 NaN                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "0                      4.0                    5.0                     5.0   \n",
       "1                      1.0                    4.0                     1.0   \n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      NaN                    4.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "0                    0.0               0.0                    0.0   \n",
       "1                    0.0               0.0                    0.0   \n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               NaN                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "0               0.0                0.0           0.0  \n",
       "1               0.0                0.0           0.0  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               NaN                0.0           NaN  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f0d1b812-1351-4fca-81b6-9263ea024446",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[6, 'human_accuracy_score'] = '2.0'\n",
    "df.at[6, 'human_relevance_score'] = '3.0'\n",
    "df.at[6, 'human_clarity_score'] = '3.0'\n",
    "df.at[6, 'human_completeness_score'] = '2.0'\n",
    "df.at[6, 'human_readability_score'] = '5.0'\n",
    "df.at[6, 'human_helpfulness'] = '0.0'\n",
    "df.at[6, 'human_correctness'] = '0.0'\n",
    "df.at[6, 'human_logical'] = '0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b4738766-3be1-4f3e-ac50-212864fd80cd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa8213cb-ff25-421b-80bd-e87a40687aaf",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Interpretation**\n",
    "\n",
    "This is a good example where Langchain correctly evaluates that the generated documentation is not relevant to the input Python code provided. GPT however, scored the generated output as quite high even though the generated documentation hallucinates functions that are not part of the class which is unacceptable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e003ae37-6ce2-463f-a80d-de9bbd1442c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df.to_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c3290ab-0b27-4fa9-8f7c-687a17231d24",
   "metadata": {},
   "source": [
    "### Example 8 - Do Not Rerun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b86d9300-df4c-4821-bc29-01fec96f72fc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "cf149bb3-63f9-44af-8cd3-ca20d6f10706",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ffcc7f93-a05a-43bb-9038-7a155245b523",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This class is used to represe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "2  \\nYou are an AI system specialized at generati...   \n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "2  1. Introduction: This class is used to represe...                 5.0   \n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "2                  2.0                  5.0                   3.0   \n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "2                5.0                 3.0                     5.0   \n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "2                      2.0                    5.0                     3.0   \n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "2                    1.0               1.0                    1.0   \n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "2               0.0                1.0           1.0  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "075c7cca-5b02-4530-81a2-6de4b7051c47",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Introduction:\n",
      "The API provides a set of classes and functions to perform X.509 certificate verification. X.509 is a standard format for public key certificates used in various security protocols. This API allows users to define their own verification policies by using different classes and functions provided.\n",
      "\n",
      "2. Functions:\n",
      "\n",
      "2.1. `_SingleX509ExtPolicy` class:\n",
      "   - Description: This abstract base class represents a single X.509 certificate extension policy. It provides a common structure for validating a specific extension in an X.509 certificate.\n",
      "   - Parameters:\n",
      "     - `value` (str): The expected value of the extension.\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.2. `OIDCIssuer` class:\n",
      "   - Description: This class represents an X.509 extension policy for the OIDC Issuer extension. It verifies that the OIDC Issuer extension in the X.509 certificate matches the expected value.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.3. `GitHubWorkflowTrigger` class:\n",
      "   - Description: This class represents an X.509 extension policy for the GitHub Workflow Trigger extension. It verifies that the GitHub Workflow Trigger extension in the X.509 certificate matches the expected value.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.4. `GitHubWorkflowSHA` class:\n",
      "   - Description: This class represents an X.509 extension policy for the GitHub Workflow SHA extension. It verifies that the GitHub Workflow SHA extension in the X.509 certificate matches the expected value.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.5. `GitHubWorkflowName` class:\n",
      "   - Description: This class represents an X.509 extension policy for the GitHub Workflow Name extension. It verifies that the GitHub Workflow Name extension in the X.509 certificate matches the expected value.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.6. `GitHubWorkflowRepository` class:\n",
      "   - Description: This class represents an X.509 extension policy for the GitHub Workflow Repository extension. It verifies that the GitHub Workflow Repository extension in the X.509 certificate matches the expected value.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.7. `GitHubWorkflowRef` class:\n",
      "   - Description: This class represents an X.509 extension policy for the GitHub Workflow Ref extension. It verifies that the GitHub Workflow Ref extension in the X.509 certificate matches the expected value.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.8. `VerificationPolicy` protocol:\n",
      "   - Description: This protocol defines the interface for X.509 certificate verification policies. It provides the `verify` method that must be implemented by all verification policies.\n",
      "   - Parameters:\n",
      "     - `cert` (Certificate): The X.509 certificate to verify.\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.9. `AnyOf` class:\n",
      "   - Description: This class represents a composite X.509 certificate verification policy that combines multiple verification policies using the logical OR operator. It verifies the certificate against any of the provided child policies.\n",
      "   - Parameters:\n",
      "     - `children` (list[VerificationPolicy]): The list of child policies to verify the certificate against.\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.10. `AllOf` class:\n",
      "   - Description: This class represents a composite X.509 certificate verification policy that combines multiple verification policies using the logical AND operator. It verifies the certificate against all of the provided child policies.\n",
      "   - Parameters:\n",
      "     - `children` (list[VerificationPolicy]): The list of child policies to verify the certificate against.\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.11. `UnsafeNoOp` class:\n",
      "   - Description: This class represents a no-op verification policy that performs no verification. It is intended for testing or debugging purposes and should not be used in production.\n",
      "   - Parameters:\n",
      "     - None\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "2.12. `Identity` class:\n",
      "   - Description: This class represents an X.509 certificate verification policy based on the identity of the certificate subject. It verifies that the certificate's subject alternative names (SANs) match the provided identity.\n",
      "   - Parameters:\n",
      "     - `identity` (str): The expected identity value.\n",
      "     - `issuer` (str): The expected OIDC issuer value.\n",
      "   - Return Value: An instance of the `VerificationResult` class.\n",
      "\n",
      "3. Error Handling:\n",
      "   - If the X.509 certificate does not contain a required extension, a `VerificationFailure` is returned with a corresponding error message.\n",
      "   - If the value of an extension in the X.509 certificate does not match the expected value, a `VerificationFailure` is returned with a corresponding error message.\n",
      "   - If none of the child policies of `AnyOf` or `AllOf` verify the certificate, a `VerificationFailure` is returned with a corresponding error message.\n",
      "   - If there are no child policies to verify in `AllOf`, a `VerificationFailure` is returned with a corresponding error message.\n",
      "   - If the `UnsafeNoOp` verification policy is used, a warning message is logged, and a `VerificationSuccess` is returned.\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('OpenAI/gpt3.5', api_key, openai_key, 'verify_policy', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f12707aa-fcf8-4886-88bb-1f472b3e6fe8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class _SingleX509ExtPolicy(ABC):\n",
      "    \n",
      "\n",
      "    oid: ObjectIdentifier\n",
      "    \n",
      "\n",
      "    def __init__(self, value: str) -> None:\n",
      "        \n",
      "        self._value = value\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "        try:\n",
      "            ext = cert.extensions.get_extension_for_oid(self.oid).value\n",
      "        except ExtensionNotFound:\n",
      "            return VerificationFailure(\n",
      "                reason=(\n",
      "                    f\"Certificate does not contain {self.__class__.__name__} \"\n",
      "                    f\"({self.oid.dotted_string}) extension\"\n",
      "                )\n",
      "            )\n",
      "\n",
      "        # NOTE(ww): mypy is confused by the `Extension[ExtensionType]` returned\n",
      "        # by `get_extension_for_oid` above.\n",
      "        ext_value = ext.value.decode()  # type: ignore[attr-defined]\n",
      "        if ext_value != self._value:\n",
      "            return VerificationFailure(\n",
      "                reason=(\n",
      "                    f\"Certificate's {self.__class__.__name__} does not match \"\n",
      "                    f\"(got {ext_value}, expected {self._value})\"\n",
      "                )\n",
      "            )\n",
      "\n",
      "        return VerificationSuccess()\n",
      "class OIDCIssuer(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_ISSUER_OID\n",
      "class GitHubWorkflowTrigger(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_TRIGGER_OID\n",
      "class GitHubWorkflowSHA(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_SHA_OID\n",
      "class GitHubWorkflowName(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_NAME_OID\n",
      "class GitHubWorkflowRepository(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_REPOSITORY_OID\n",
      "class GitHubWorkflowRef(_SingleX509ExtPolicy):\n",
      "    \n",
      "\n",
      "    oid = _OIDC_GITHUB_WORKFLOW_REF_OID\n",
      "class VerificationPolicy(Protocol):\n",
      "    \n",
      "\n",
      "    @abstractmethod\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "        raise NotImplementedError  # pragma: no cover\n",
      "class AnyOf:\n",
      "    \n",
      "\n",
      "    def __init__(self, children: list[VerificationPolicy]):\n",
      "        \n",
      "        self._children = children\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "        verified = any(child.verify(cert) for child in self._children)\n",
      "        if verified:\n",
      "            return VerificationSuccess()\n",
      "        else:\n",
      "            return VerificationFailure(\n",
      "                reason=f\"0 of {len(self._children)} policies succeeded\"\n",
      "            )\n",
      "class AllOf:\n",
      "    \n",
      "\n",
      "    def __init__(self, children: list[VerificationPolicy]):\n",
      "        \n",
      "\n",
      "        self._children = children\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "\n",
      "        # Without this, we'd consider empty lists of child policies trivially valid.\n",
      "        # This is almost certainly not what the user wants and is a potential\n",
      "        # source of API misuse, so we explicitly disallow it.\n",
      "        if len(self._children) < 1:\n",
      "            return VerificationFailure(reason=\"no child policies to verify\")\n",
      "\n",
      "        # NOTE(ww): We need the cast here because MyPy can't tell that\n",
      "        # `VerificationResult.__bool__` is invariant with\n",
      "        # `VerificationSuccess | VerificationFailure`.\n",
      "        results = [child.verify(cert) for child in self._children]\n",
      "        failures = [\n",
      "            cast(VerificationFailure, result).reason for result in results if not result\n",
      "        ]\n",
      "        if len(failures) > 0:\n",
      "            inner_reasons = \", \".join(failures)\n",
      "            return VerificationFailure(\n",
      "                reason=f\"{len(failures)} of {len(self._children)} policies failed: {inner_reasons}\"\n",
      "            )\n",
      "        return VerificationSuccess()\n",
      "class UnsafeNoOp:\n",
      "    \n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "\n",
      "        logger.warning(\n",
      "            \"unsafe (no-op) verification policy used! no verification performed!\"\n",
      "        )\n",
      "        return VerificationSuccess()\n",
      "class Identity:\n",
      "    \n",
      "\n",
      "    def __init__(self, *, identity: str, issuer: str):\n",
      "        \n",
      "\n",
      "        self._identity = identity\n",
      "        self._issuer = OIDCIssuer(issuer)\n",
      "\n",
      "    def verify(self, cert: Certificate) -> VerificationResult:\n",
      "        \n",
      "\n",
      "        issuer_verified: VerificationResult = self._issuer.verify(cert)\n",
      "        if not issuer_verified:\n",
      "            return issuer_verified\n",
      "\n",
      "        # Build a set of all valid identities.\n",
      "        san_ext = cert.extensions.get_extension_for_class(SubjectAlternativeName).value\n",
      "        all_sans = set(san_ext.get_values_for_type(RFC822Name))\n",
      "        all_sans.update(san_ext.get_values_for_type(UniformResourceIdentifier))\n",
      "        all_sans.update(\n",
      "            [\n",
      "                on.value.decode()\n",
      "                for on in san_ext.get_values_for_type(OtherName)\n",
      "                if on.type_id == _OTHERNAME_OID\n",
      "            ]\n",
      "        )\n",
      "\n",
      "        verified = self._identity in all_sans\n",
      "        if not verified:\n",
      "            return VerificationFailure(\n",
      "                reason=f\"Certificate's SANs do not match {self._identity}; actual SANs: {all_sans}\"\n",
      "            )\n",
      "\n",
      "        return VerificationSuccess()\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c1890670-07e1-40f8-bab8-17e52cfb9374",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately describes the purpose, functionality, parameters, and return values of each class and function. The information provided is consistent with the code.\n",
      "\n",
      "Relevance: 5 - The generated documentation is relevant as it provides insights into the purpose and usage of each class and function in the provided code.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear and explains the purpose, parameters, and return values of each class and function. The descriptions are concise and easy to understand.\n",
      "\n",
      "Completeness: 3 - The generated documentation covers the main classes and functions in the code. However, it could be improved by providing more details about the specific requirements and constraints of each parameter, as well as the possible values returned by each function.\n",
      "\n",
      "Readability: 4 - The generated documentation is readable and well-structured. The use of headings and bullet points helps organize the information, making it easier to navigate and understand.\n",
      "\n",
      "Overall Score: 4\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9da4334f-1323-45e4-8210-a0cb73847e5c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "de17a5ec-f588-48a5-9f28-9bc2f8c6a58f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission is evaluated based on whether it is helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides a detailed explanation of the API based on the provided Python code. It follows the structure outlined in the task, providing an introduction, documenting each function, and describing error handling.\\n\\nThe introduction gives a brief overview of the API\\'s purpose and its intended use. It explains that the API is for performing X.509 certificate verification, which is a standard format for public key certificates used in various security protocols.\\n\\nThe function documentation is thorough and detailed. Each class and function is explained, including its purpose, parameters, and return values. This information is crucial for understanding how to use the API and what to expect from each function.\\n\\nThe error handling section describes the possible error responses and their meanings. This is important for users to understand what could go wrong and how to handle these situations.\\n\\nOverall, the submission is helpful as it provides a comprehensive overview of the API, insightful as it explains the purpose and use of each function, and appropriate as it follows the structure outlined in the task and provides accurate information based on the provided Python code.\\n\\nTherefore, the submission meets the criterion. \\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The submission is being evaluated on the criterion of correctness, which includes accuracy and factualness.\\n\\n1. The introduction provided in the submission accurately describes the purpose of the API and its intended use. It correctly identifies the API as a set of classes and functions for performing X.509 certificate verification.\\n\\n2. The submission correctly documents each class and function in the provided Python code. For each class, it provides a clear description of what the class does, lists and describes the parameters, and specifies the return value. The descriptions are accurate and align with the provided Python code.\\n\\n3. The submission accurately describes the error handling in the provided Python code. It correctly identifies the different types of errors that can occur and how they are handled.\\n\\nBased on this analysis, the submission meets the criterion of correctness. It is accurate and factual, correctly documenting the provided Python code and its error handling. \\n\\nTherefore, the answer is:', 'value': 'Therefore, the answer is:', 'score': None}\n",
      "{'reasoning': 'The criteria for this task is to assess whether the output is logical and complete, and if it captures all required fields. \\n\\nLooking at the submission, it appears to be logically structured and complete. The submission has followed the structure provided in the input, including an introduction, function documentation, and error handling. \\n\\nIn the introduction, the submission provides a brief description of the purpose of the API and its intended use. \\n\\nIn the function documentation, the submission documents each class and function in the provided Python code. For each class, the submission includes a description explaining what the class does, lists and describes each parameter, and specifies the data type and possible values returned. \\n\\nIn the error handling section, the submission describes possible error responses and their meanings. \\n\\nTherefore, the submission meets the criteria as it is logical, complete, and captures all required fields. \\n\\nThe answer is: \\nY', 'value': 'Y', 'score': 1}\n"
     ]
    }
   ],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5cf8b3f2-265f-4a6f-9a27-0d307fb1a84f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/app-root/src/api-docs-generation/notebooks/evaluation/helper_functions.ipynb:140: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  \"def langchain_scores(generated_patch, prompt, actual_doc):\\n\",\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6772c540-be21-4a5e-b3dc-6f3d7e12cae8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 NaN                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      NaN                    4.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               NaN                    NaN   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               NaN                1.0           NaN  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cc016f0f-71da-4229-a196-3513c87d2c3a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[7, 'human_accuracy_score'] = '4.0'\n",
    "df.at[7, 'human_relevance_score'] = '4.0'\n",
    "df.at[7, 'human_clarity_score'] = '3.0'\n",
    "df.at[7, 'human_completeness_score'] = '4.0'\n",
    "df.at[7, 'human_readability_score'] = '5.0'\n",
    "df.at[7, 'human_helpfulness'] = '1.0'\n",
    "df.at[7, 'human_correctness'] = '1.0'\n",
    "df.at[7, 'human_logical'] = '1.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4ef7f56d-42e4-4d5b-98bc-78e8e3eb2ad1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2948f453-d053-44f1-b609-400255ed9bcb",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Interpretation**\n",
    "\n",
    "The generated documentation does a pretty good job at describing the classes/functions provided. Langchain scores 1 for helpfulness and logicalness however we see an undefined value for correctness (even though it considers it as fairly correct). These scores are in alignment with our human evaluated scores as well as the GPT evaluated scores. This is a good example where all the 3 scoring systems were similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d6aaf1f2-ab37-4ba4-a1ba-ffc62361454b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# save the newly added example\n",
    "df.to_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d67892-3e14-4666-9888-1d14a9530615",
   "metadata": {},
   "source": [
    "### Example 9 - Do Not Rerun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "279e9775-08fe-4129-999d-9ee5f493ebd6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "7a81974c-f0bc-487a-a73c-f2aea4355fa5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4b345b2e-6dc5-41a3-a95f-5993d7215858",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is used to sign and ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "3  \\nYou are an AI system specialized at generati...   \n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "3  1. Introduction: This API is used to sign and ...                 4.0   \n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "3                  1.0                  5.0                   1.0   \n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "3                3.0                 1.0                     4.0   \n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "3                      1.0                    5.0                     2.0   \n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "3                    0.0               0.0                    0.0   \n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "3               0.0                0.0           0.0  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e24d057b-1209-4e74-ac64-2275379b16e4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Class Signer**\n",
      "\n",
      "**Introduction:**\n",
      "The Signer class is used for signing artifacts using an elliptic curve private key. It interacts with the FulcioCertificateSigningResponse and SigningContext to generate and retrieve certificates for signing. The signing process involves verifying the identity token, retrieving the certificate, and signing the artifact. The Signer class also provides a method to create a transparency log entry for the signed artifact.\n",
      "\n",
      "**Methods:**\n",
      "\n",
      "1. \\_\\_init\\_\\_:\n",
      "   - Description: Initializes the Signer class with the identity token, signing context, and an optional cache flag.\n",
      "   - Parameters:\n",
      "     - identity_token (IdentityToken): An object representing the identity token.\n",
      "     - signing_ctx (SigningContext): An object representing the signing context.\n",
      "     - cache (bool, optional): Flag indicating whether to cache the private key and signing certificate. Defaults to True.\n",
      "   - Return: None\n",
      "\n",
      "2. \\_private_key:\n",
      "   - Description: Get the private key used for signing. If a cached private key exists, it returns the cached key; otherwise, it generates a new ephemeral key.\n",
      "   - Parameters: None\n",
      "   - Return: EllipticCurvePrivateKey\n",
      "\n",
      "3. \\_signing_cert:\n",
      "   - Description: Retrieves the signing certificate. If a cached signing certificate exists and is not expired, it returns the cached certificate; otherwise, it retrieves a new certificate by sending a signing certificate request to Fulcio service.\n",
      "   - Parameters:\n",
      "     - private_key (EllipticCurvePrivateKey): The private key used to sign the certificate request.\n",
      "   - Return: FulcioCertificateSigningResponse\n",
      "\n",
      "4. sign:\n",
      "   - Description: Signs the provided input artifact.\n",
      "   - Parameters:\n",
      "     - input_ (IO[bytes]): The input artifact to sign.\n",
      "   - Return: SigningResult\n",
      "\n",
      "\n",
      "**Class SigningContext**\n",
      "\n",
      "**Introduction:**\n",
      "The SigningContext class represents the context for artifact signing. It contains references to the FulcioClient and RekorClient, which are used for interacting with the Fulcio and Rekor services, respectively.\n",
      "\n",
      "**Methods:**\n",
      "\n",
      "1. \\_\\_init\\_\\_:\n",
      "   - Description: Initializes the SigningContext class with the FulcioClient and RekorClient.\n",
      "   - Parameters:\n",
      "     - fulcio (FulcioClient): A client for interacting with the Fulcio service.\n",
      "     - rekor (RekorClient): A client for interacting with the Rekor service.\n",
      "   - Return: None\n",
      "\n",
      "2. production:\n",
      "   - Description: Creates a production SigningContext instance with production versions of FulcioClient and RekorClient.\n",
      "   - Parameters: None\n",
      "   - Return: SigningContext\n",
      "\n",
      "3. staging:\n",
      "   - Description: Creates a staging SigningContext instance with staging versions of FulcioClient and RekorClient.\n",
      "   - Parameters: None\n",
      "   - Return: SigningContext\n",
      "\n",
      "4. signer:\n",
      "   - Description: Creates a Signer instance within the signing context for signing artifacts.\n",
      "   - Parameters:\n",
      "     - identity_token (IdentityToken): An object representing the identity token.\n",
      "     - cache (bool, optional): Flag indicating whether to cache the private key and signing certificate. Defaults to True.\n",
      "   - Yield: Signer\n",
      "\n",
      "\n",
      "**Class SigningResult**\n",
      "\n",
      "**Introduction:**\n",
      "The SigningResult class represents the result of the signing process. It contains information about the signed artifact, including the artifact's input digest, the certificate PEM, the base64-encoded signature, and the transparency log entry.\n",
      "\n",
      "**Properties:**\n",
      "1. input_digest (HexStr): The hex-encoded input digest of the signed artifact.\n",
      "2. cert_pem (PEMCert): The PEM-encoded signing certificate.\n",
      "3. b64_signature (B64Str): The base64-encoded signature of the signed artifact.\n",
      "4. log_entry (LogEntry): The transparency log entry for the signed artifact.\n",
      "\n",
      "**Methods:**\n",
      "1. to_bundle:\n",
      "   - Description: Converts the SigningResult into a Bundle object, which can be used for verification and validation purposes.\n",
      "   - Parameters: None\n",
      "   - Return: Bundle\n",
      "\n",
      "The Bundle class and other referenced classes are not provided in the code above, so their documentation is omitted.\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('OpenAI/gpt3.5', api_key, openai_key, 'sign', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ebe7d73f-638b-4065-9396-23d32f8c1ead",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class Signer:\n",
      "    \n",
      "\n",
      "    def __init__(\n",
      "        self,\n",
      "        identity_token: IdentityToken,\n",
      "        signing_ctx: SigningContext,\n",
      "        cache: bool = True,\n",
      "    ) -> None:\n",
      "        \n",
      "        self._identity_token = identity_token\n",
      "        self._signing_ctx: SigningContext = signing_ctx\n",
      "        self.__cached_private_key: Optional[ec.EllipticCurvePrivateKey] = None\n",
      "        self.__cached_signing_certificate: Optional[\n",
      "            FulcioCertificateSigningResponse\n",
      "        ] = None\n",
      "        if cache:\n",
      "            logger.debug(\"Generating ephemeral keys...\")\n",
      "            self.__cached_private_key = ec.generate_private_key(ec.SECP256R1())\n",
      "            logger.debug(\"Requesting ephemeral certificate...\")\n",
      "            self.__cached_signing_certificate = self._signing_cert(self._private_key)\n",
      "\n",
      "    @property\n",
      "    def _private_key(self) -> ec.EllipticCurvePrivateKey:\n",
      "        \n",
      "        if self.__cached_private_key is None:\n",
      "            logger.debug(\"no cached key; generating ephemeral key\")\n",
      "            return ec.generate_private_key(ec.SECP256R1())\n",
      "        return self.__cached_private_key\n",
      "\n",
      "    def _signing_cert(\n",
      "        self,\n",
      "        private_key: ec.EllipticCurvePrivateKey,\n",
      "    ) -> FulcioCertificateSigningResponse:\n",
      "        \n",
      "        # If it exists, verify if the current certificate is expired\n",
      "        if self.__cached_signing_certificate:\n",
      "            not_valid_after = self.__cached_signing_certificate.cert.not_valid_after\n",
      "            not_valid_after_tzutc = not_valid_after.replace(tzinfo=timezone.utc)\n",
      "            if datetime.now(timezone.utc) > not_valid_after_tzutc:\n",
      "                raise ExpiredCertificate\n",
      "            return self.__cached_signing_certificate\n",
      "\n",
      "        else:\n",
      "            logger.debug(\"Retrieving signed certificate...\")\n",
      "\n",
      "            # Build an X.509 Certificiate Signing Request\n",
      "            builder = (\n",
      "                x509.CertificateSigningRequestBuilder()\n",
      "                .subject_name(\n",
      "                    x509.Name(\n",
      "                        [\n",
      "                            x509.NameAttribute(\n",
      "                                NameOID.EMAIL_ADDRESS, self._identity_token._identity\n",
      "                            ),\n",
      "                        ]\n",
      "                    )\n",
      "                )\n",
      "                .add_extension(\n",
      "                    x509.BasicConstraints(ca=False, path_length=None),\n",
      "                    critical=True,\n",
      "                )\n",
      "            )\n",
      "            certificate_request = builder.sign(private_key, hashes.SHA256())\n",
      "\n",
      "            certificate_response = self._signing_ctx._fulcio.signing_cert.post(\n",
      "                certificate_request, self._identity_token\n",
      "            )\n",
      "\n",
      "            return certificate_response\n",
      "\n",
      "    def sign(\n",
      "        self,\n",
      "        input_: IO[bytes],\n",
      "    ) -> SigningResult:\n",
      "        \n",
      "        input_digest = sha256_streaming(input_)\n",
      "        private_key = self._private_key\n",
      "\n",
      "        if not self._identity_token.in_validity_period():\n",
      "            raise ExpiredIdentity\n",
      "\n",
      "        try:\n",
      "            certificate_response = self._signing_cert(private_key)\n",
      "        except ExpiredCertificate as e:\n",
      "            raise e\n",
      "\n",
      "        # TODO(alex): Retrieve the public key via TUF\n",
      "        #\n",
      "        # Verify the SCT\n",
      "        sct = certificate_response.sct  # noqa\n",
      "        cert = certificate_response.cert  # noqa\n",
      "        chain = certificate_response.chain\n",
      "\n",
      "        verify_sct(sct, cert, chain, self._signing_ctx._rekor._ct_keyring)\n",
      "\n",
      "        logger.debug(\"Successfully verified SCT...\")\n",
      "\n",
      "        # Sign artifact\n",
      "        artifact_signature = private_key.sign(\n",
      "            input_digest, ec.ECDSA(Prehashed(hashes.SHA256()))\n",
      "        )\n",
      "        b64_artifact_signature = B64Str(base64.b64encode(artifact_signature).decode())\n",
      "\n",
      "        # Prepare inputs\n",
      "        b64_cert = base64.b64encode(\n",
      "            cert.public_bytes(encoding=serialization.Encoding.PEM)\n",
      "        )\n",
      "\n",
      "        # Create the transparency log entry\n",
      "        proposed_entry = sigstore_rekor_types.Hashedrekord(\n",
      "            kind=\"hashedrekord\",\n",
      "            api_version=\"0.0.1\",\n",
      "            spec=sigstore_rekor_types.HashedrekordV001Schema(\n",
      "                signature=sigstore_rekor_types.Signature1(\n",
      "                    content=b64_artifact_signature,\n",
      "                    public_key=sigstore_rekor_types.PublicKey1(\n",
      "                        content=b64_cert.decode()\n",
      "                    ),\n",
      "                ),\n",
      "                data=sigstore_rekor_types.Data(\n",
      "                    hash=sigstore_rekor_types.Hash(\n",
      "                        algorithm=sigstore_rekor_types.Algorithm.SHA256,\n",
      "                        value=input_digest.hex(),\n",
      "                    )\n",
      "                ),\n",
      "            ),\n",
      "        )\n",
      "        entry = self._signing_ctx._rekor.log.entries.post(proposed_entry)\n",
      "\n",
      "        logger.debug(f\"Transparency log entry created with index: {entry.log_index}\")\n",
      "\n",
      "        return SigningResult(\n",
      "            input_digest=HexStr(input_digest.hex()),\n",
      "            cert_pem=PEMCert(\n",
      "                cert.public_bytes(encoding=serialization.Encoding.PEM).decode()\n",
      "            ),\n",
      "            b64_signature=B64Str(b64_artifact_signature),\n",
      "            log_entry=entry,\n",
      "        )\n",
      "class SigningContext:\n",
      "    \n",
      "\n",
      "    def __init__(\n",
      "        self,\n",
      "        *,\n",
      "        fulcio: FulcioClient,\n",
      "        rekor: RekorClient,\n",
      "    ):\n",
      "        \n",
      "        self._fulcio = fulcio\n",
      "        self._rekor = rekor\n",
      "\n",
      "    @classmethod\n",
      "    def production(cls) -> SigningContext:\n",
      "        \n",
      "        updater = TrustUpdater.production()\n",
      "        rekor = RekorClient.production(updater)\n",
      "        return cls(\n",
      "            fulcio=FulcioClient.production(),\n",
      "            rekor=rekor,\n",
      "        )\n",
      "\n",
      "    @classmethod\n",
      "    def staging(cls) -> SigningContext:\n",
      "        \n",
      "        updater = TrustUpdater.staging()\n",
      "        rekor = RekorClient.staging(updater)\n",
      "        return cls(\n",
      "            fulcio=FulcioClient.staging(),\n",
      "            rekor=rekor,\n",
      "        )\n",
      "\n",
      "    @contextmanager\n",
      "    def signer(\n",
      "        self, identity_token: IdentityToken, *, cache: bool = True\n",
      "    ) -> Iterator[Signer]:\n",
      "        \n",
      "        yield Signer(identity_token, self, cache)\n",
      "class SigningResult(BaseModel):\n",
      "    \n",
      "\n",
      "    input_digest: HexStr\n",
      "    \n",
      "\n",
      "    cert_pem: PEMCert\n",
      "    \n",
      "\n",
      "    b64_signature: B64Str\n",
      "    \n",
      "\n",
      "    log_entry: LogEntry\n",
      "    \n",
      "\n",
      "    def to_bundle(self) -> Bundle:\n",
      "        \n",
      "\n",
      "        # NOTE: We explicitly only include the leaf certificate in the bundle's \"chain\"\n",
      "        # here: the specs explicitly forbid the inclusion of the root certificate,\n",
      "        # and discourage inclusion of any intermediates (since they're in the root of\n",
      "        # trust already).\n",
      "        cert = x509.load_pem_x509_certificate(self.cert_pem.encode())\n",
      "        cert_der = cert.public_bytes(encoding=serialization.Encoding.DER)\n",
      "        chain = X509CertificateChain(certificates=[X509Certificate(raw_bytes=cert_der)])\n",
      "\n",
      "        inclusion_proof: InclusionProof | None = None\n",
      "        if self.log_entry.inclusion_proof is not None:\n",
      "            inclusion_proof = InclusionProof(\n",
      "                log_index=self.log_entry.inclusion_proof.log_index,\n",
      "                root_hash=bytes.fromhex(self.log_entry.inclusion_proof.root_hash),\n",
      "                tree_size=self.log_entry.inclusion_proof.tree_size,\n",
      "                hashes=[\n",
      "                    bytes.fromhex(h) for h in self.log_entry.inclusion_proof.hashes\n",
      "                ],\n",
      "                checkpoint=Checkpoint(\n",
      "                    envelope=self.log_entry.inclusion_proof.checkpoint\n",
      "                ),\n",
      "            )\n",
      "\n",
      "        tlog_entry = TransparencyLogEntry(\n",
      "            log_index=self.log_entry.log_index,\n",
      "            log_id=LogId(key_id=bytes.fromhex(self.log_entry.log_id)),\n",
      "            kind_version=KindVersion(kind=\"hashedrekord\", version=\"0.0.1\"),\n",
      "            integrated_time=self.log_entry.integrated_time,\n",
      "            inclusion_promise=InclusionPromise(\n",
      "                signed_entry_timestamp=base64.b64decode(\n",
      "                    self.log_entry.inclusion_promise\n",
      "                )\n",
      "            )\n",
      "            if self.log_entry.inclusion_promise\n",
      "            else None,\n",
      "            inclusion_proof=inclusion_proof,\n",
      "            canonicalized_body=base64.b64decode(self.log_entry.body),\n",
      "        )\n",
      "\n",
      "        material = VerificationMaterial(\n",
      "            x509_certificate_chain=chain,\n",
      "            tlog_entries=[tlog_entry],\n",
      "        )\n",
      "\n",
      "        bundle = Bundle(\n",
      "            media_type=\"application/vnd.dev.sigstore.bundle+json;version=0.2\",\n",
      "            verification_material=material,\n",
      "            message_signature=MessageSignature(\n",
      "                message_digest=HashOutput(\n",
      "                    algorithm=HashAlgorithm.SHA2_256,\n",
      "                    digest=bytes.fromhex(self.input_digest),\n",
      "                ),\n",
      "                signature=base64.b64decode(self.b64_signature),\n",
      "            ),\n",
      "        )\n",
      "\n",
      "        return bundle\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "ced69d21-f85d-4e16-bcc8-0f6db0d3f684",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 4 - The generated documentation accurately describes the purpose and functionality of the Signer, SigningContext, and SigningResult classes. It correctly identifies the methods and properties of each class and provides accurate descriptions of their purposes.\n",
      "\n",
      "Relevance: 5 - The generated documentation is highly relevant to the code provided. It provides an overview of the purpose of each class and its intended use, as well as detailed descriptions of the methods and properties.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear and concise. It uses clear language to describe the purpose of each class and its methods. The descriptions of parameters and return values are also clear and provide relevant details.\n",
      "\n",
      "Completeness: 3 - The generated documentation is mostly complete, but there are some missing details. For example, the documentation could provide more information about the parameters and return values of each method, including their data types and any constraints. Additionally, the documentation mentions the existence of other referenced classes like Bundle, but does not provide any information about them.\n",
      "\n",
      "Readability: 4 - The generated documentation is readable and easy to understand. It uses a consistent structure and formatting, making it easy to navigate and find the relevant information.\n",
      "\n",
      "Overall Score: 4\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "e47a05e3-0e66-4798-98ce-84af2d286e4b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2a7bd370-3958-4ed9-85d4-48741e39b9ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission is to be evaluated based on whether it is helpful, insightful, and appropriate.\\n\\nLooking at the submission, it is clear that it provides a detailed explanation of the provided Python code. The submission has broken down the code into its individual classes and methods, and provided a detailed explanation for each. This includes the purpose of each class and method, the parameters they take, and the values they return. This is very helpful for someone trying to understand the code.\\n\\nThe submission is also insightful. It not only explains what the code does, but also provides context on how the different parts of the code interact with each other. For example, it explains how the Signer class interacts with the FulcioCertificateSigningResponse and SigningContext to generate and retrieve certificates for signing.\\n\\nThe submission is also appropriate. It follows the output structure provided in the task, and provides clear, concise, accurate, and user-centric documentation. It avoids speculative information and prioritizes accuracy and completeness.\\n\\nBased on this analysis, the submission meets the criterion of being helpful, insightful, and appropriate. Therefore, the answer is \"Y\". \\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The submission provides a detailed and accurate documentation for the provided Python classes: Signer, SigningContext, and SigningResult. It follows the required structure, providing an introduction for each class, and documenting each method including its description, parameters, and return values. The submission also correctly identifies and describes the properties of the SigningResult class. \\n\\nThe submission does not include speculative information and prioritizes accuracy and completeness. It correctly identifies and describes the data types and constraints of the parameters and return values. \\n\\nThe submission also correctly identifies and describes the possible error responses, such as ExpiredCertificate and ExpiredIdentity, in the Signer class.\\n\\nThe submission does not include documentation for the logger, which is used in the classes but is not a part of the classes themselves. This omission does not affect the correctness of the documentation for the provided classes.\\n\\nBased on these observations, the submission meets the criteria of correctness, accuracy, and factualness.\\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': 'The criteria for this task is to assess whether the output is logical and complete, and whether it captures all required fields. \\n\\nLooking at the submission, it appears to be logically structured and complete. The submission has provided documentation for all the classes provided in the input, namely the Signer, SigningContext, and SigningResult classes. \\n\\nFor each class, the submission has provided an introduction that describes the purpose of the class and its intended use. This meets the first requirement of the task.\\n\\nThe submission has also documented each method within the classes, including a description of what the method does, a list and description of each parameter, and the return values. This meets the second requirement of the task.\\n\\nThe submission does not include any error handling documentation. However, the provided Python code does not include any explicit error handling that would need to be documented, so this is not a shortcoming of the submission.\\n\\nThe submission has also avoided speculative information and prioritized accuracy and completeness, as per the task instructions.\\n\\nTherefore, the submission meets the criteria. \\n\\nY', 'value': 'Y', 'score': 1}\n"
     ]
    }
   ],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e3b19673-35aa-46af-b894-f7588e45b9dc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/app-root/src/api-docs-generation/notebooks/evaluation/helper_functions.ipynb:140: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  \"def langchain_scores(generated_patch, prompt, actual_doc):\\n\",\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5fee551f-0fd7-4cdb-9df0-110ecce8ff50",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Class Signer**\\n\\n**Introduction:**\\nThe Sig...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "8  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "8  **Class Signer**\\n\\n**Introduction:**\\nThe Sig...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "8                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "8                4.0                 NaN                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "8                      NaN                    4.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "8                    1.0               NaN                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  \n",
       "8               NaN                1.0           NaN  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b1239a92-e9df-47c4-ade4-5157636daaac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[8, 'human_accuracy_score'] = '3.0'\n",
    "df.at[8, 'human_relevance_score'] = '4.0'\n",
    "df.at[8, 'human_clarity_score'] = '3.0'\n",
    "df.at[8, 'human_completeness_score'] = '3.0'\n",
    "df.at[8, 'human_readability_score'] = '4.0'\n",
    "df.at[8, 'human_helpfulness'] = '1.0'\n",
    "df.at[8, 'human_correctness'] = '1.0'\n",
    "df.at[8, 'human_logical'] = '1.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3a792a43-aa20-41a5-adbc-8b9312ca2fe3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Class Signer**\\n\\n**Introduction:**\\nThe Sig...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "8  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "8  **Class Signer**\\n\\n**Introduction:**\\nThe Sig...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "8                  3.0                  5.0                   4.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "8                4.0                 3.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "8                      3.0                    4.0                     4.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "8                    1.0               1.0                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  \n",
       "8               1.0                1.0           1.0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f30fec4-f5da-4510-b955-35f47a694e3c",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Interpretation**\n",
    "\n",
    "Here, again we see that the generated documentation seems fairly accurate in describing the API code provided. Langchain scores are 1 for helpfulness, correctness and logicalness. Similarly, the human evaluated scores are in alignment with Langchain scores. However, it is important to note that the human evaluation scores for clarity/completeness are lower compare to the GPT evaluated scores. This indicates that the human scores take into consideration the SMEs perspective when evaluating the generated documentation which GPT does not consider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "11d4529d-dd23-4692-8c46-1651bb1da521",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# save the newly added example\n",
    "df.to_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b87ad4-6a30-4fda-a2b1-afd898d3eecf",
   "metadata": {},
   "source": [
    "### Example 10 - Do Not Rerun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "3f1155a2-ee0b-45cc-9fc2-4d83809379a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.read_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "92350543-67c9-4f64-945c-ec58a174c125",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "0696ef36-cec6-4d98-944b-2de15435cb1c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Introduction:**\\n\\nThe `LogInclusionProof` c...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Class Signer**\\n\\n**Introduction:**\\nThe Sig...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "4  \\nYou are an AI system specialized at generati...   \n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "8  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "4  **Introduction:**\\n\\nThe `LogInclusionProof` c...                 5.0   \n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "8  **Class Signer**\\n\\n**Introduction:**\\nThe Sig...                 4.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "4                  2.0                  5.0                   2.0   \n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "8                  3.0                  5.0                   4.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "4                4.0                 3.0                     5.0   \n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "8                4.0                 3.0                     3.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "4                      2.0                    4.0                     4.0   \n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "8                      3.0                    4.0                     4.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "4                    1.0               0.0                    0.0   \n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "8                    1.0               1.0                    1.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "4               0.0                0.0           1.0  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  \n",
       "8               1.0                1.0           1.0  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "deaa82a6-df12-4054-bdf6-71c9aeb6e0d9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated_text='1. Introduction: This API is designed to verify the integrity of the materials used in the creation of a digital signature. It provides a simple and secure way to verify the authenticity and integrity of the materials used in the creation of a digital signature.\\n\\n2. Functions:\\n\\nverify_signature:\\n\\nDescription: Verifies the integrity of the materials used in the creation of a digital signature.\\n\\nParameters:\\n\\n- materials: The materials used in the creation of the digital signature, including the signature itself, the signing key, and the certificate.\\n- signature_format: The format of the signature, such as \"JWS\" or \"COSE\".\\n- signature_algorithm: The algorithm used to create the signature, such as \"RS256\" or \"ES256\".\\n- signing_key: The public key used to verify the signature.\\n- certificate: The certificate used to verify the signature.\\n- rekor_entry: The Rekor entry associated with the signature, if available.\\n\\nReturn Values:\\n\\n- A VerificationResult object, which can be either a VerificationSuccess or a VerificationFailure object.\\n\\nError Handling:\\n\\n- InvalidMaterials: Raised when the materials provided are invalid or malformed.\\n- RekorEntryMissing: Raised when the Rekor entry is missing or unavailable.\\n- InvalidRekorEntry: Raised when the Rekor entry is invalid or malformed.\\n\\n3. Error Handling:\\n\\nInvalidMaterials: This error is raised when the materials provided are invalid or malformed. This could be due to a variety of reasons, such as a missing or malformed signature, signing key, or certificate.\\n\\nRekorEntryMissing: This error is raised when the Rekor entry is missing or unavailable. This could be due to a variety of reasons, such as the Rekor entry not being available or the materials not being signed with a valid Rekor entry.\\n\\nInvalidRekorEntry: This error is raised when the Rekor entry is invalid or malformed. This could be due to a variety of reasons, such as a missing or malformed signature, signing key, or certificate.' generated_token_count=444 generated_tokens=None input_text=None input_token_count=322 input_tokens=None moderation=None seed=3643591800.0 stop_reason='eos_token' stop_sequence=None\n"
     ]
    }
   ],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1', api_key, openai_key, 'verify_models', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "105e4496-3d23-4cec-877f-08f735ffd22a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Prompt \n",
      " \n",
      "You are an AI system specialized at generating API documentation for the provided Python code. You will be provided functions, classes, or Python scripts. Your documentation should include:\n",
      "\n",
      "1. Introduction: Briefly describe the purpose of the API and its intended use.\n",
      "2. Functions: Document each API function, including:\n",
      "    - Description: Clearly explain what the endpoint or function does.\n",
      "    - Parameters: List and describe each parameter, including data types and any constraints.\n",
      "    - Return Values: Specify the data type and possible values returned.\n",
      "\n",
      "3. Error Handling: Describe possible error responses and their meanings.\n",
      "\n",
      "Make sure to follow this output structure to create API documentation that is clear, concise, accurate, and user-centric. Avoid speculative information and prioritize accuracy and completeness.\n",
      "\n",
      "\n",
      "        \n",
      "Class code:\n",
      "\n",
      "class VerificationResult(BaseModel):\n",
      "    \n",
      "\n",
      "    success: bool\n",
      "    \n",
      "\n",
      "    def __bool__(self) -> bool:\n",
      "        \n",
      "        return self.success\n",
      "class VerificationSuccess(VerificationResult):\n",
      "    \n",
      "\n",
      "    success: bool = True\n",
      "    \n",
      "class VerificationFailure(VerificationResult):\n",
      "    \n",
      "\n",
      "    success: bool = False\n",
      "    \n",
      "\n",
      "    reason: str\n",
      "    \n",
      "class InvalidMaterials(Error):\n",
      "    \n",
      "\n",
      "    def diagnostics(self) -> str:\n",
      "        \n",
      "\n",
      "        return dedent(\n",
      "            f\\\n",
      "        An issue occurred while parsing the verification materials.\n",
      "\n",
      "        The provided verification materials are malformed and may have been\n",
      "        modified maliciously.\n",
      "\n",
      "        Additional context:\n",
      "\n",
      "        {self}\n",
      "        \n",
      "        )\n",
      "class RekorEntryMissing(Exception):\n",
      "    \n",
      "\n",
      "    pass\n",
      "class InvalidRekorEntry(InvalidMaterials):\n",
      "    \n",
      "\n",
      "    pass\n",
      "\n",
      "Class Documentation:\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "aa00b64f-3a9b-4151-b60b-15a56bc78db5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 5 - The generated documentation accurately describes the purpose, functions, and error handling of the API. It correctly identifies the parameters, return values, and possible error conditions based on the code provided.\n",
      "\n",
      "Relevance: 5 - The generated documentation is relevant to the code provided. It covers all the necessary information about the API, including its purpose, functions, parameters, return values, and possible error conditions.\n",
      "\n",
      "Clarity: 4 - The generated documentation is clear and easy to understand. It provides clear descriptions of the API's functions, parameters, return values, and error conditions. However, there are a few minor formatting issues and inconsistencies in indentation that could be improved for better clarity.\n",
      "\n",
      "Completeness: 4 - The generated documentation is mostly complete and includes all the necessary information about the API. It covers the introduction, functions, parameters, return values, and error handling. However, some additional details could be provided, such as more specific descriptions of the parameters and possible values for the return values.\n",
      "\n",
      "Readability: 5 - The generated documentation is readable and well-structured. It follows a logical order and uses clear language to explain the purpose, functions, and error handling of the API. The documentation is properly formatted and organized, making it easy to read and understand.\n",
      "\n",
      "Overall Score: 4.5\n"
     ]
    }
   ],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "0a153cad-0bad-4ea1-80ab-802c7da0d96a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a5c7a67b-0dd1-451b-b62e-255edb45fccd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reasoning': 'The criterion for this task is \"helpfulness\". The submission is to be evaluated based on whether it is helpful, insightful, and appropriate.\\n\\nLooking at the submission, it provides a detailed explanation of the API, its functions, parameters, return values, and error handling. It gives a clear understanding of what the API does, how it works, and what to expect in different scenarios. \\n\\nThe introduction gives a brief overview of the API\\'s purpose and its intended use. The functions section provides a detailed explanation of the function, its parameters, and return values. The error handling section explains the possible errors that can occur and their meanings. \\n\\nThe submission is insightful as it provides a deep understanding of the API, its functions, and possible errors. It is also appropriate as it follows the output structure provided in the task.\\n\\nTherefore, the submission meets the criterion of being helpful, insightful, and appropriate.\\n\\nY', 'value': 'Y', 'score': 1}\n",
      "{'reasoning': \"The submission is being evaluated on the criterion of correctness, which includes accuracy and factualness. \\n\\nThe submission provides an introduction to the API, explaining its purpose and intended use. It describes the API as being designed to verify the integrity of materials used in the creation of a digital signature. This seems to align with the provided Python code, which includes classes for verification results and errors related to verification materials.\\n\\nThe submission also documents a function, 'verify_signature', which is not present in the provided Python code. The parameters and return values described for this function do not match any of the classes or methods in the provided code. This is a factual inaccuracy.\\n\\nThe submission describes error handling for three errors: 'InvalidMaterials', 'RekorEntryMissing', and 'InvalidRekorEntry'. These errors are present in the provided Python code, and the descriptions given in the submission align with their implementation in the code. This part of the submission is accurate and factual.\\n\\nHowever, the submission's documentation of a function that is not present in the provided code is a significant inaccuracy. This means the submission does not meet the criterion of correctness.\\n\\nN\", 'value': 'N', 'score': 0}\n",
      "{'reasoning': \"The criterion is to assess if the output is logical and complete, capturing all required fields. \\n\\nLooking at the submission, the output is logical as it provides a clear and concise explanation of the API, its functions, parameters, return values, and error handling. \\n\\nHowever, the submission does not capture all required fields. The provided Python code includes several classes (VerificationResult, VerificationSuccess, VerificationFailure, InvalidMaterials, RekorEntryMissing, InvalidRekorEntry) and their methods, but the submission only documents a single function 'verify_signature' which is not present in the provided code. \\n\\nThe submission does not document the classes, their attributes, and methods as required by the task. Therefore, the submission is not complete.\\n\\nN\", 'value': 'N', 'score': 0}\n"
     ]
    }
   ],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "7b00f84c-5a21-4b9a-890e-7bf0b6a9a0a2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/app-root/src/api-docs-generation/notebooks/evaluation/helper_functions.ipynb:140: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.\n",
      "  \"def langchain_scores(generated_patch, prompt, actual_doc):\\n\",\n"
     ]
    }
   ],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "325ab810-fb7b-4c5e-9d4a-d823329ebe2d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Class Signer**\\n\\n**Introduction:**\\nThe Sig...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is designed to verif...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "8  \\nYou are an AI system specialized at generati...   \n",
       "9  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "8  **Class Signer**\\n\\n**Introduction:**\\nThe Sig...                 4.0   \n",
       "9  1. Introduction: This API is designed to verif...                 5.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "8                  3.0                  5.0                   4.0   \n",
       "9                  NaN                  5.0                   NaN   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "8                4.0                 3.0                     3.0   \n",
       "9                4.0                 NaN                     4.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "8                      3.0                    4.0                     4.0   \n",
       "9                      NaN                    5.0                     NaN   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "8                    1.0               1.0                    1.0   \n",
       "9                    1.0               NaN                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  \n",
       "8               1.0                1.0           1.0  \n",
       "9               NaN                0.0           NaN  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "8a740113-e0b2-4152-96de-f25b6e81f364",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[9, 'human_accuracy_score'] = '2.0'\n",
    "df.at[9, 'human_relevance_score'] = '2.0'\n",
    "df.at[9, 'human_clarity_score'] = '2.0'\n",
    "df.at[9, 'human_completeness_score'] = '2.0'\n",
    "df.at[9, 'human_readability_score'] = '4.0'\n",
    "df.at[9, 'human_helpfulness'] = '0.0'\n",
    "df.at[9, 'human_correctness'] = '0.0'\n",
    "df.at[9, 'human_logical'] = '0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "f30563af-b8de-4956-8345-59e4f5e81b68",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>response</th>\n",
       "      <th>gpt_accuracy_score</th>\n",
       "      <th>human_accuracy_score</th>\n",
       "      <th>gpt_relevance_score</th>\n",
       "      <th>human_relevance_score</th>\n",
       "      <th>gpt_clarity_score</th>\n",
       "      <th>human_clarity_score</th>\n",
       "      <th>gpt_completeness_score</th>\n",
       "      <th>human_completeness_score</th>\n",
       "      <th>gpt_readability_score</th>\n",
       "      <th>human_readability_score</th>\n",
       "      <th>langchain_helpfulness</th>\n",
       "      <th>human_helpfulness</th>\n",
       "      <th>langchain_correctness</th>\n",
       "      <th>human_correctness</th>\n",
       "      <th>langchain_logical</th>\n",
       "      <th>human_logical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe Error class is a base cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>\\nIntroduction:\\n\\nThis API is used to verify ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction:\\nThe API provides a set of cl...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>**Class Signer**\\n\\n**Introduction:**\\nThe Sig...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>\\nYou are an AI system specialized at generati...</td>\n",
       "      <td>1. Introduction: This API is designed to verif...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              prompt  \\\n",
       "5  \\nYou are an AI system specialized at generati...   \n",
       "6  \\nYou are an AI system specialized at generati...   \n",
       "7  \\nYou are an AI system specialized at generati...   \n",
       "8  \\nYou are an AI system specialized at generati...   \n",
       "9  \\nYou are an AI system specialized at generati...   \n",
       "\n",
       "                                            response  gpt_accuracy_score  \\\n",
       "5  1. Introduction:\\nThe Error class is a base cl...                 4.0   \n",
       "6  \\nIntroduction:\\n\\nThis API is used to verify ...                 4.0   \n",
       "7  1. Introduction:\\nThe API provides a set of cl...                 4.0   \n",
       "8  **Class Signer**\\n\\n**Introduction:**\\nThe Sig...                 4.0   \n",
       "9  1. Introduction: This API is designed to verif...                 5.0   \n",
       "\n",
       "  human_accuracy_score  gpt_relevance_score human_relevance_score  \\\n",
       "5                  5.0                  5.0                   5.0   \n",
       "6                  2.0                  4.0                   3.0   \n",
       "7                  4.0                  5.0                   4.0   \n",
       "8                  3.0                  5.0                   4.0   \n",
       "9                  2.0                  5.0                   2.0   \n",
       "\n",
       "   gpt_clarity_score human_clarity_score  gpt_completeness_score  \\\n",
       "5                4.0                 5.0                     5.0   \n",
       "6                4.0                 3.0                     3.0   \n",
       "7                4.0                 3.0                     3.0   \n",
       "8                4.0                 3.0                     3.0   \n",
       "9                4.0                 2.0                     4.0   \n",
       "\n",
       "  human_completeness_score  gpt_readability_score human_readability_score  \\\n",
       "5                      5.0                    5.0                     5.0   \n",
       "6                      2.0                    4.0                     5.0   \n",
       "7                      4.0                    4.0                     5.0   \n",
       "8                      3.0                    4.0                     4.0   \n",
       "9                      2.0                    5.0                     4.0   \n",
       "\n",
       "   langchain_helpfulness human_helpfulness  langchain_correctness  \\\n",
       "5                    1.0               1.0                    1.0   \n",
       "6                    NaN               0.0                    0.0   \n",
       "7                    1.0               1.0                    NaN   \n",
       "8                    1.0               1.0                    1.0   \n",
       "9                    1.0               0.0                    0.0   \n",
       "\n",
       "  human_correctness  langchain_logical human_logical  \n",
       "5               1.0                1.0           1.0  \n",
       "6               0.0                0.0           0.0  \n",
       "7               1.0                1.0           1.0  \n",
       "8               1.0                1.0           1.0  \n",
       "9               0.0                0.0           0.0  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d090abd-7760-4deb-9c8f-d332273c777f",
   "metadata": {},
   "source": [
    "**Interpretation**\n",
    "\n",
    "This is an interesting example since the generated documentation is partially correct. It seems to provide a pretty good description of the API in general, however it fails to capture all the functions/classes provided in the input Python code. Langchain was able to catch this and evaluate the output accordingly by assigning a score of 0 for both correctness and logicalness. It scores 1 for helpfulness as it takes into consideration that the document provides a fairly good overview of the input code provided. The human evaluated scores are compartively similar to Langchain however lower when compared to the GPT evaluated scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "1950d412-11ea-4d86-9b19-67f35f67f3fb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# save the newly added example\n",
    "df.to_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "812910d9-9b4f-4430-bffe-d58bb4b67083",
   "metadata": {},
   "source": [
    "## Copy this section, modify and run from here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ecea4ae-90bd-4932-b52d-968c02d28d28",
   "metadata": {},
   "source": [
    "### Example X "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fea4b1a-df7a-4ff4-bc16-75c5d334caf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_pickle('eval_df.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "828ae7e4-43ff-4db7-9eca-2f5daf0e51ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be88b8ab-daef-4017-a66d-6b39ceb2a442",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49bc46c2-46f1-4465-8d36-94d5f25a5614",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt, generated_text, actual_doc = get_response('ibm/granite-20b-code-instruct-v1', api_key, openai_key, '<file-name>', instruction, functions=False, classes=False, documentation=False, imports=False, other=False, functions_code=False, functions_doc=False, classes_code=True, classes_doc=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e2b6e37-bdee-4778-904b-252e2d7a7c9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\n Prompt \\n\", prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3913dd7-44c5-4a7f-8476-b52b6dc68f27",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\n Generated Text \\n\", generated_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10065769-7d8a-49d9-83ec-192900e9e19c",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt_score = eval_using_model(generated_text, openai_key=openai_key, initial_prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0510cc8-b2bb-46a9-8ebb-ca3250ffa2e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score = extract_scores(gpt_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc4151f2-b35f-44a7-9935-6350a2eb8cf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain_helpfulness, langchain_correctness, langchain_logical = langchain_scores(generated_text, prompt, actual_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42ec80f0-8724-474b-a7ac-615c9fa7b030",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = append_row_to_dataframe(df, prompt, generated_text, gpt_accuracy_score, gpt_relevance_score, gpt_clarity_score, gpt_completeness_score, gpt_readability_score, langchain_helpfulness, langchain_correctness, langchain_logical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18a37b82-ae78-492f-9900-28ad898c0643",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d557d78a-7e7a-4954-b5bc-e69d5bdfe450",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Append Human Scores\n",
    "\n",
    "df.at[X, 'human_accuracy_score'] = '2.0'\n",
    "df.at[X, 'human_relevance_score'] = '3.0'\n",
    "df.at[X, 'human_clarity_score'] = '4.0'\n",
    "df.at[X, 'human_completeness_score'] = '4.0'\n",
    "df.at[X, 'human_readability_score'] = '5.0'\n",
    "df.at[X, 'human_helpfulness'] = '0.0'\n",
    "df.at[X, 'human_correctness'] = '0.0'\n",
    "df.at[X, 'human_logical'] = '0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abd613e7-b92c-44c7-93f7-12be30c3f693",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ce1d099-9c48-48e2-8fe3-78f24ade821a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the newly added example\n",
    "df.to_pickle('eval_df.pkl')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}