chapter_07_trust_and_transparency.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 07: Building Trust and Transparency in LLMs\n",
    "\n",
    "In an era where LLMs are becoming increasingly embedded in various facets of our daily lives, the need for trust and transparency has never been more critical. As these sophisticated systems take on roles ranging from customer service representatives to content creators, their influence grows, raising important questions about their reliability, decision-making processes, and ethical implications. We will begin by exploring how users can build trust in LLMs through transparency, allowing users to have confidence in the outputs and behaviors of LLMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from assets.tools.earthquake import count_earthquakes, query_earthquakes, USGeopoliticalSurveyEarthquakeAPI\n",
    "from language_models.agent import (\n",
    "    Agent,\n",
    "    OutputType,\n",
    "    PromptingStrategy,\n",
    "    Step,\n",
    ")\n",
    "from language_models.tools import Tool, current_date\n",
    "from language_models.models.llm import OpenAILanguageModel\n",
    "from language_models.proxy_client import ProxyClient\n",
    "from language_models.settings import settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "proxy_client = ProxyClient(\n",
    "    client_id=settings.CLIENT_ID,\n",
    "    client_secret=settings.CLIENT_SECRET,\n",
    "    auth_url=settings.AUTH_URL,\n",
    "    api_base=settings.API_BASE,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAILanguageModel(\n",
    "    proxy_client=proxy_client,\n",
    "    model=\"gpt-4\",\n",
    "    max_tokens=500,\n",
    "    temperature=0.2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Governed Tool Access\n",
    "\n",
    "Governed tool access entails implementing tiered access levels for tools, delineating permissions based on the tool’s nature and the associated risks. Tools may be classified into categories such as executable, requiring approval before execution, or merely suggestible. Achieving governed tool access involves embedding checks within the AI agent’s code to regulate tool usage. When an LLM opts to employ a particular tool, the code verifies the tool’s permissions and determines the appropriate course of action based on the access level. For our purposes, we simply add a property to the tool object that specifies whether the tool requires approval or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_earthquakes_tool = Tool(\n",
    "    function=query_earthquakes,\n",
    "    name=\"Query Earthquakes\",\n",
    "    description=\"Use this tool to search recent earthquakes\",\n",
    "    args_schema=USGeopoliticalSurveyEarthquakeAPI,\n",
    "    requires_approval=True,\n",
    ")\n",
    "\n",
    "count_earthquakes_tool = Tool(\n",
    "    function=count_earthquakes,\n",
    "    name=\"Count Earthquakes\",\n",
    "    description=\"Use this tool to count and aggregate recent earthquakes\",\n",
    "    args_schema=USGeopoliticalSurveyEarthquakeAPI,\n",
    "    requires_approval=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are an United States Geological Survey expert who can answer questions regarding earthquakes.\"\n",
    "\n",
    "agent = Agent.create(\n",
    "    llm=llm,\n",
    "    system_prompt=system_prompt,\n",
    "    prompt=\"{question}\",\n",
    "    prompt_variables=[\"question\"],\n",
    "    output_type=OutputType.STRING,\n",
    "    tools=[current_date, count_earthquakes_tool, query_earthquakes_tool],\n",
    "    prompting_strategy=PromptingStrategy.CHAIN_OF_THOUGHT,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of approving the tool use: Count Earthquakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: To answer this question, I need to count the number of earthquakes that occurred today. I can use the \"Count Earthquakes\" tool for this. I need to provide the start time as the beginning of today and the end time as the current time.\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool\u001b[0m\u001b[1m\u001b[0m: Current Date\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Input\u001b[0m\u001b[1m\u001b[0m: {}\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Output\u001b[0m\u001b[1m\u001b[0m: 2024-08-22 20:08:55.705812\n",
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: Now that I have the current date and time, I can use this information to set the start time as the beginning of today (2024-08-22 00:00:00) and the end time as the current time (2024-08-22 20:08:55.705812) in the \"Count Earthquakes\" tool.\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool\u001b[0m\u001b[1m\u001b[0m: Count Earthquakes\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Input\u001b[0m\u001b[1m\u001b[0m: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:08:55.705812'}\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Use Approved\u001b[0m\u001b[1m\u001b[0m: Yes\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Output\u001b[0m\u001b[1m\u001b[0m: {'count': 146, 'maxAllowed': 20000}\n",
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: The tool output shows that there were 146 earthquakes today.\n",
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: There were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"question\": \"How many earthquakes occurred today?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of not approving the tool use: Query Earthquakes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: To show the details of 3 earthquakes that occurred today, I can use the \"Query Earthquakes\" tool. I will set the start time as the beginning of today, the end time as the current time, and the limit as 3.\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool\u001b[0m\u001b[1m\u001b[0m: Query Earthquakes\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Input\u001b[0m\u001b[1m\u001b[0m: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:08:55.705812', 'limit': 3}\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Use Approved\u001b[0m\u001b[1m\u001b[0m: No\n",
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: Since the user did not approve the use of the Query Earthquakes tool, I cannot provide specific details about the earthquakes. I can only provide the information that was previously gathered, which is the count of earthquakes that occurred today.\n",
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"question\": \"Show me 3.\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Displaying the Chain-of-Thought\n",
    "\n",
    "In order to maintain transparency and provide users with insights into the AI agent’s decision-making process, it is essential to display the reasoning process of the LLM. This includes thoughts, tools used, and observations made during the interaction. By visualizing the thought-tool-observation loop, users can gain a better understanding of how the AI agent arrived at its final response. This transparency not only builds trust but also enables users to assess the reliability and effectiveness of the AI agent’s decision-making process. For instance, when viewing the Chain-of-Thought, users can examine whether the AI agent correctly provides tool inputs. If a user requests sales data for the past week, the input date range should ideally span from the start date of today minus 7 days to the end date of today, ensuring accurate data retrieval. Observing such details within the reasoning process empowers users to evaluate the AI agent’s adherence to input requirements and adjust the system prompt and task prompt accordingly. The same applies to workflows. On the other hand, when using Single Completion, agents generate a direct output in response to a user query, bypassing the reasoning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def render_reasoning(steps: list[Step]):\n",
    "    names = {\n",
    "        \"system_prompt\": \"System Prompt\",\n",
    "        \"prompting_strategy\": \"Prompting Strategy\",\n",
    "        \"prompt\": \"Prompt\",\n",
    "        \"raw_output\": \"Raw Output\",\n",
    "        \"observation\": \"Observation\",\n",
    "        \"final_answer\": \"Final Answer\",\n",
    "        \"tool_use\": \"Tool Use\",\n",
    "        \"tool_output\": \"Tool Output\",\n",
    "    }\n",
    "    for step in steps:\n",
    "        print(names[step.name])\n",
    "        print(\"-\" * len(step.name))\n",
    "        if step.name == \"tool_use\":\n",
    "            print(f\"Thought: {step.content.thought}\")\n",
    "            print()\n",
    "            print(f\"Used: {step.content.used}\")\n",
    "            print()\n",
    "            print(f\"Arguments: {step.content.arguments}\")\n",
    "        elif step.name == 'final_answer':\n",
    "            print(f\"Thought: {step.content.thought}\")\n",
    "            print()\n",
    "            print(f\"Output: {step.content.output}\")\n",
    "        else:\n",
    "            print(step.content)\n",
    "\n",
    "        if step.name != \"final_answer\":\n",
    "            print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In earlier chapters, we observed the LLM's activities through the logs, focusing mainly on key aspects. However, in the Chain-of-Thought details, we gain access to more granular information, such as the prompt, raw output, and observations. Here is an example of the AI ​​agent using the Count Earthquakes tool, with the tool being approved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: To answer this question, I need to count the number of earthquakes that occurred today. I can use the \"Count Earthquakes\" tool for this. I need to provide the start time as the beginning of today and the end time as the current time.\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool\u001b[0m\u001b[1m\u001b[0m: Current Date\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Input\u001b[0m\u001b[1m\u001b[0m: {}\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Output\u001b[0m\u001b[1m\u001b[0m: 2024-08-22 20:09:33.246955\n",
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: The user is asking for the number of earthquakes that occurred today. I need to use the \"Count Earthquakes\" tool to get this information. The start time should be the start of today and the end time should be the current time.\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool\u001b[0m\u001b[1m\u001b[0m: Count Earthquakes\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Input\u001b[0m\u001b[1m\u001b[0m: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:09:33.246955'}\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Use Approved\u001b[0m\u001b[1m\u001b[0m: Yes\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Output\u001b[0m\u001b[1m\u001b[0m: {'count': 146, 'maxAllowed': 20000}\n",
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: The tool has returned the count of earthquakes that occurred today.\n",
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: There were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"question\": \"How many earthquakes occurred today?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System Prompt\n",
      "-------------\n",
      "You are an United States Geological Survey expert who can answer questions regarding earthquakes.\n",
      "\n",
      "Prompting Strategy\n",
      "------------------\n",
      "chain_of_thought\n",
      "\n",
      "Prompt\n",
      "------\n",
      "How many earthquakes occurred today?\n",
      "\n",
      "Raw Output\n",
      "----------\n",
      "Thought: To answer this question, I need to count the number of earthquakes that occurred today. I can use the \"Count Earthquakes\" tool for this. I need to provide the start time as the beginning of today and the end time as the current time.\n",
      "\n",
      "Tool: Current Date\n",
      "\n",
      "Tool Input: {}\n",
      "\n",
      "Tool Use\n",
      "--------\n",
      "Thought: To answer this question, I need to count the number of earthquakes that occurred today. I can use the \"Count Earthquakes\" tool for this. I need to provide the start time as the beginning of today and the end time as the current time.\n",
      "\n",
      "Used: Current Date\n",
      "\n",
      "Arguments: {}\n",
      "\n",
      "Tool Output\n",
      "-----------\n",
      "2024-08-22 20:09:33.246955\n",
      "\n",
      "Raw Output\n",
      "----------\n",
      "Thought: The user is asking for the number of earthquakes that occurred today. I need to use the \"Count Earthquakes\" tool to get this information. The start time should be the start of today and the end time should be the current time.\n",
      "\n",
      "Tool: Count Earthquakes\n",
      "\n",
      "Tool Input: {\"start_time\": \"2024-08-22 00:00:00\", \"end_time\": \"2024-08-22 20:09:33.246955\"}\n",
      "\n",
      "Tool Use\n",
      "--------\n",
      "Thought: The user is asking for the number of earthquakes that occurred today. I need to use the \"Count Earthquakes\" tool to get this information. The start time should be the start of today and the end time should be the current time.\n",
      "\n",
      "Used: Count Earthquakes\n",
      "\n",
      "Arguments: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:09:33.246955'}\n",
      "\n",
      "Tool Output\n",
      "-----------\n",
      "{'count': 146, 'maxAllowed': 20000}\n",
      "\n",
      "Raw Output\n",
      "----------\n",
      "Thought: The tool has returned the count of earthquakes that occurred today. \n",
      "\n",
      "Final Answer: There were 146 earthquakes today.\n",
      "\n",
      "Final Answer\n",
      "------------\n",
      "Thought: The tool has returned the count of earthquakes that occurred today.\n",
      "\n",
      "Output: There were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "render_reasoning(output.steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of the AI ​​agent using the Query Earthquakes tool, with the tool being approved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: To show the details of 3 earthquakes that occurred today, I can use the \"Query Earthquakes\" tool. I will set the start time as the beginning of today, the end time as the current time, and the limit as 3.\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool\u001b[0m\u001b[1m\u001b[0m: Query Earthquakes\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Input\u001b[0m\u001b[1m\u001b[0m: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:09:33.246955', 'limit': 3}\n",
      "\u001b[1m\u001b[38;2;236;154;60mTool Use Approved\u001b[0m\u001b[1m\u001b[0m: No\n",
      "\u001b[1m\u001b[38;2;45;114;210mThought\u001b[0m\u001b[1m\u001b[0m: Since the user did not approve the use of the Query Earthquakes tool, I am unable to provide specific details about the earthquakes.\n",
      "\u001b[1m\u001b[38;2;50;164;103mFinal Answer\u001b[0m\u001b[1m\u001b[0m: I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "output = agent.invoke({\"question\": \"Show me 3.\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "print(output.final_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System Prompt\n",
      "-------------\n",
      "You are an United States Geological Survey expert who can answer questions regarding earthquakes.\n",
      "\n",
      "Prompting Strategy\n",
      "------------------\n",
      "chain_of_thought\n",
      "\n",
      "Prompt\n",
      "------\n",
      "Show me 3.\n",
      "\n",
      "Raw Output\n",
      "----------\n",
      "Thought: To show the details of 3 earthquakes that occurred today, I can use the \"Query Earthquakes\" tool. I will set the start time as the beginning of today, the end time as the current time, and the limit as 3.\n",
      "\n",
      "Tool: Query Earthquakes\n",
      "\n",
      "Tool Input: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:09:33.246955', 'limit': 3}\n",
      "\n",
      "Tool Use\n",
      "--------\n",
      "Thought: To show the details of 3 earthquakes that occurred today, I can use the \"Query Earthquakes\" tool. I will set the start time as the beginning of today, the end time as the current time, and the limit as 3.\n",
      "\n",
      "Used: Query Earthquakes\n",
      "\n",
      "Arguments: {'start_time': '2024-08-22 00:00:00', 'end_time': '2024-08-22 20:09:33.246955', 'limit': 3}\n",
      "\n",
      "Observation\n",
      "-----------\n",
      "The user did not approve the use of the tool: Query Earthquakes\n",
      "\n",
      "You should respond with:\n",
      "```\n",
      "Thought: <thought process on how to respond to the prompt>\n",
      "\n",
      "Final Answer: <response to the prompt>\n",
      "```\n",
      "\n",
      "Raw Output\n",
      "----------\n",
      "I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n",
      "\n",
      "Observation\n",
      "-----------\n",
      "Could not parse your response:\n",
      "I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n",
      "\n",
      "You should respond with:\n",
      "```\n",
      "Thought: <thought process on how to respond to the prompt>\n",
      "\n",
      "Tool: <name of the tool to use>\n",
      "\n",
      "Tool Input: <input of the tool to use>\n",
      "```\n",
      "\n",
      "Your <input of the tool to use> should be a JSON format and must be keyword arguments of the properties of <name of the tool to use>\n",
      "\n",
      "When you know the final answer to the user's query you should respond with:\n",
      "```\n",
      "Thought: <thought process on how to respond to the prompt>\n",
      "\n",
      "Final Answer: <response to the prompt>\n",
      "```\n",
      "\n",
      "Your <response to the prompt> should be the final answer to the user's query and must be a string\n",
      "\n",
      "Raw Output\n",
      "----------\n",
      "Thought: Since the user did not approve the use of the Query Earthquakes tool, I am unable to provide specific details about the earthquakes. \n",
      "\n",
      "Final Answer: I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n",
      "\n",
      "Final Answer\n",
      "------------\n",
      "Thought: Since the user did not approve the use of the Query Earthquakes tool, I am unable to provide specific details about the earthquakes.\n",
      "\n",
      "Output: I'm sorry, but without using the Query Earthquakes tool, I can't provide specific details about the earthquakes. I can tell you that there were 146 earthquakes today.\n"
     ]
    }
   ],
   "source": [
    "render_reasoning(output.steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitoring\n",
    "\n",
    "Monitoring plays a pivotal role in ensuring the effective operation and reliability of LLMs. Given the intricate and evolving nature of these systems, continuous oversight is essential to detect anomalies, resolve performance issues, and maintain user trust. One fundamental aspect of monitoring involves logging every interaction between users and the AI agent. This comprehensive logging provides transparency, allowing users to review past decisions and understand the context in which they were made and by whom. It also facilitates accountability, allowing developers to identify and address misuse or violations of the system.\n",
    "\n",
    "Additionally, monitoring encompasses the evaluation of tool utilization within the AI agent. By tracking which tools are frequently or infrequently used, developers can make informed decisions about system optimization. Tools that are rarely utilized may be removed to streamline the system and reduce costs, while those frequently used can be prioritized for enhancements to improve their effectiveness. Visualizing tool usage through graphical representations provides further insights into the AI agent's behavior, helping to identify whether the system is functioning as expected or if performance issues need to be addressed. This approach allows developers to fine-tune the AI’s operations based on actual usage patterns and user interactions.\n",
    "\n",
    "Moreover, monitoring also involves assessing the economic aspects of operating LLMs. By tracking the consumption of tokens and other resources, users can evaluate the cost-effectiveness of their AI agents. This financial oversight helps in making strategic decisions about transitioning to more cost-efficient models while balancing performance needs. Overall, a comprehensive monitoring strategy, including tool usage analysis, financial tracking, and transparent logging, is essential for maintaining trust and transparency in LLMs. These practices not only uphold ethical standards but also enable developers to continuously improve and adapt their AI solutions to meet user expectations and operational requirements."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-powered-ai-agents",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}