Misc notebook changes (#378)

* WIP * Ignoring agentchat_agentops in testing automation bc it's not executable top down * Tidied autogen examples and moved demo notebook for clarity * Prettier readme * Updated API Key messaging * Missed one * idk what these changes are * Updating notebook automation triggers * black formatting
AgentOps-AI · Sep 9, 2024 · 3995834 · 3995834
1 parent eb2f2ef
commit 3995834
Show file tree

Hide file tree

Showing 19 changed files with 236 additions and 88 deletions.
diff --git a/.github/workflows/test-notebooks.yml b/.github/workflows/test-notebooks.yml
@@ -1,11 +1,22 @@
 name: Test Notebooks
 on:
-  pull_request_target: # Allows manual triggering from PR UI
   push:
     branches:
       - main
-    paths-ignore:
-      - 'docs/**'
+    paths:
+      - "agentops/**"
+      - "examples/**"
+      - "tests/**"
+      - ".github/workflows/test-notebooks.yml"
+  pull_request_target:
+    branches:
+      - main
+    paths:
+      - "agentops/**"
+      - "examples/**"
+      - "tests/**"
+      - ".github/workflows/test-notebooks.yml"
+  workflow_dispatch:
 jobs:
   test-notebooks:
     runs-on: ubuntu-latest
@@ -38,7 +49,8 @@ jobs:
         exit_code=0
 
         exclude_notebooks=(
-          "./examples/crew/job_posting.ipynb"
+          "./examples/crew/job_posting.ipynb",
+          "./examples/demos/agentchat_agentops.ipynb"
         )
 
         for notebook in $(find . -name '*.ipynb'); do

diff --git a/examples/README.md b/examples/README.md
@@ -1,5 +1,20 @@
 # AgentOps Examples
 
+<img src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/logo/banner-badge.png?raw=true" width="25%"/>
+
+[AgentOps](https://agentops.ai/?=autogen) provides session replays, metrics, and monitoring for AI agents.
+
+At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the [AgentOps Repo](https://github.com/AgentOps-AI/agentops).
+
+### Overview Dashboard
+<img src="https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/overview.gif"  width="25%"/>
+
+### Session Replays
+<img src="https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/app_screenshots/drilldown.gif"  width="25%"/>
+
+### Session Drilldown
+<img src="https://github.com/AgentOps-AI/agentops/blob/main/docs/images/external/app_screenshots/session-replay.png?raw=true"  width="25%"/>
+
 ## Quickstart
 - [Single LLM](./openai-gpt.ipynb)
 - [Recording Events](./recording-events.ipynb)

diff --git a/examples/autogen/AgentChat.ipynb b/examples/autogen/AgentChat.ipynb
@@ -6,12 +6,10 @@
    "id": "bb6538d8-2a5d-4a99-b2c1-7130963e4f7b",
    "metadata": {},
    "source": [
-    "# Adding AgentOps to an existing Autogen service\n",
-    "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/autogen/autogen-integration.png?raw=true\"/>\n",
+    "# AutoGen Chat Example\n",
+    "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/autogen/autogen-integration.png?raw=true\" width=\"25%\"/>\n",
     "\n",
-    "To get started, you'll need to install the AgentOps package and [set an API key](app.agentops.ai).\n",
-    "\n",
-    "AgentOps automatically configures itself when it's initialized meaning your agent run data will be tracked and logged to your AgentOps account right away."
+    "AgentOps automatically configures itself when it's initialized meaning your agent run data will be tracked and logged to your AgentOps dashboard right away."
    ]
   },
   {
@@ -53,15 +51,23 @@
     "import agentops\n",
     "import os\n",
     "from dotenv import load_dotenv\n",
-    "from IPython.core.error import StdinNotImplementedError"
+    "from IPython.core.error import (\n",
+    "    StdinNotImplementedError,\n",
+    ")  # only needed by AgentOps testing automation"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "70b502a2",
    "metadata": {},
    "source": [
-    "Next, we'll grab our API keys. You can use dotenv like below or however else you like to load environment variables"
+    "Next, we'll set our API keys. There are several ways to do this, the code below is just the most foolproof way for the purposes of this notebook. It accounts for both users who use environment variables and those who just want to set the API Key here in this notebook.\n",
+    "\n",
+    "[Get an AgentOps API key](https://agentops.ai/settings/projects)\n",
+    "\n",
+    "1. Create an environment variable in a .env file or other method. By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Or...\n",
+    "\n",
+    "2. Replace `<your_agentops_key>` below and pass in the optional `api_key` parameter to the AgentOps `init(api_key=...)` function. Remember not to commit your API key to a public repo!"
    ]
   },
   {
@@ -84,7 +90,7 @@
    "outputs": [],
    "source": [
     "# When initializing AgentOps, you can pass in optional tags to help filter sessions\n",
-    "agentops.init(AGENTOPS_API_KEY, default_tags=[\"simple-autogen-example\"])\n",
+    "agentops.init(api_key=AGENTOPS_API_KEY, default_tags=[\"simple-autogen-example\"])\n",
     "\n",
     "print(\"AgentOps is now running. You can view your session in the link above\")"
    ]
@@ -104,6 +110,14 @@
     "* Errors"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "e875dd0c",
+   "metadata": {},
+   "source": [
+    "# Simple Chat Example"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -113,7 +127,11 @@
    "source": [
     "# Define model, openai api key, tags, etc in the agent configuration\n",
     "config_list = [\n",
-    "    {\"model\": \"gpt-4-turbo\", \"api_key\": OPENAI_API_KEY, \"tags\": [\"gpt-4\", \"tool\"]}\n",
+    "    {\n",
+    "        \"model\": \"gpt-4-turbo\",\n",
+    "        \"api_key\": OPENAI_API_KEY,\n",
+    "        \"tags\": [\"agentchat-example\", \"chat\"],\n",
+    "    }\n",
     "]\n",
     "\n",
     "# Create the agent that uses the LLM.\n",
@@ -161,7 +179,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.3"
+   "version": "3.12.5"
   }
  },
  "nbformat": 4,

diff --git a/examples/autogen/MathAgent.ipynb b/examples/autogen/MathAgent.ipynb
@@ -6,8 +6,8 @@
    "id": "bb6538d8-2a5d-4a99-b2c1-7130963e4f7b",
    "metadata": {},
    "source": [
-    "# Adding AgentOps to an existing Autogen service\n",
-    "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/autogen/autogen-integration.png?raw=true\"/>\n",
+    "# AutoGen Tool Example\n",
+    "<img src=\"https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/autogen/autogen-integration.png?raw=true\" width=\"25%\"/>\n",
     "\n",
     "To get started, you'll need to install the AgentOps package and [set an API key](app.agentops.ai).\n",
     "\n",
@@ -54,15 +54,24 @@
     "from autogen import ConversableAgent, register_function\n",
     "import agentops\n",
     "import os\n",
-    "from dotenv import load_dotenv"
+    "from dotenv import load_dotenv\n",
+    "from IPython.core.error import (\n",
+    "    StdinNotImplementedError,\n",
+    ")  # only needed by AgentOps testing automation"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "24f8bd70",
    "metadata": {},
    "source": [
-    "Next, we'll grab our API keys. You can use dotenv like below or however else you like to load environment variables"
+    "Next, we'll set our API keys. There are several ways to do this, the code below is just the most foolproof way for the purposes of this notebook. It accounts for both users who use environment variables and those who just want to set the API Key here in this notebook.\n",
+    "\n",
+    "[Get an AgentOps API key](https://agentops.ai/settings/projects)\n",
+    "\n",
+    "1. Create an environment variable in a .env file or other method. By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Or...\n",
+    "\n",
+    "2. Replace `<your_agentops_key>` below and pass in the optional `api_key` parameter to the AgentOps `init(api_key=...)` function. Remember not to commit your API key to a public repo!"
    ]
   },
   {
@@ -105,16 +114,12 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2962d990-f7ef-43d8-ba09-d29bd8356d9f",
+   "cell_type": "markdown",
+   "id": "dc592637",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "# Define model, openai api key, tags, etc in the agent configuration\n",
-    "config_list = [\n",
-    "    {\"model\": \"gpt-4-turbo\", \"api_key\": OPENAI_API_KEY, \"tags\": [\"gpt-4\", \"tool\"]}\n",
-    "]"
+    "# Tool Example\n",
+    "AgentOps tracks when Autogen agents use tools. You can find more information on this example in [tool-use.ipynb](https://github.com/microsoft/autogen/blob/main/website/docs/tutorial/tool-use.ipynb)"
    ]
   },
   {
@@ -124,6 +129,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Define model, openai api key, tags, etc in the agent configuration\n",
+    "config_list = [\n",
+    "    {\n",
+    "        \"model\": \"gpt-4-turbo\",\n",
+    "        \"api_key\": OPENAI_API_KEY,\n",
+    "        \"tags\": [\"mathagent-example\", \"tool\"],\n",
+    "    }\n",
+    "]\n",
+    "\n",
     "Operator = Literal[\"+\", \"-\", \"*\", \"/\"]\n",
     "\n",
     "\n",
@@ -174,7 +188,13 @@
     ")\n",
     "\n",
     "# Let the assistant start the conversation.  It will end when the user types \"exit\".\n",
-    "user_proxy.initiate_chat(assistant, message=\"What is (1423 - 123) / 3 + (32 + 23) * 5?\")\n",
+    "try:\n",
+    "    user_proxy.initiate_chat(\n",
+    "        assistant, message=\"What is (1423 - 123) / 3 + (32 + 23) * 5?\"\n",
+    "    )\n",
+    "except StdinNotImplementedError:\n",
+    "    # This is only necessary for AgentOps testing automation which is headless and will not have user input\n",
+    "    print(\"Stdin not implemented. Skipping initiate_chat\")\n",
     "\n",
     "agentops.end_session(\"Success\")"
    ]
@@ -187,7 +207,7 @@
     "You can see your run in action at [app.agentops.ai](app.agentops.ai). In this example, the AgentOps dashboard will show:\n",
     "\n",
     "* Agents talking to each other\n",
-    "* Each use of the calculator tool\n",
+    "* Each use of the `calculator` tool\n",
     "* Each call to OpenAI for LLM use"
    ]
   }

diff --git a/examples/cohere/cohere_example.ipynb b/examples/cohere/cohere_example.ipynb
@@ -47,7 +47,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next, we'll grab our API keys. You can use dotenv like below or however else you like to load environment variables"
+    "Next, we'll set our API keys. There are several ways to do this, the code below is just the most foolproof way for the purposes of this notebook. It accounts for both users who use environment variables and those who just want to set the API Key here in this notebook.\n",
+    "\n",
+    "[Get an AgentOps API key](https://agentops.ai/settings/projects)\n",
+    "\n",
+    "1. Create an environment variable in a .env file or other method. By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Or...\n",
+    "\n",
+    "2. Replace `<your_agentops_key>` below and pass in the optional `api_key` parameter to the AgentOps `init(api_key=...)` function. Remember not to commit your API key to a public repo!"
    ]
   },
   {
@@ -144,7 +150,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.3"
+   "version": "3.12.5"
   }
  },
  "nbformat": 4,

diff --git a/examples/crew/job_posting.ipynb b/examples/crew/job_posting.ipynb
@@ -7,6 +7,13 @@
     "First let's install the required packages"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Crew Job Posting"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -41,7 +48,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next, we'll grab our API keys. You can use dotenv like below or however else you like to load environment variables"
+    "Next, we'll set our API keys. There are several ways to do this, the code below is just the most foolproof way for the purposes of this notebook. It accounts for both users who use environment variables and those who just want to set the API Key here in this notebook.\n",
+    "\n",
+    "[Get an AgentOps API key](https://agentops.ai/settings/projects)\n",
+    "\n",
+    "1. Create an environment variable in a .env file or other method. By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Or...\n",
+    "\n",
+    "2. Replace `<your_agentops_key>` below and pass in the optional `api_key` parameter to the AgentOps `init(api_key=...)` function. Remember not to commit your API key to a public repo!"
    ]
   },
   {
@@ -258,7 +271,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.3"
+   "version": "3.12.5"
   }
  },
  "nbformat": 4,

diff --git a/examples/crew/markdown_validator.ipynb b/examples/crew/markdown_validator.ipynb
@@ -65,7 +65,13 @@
       "id": "0e307923",
       "metadata": {},
       "source": [
-        "Next, we'll grab our API keys. You can use dotenv like below or however else you like to load environment variables"
+        "Next, we'll set our API keys. There are several ways to do this, the code below is just the most foolproof way for the purposes of this notebook. It accounts for both users who use environment variables and those who just want to set the API Key here in this notebook.\n",
+        "\n",
+        "[Get an AgentOps API key](https://agentops.ai/settings/projects)\n",
+        "\n",
+        "1. Create an environment variable in a .env file or other method. By default, the AgentOps `init()` function will look for an environment variable named `AGENTOPS_API_KEY`. Or...\n",
+        "\n",
+        "2. Replace `<your_agentops_key>` below and pass in the optional `api_key` parameter to the AgentOps `init(api_key=...)` function. Remember not to commit your API key to a public repo!"
       ]
     },
     {
@@ -298,7 +304,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.12.3"
+      "version": "3.12.5"
     }
   },
   "nbformat": 4,