Improved MultiOn examples (#243)

AgentOps-AI · Jun 11, 2024 · 15a5c7a · 15a5c7a
1 parent 267d0df
commit 15a5c7a
Show file tree

Hide file tree

Showing 14 changed files with 501 additions and 105 deletions.
diff --git a/docs/images/checks-passed.png b/docs/images/checks-passed.png
diff --git a/docs/images/external/README.md b/docs/images/external/README.md
@@ -0,0 +1,22 @@
+# External Images Directory
+
+## Purpose
+This directory serves as a centralized repository for images that are used not only within our documentation but also embedded in external websites via direct links. 
+
+## Automatic Synchronization
+Any updates made to the images in this directory (such as replacing an image file with a new version) will be reflected across all external sites that display these images. This is because the image URLs remain constant, and the external sites dynamically fetch the latest version of the image from this directory.
+
+To create a URL that consistenly updates, use the pattern:
+```
+https://raw.githubusercontent.com/AgentOps-AI/agentops/main/docs/images/external/<filename>.<extension>
+```
+
+## Usage Guidelines
+- **Updating Images**: To update an image, upload the new file with the exact same name as the one it's replacing. This ensures that the image update is seamless across all platforms.
+- **Consistency**: Regularly check that the images are up-to-date and accurately represent our brand and documentation standards.
+- **Do Not Remove**: Avoid deleting images from this directory. Removal of images can lead to broken links and missing visuals on external sites that rely on these images.
+
+## Example
+If you update the `logo.png` file in this directory, the new logo will automatically appear in place of the old one on all external documentation pages that link to this image.
+
+**Important**: Deletions and modifications can have immediate and widespread effects. Always verify that changes made here are intentional and correct before proceeding.
diff --git a/docs/images/external/app_screenshots/overview.png b/docs/images/external/app_screenshots/overview.png
diff --git a/docs/images/external/app_screenshots/session-overview.png b/docs/images/external/app_screenshots/session-overview.png
diff --git a/docs/images/external/app_screenshots/session-replay.png b/docs/images/external/app_screenshots/session-replay.png
diff --git a/docs/images/external/logo/rounded-icon.svg b/docs/images/external/logo/rounded-icon.svg
diff --git a/docs/images/external/multion/multion_integration.png b/docs/images/external/multion/multion_integration.png
diff --git a/docs/logo/dark.svg b/docs/logo/dark.svg
diff --git a/docs/logo/light.svg b/docs/logo/light.svg
diff --git a/examples/multion/Autonomous_web_browsing.ipynb b/examples/multion/Autonomous_web_browsing.ipynb
diff --git a/examples/multion/Sample_browsing_agent.ipynb b/examples/multion/Sample_browsing_agent.ipynb
@@ -0,0 +1,165 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h1>MultiOn Tracking Web Browse Actions</h1>\n",
+    "<img src=\"https://github.com/AgentOps-AI/agentops/blob/b4aac2d4b9fb16d6aa0a25aa9018210a94f1bef2/docs/logo/multion_integration.png?raw=true\" width=\"250px\" style=\"max-width: 100%; height: auto;\"/>\n",
+    "    \n",
+    "Agents using MultiOn can launch and control remote or local web browsers to perform actions and retrieve context using natural language commands. With AgentOps, MultiOn evens such as browse, retrieve, and step are automatically tracked.\n",
+    "\n",
+    "\n",
+    "![AgentOps MultiOn Browse](https://github.com/AgentOps-AI/agentops/blob/main/docs/images/agentops-multion-browse.gif?raw=true)\n",
+    "\n",
+    "Furthermore, events and LLM calls in your Python program will be tracked as well."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install dependencies\n",
+    "%pip install multion\n",
+    "%pip install agentops\n",
+    "%pip install openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from multion.client import MultiOn\n",
+    "from multion.core.request_options import RequestOptions\n",
+    "import openai\n",
+    "import agentops"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set your API keys\n",
+    "import os\n",
+    "os.environ[\"MULTION_API_KEY\"] = \"multion_key\"\n",
+    "os.environ[\"AGENTOPS_API_KEY\"] = \"agentops_key\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create an AgentOps session\n",
+    "When running `agentops.init()`, be sure to set `auto_start_session=False`. MultiOn will automatically launch AgentOps sessions by default, but by setting auto start to false, you can configure your AgentOps client independently."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agentops.init(auto_start_session=False, tags=['MultiOn browse example'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we can launch a MultiOn browse event. This event will automatically get added to your AgentOps session."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "multion = MultiOn(\n",
+    "    api_key=os.environ.get(\"MULTION_API_KEY\"),\n",
+    "    agentops_api_key=os.environ.get(\"AGENTOPS_API_KEY\"),\n",
+    ")\n",
+    "cmd = \"what three things do i get with agentops\"\n",
+    "request_options = RequestOptions(\n",
+    "    timeout_in_seconds=60, max_retries=4, additional_headers={\"test\": \"ing\"}\n",
+    ")\n",
+    "\n",
+    "browse_response = multion.browse(\n",
+    "    cmd=\"what three things do i get with agentops\",\n",
+    "    url=\"https://www.agentops.ai/\",\n",
+    "    max_steps=4,\n",
+    "    include_screenshot=True,\n",
+    "    request_options=request_options,\n",
+    ")\n",
+    "\n",
+    "print(browse_response.message)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's use OpenAI to summarize our output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages = [{\"role\": \"user\", \"content\": f\"Format this data as a markdown table: {browse_response.message}\"}]\n",
+    "client = openai.OpenAI()\n",
+    "response = client.chat.completions.create(messages=messages, model=\"gpt-3.5-turbo\")\n",
+    "\n",
+    "print(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# End session to see your dashboard\n",
+    "agentops.end_session('Success')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Check your session\n",
+    "Check your session on [AgentOps](https://app.agentops.ai). This session should include the MultiOn browse action and the OpenAI call."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/examples/multion/step.ipynb → ...s/multion/Step_by_step_web_browsing.ipynb b/examples/multion/step.ipynb → ...s/multion/Step_by_step_web_browsing.ipynb
@@ -1,5 +1,17 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# MultiOn Step Tracing\n",
+    "<img src=\"https://github.com/AgentOps-AI/agentops/blob/b4aac2d4b9fb16d6aa0a25aa9018210a94f1bef2/docs/logo/multion_integration.png?raw=true\" width=\"250px\" style=\"max-width: 100%; height: auto;\"/>\n",
+    "    \n",
+    "With AgentOps, agent actions as well as MultiOn browse events will get tracked. MultiOn browse events automatically trace screenshots as well.\n",
+    "\n",
+    "This example shows how to use MultiOn's session creator to launch a self-directed browser agent that accomplishes a specified objective using Step Mode. MultiOn agents can either accomplish tasks fully autonomously or managed one step at a time. In this example, we will launch a MutliOn agent and manage each individual step. "
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -14,7 +26,22 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Stream"
+    "# Session streaming\n",
+    "In this example, we'll use MultiOn to stream individual steps to accomplish a task. To track your runs in the AgentOps dashboard, specify an `agentops_api_key` when initializing `MultiOn()`\n",
+    "\n",
+    "You can run MultiOn without running `agentops.init()`. However, you will only see events from MultiOn, and not any from your own agent.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set your API keys\n",
+    "import os\n",
+    "os.environ[\"MULTION_API_KEY\"] = \"multion_key\"\n",
+    "os.environ[\"AGENTOPS_API_KEY\"] = \"agentops_key\""
    ]
   },
   {
@@ -29,7 +56,6 @@
     "    SessionsStepRequestBrowserParams,\n",
     ")\n",
     "from multion.core.request_options import RequestOptions\n",
-    "import os\n",
     "\n",
     "multion = MultiOn(\n",
     "    api_key=os.environ.get(\"MULTION_API_KEY\"),\n",
@@ -75,14 +101,29 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Step_Stream"
+    "# Step Stream\n",
+    "Step stream is just like step, but it streams responses in the same way a streamed LLM response is received. Instead of waiting for the entire step to complete, MultiOn will return work in progress. To track your runs in the AgentOps dashboard, specify an `agentops_api_key` when initializing `MultiOn()`"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"MULTION_API_KEY\"] = \"e8cbbd0f8fa042f49f267a44bf97425c\"\n",
+    "os.environ[\"AGENTOPS_API_KEY\"] =\"a640373b-30ae-4655-a1f3-5caa882a8721\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
    "source": [
     "import multion\n",
     "from multion.client import MultiOn\n",
@@ -154,7 +195,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -168,9 +209,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.11.5"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }
diff --git a/examples/multion/retrieve.ipynb → ...ples/multion/Webpage_data_retrieval.ipynb b/examples/multion/retrieve.ipynb → ...ples/multion/Webpage_data_retrieval.ipynb
@@ -1,5 +1,17 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# MultiOn Autonomous Session Tracing\n",
+    "<img src=\"https://github.com/AgentOps-AI/agentops/blob/b4aac2d4b9fb16d6aa0a25aa9018210a94f1bef2/docs/logo/multion_integration.png?raw=true\" width=\"250px\" style=\"max-width: 100%; height: auto;\"/>\n",
+    "    \n",
+    "With AgentOps, agent actions as well as MultiOn retrieve events will get tracked. MultiOn browse events automatically trace screenshots as well.\n",
+    "\n",
+    "This example shows how to use MultiOn's session creator to launch a self-directed browser agent that accomplishes a specified objective."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -11,10 +23,15 @@
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "metadata": {},
+   "outputs": [],
    "source": [
-    "# Retrieve"
+    "# Set your API keys\n",
+    "import os\n",
+    "os.environ[\"MULTION_API_KEY\"] = \"multion_key\"\n",
+    "os.environ[\"AGENTOPS_API_KEY\"] = \"agentops_key\""
    ]
   },
   {
@@ -25,7 +42,6 @@
    "source": [
     "import multion\n",
     "from multion.client import MultiOn\n",
-    "import os\n",
     "\n",
     "multion = MultiOn(\n",
     "    api_key=os.environ.get(\"MULTION_API_KEY\"),\n",
@@ -38,7 +54,7 @@
     "cmd = \"find the cheapest satellite phone on REI\"\n",
     "print(create_session_response.message)\n",
     "\n",
-    "retrieve_response = multion.sessions.retrieve(session_id=session_id, cmd=cmd, url=\"https://www.rei.com/\", fields=[\"price\"], page_number=2, include_screenshot=True)\n",
+    "retrieve_response = multion.sessions.retrieve(session_id=session_id, cmd=cmd, url=\"https://www.rei.com/\", fields=[\"price\"], include_screenshot=True)\n",
     "print(retrieve_response.message)\n",
     "\n",
     "while retrieve_response.status == \"CONTINUE\":\n",
@@ -47,7 +63,6 @@
     "        cmd=cmd,\n",
     "        url=\"https://www.rei.com/\",\n",
     "        fields=[\"price\"],\n",
-    "        page_number=2,\n",
     "        include_screenshot=True,\n",
     "    )\n",
     "    print(retrieve_response.message)\n",
@@ -68,11 +83,18 @@
     "\n",
     "![AgentOps Multion Retrieve](https://github.com/AgentOps-AI/agentops/blob/main/docs/images/agentops-multion-retrieve.gif?raw=true)"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -86,9 +108,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.10"
+   "version": "3.11.5"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }