Merge branch 'update-multion-retrieval' of https://github.com/AgentOp…

…s-AI/agentops into update-multion-retrieval

.github/workflows/black-formatter.yml

@@ -0,0 +1,25 @@
+
+name: Black Code Formatter Check
+
+on: [pull_request]
+
+jobs:
+  black-check:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.11'  # Specify the version of Python you need
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install black
+        pip install "black[jupyter]"
+    - name: Run Black
+      run: black --diff --check .

.github/workflows/codecov.yml

@@ -0,0 +1,37 @@
+name: Codecov
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.11
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install tox
+
+    - name: Run tests with tox
+      run: tox
+
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v2
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        files: ./coverage.xml
+        flags: unittests
+        name: codecov-umbrella
+        fail_ci_if_error: true

README.md

@@ -1,6 +1,6 @@
 <div align="center">
   <a href="https://agentops.ai?ref=gh">
-    <img src="https://raw.githubusercontent.com/AgentOps-AI/agentops/35d5682866921a9e28d8ef66ae3c3b3d92d8fa6b/img/logo.png" style="margin: 15px; max-width: 300px" width="50%" alt="Logo">
+    <img src="docs/images/external/logo/banner-badge.png" style="max-width: 500px" width="50%" alt="Logo">
   </a>
 </div>
 <p align="center">
@@ -82,21 +82,21 @@ All your sessions are available on the [AgentOps dashboard](https://app.agentops
 <details open>
   <summary>Agent Dashboard</summary>
   <a href="https://app.agentops.ai?ref=gh">
-   <img src="https://github.com/AgentOps-AI/agentops/assets/14807319/158e082a-9a7d-49b7-9b41-51a49a1f7d3d" style="width: 90%;" alt="Agent Dashboard"/>
+   <img src="docs/images/external/app_screenshots/overview.png" style="width: 90%;" alt="Agent Dashboard"/>
   </a>
 </details>
 
 <details>
   <summary>Session Analytics</summary>
   <a href="https://app.agentops.ai?ref=gh">
-    <img src="https://github.com/AgentOps-AI/agentops/assets/14807319/d7228019-1488-40d3-852f-a61e998658ad" style="width: 90%;" alt="Session Analytics"/>
+    <img src="docs/images/external/app_screenshots/session-overview.png" style="width: 90%;" alt="Session Analytics"/>
   </a>
 </details>
 
 <details>
   <summary>Session Replays</summary>
   <a href="https://app.agentops.ai?ref=gh">
-    <img src="https://github.com/AgentOps-AI/agentops/assets/14807319/561d59f3-c441-4066-914b-f6cfe32a598c" style="width: 90%;" alt="Session Replays"/>
+    <img src="docs/images/external/app_screenshots/session-replay.png" style="width: 90%;" alt="Session Replays"/>
   </a>
 </details>
 
@@ -212,6 +212,35 @@ agentops.end_session('Success')
 </details>
 
 
+### LiteLLM
+
+AgentOps provides support for LiteLLM(>=1.3.1), allowing you to call 100+ LLMs using the same Input/Output Format. 
+
+- [AgentOps integration example](https://docs.agentops.ai/v1/integrations/litellm)
+- [Official LiteLLM documentation](https://docs.litellm.ai/docs/providers)
+
+<details>
+  <summary>Installation</summary>
+
+```bash
+pip install litellm
+```
+
+```python python
+# Do not use LiteLLM like this
+# from litellm import completion
+# ...
+# response = completion(model="claude-3", messages=messages)
+
+# Use LiteLLM like this
+import litellm
+...
+response = litellm.completion(model="claude-3", messages=messages)
+# or
+response = await litellm.acompletion(model="claude-3", messages=messages)
+```
+</details>
+
 ### LlamaIndex 🦙
 
 (Coming Soon)
@@ -245,8 +274,20 @@ agentops.end_session('Success')
 
 ### Why AgentOps? 🤔
 
-Our mission is to bring your agent from prototype to production.
+Without the right tools, AI agents are slow, expensive, and unreliable. Our mission is to bring your agent from prototype to production. Here's why AgentOps stands out:
+
+- **Comprehensive Observability**: Track your AI agents' performance, user interactions, and API usage.
+- **Real-Time Monitoring**: Get instant insights with session replays, metrics, and live monitoring tools.
+- **Cost Control**: Monitor and manage your spend on LLM and API calls.
+- **Failure Detection**: Quickly identify and respond to agent failures and multi-agent interaction issues.
+- **Tool Usage Statistics**: Understand how your agents utilize external tools with detailed analytics.
+- **Session-Wide Metrics**: Gain a holistic view of your agents' sessions with comprehensive statistics.
+
+AgentOps is designed to make agent observability, testing, and monitoring easy.
+
+
+## Star History
 
-Agent developers often work with little to no visibility into agent testing performance. This means their agents never leave the lab. We're changing that.
+Check out our growth in the community:
 
-AgentOps is the easiest way to evaluate, grade, and test agents. Is there a feature you'd like to see AgentOps cover? Just raise it in the issues tab, and we'll work on adding it to the roadmap.
+<img src="https://api.star-history.com/svg?repos=AgentOps-AI/agentops&type=Date" style="max-width: 500px" width="50%" alt="Logo">

agentops/__init__.py

@@ -6,7 +6,6 @@
 from .client import Client
 from .config import Configuration
 from .event import Event, ActionEvent, LLMEvent, ToolEvent, ErrorEvent
-from .enums import Models
 from .decorators import record_function
 from .agent import track_agent
 from .log_config import logger

agentops/client.py

@@ -1,8 +1,8 @@
 """
-    AgentOps client module that provides a client class with public interfaces and configuration.
+AgentOps client module that provides a client class with public interfaces and configuration.
 
-    Classes:
-        Client: Provides methods to interact with the AgentOps service.
+Classes:
+    Client: Provides methods to interact with the AgentOps service.
 """
 
 import os
@@ -80,7 +80,6 @@ def __init__(
         inherited_session_id: Optional[str] = None,
         skip_auto_end_session: Optional[bool] = False,
     ):
-
         if override is not None:
             logger.warning(
                 "The 'override' parameter is deprecated. Use 'instrument_llm_calls' instead.",
@@ -389,7 +388,7 @@ def end_session(
         end_state_reason: Optional[str] = None,
         video: Optional[str] = None,
         is_auto_end: Optional[bool] = None,
-    ):
+    ) -> Decimal:
         """
         End the current session with the AgentOps service.
 
@@ -398,6 +397,9 @@ def end_session(
             end_state_reason (str, optional): The reason for ending the session.
             video (str, optional): The video screen recording of the session
             is_auto_end (bool, optional): is this an automatic use of end_session and should be skipped with skip_auto_end_session
+
+        Returns:
+            Decimal: The token cost of the session. Returns 0 if the cost is unknown.
         """
 
         if is_auto_end and self.config.skip_auto_end_session:
@@ -420,6 +422,7 @@ def end_session(
 
         if token_cost is None or token_cost == "unknown":
             logger.info("Could not determine cost of run.")
+            token_cost_d = Decimal(0)
         else:
             token_cost_d = Decimal(token_cost)
             logger.info(
@@ -439,6 +442,7 @@ def end_session(
 
         self._session = None
         self._worker = None
+        return token_cost_d
 
     def create_agent(self, name: str, agent_id: Optional[str] = None):
         if agent_id is None:

agentops/enums.py

@@ -9,20 +9,6 @@ class EventType(Enum):
     ERROR = "errors"
 
 
-class Models(Enum):
-    GPT_3_5_TURBO = "gpt-3.5-turbo"
-    GPT_3_5_TURBO_0301 = "gpt-3.5-turbo-0301"
-    GPT_3_5_TURBO_0613 = "gpt-3.5-turbo-0613"
-    GPT_3_5_TURBO_16K = "gpt-3.5-turbo-16k"
-    GPT_3_5_TURBO_16K_0613 = "gpt-3.5-turbo-16k-0613"
-    GPT_4_0314 = "gpt-4-0314"
-    GPT_4 = "gpt-4"
-    GPT_4_32K = "gpt-4-32k"
-    GPT_4_32K_0314 = "gpt-4-32k-0314"
-    GPT_4_0613 = "gpt-4-0613"
-    TEXT_EMBEDDING_ADA_002 = "text-embedding-ada-002"
-
-
 class EndState(Enum):
     SUCCESS = "Success"
     FAIL = "Fail"

agentops/event.py

@@ -5,10 +5,10 @@
     Event: Represents discrete events to be recorded.
 """
 
-from dataclasses import asdict, dataclass, field
+from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional, Sequence, Union
 from .helpers import get_ISO_time, check_call_stack_for_agent_id
-from .enums import EventType, Models
+from .enums import EventType
 from uuid import UUID, uuid4
 import traceback
 
@@ -72,7 +72,7 @@ class LLMEvent(Event):
     prompt_tokens(int, optional): The number of tokens in the prompt message.
     completion(str, object, optional): The message or returned by the LLM. Preferably in ChatML format which is more fully supported by AgentOps.
     completion_tokens(int, optional): The number of tokens in the completion message.
-    model(Models, str, optional): LLM model e.g. "gpt-4". Models defined in enums.Models are more fully supported by AgentOps e.g. extra features in dashboard.
+    model(str, optional): LLM model e.g. "gpt-4", "gpt-3.5-turbo".
 
     """
 
@@ -82,7 +82,7 @@ class LLMEvent(Event):
     prompt_tokens: Optional[int] = None
     completion: Union[str, object] = None
     completion_tokens: Optional[int] = None
-    model: Optional[Union[Models, str]] = None
+    model: Optional[str] = None
 
 
 @dataclass

agentops/host_env.py

@@ -127,7 +127,9 @@ def get_disk_details():
             }
         except OSError as inaccessible:
             # Skip inaccessible partitions, such as removable drives with no media
-            logger.debug("Mountpoint %s inaccessible: %s", partition.mountpoint, inaccessible)
+            logger.debug(
+                "Mountpoint %s inaccessible: %s", partition.mountpoint, inaccessible
+            )
 
     return disk_info