[Bug]: LangChain callback handler on_llm_start/end issue

[areibman]

Contact Details

No response

📦 Package Version

0.3.24

🎞️ Framework Version

Langchain

🔎 Describe the Bug

Summary:
This issue pertains to the integration of Langchain callbacks into the Agency-Swarm framework. Despite successful implementation of observability, some key messages from the agent team are missing in the logs. Additionally, there are errors related to unsupported chat model callbacks.

Details:
1. Integration Overview:
• Langchain callbacks (on_llm_start, on_llm_end) have been integrated into the Agency-Swarm framework (not to be confused with the “Swarms” framework).
• Relevant changes can be reviewed in thread.py, particularly around the on_llm_start and on_llm_end methods, as well as agency.py and other associated files.
2. Observability Issue:
• First and last messages from the agent team are not appearing in the logs despite being shown correctly in Langfuse.
• Root cause of this issue is unclear and requires investigation.
3. Error with AgentOps Callback Handler:
• Using the AgentOps callback handler throws the following error:

NotImplementedError: LangchainCallbackHandler does not implement on_chat_model_start

•	It seems chat models are not currently supported. The assumption is that on_llm_start should be used instead, but confirmation is needed.

Questions/Next Steps:
1. Investigate why the first and last messages from the agent team are missing, even though they appear in Langfuse.
2. Clarify the correct implementation of on_llm_start for observability in the absence of support for chat models.
3. Determine if additional support for on_chat_model_start needs to be added or if existing methods can handle chat models.

Relevant Files:
• thread.py (focus: on_llm_start, on_llm_end)
• agency.py and other files where these methods are defined.

Context:
• User implementing observability for agent-based frameworks during holiday season.
• Question originated from a user attempting to use AgentOps callback handler with Langchain.

🤝 Contribution

• Yes, I'd be happy to submit a pull request with these changes.
• I need some guidance on how to contribute.
• I'd prefer the AgentOps team to handle this update.

[the-praxs]

@areibman can I get the exact code to test this?

The langchain_callback_handler demands an update from our side for the latest LangChain versions. Our code is compatible with 0.1 but not >=0.2.

I have started working on this.

[areibman]

Additional context: @the-praxs https://github.com/VRSEN/agency-swarm/pull/199/files#diff-bde4ac85e74f65b45eb470ec7dbb717d6a7b9f9693de827f0abbd3c2e385504f

[the-praxs]

PR here: #659