[Bug]: LangChain callback handler on_llm_start/end issue #656
Description
Contact Details
No response
📦 Package Version
0.3.24
🎞️ Framework Version
Langchain
🔎 Describe the Bug
Summary:
This issue pertains to the integration of Langchain callbacks into the Agency-Swarm framework. Despite successful implementation of observability, some key messages from the agent team are missing in the logs. Additionally, there are errors related to unsupported chat model callbacks.
Details:
1. Integration Overview:
• Langchain callbacks (on_llm_start, on_llm_end) have been integrated into the Agency-Swarm framework (not to be confused with the “Swarms” framework).
• Relevant changes can be reviewed in thread.py, particularly around the on_llm_start and on_llm_end methods, as well as agency.py and other associated files.
2. Observability Issue:
• First and last messages from the agent team are not appearing in the logs despite being shown correctly in Langfuse.
• Root cause of this issue is unclear and requires investigation.
3. Error with AgentOps Callback Handler:
• Using the AgentOps callback handler throws the following error:
NotImplementedError: LangchainCallbackHandler does not implement on_chat_model_start
• It seems chat models are not currently supported. The assumption is that on_llm_start should be used instead, but confirmation is needed.
Questions/Next Steps:
1. Investigate why the first and last messages from the agent team are missing, even though they appear in Langfuse.
2. Clarify the correct implementation of on_llm_start for observability in the absence of support for chat models.
3. Determine if additional support for on_chat_model_start needs to be added or if existing methods can handle chat models.
Relevant Files:
• thread.py (focus: on_llm_start, on_llm_end)
• agency.py and other files where these methods are defined.
Context:
• User implementing observability for agent-based frameworks during holiday season.
• Question originated from a user attempting to use AgentOps callback handler with Langchain.
🤝 Contribution
- Yes, I'd be happy to submit a pull request with these changes.
- I need some guidance on how to contribute.
- I'd prefer the AgentOps team to handle this update.