Skip to content

Comments

feat(llm-observability): $ai_tools capture in Langchain#199

Merged
Twixes merged 5 commits intomasterfrom
ai-callback-tools
Feb 27, 2025
Merged

feat(llm-observability): $ai_tools capture in Langchain#199
Twixes merged 5 commits intomasterfrom
ai-callback-tools

Conversation

@Twixes
Copy link
Member

@Twixes Twixes commented Feb 26, 2025

Tools are part of the LLM input prompt, and we haven't been capturing them at all in LLM observability!
This PR adds capturing of tools as $ai_tools to the Langchain callback handler. Does not touch SDK wrappers.

@Twixes Twixes requested a review from k11kirky February 26, 2025 23:41
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

This PR enhances LLM observability in the Langchain callback handler by adding support for capturing tools as $ai_tools, which are important components of LLM input prompts that were previously not being tracked.

  • Added tools field to GenerationMetadata class in posthog/ai/langchain/callbacks.py to store tool configurations provided to the model
  • Implemented tool extraction from invocation parameters in _set_llm_metadata method to capture tool data during LLM calls
  • Added $ai_tools property to event properties when capturing generation events for complete tool visibility
  • Created new test case test_metadata_tools in posthog/test/ai/langchain/test_callbacks.py to verify proper tool data capture and retrieval
  • Focused implementation only affects Langchain callback handler without modifying SDK wrappers

2 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile

"type": "string",
},
},
"required": ["query_description", "query_kind"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: The 'required' array references 'query_description' and 'query_kind' parameters that don't exist in the properties object, which only has 'bar'.

Suggested change
"required": ["query_description", "query_kind"],
"required": ["bar"],

@Twixes Twixes merged commit 4cda646 into master Feb 27, 2025
6 checks passed
@Twixes Twixes deleted the ai-callback-tools branch February 27, 2025 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants