-
-
Notifications
You must be signed in to change notification settings - Fork 353
Add OpenTelemetry observability support #556
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
pcomans
wants to merge
19
commits into
crmne:main
Choose a base branch
from
chime:otel-contrib
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add instrumentation module with span building for chat completions and tool calls - Support session tracking across conversation turns - Add configurable metadata prefix for custom trace attributes - Implement NullTracer pattern for graceful fallback when OTel not installed - Fail loudly with helpful error when tracing enabled but OTel unavailable - Add comprehensive documentation for LangSmith and other backends - Include full test coverage for instrumentation
Instrumentation.enabled? and Instrumentation.tracer now accept an optional config parameter, allowing Chat to pass its context-specific config instead of always using the global RubyLLM.config. This ensures that per-context tracing configuration is respected.
Convert AR record ID to string when passing to Chat#session_id to ensure type consistency. Chat defaults to UUID (string) when no session_id is provided, so AR adapters should also pass strings.
OTel supports string, int, float, and bool attribute values natively. Only stringify complex objects that don't fit these types. This improves filtering and querying capabilities in observability backends.
Replace attr_reader :metadata with a method that returns @metadata.dup to prevent external mutation of internal state. Callers must use with_metadata to modify the hash.
Add tracing_langsmith_compat config option (default: false) that: - Adds langsmith.span.kind attribute to spans - Adds input.value/output.value for LangSmith panels - Auto-sets tracing_metadata_prefix to 'langsmith.metadata' Standard gen_ai.* attributes are always emitted regardless of this setting. This allows users of other OTel backends to avoid the LangSmith-specific noise while LangSmith users get full integration. Also replaces the made-up gen_ai.prompt/gen_ai.completion fallback attributes with proper gen_ai.tool.input/gen_ai.tool.output for tool spans.
Use "#{self.class.name}:#{id}" format (e.g., "ChatSession:123")
instead of just the ID. This prevents session_id collisions when
multiple models use acts_as_chat, ensuring traces are correctly
grouped per-model in observability backends.
Update What Gets Traced section to show gen_ai.tool.input/output (always emitted) vs input.value/output.value (LangSmith-only). Add LangSmith-specific attribute tables to both Chat and Tool sections.
- Remove tracer memoization to handle OTel reconfiguration - Warn and disable (not crash) when OTel gems missing (follows OTel's 'safe by default' pattern) - Revert metadata prefix when disabling langsmith_compat - JSON encode Hash/Array metadata values instead of .to_s garbage
Remove otel_safe_value method - just pass values through and let the OpenTelemetry SDK handle type conversion. No need to reinvent the wheel.
- Use attr_reader for tracing_langsmith_compat (custom setter defined) - Consolidate build_request_attributes params into config hash - Use allow/have_received pattern for logger spy tests - Fix Model::Info double in tests
74e06eb to
62ee78b
Compare
- Unified complete() to always wrap in span (streaming + non-streaming) - Extracted add_request_span_attributes and add_response_span_attributes helpers - Added streaming documentation section - Added 4 streaming instrumentation tests
- Rename gen_ai.system to gen_ai.provider.name - Rename gen_ai.tool.input to gen_ai.tool.call.arguments - Rename gen_ai.tool.output to gen_ai.tool.call.result - Change tool operation name from 'tool' to 'execute_tool' - Add links to OTEL GenAI spec in observability docs See: https://opentelemetry.io/docs/specs/semconv/gen-ai/
- Introduced 'json' library to enhance metadata handling capabilities.
- Updated content logging to output prompts and completions as JSON arrays. - Introduced `gen_ai.input.messages` and `gen_ai.output.messages` attributes for structured message logging. - Refactored `build_message_attributes` and `build_completion_attributes` methods to generate OTEL-compliant message formats. - Updated tests to validate new JSON structure for input and output messages.
…observability - Updated span names in the complete and execute_tool methods to include specific identifiers for better traceability. - Enhanced logging clarity by dynamically generating span names based on model ID and tool call name.
Author
|
@crmne I'd love to get this merged. Let me know how I can support you. |
- Revised span naming conventions in the documentation for `chat.ask()` and tool invocation to include specific identifiers, enhancing traceability and alignment with industry standards.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add OpenTelemetry-based observability to RubyLLM, enabling users to send traces to any compatible backend (LangSmith, Honeycomb, New Relic, etc.).
Follows the OpenTelemetry Semantic Conventions for GenAI.
What's Included
chat {model_id}spans (e.g.,chat gpt-4o) with model, provider, token counts, and GenAI semantic convention attributesexecute_tool {tool_name}child spans (e.g.,execute_tool get_weather) with tool name, arguments, and resultsClassName:id) to prevent collisions across modelschat.with_metadata(user_id: 123)for filtering/debugging. Native types (int, float, bool) preserved for better backend filteringtracing_langsmith_compatmode adds LangSmith-specific attributes (langsmith.span.kind,input.value/output.valuepanels)RubyLLM.contextConfigurationErrorwith helpful message if tracing enabled but OpenTelemetry gems not installedConfiguration
Not in Scope
Intentionally deferred to keep this PR focused: