Skip to content

Conversation

@pcomans
Copy link

@pcomans pcomans commented Jan 6, 2026

Summary

Add OpenTelemetry-based observability to RubyLLM, enabling users to send traces to any compatible backend (LangSmith, Honeycomb, New Relic, etc.).

Follows the OpenTelemetry Semantic Conventions for GenAI.

What's Included

  • Chat completion tracing: chat {model_id} spans (e.g., chat gpt-4o) with model, provider, token counts, and GenAI semantic convention attributes
  • Streaming support: Single span wraps entire streaming operation; token counts and content recorded on completion
  • Tool call tracing: execute_tool {tool_name} child spans (e.g., execute_tool get_weather) with tool name, arguments, and results
  • Session tracking: UUID per Chat instance for grouping multi-turn conversations. ActiveRecord chats use namespaced record IDs (ClassName:id) to prevent collisions across models
  • Custom metadata: Attach arbitrary metadata via chat.with_metadata(user_id: 123) for filtering/debugging. Native types (int, float, bool) preserved for better backend filtering
  • Content logging: Opt-in prompt/completion logging with configurable truncation
  • LangSmith compatibility: Optional tracing_langsmith_compat mode adds LangSmith-specific attributes (langsmith.span.kind, input.value/output.value panels)
  • Config scoping: Tracing respects per-context configuration when using RubyLLM.context
  • Fail-fast validation: Raises ConfigurationError with helpful message if tracing enabled but OpenTelemetry gems not installed
  • NullTracer pattern: Minimal overhead when tracing disabled

Configuration

RubyLLM.configure do |config|
  config.tracing_enabled = true                  # Explicit opt-in required
  config.tracing_log_content = true              # Log prompts/completions (default: false)
  config.tracing_max_content_length = 10_000     # Truncation limit (default: 10000)
  config.tracing_langsmith_compat = true         # Add LangSmith-specific attributes (auto-sets metadata prefix)
end

Not in Scope

Intentionally deferred to keep this PR focused:

  • Other operations (embed, paint, transcribe, moderate): Lower priority; can add incrementally based on demand

- Add instrumentation module with span building for chat completions and tool calls
- Support session tracking across conversation turns
- Add configurable metadata prefix for custom trace attributes
- Implement NullTracer pattern for graceful fallback when OTel not installed
- Fail loudly with helpful error when tracing enabled but OTel unavailable
- Add comprehensive documentation for LangSmith and other backends
- Include full test coverage for instrumentation
Instrumentation.enabled? and Instrumentation.tracer now accept an
optional config parameter, allowing Chat to pass its context-specific
config instead of always using the global RubyLLM.config.

This ensures that per-context tracing configuration is respected.
Convert AR record ID to string when passing to Chat#session_id to
ensure type consistency. Chat defaults to UUID (string) when no
session_id is provided, so AR adapters should also pass strings.
OTel supports string, int, float, and bool attribute values natively.
Only stringify complex objects that don't fit these types. This
improves filtering and querying capabilities in observability backends.
Replace attr_reader :metadata with a method that returns @metadata.dup
to prevent external mutation of internal state. Callers must use
with_metadata to modify the hash.
Add tracing_langsmith_compat config option (default: false) that:
- Adds langsmith.span.kind attribute to spans
- Adds input.value/output.value for LangSmith panels
- Auto-sets tracing_metadata_prefix to 'langsmith.metadata'

Standard gen_ai.* attributes are always emitted regardless of this
setting. This allows users of other OTel backends to avoid the
LangSmith-specific noise while LangSmith users get full integration.

Also replaces the made-up gen_ai.prompt/gen_ai.completion fallback
attributes with proper gen_ai.tool.input/gen_ai.tool.output for
tool spans.
Use "#{self.class.name}:#{id}" format (e.g., "ChatSession:123")
instead of just the ID. This prevents session_id collisions when
multiple models use acts_as_chat, ensuring traces are correctly
grouped per-model in observability backends.
Update What Gets Traced section to show gen_ai.tool.input/output
(always emitted) vs input.value/output.value (LangSmith-only).
Add LangSmith-specific attribute tables to both Chat and Tool sections.
- Remove tracer memoization to handle OTel reconfiguration
- Warn and disable (not crash) when OTel gems missing (follows OTel's
  'safe by default' pattern)
- Revert metadata prefix when disabling langsmith_compat
- JSON encode Hash/Array metadata values instead of .to_s garbage
Remove otel_safe_value method - just pass values through and let the
OpenTelemetry SDK handle type conversion. No need to reinvent the wheel.
- Use attr_reader for tracing_langsmith_compat (custom setter defined)
- Consolidate build_request_attributes params into config hash
- Use allow/have_received pattern for logger spy tests
- Fix Model::Info double in tests
@pcomans pcomans marked this pull request as ready for review January 6, 2026 21:47
@pcomans pcomans force-pushed the otel-contrib branch 2 times, most recently from 74e06eb to 62ee78b Compare January 6, 2026 22:08
- Unified complete() to always wrap in span (streaming + non-streaming)
- Extracted add_request_span_attributes and add_response_span_attributes helpers
- Added streaming documentation section
- Added 4 streaming instrumentation tests
- Rename gen_ai.system to gen_ai.provider.name
- Rename gen_ai.tool.input to gen_ai.tool.call.arguments
- Rename gen_ai.tool.output to gen_ai.tool.call.result
- Change tool operation name from 'tool' to 'execute_tool'
- Add links to OTEL GenAI spec in observability docs

See: https://opentelemetry.io/docs/specs/semconv/gen-ai/
- Introduced 'json' library to enhance metadata handling capabilities.
- Updated content logging to output prompts and completions as JSON arrays.
- Introduced `gen_ai.input.messages` and `gen_ai.output.messages` attributes for structured message logging.
- Refactored `build_message_attributes` and `build_completion_attributes` methods to generate OTEL-compliant message formats.
- Updated tests to validate new JSON structure for input and output messages.
…observability

- Updated span names in the complete and execute_tool methods to include specific identifiers for better traceability.
- Enhanced logging clarity by dynamically generating span names based on model ID and tool call name.
@pcomans
Copy link
Author

pcomans commented Jan 6, 2026

@crmne I'd love to get this merged. Let me know how I can support you.

- Revised span naming conventions in the documentation for `chat.ask()` and tool invocation to include specific identifiers, enhancing traceability and alignment with industry standards.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant