Skip to content

Commit b972d72

Browse files
authored
Enhance CrewAI LLO Support in Genesis ADOT SDK (#365)
## What does this pull request do? Handles additional CrewAI LLO attributes not originally [in scope](https://quip-amazon.com/Dni9AztXMB2x/Genesis-observability-trace-data-attributes). Certain configurations of CrewAI Agents in customer applications can produce the following LLO attributes: - `gen_ai.agent.human_input` -> generated by [OpenLit](https://github.com/openlit/openlit/blob/9f285555330ae7c92f3382c105c44373b2c9a77d/sdk/python/src/openlit/semcov/__init__.py#L269C33-L269C57) - `gen_ai.agent.actual_output` -> generated by [OpenLit](https://github.com/openlit/openlit/blob/9f285555330ae7c92f3382c105c44373b2c9a77d/sdk/python/src/openlit/semcov/__init__.py#L268) - `crewai.crew.tasks_output` -> generated by [Traceloop/Openllmetry](https://github.com/traceloop/openllmetry/blob/de23561e4e45fc63a8f0020f15e68df525cf29c1/packages/opentelemetry-instrumentation-crewai/opentelemetry/instrumentation/crewai/crewai_span_attributes.py#L31) - `crewai.crew.result` -> generated by [Traceloop/Openllmetry](https://github.com/traceloop/openllmetry/blob/de23561e4e45fc63a8f0020f15e68df525cf29c1/packages/opentelemetry-instrumentation-crewai/opentelemetry/instrumentation/crewai/crewai_span_attributes.py#L31) Example CrewAI Agent Configurations: ``` assistant_agent = Agent( role="Assistant", goal="Provide helpful responses to user queries", backstory="You are a helpful assistant that provides accurate and useful information.", verbose=True, llm=llm, ) crew = Crew( agents=[assistant_agent], tasks=[response_task], verbose=True, process=Process.sequential ) ``` Related PR: #361 ## Test plan Built this custom ADOT SDK into various sample apps and exported the span and logs data to the OTLP X-Ray and Logs endpoint, respectively, to validate the LLO extraction and transformation to Gen AI Events. Configurations tested: - CrewAI + Traceloop/Openllmetry - CrewAI + OpenInference - CrewAI + OpenLit Environment variable configuration: ``` λ env OTEL_METRICS_EXPORTER=none \ OTEL_TRACES_EXPORTER=otlp \ OTEL_LOGS_EXPORTER=otlp \ OTEL_PYTHON_DISTRO=aws_distro \ OTEL_PYTHON_CONFIGURATOR=aws_configurator \ OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \ OTEL_EXPORTER_OTLP_LOGS_HEADERS="x-aws-log-group=test,x-aws-log-stream=default" \ OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \ OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs \ OTEL_RESOURCE_ATTRIBUTES="service.name=langchain-app" \ OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED="true" \ OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="true" \ OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="http,sqlalchemy,psycopg2,pymysql,sqlite3,aiopg,asyncpg,mysql_connector,botocore,boto3,urllib3,requests,starlette" \ AGENT_OBSERVABILITY_ENABLED="true" \ python app.py ``` By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
1 parent 409cb6a commit b972d72

File tree

2 files changed

+372
-112
lines changed

2 files changed

+372
-112
lines changed

aws-opentelemetry-distro/src/amazon/opentelemetry/distro/llo_handler.py

Lines changed: 61 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,15 @@
1717
# Framework-specific attribute keys
1818
TRACELOOP_ENTITY_INPUT = "traceloop.entity.input"
1919
TRACELOOP_ENTITY_OUTPUT = "traceloop.entity.output"
20+
TRACELOOP_CREW_TASKS_OUTPUT = "crewai.crew.tasks_output"
21+
TRACELOOP_CREW_RESULT = "crewai.crew.result"
2022
OPENINFERENCE_INPUT_VALUE = "input.value"
2123
OPENINFERENCE_OUTPUT_VALUE = "output.value"
2224
OPENLIT_PROMPT = "gen_ai.prompt"
2325
OPENLIT_COMPLETION = "gen_ai.completion"
2426
OPENLIT_REVISED_PROMPT = "gen_ai.content.revised_prompt"
27+
OPENLIT_AGENT_ACTUAL_OUTPUT = "gen_ai.agent.actual_output"
28+
OPENLIT_AGENT_HUMAN_INPUT = "gen_ai.agent.human_input"
2529

2630
# Roles
2731
ROLE_SYSTEM = "system"
@@ -51,11 +55,14 @@ class LLOHandler:
5155
- traceloop.entity.input: Input text for LLM operations
5256
- traceloop.entity.output: Output text from LLM operations
5357
- traceloop.entity.name: Name of the entity processing the LLO
58+
- crewai.crew.tasks_output: Tasks output data from CrewAI (uses gen_ai.system if available)
59+
- crewai.crew.result: Final result from CrewAI crew (uses gen_ai.system if available)
5460
5561
- OpenLit:
5662
- gen_ai.prompt: Direct prompt text (treated as user message)
5763
- gen_ai.completion: Direct completion text (treated as assistant message)
5864
- gen_ai.content.revised_prompt: Revised prompt text (treated as system message)
65+
- gen_ai.agent.actual_output: Output from CrewAI agent (treated as assistant message)
5966
6067
- OpenInference:
6168
- input.value: Direct input prompt
@@ -87,9 +94,13 @@ def __init__(self, logger_provider: LoggerProvider):
8794
self._exact_match_patterns = [
8895
TRACELOOP_ENTITY_INPUT,
8996
TRACELOOP_ENTITY_OUTPUT,
97+
TRACELOOP_CREW_TASKS_OUTPUT,
98+
TRACELOOP_CREW_RESULT,
9099
OPENLIT_PROMPT,
91100
OPENLIT_COMPLETION,
92101
OPENLIT_REVISED_PROMPT,
102+
OPENLIT_AGENT_ACTUAL_OUTPUT,
103+
OPENLIT_AGENT_HUMAN_INPUT,
93104
OPENINFERENCE_INPUT_VALUE,
94105
OPENINFERENCE_OUTPUT_VALUE,
95106
]
@@ -213,8 +224,8 @@ def _emit_llo_attributes(
213224
214225
Supported frameworks:
215226
- Standard Gen AI: Structured prompt/completion with roles
216-
- Traceloop: Entity input/output
217-
- OpenLit: Direct prompt/completion/revised prompt
227+
- Traceloop: Entity input/output and CrewAI outputs
228+
- OpenLit: Direct prompt/completion/revised prompt and agent outputs
218229
- OpenInference: Direct values and structured messages
219230
220231
Args:
@@ -408,9 +419,15 @@ def _extract_traceloop_events(
408419
Processes Traceloop-specific attributes:
409420
- `traceloop.entity.input`: Input data (uses span.start_time)
410421
- `traceloop.entity.output`: Output data (uses span.end_time)
411-
- `traceloop.entity.name`: Used as the gen_ai.system value
422+
- `traceloop.entity.name`: Used as the gen_ai.system value when gen_ai.system isn't available
423+
- `crewai.crew.tasks_output`: Tasks output data from CrewAI (uses span.end_time)
424+
- `crewai.crew.result`: Final result from CrewAI crew (uses span.end_time)
412425
413-
Creates generic `gen_ai.{entity_name}.message` events for both input and output.
426+
Creates generic `gen_ai.{entity_name}.message` events for both input and output,
427+
and assistant message events for CrewAI outputs.
428+
429+
For CrewAI-specific attributes (crewai.crew.tasks_output and crewai.crew.result),
430+
uses span's gen_ai.system attribute if available, otherwise falls back to traceloop.entity.name.
414431
415432
Args:
416433
span: The source ReadableSpan containing the attributes
@@ -422,12 +439,14 @@ def _extract_traceloop_events(
422439
"""
423440
events = []
424441
span_ctx = span.context
442+
# Use traceloop.entity.name for the gen_ai.system value
425443
gen_ai_system = span.attributes.get("traceloop.entity.name", "unknown")
426444

427445
# Use helper methods to get appropriate timestamps
428446
input_timestamp = self._get_timestamp(span, event_timestamp, is_input=True)
429447
output_timestamp = self._get_timestamp(span, event_timestamp, is_input=False)
430448

449+
# Standard Traceloop entity attributes
431450
traceloop_attrs = [
432451
(TRACELOOP_ENTITY_INPUT, input_timestamp, ROLE_USER), # Treat input as user role
433452
(TRACELOOP_ENTITY_OUTPUT, output_timestamp, ROLE_ASSISTANT), # Treat output as assistant role
@@ -450,6 +469,32 @@ def _extract_traceloop_events(
450469
)
451470
events.append(event)
452471

472+
# CrewAI-specific Traceloop attributes
473+
# For CrewAI attributes, prefer gen_ai.system if available, otherwise use traceloop.entity.name
474+
crewai_gen_ai_system = span.attributes.get("gen_ai.system", gen_ai_system)
475+
476+
crewai_attrs = [
477+
(TRACELOOP_CREW_TASKS_OUTPUT, output_timestamp, ROLE_ASSISTANT),
478+
(TRACELOOP_CREW_RESULT, output_timestamp, ROLE_ASSISTANT),
479+
]
480+
481+
for attr_key, timestamp, role in crewai_attrs:
482+
if attr_key in attributes:
483+
event_attributes = {"gen_ai.system": crewai_gen_ai_system, "original_attribute": attr_key}
484+
body = {"content": attributes[attr_key], "role": role}
485+
486+
# For CrewAI outputs, use the assistant message event
487+
event_name = GEN_AI_ASSISTANT_MESSAGE
488+
489+
event = self._get_gen_ai_event(
490+
name=event_name,
491+
span_ctx=span_ctx,
492+
timestamp=timestamp,
493+
attributes=event_attributes,
494+
body=body,
495+
)
496+
events.append(event)
497+
453498
return events
454499

455500
def _extract_openlit_span_event_attributes(
@@ -462,10 +507,11 @@ def _extract_openlit_span_event_attributes(
462507
- `gen_ai.prompt`: Direct prompt text (treated as user message)
463508
- `gen_ai.completion`: Direct completion text (treated as assistant message)
464509
- `gen_ai.content.revised_prompt`: Revised prompt text (treated as system message)
510+
- `gen_ai.agent.actual_output`: Output from CrewAI agent (treated as assistant message)
465511
466512
The event timestamps are set based on attribute type:
467513
- Prompt and revised prompt: span.start_time
468-
- Completion: span.end_time
514+
- Completion and agent output: span.end_time
469515
470516
Args:
471517
span: The source ReadableSpan containing the attributes
@@ -487,6 +533,16 @@ def _extract_openlit_span_event_attributes(
487533
(OPENLIT_PROMPT, prompt_timestamp, ROLE_USER), # Assume user role for direct prompts
488534
(OPENLIT_COMPLETION, completion_timestamp, ROLE_ASSISTANT), # Assume assistant role for completions
489535
(OPENLIT_REVISED_PROMPT, prompt_timestamp, ROLE_SYSTEM), # Assume system role for revised prompts
536+
(
537+
OPENLIT_AGENT_ACTUAL_OUTPUT,
538+
completion_timestamp,
539+
ROLE_ASSISTANT,
540+
), # Assume assistant role for agent output
541+
(
542+
OPENLIT_AGENT_HUMAN_INPUT,
543+
prompt_timestamp,
544+
ROLE_USER,
545+
), # Assume user role for agent human input
490546
]
491547

492548
for attr_key, timestamp, role in openlit_event_attrs:

0 commit comments

Comments
 (0)