Bug Report: LangChain Metrics issues

[nirga]

1. I see that we are adding custom attributes to metrics, However, TRACELOOP_ENTITY_NAME (traceloop.entity.name) is a very high cardinality attribute - https://github.com/traceloop/openllmetry/pull/2154/files#diff-caff857eafc9121a585e40cdac1c0815d97f474cd3d0bf1a23755572e3b38105R473

• Most destinations would have limits on such fields with very high cardinality for metrics.
• I feel this kind of information is best understood with traces/spans instead or at-least some control should be given to enable/disable this metric as required or limit attributes.

2. Seeing this error generated for gen_ai.response.model when invoked as a agent.

Failed to encode key gen_ai.response.model: Invalid type <class 'NoneType'> of value None
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py", line 113, in _encode_attributes
    pb2_attributes.append(_encode_key_value(key, value))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py", line 94, in _encode_key_value
    return PB2KeyValue(key=key, value=_encode_value(value))
                                      ^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py", line 90, in _encode_value
    raise Exception(f"Invalid type {type(value)} of value {value}")
Exception: Invalid type <class 'NoneType'> of value None

3. I do not see gen_ai.client.token.usage being generated when used like below, I do see it generated with simple LLMChain but not when invoked via agent.

    llm = AzureChatOpenAI(
        openai_api_version=os.environ.get("AZURE_OPENAI_VERSION"),
        azure_deployment=os.environ.get("AZURE_OPENAI_DEPLOYMENT"),
        azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
        api_key=os.environ.get("AZURE_OPENAI_KEY"),
        model="gpt-4"
        )
    event_message = "this is a test"
    tools = [Tool().useless_tool]
    
    agent_template = categorise_event_prompt
    input_template = "{input}"
    
    prompt = ChatPromptTemplate.from_messages([
        ("system", agent_template),
        ("human", input_template),
        MessagesPlaceholder("agent_scratchpad")
    ])
    agent = create_openai_tools_agent(llm=llm, tools=tools, prompt=prompt)
    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
    result = agent_executor.invoke({"input": event_message})

Originally posted by @nitin302 in #2154 (comment)

[nirga]

@nitin302 - 1 and 2 are resolved. 3 is a bit more tricky, there seem to be a bug with LangChain specifically for agents. I've printed everything that is returned on the callback mechanism and apparently they're not returning any token usage data for some reason:

response generations=[[Generation(text=" I should use the Search tool to search for the artist's name and their album in the FooBar database.\nAction: Search\nAction Input: tool_input: 'Union[str, dict[str, Any]]' = 'The Storm Before the Calm'", generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {}, 'model_name': 'gpt-3.5-turbo-instruct'} run=None type='LLMResult'
response.llm_output {'token_usage': {}, 'model_name': 'gpt-3.5-turbo-instruct'}

This is when running this for example. I'll continue investigating so keeping this open (and merging the PR that solves 1 and 2).

[nitin302]

sure @nirga

1. I've tested fix for 1 and 2 and it works as expected, although I am unclear why model_name would not be present.
2. Do you think we should raise a bug on the upstream LangChain repository for 3?

[nirga]

Yes @nitin302 - let me know if you plan on doing that and if not I'll do it later. Same goes for model_name - looks like a langchain bug.

[nitin302]

@nirga could you do it, I think you can give more context on the exact line of the issue.

[nirga]

Sorry for the delay here @nitin302 was about to do that but I see you've already done that. Let's see if they follow up there