LangFuse not logging LlamaIndex events

[adieyal]

I've read through the documentation and FAQs but I can't figure out why I am not managing to get LangFuse to log LlamaIndex events. Here is an example session that shows that my LangFuse configuration is correctly set-up while events are not being recorded.

from llama_index.core.callbacks import CallbackManager
from langfuse.llama_index import LlamaIndexCallbackHandler
from llama_index.llms.openai import OpenAI

langfuse_handler = LlamaIndexCallbackHandler(user_id="test")
callback_manager = CallbackManager([langfuse_handler])
llm = OpenAI(model="gpt-4o", api_key=OPENAI_API_KEY, callback_manager=callback_manager)

# No events are logged in LangFuse after the completion
result = llm.complete("Hello world")
print(result)
# CompletionResponse(text='Hello! How can I assist you today?', additional_kwargs={}, raw=ChatCompletion(id='chatcmpl-9tXX7Xnu2Xu5mxblaPF3tHOwdAeRr', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Hello! How can I assist you today?', role='assistant', function_call=None, tool_calls=None, refusal=None))], created=1723023969, model='gpt-4o-2024-05-13', object='chat.completion', system_fingerprint='fp_3aa7262c27', usage=CompletionUsage(completion_tokens=9, prompt_tokens=9, total_tokens=18)), logprobs=None, delta=None)

# Checking to ensure that the calllback handler is set
handler = llm.callback_manager.handlers[0]
type(handler)
# <class 'langfuse.llama_index.llama_index.LlamaIndexCallbackHandler'>

# Triggering a trace manually results in an event being logged by LangFuse.
handler.langfuse.trace().update()
# <langfuse.client.StatefulTraceClient object at 0x7f8423a86b20>

I have also tried the same procedure using the global LlamaIndex Settings object. The observe decorator also works, but passing the handler from the langfuse_context to the Callback Manager only records the observe logs and not LlamaIndex.

Here are my library versions:

langfuse==2.42.1
llama-index-callbacks-langfuse==0.1.4
llama-index==0.10.62

Any advice would be appreciated.

[dosubot[bot]]

It looks like you might be missing a crucial step to ensure that all events are flushed before your application exits, especially in short-lived applications. You need to call langfuse_handler.flush() to ensure that all events are properly logged. Here’s how you can modify your code:

from llama_index.core.callbacks import CallbackManager
from langfuse.llama_index import LlamaIndexCallbackHandler
from llama_index.llms.openai import OpenAI

langfuse_handler = LlamaIndexCallbackHandler(user_id="test")
callback_manager = CallbackManager([langfuse_handler])
llm = OpenAI(model="gpt-4o", api_key=OPENAI_API_KEY, callback_manager=callback_manager)

# No events are logged in LangFuse after the completion
result = llm.complete("Hello world")
print(result)

# Ensure all events are flushed before the application exits
langfuse_handler.flush()

Additionally, ensure that your LlamaIndexCallbackHandler is correctly initialized and integrated with your application. Here’s a quick checklist:

1. Initialization:
   Ensure that the LlamaIndexCallbackHandler is initialized with the correct parameters.

   handler = LlamaIndexCallbackHandler(
       public_key="your_public_key",
       secret_key="your_secret_key",
       host="your_host",
       user_id="test"
   )

2. Setting Root and Trace Parameters:
   Use the set_root and set_trace_params methods appropriately.

   handler.set_root(root_trace)
   handler.set_trace_params(name="trace_name", user_id="user_id", session_id="session_id")

3. Imports:
   Ensure you have the necessary imports.

   from llama_index.core.callbacks.schema import CBEventType, CBEvent
   from langfuse.llama_index.utils import CallbackEvent, ParsedLLMEndPayload

By following these steps, you should be able to ensure that LlamaIndex events are logged correctly in LangFuse ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[marcklingen]

Hi @adieyal, do you run this locally (script / web server) or on a long-running instance?

Depending on how you run Langfuse, you need to manually flush as mentioned above. Have a look here for more hints on how to debug this: https://langfuse.com/faq/all/missing-traces

[adieyal]

Thanks for the quick response. I had tried flushing previously but it did not have any effect.
I just tried again from both a command line script as well as inside a web request. Both times calling flush. I also set the root_trace as suggested by dosubot. The behaviour hasn't changed.

When creating the handler with debug enabled, ie.:

LlamaIndexCallbackHandler(public_key=LANGFUSE_PUBLIC_KEY, secret_key=LANGFUSE_SECRET_KEY, host=LANGFUSE_HOST, debug=True)

I get a stream of the following

DEBUG 2024-08-07 08:55:30,820 langfuse(121) ~0 items in the Langfuse queue

At no time is anything added to the queue.

I have also looked through the missing-traces debug advice - I don't think there is anything new there that I haven't tried. The fact that I can manually trigger a trace suggests that my configuration of the Langfuse API is correct.

Given the message above about 0 items in the queue, it seems like Langfuse is not being notified of the events by LlamaIndex. In case there was some issue with the completion event, I also tried the chat api with the same result.