Self-hosted Langfuse running as a Google cloud service not showing traces

[SectumPsempra]

Describe the bug

This is from a production setup running on google cloud as cloud run services.

Setup:

• Languse is running as a cloud run service. Lets call it lang-service.
• There is another chatbot LLM service which is running on cloud run. Lets call it llm-service.
• Whenever llm-service receives a new message it calls graph.inoke with a config in which langfuse callback handler is passed. So for every LLM generation the trace should get recorded.
• llm-service have these environment variables:
  • LANGFUSE_HOST
  • NEXTAUTH_URL
  • DATABASE_URL
  • DIRECT_URL
  • LANGFUSE_PUBLIC_KEY
  • LANGFUSE_SECRET_KEY
  • LANGFUSE_DEBUG
  • NEXTAUTH_SECRET
  • SALT
  • LANGFUSE_FLUSH_AT

What is working:

• I am able to run langfuse locally as docker services and see the traces/generations in the langfuse dashboard running locally.
• Able to run and see the traces on cloud service(lang-service) when running llm service from local as a docker container. Basically the cloud langfuse is accessible and working when calling it from local by just modifying NEXTAUTH_URL, SECRET_KEY and PUBLIC_KEY.

What is not working:

• When using lang-service from llm-service both deployed on cloud run, the trace, span generation is happening but it is not printing the ‘uploading data’ in logs in DEBUG mode as it does for local. It is not calling the ingestion API. Basically nothing in the task_manager.Consumer.run is getting logged or called.
• In DEBUG mode, there are no error messages or exceptions. After printing ‘Creating span’ it just goes off.

For debugging purposes we have added langfuse_context.flush() in the code and also set the LANGFUSE_FLUSH_AT=1.

Any help or suggestion will be greatly appreciated. Thank you.

To reproduce

• Follow the self-hosting langfuse guide
• Deploy a service on google cloud run - main code, Region: us-central1
• Deploy langfuse service as a cloud run service, Region: us-central1
• Set the environment variables defined here - https://langfuse.com/docs/deployment/self-host#configuring-environment-variables in the main service
• Submit a message to the main service
• Check the Langfuse dashboard running on LANGFUSE_HOST or NEXTAUTH_URL

SDK and container versions

Image for langfuse service - langfuse/langfuse:2.69.0
Gcloud Postgresql version - 16.3
langgraph==0.1.19
langfuse==2.39.3
langchain-openai - latest
langchain - latest

Additional information

No response

Are you interested to contribute a fix for this bug?

No

[marcklingen]

which python version do you use? afaik langgraph tarcing only works on more recent python versions, more on this here: #1926

moving this to a support thread

[SectumPsempra]

We are passing it as defined in the langchain integration on the langfuse docs page, like: https://langfuse.com/docs/integrations/langchain/example-python-langgraph#add-langfuse-as-callback-to-the-invocation passing it to graph.invoke as part of config parameter and the langfuse handler is created using secret_key, public_key and host.

Yes to your second question, it is part of the same cloud service.

[marcklingen]

this sounds good, i'd try to call handler.flush() after the graph response is completed

[SectumPsempra]

Yeah I am flushing at the end of graph already.

Updates:

• Tried with Python 3.11 but no change in behavior.
• I noticed this, the last message is coming from langfuse.client.StatefulSpanClient’s update method self.log.debug(f"Update span {span_body}..."). After this line it again goes into the ever printing langfuse - DEBUG - ~0 items in the Langfuse queue messages.

One question about flush, Do we need to do it like:

from langfuse.decorators import langfuse_context
langfuse_context.flush()

or use the same handler that was passed to config and do .flush() on that? Because I have tried both ways and that is not making any difference.

Are there any consumer timeouts? or any other timeouts that can be increased related to uploads, etc?

[marcklingen]

Thanks for the quick call, the issue here was that the CallbackHandler was initialized independently from the @observe decorator. We changed to langfuse_context.get_current_langchain_handler() whcih should resolve the issue as langfuse_context.flush() will flush the events created by this handler then.

[SectumPsempra]

@marcklingen Thank you so much, this indeed resolved the issue and I am able to see the traces finally in the dashboard. We deeply appreciate your help and quick response.

Thanks again.

[marcklingen]

awesome, thanks for confirming, glad that this works as expected!

[SectumPsempra]

Resolved.