Strategy to build a analytics storage from langfuse data.

[dahnny012]

Describe your question

Hello,

We want to integrate langfuse within out data pipelines. The only issue is the processors we use for data pipelines only accept s3/glue ice berg. We are thinking of building a data pipeline that exports data from langfuse observations, traces, scores every hour.

For this export what is the best strategy to export this data?

1. Should we use the API?
2. Should we pull from Clickhouse directly?
3. Should we use the export feature and build a reactive system based on that?

Another question about this.

I know there will be changes to observations and a new events table. How should we navigate this upgrade when designing this workflow.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.148

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.