Feature Request & Performance Concerns: Trace Deletion, Dataset Retrieval, and SDK Filtering

[dadastory]

Describe the feature or potential improvement

First of all, thanks for building Langfuse — it’s a very promising project and we really hope it continues to grow into a more robust and production-ready observability platform for LLM systems.
While using it at scale, we’ve encountered several limitations that significantly affect usability and performance. We’d like to share them here in the hope of making the project even better.

---

1. Trace Deletion Limitations (UI & Python SDK)

Currently, trace deletion is quite limited and slow:

• ❌ Web UI does not support batch deletion of traces
• ❌ Python SDK does not support batch deletion, nor reliable single-trace deletion
• 🐌 Deletion operations are extremely slow when traces scale to large volumes

This becomes a serious issue in long-running or high-throughput environments where traces accumulate rapidly.

Questions / Suggestions:

• Are there plans to support batch delete APIs for traces?
• Would you consider storing traces in Elasticsearch (or similar OLAP / log-oriented storage) instead of (or in addition to) the current database backend, to improve deletion and query performance?

---

2. Dataset Retrieval Is Not Streamed / Paginated

We noticed that:

• client.get_dataset() fetches the entire dataset at once
• There is no pagination or streaming mechanism
• With datasets containing tens of thousands of rows, this easily leads to:
  • request timeouts
  • excessive memory usage
  • the client completely hanging or crashing

This behavior makes it very difficult to manage or inspect large datasets programmatically.

Suggestion:

• Add pagination / streaming / iterator-style APIs for dataset retrieval
• Or at least provide a way to specify limits, offsets, or chunk sizes

---

3. Missing Metadata-Based Filtering in Python SDK

In the Python SDK:

• ❌ No support for filtering traces by metadata
• ❌ No support for filtering datasets by metadata
• This makes it hard to:
  • select subsets of data
  • build automated pipelines
  • implement conditional analysis or cleanup logic

Compared to the UI and common observability tooling expectations, this feels like a major functional gap.

Suggestion:

• Add metadata-based filtering options to SDK methods for:
  • traces
  • datasets
  • observations / spans (if applicable)

---

Closing Thoughts

We genuinely believe Langfuse has great potential and is already very useful.
Addressing the issues above would significantly improve its scalability, developer experience, and suitability for real-world production workloads.

Thanks a lot for your work on the project — looking forward to future improvements! 🚀

Additional information

No response

[jannikmaierhoefer]

Hi @dadastory, thanks for adding these points.

1. thanks for flagging - we are currently working on improving the performance of this feature. For scheduled deletions, we would recommend using the data retention feature (https://langfuse.com/docs/administration/data-retention) (available with an enterprise license).
2. Agree that this would be great to have - I added this as a separate roadmap item here and will forward this to the team.
3. The observations v2 API supports filtering for metadata and will be available for self-hosters soon.