Pagination Issue Due to Missing Tag Filtering in Observations API

[jayapriyaperumal]

Describe the feature or potential improvement

Issue Description:

Our application retrieves observations using the observation method and filters them locally based on specific tags (e.g., local). However, since the method does not support filtering by tags directly:

1. Pagination Discrepancy:
The total count of results returned by the methods includes all observations, even those that do not match the desired tags. Consequently, the number of relevant results per page becomes inconsistent, causing pagination to either miss data or return fewer results than expected.

2. Increased Overhead:
As a workaround, we currently fetch additional observations (more than the requested limit) and filter them manually. While this ensures some consistency, it leads to higher data usage and processing overhead, especially for large datasets.

Request for Enhancement:

Would it be possible to include tag-based filtering as a supported query parameter in the fetch_observations methods? This feature would allow us to:
• Retrieve only the relevant observations directly from the method.
• Maintain accurate pagination based on the filtered results.
• Reduce unnecessary data transfer and improve performance.

Additional information

GET /observations?user_id=jayapriya&tags=local&page=1&limit=10

By allowing tag filtering in the request, the method could return only the observations matching the specified tag(s), along with accurate metadata for pagination (e.g., total_count, total_pages). https://python.reference.langfuse.com/langfuse/client#Langfuse.fetch_observations

[marcklingen]

I've created a bug report here: #4684