Missing scores after pushing scores to Langfuse

[laisalvescw]

I am following the cookbook from this link:
https://langfuse.com/guides/cookbook/evaluation_of_rag_with_ragas#scoring-as-batch
I am facing an issue when I execute the last code snippet to push the scores back to Langfuse (in Batch):
for _, row in df.iterrows(): for metric_name in ["faithfulness", "answer_relevancy"]: langfuse.score( name=metric_name, value=row[metric_name], trace_id=row["trace_id"] )
The df contains the following columns: "user_input (str),retrieved_contexts (list(str)),response (str),faithfulness(float),answer_relevancy(float),trace_id"
"

The first time I have scored around 30 traces, but when I executed this code, but I only see one trace scored. The second time, I scored all the traces and fetched the scores back to Langfuse, but only a few showed the scores.
The code doesn't show any error.
The df contains all the scores, with just a few missings in the faithfulness column.

[dosubot[bot]]

There are a few potential reasons why scores might not be recorded as expected in Langfuse:

1. Rate Limiting: The Langfuse API has rate limits that could affect the scoring process. If you exceed the limit of 1000 batches per minute for Hobby/Pro users or 5000 batches per minute for Team users, the API will respond with a status code 429, and you should retry after the specified time ^[1].

2. Trace ID Issues: Ensure that the trace_id is correctly set for each score. The trace_id is a required parameter, and if it's missing or incorrect, the score won't be attached to the correct trace ^[2].

3. Data Type and Format: Verify that the value parameter for each score is in the correct format. For numeric scores, the value should be a number, often standardized to a range of 0 to 1 ^[3][4].

4. Debugging and Error Handling: Enable debugging in the Langfuse SDK to get detailed logs of the operations. This can help identify any issues with the SDK or network problems. You can also set up an error handler to catch any errors that might not be immediately visible in the code ^[5][6].

5. Batch Processing: Ensure that the batch processing logic is correctly implemented and that all iterations over the DataFrame are executed as expected.

By checking these aspects, you should be able to diagnose and resolve the issue with missing scores.

To continue talking to Dosu, mention @dosu.

[jannikmaierhoefer]

Hi @laisalvescw, can you see those scores in the Scores table in the Langfuse UI?

[laisalvescw]

I identified that the issue was due to missing values in the 'faithfulness' column, which were causing JSON formatting errors. After filling in these missing values, the API began working smoothly. This wasn’t immediately obvious because the number of missing values differed from the number of scores that appeared missing in the Langfuse UI. With the values now updated, the issue appears to be resolved.

[jannikmaierhoefer]

Thanks for letting us know!