DataSet Scores are not being displayed

[MaxYaz]

Hi!
I have a Dataset with two items in it. After I run an evaluation I can see scores for every item:

But the problem is the aggregated metric for a dataset run is not being displayed:

Any ideas why this can be the case?

[dosubot[bot]]

[MaxYaz]

@dosu The output was evaluated and scored as it is shown in the screenshot

[dosubot[bot]]

The aggregated metric for a dataset run might not be displayed because the Langfuse client was not flushed at the end of the experiment run. According to the documentation, it is essential to flush the Langfuse client to ensure all data is sent to the server at the end of the experiment run.

Here is the relevant code snippet:

# Flush the langfuse client to ensure all data is sent to the server at the end of the experiment run
langfuse.flush()

Additionally, ensure that your application function is decorated with the @observe decorator to automatically link the trace, and that you are properly scoring the output during the experiment run. This allows Langfuse to compare different runs more easily and aggregate the scores.

dataset = langfuse.get_dataset("<dataset_name>")

for item in dataset.items:
    with item.observe(
        run_name="<run_name>",
        run_description="My first run",
        run_metadata={"model": "llama3"},
    ) as trace_id:
        output = my_llm_application.run(item.input)
        langfuse.score(
            trace_id=trace_id,
            name="<example_eval>",
            value=my_eval_fn(item.input, output, item.expected_output),
            comment="This is a comment",
        )

langfuse.flush()

Make sure you include these steps in your experiment run to ensure that all data, including scores, is properly sent and aggregated ^[1][2].

To continue talking to Dosu, mention @dosu.

[MaxYaz]

@dosu I do have langfuse.flush() at the end of the run.

[dosubot[bot]]

The aggregated metric for a dataset run might not be displayed even if you have langfuse.flush() at the end of the run if the output of the application is not being evaluated and scored correctly. Ensure that you are using the langfuse.score function to evaluate the output and compare different runs. Here is an example of how to do this:

dataset = langfuse.get_dataset("<dataset_name>")

for item in dataset.items:
    with item.observe(
        run_name="<run_name>",
        run_description="My first run",
        run_metadata={"model": "llama3"},
    ) as trace_id:
        output = my_llm_application.run(item.input)
        
        langfuse.score(
            trace_id=trace_id,
            name="<example_eval>",
            value=my_eval_fn(item.input, output, item.expected_output),
            comment="This is a comment",
        )

langfuse.flush()

Ensure that the langfuse.score function is called with appropriate parameters to evaluate the output. Additionally, make sure that you are properly linking the dataset items to the traces and runs, as shown in the integration tests. This will allow the aggregated metric to be displayed in the dataset runs table ^[1][2][3].

I've analyzed the situation and it looks like we need your specialized knowledge, @marcklingen.

To continue talking to Dosu, mention @dosu.

[MaxYaz]

I have everything linked:
Item Scores in a Run:

Traces:

Scores:

[marcklingen]

thanks for sharing @MaxYaz, i think the issue here is that these are observation level scores that do not get aggregated on the dataset run level, correct?

[MaxYaz]

@marcklingen Hi!
Yes, you are absolutely correct!

[marcklingen]

thanks for confirming, i'll create an issue to fix this

[MaxYaz]

Thank you

[marcklingen]

#3310

[marcklingen]

Fixed: #3310

Thanks again for reporting this.