Skip to content

Commit d7e208d

Browse files
committed
Fix double counting of request tokens in evals
1 parent de0aaa8 commit d7e208d

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

pydantic_evals/pydantic_evals/dataset.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -947,6 +947,8 @@ async def _run_once():
947947
# That way users can customize this logic. We'd default to a function that does the current thing but also
948948
# allow `None` to disable it entirely.
949949
for node in span_tree:
950+
if 'gen_ai.request.model' not in node.attributes:
951+
continue # we only want to count the below specifically for the individual LLM requests, not agent runs
950952
for k, v in node.attributes.items():
951953
if k == 'gen_ai.operation.name' and v == 'chat':
952954
task_run.increment_metric('requests', 1)

0 commit comments

Comments
 (0)