How to limit test case execution time in pydantic-evals? / How to handle long-running test cases?

[baek54321]

Question

I have a test set with 40+ test cases. I primarily use custom evaluators, some of which also use LLM calls to evaluate an output. Some cases take a very long time to execute and potentially even time out. This regularly causes my whole experiment / test set evaluation to crash.

Is there a way how to limit the execution time for a test case? Or some other strategy to handle long-running test cases and prevent them from crashing the whole evolution?

Additional Context

• Python 3.13.3
• pydantic-ai 1.0.0

[DouweM]

@baek54321 Hmm, as of #2295 an exception in one case or evaluator should no longer be causing the entire evaluation to fail. That PR also adds support for custom retry logic for tasks and evaluators: https://ai.pydantic.dev/api/pydantic_evals/dataset/#pydantic_evals.dataset.Dataset.evaluate

Pydantic Evals doesn't currently implement task/evaluator timeouts, but you can use a package like https://github.com/pnpnpn/timeout-decorator to easily apply a timeout to your task or evaluator function. You can then let the error bubble up or set up retries using those retry_task and retry_evaluators options.

If you're still seeing a failure in one task/evaluator crash the entire run, can you please share some more details like the stack trace?

[baek54321]

@DouweM , thanks for your response!

I added retry logic for tasks and evaluators and switched to an asynchronous execution (have used evaluate_sync before). The main issue I now face is that the evaluation gets stuck at around 98% with some long-running things not stopping/finishing and staying there for a long time.

My retry config:

rc = RetryConfig(
      retry=retry_if_exception_type(HTTPStatusError) | retry_if_exception_type(StopIteration),
      wait=wait_retry_after(max_wait=300),
      stop=stop_after_attempt(3) | stop_after_delay(300),
      reraise=False)

My call to run the evaluation:

test_dataset.evaluate(
          run_pipeline_dict, name='XYZ Evaluation', progress=True,
          max_concurrency=5, retry_task=rc, retry_evaluators=rc))

Also I see the following errors:

  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-56210-8w26mtpe for automatic cleanup: unknown resource type semlock
Evaluating XYZ Evaluation ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺  98% 0:00:05Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-56210-r8dliwzm for automatic cleanup: unknown resource type semlock
Evaluating XYZ Evaluation ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺  98% 0:00:05Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-56210-19xhds_9 for automatic cleanup: unknown resource type semlock
Evaluating XYZ Evaluation ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺  98% 0:00:05Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-56210-zye8pu7s for automatic cleanup: unknown resource type semlock
Evaluating XYZ Evaluation ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺  98% 0:00:05Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-56210-wz25hwfg for automatic cleanup: unknown resource type semlock

I tried to add the time-out decorator you shared but I get pickling errors, I assume the parallelization creates those issues.

[DouweM]

I tried to add the time-out decorator you shared but I get pickling errors, I assume the parallelization creates those issues.

Hmm, you're right, I hadn't seen https://github.com/pnpnpn/timeout-decorator#multithreading but it does indeed require everything to be pickleabe. Could you give anyio.fail_after a try? Just put with anyio.fail_after(timeout): in your task function around the actual work.

As for the unknown resource type semlock errors, that's strange. Do you have the full traceback by any chance?

[baek54321]

@DouweM , regarding this error, I tried to get the full traceback but so far I didn't manage. These errors come once atexit_call() is called:

-> atexit_call()
(Pdb) n
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-31330-pirldrx4 for automatic cleanup: unknown resource type semlock
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-31330-oguxxo38 for automatic cleanup: unknown resource type semlock
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-31330-p403x_60 for automatic cleanup: unknown resource type semlock
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-31330-dvm4wbw6 for automatic cleanup: unknown resource type semlock
Traceback (most recent call last):
> /opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/threading.py(1535)_shutdown()
-> for atexit_call in reversed(_threading_atexits):
(Pdb)   File "/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py", line 295, in main
    raise ValueError(
        f'Cannot register {name} for automatic cleanup: '
        f'unknown resource type {rtype}')
ValueError: Cannot register /loky-31330-utx0xpj5 for automatic cleanup: unknown resource type semlock

For context, we are using pydantic-ai inside a Python script structured with the click package.

For the timeout topic, I tried anyio.fail_after around the async part of the task but it didn't really help. But I am not sure if I also should add it around the evaluators which could time out. Generally, our application is rather complex and we also have to use nest_asyncio because not all parts as asynchronous and we also have a Streamlit app on top.

[DouweM]

These errors come once atexit_call() is called:

@baek54321 It'd be really useful to have some example code that triggers this issue, if not it's going to be almost impossible to debug.

But I am not sure if I also should add it around the evaluators which could time out.

Yeah if it's the evaluators timing out I'd add it around those.

[baek54321]

@DouweM I have good news -- the any.fail_after statements fixed the timeout issue.

And I figured out that the reported errors are coming from a parallelization part in our application unrelated to pydantic-ai. So I am closing this issue as the original timeout problem is resolved by your suggestion. :) Thanks for the support!!