Fix file handle leaks in azure-ai-evaluation when exceptions occur during evaluation

[Copilot]

Problem

The azure-ai-evaluation package had potential file handle leaks when the _evaluate function completed with errors. Specifically, temporary JSONL files created for ProxyClient usage were not being cleaned up if exceptions occurred at certain points during evaluation.

Affected Scenarios

1. Exception during _begin_aoai_evaluation: When using a target function with ProxyClient, a temporary file is created in _preprocess_data but may not be cleaned up if _begin_aoai_evaluation fails before reaching _run_callable_evaluators.

2. OAI-only evaluation failures: When only graders are used (no local evaluators), the cleanup logic in _run_callable_evaluators is never reached if OAI evaluation fails.

3. Early evaluation failures: Any exception occurring after temp file creation but before the cleanup logic in _run_callable_evaluators would leave files on disk.

Root Cause

The cleanup logic was only present in _run_callable_evaluators, creating a gap where temporary files could be leaked if exceptions occurred elsewhere in the evaluation flow.

Solution

Implemented centralized temporary file cleanup using a try-finally pattern at the _evaluate function level to ensure all temporary files are cleaned up regardless of where exceptions occur.

Changes Made

1. Added tracking infrastructure: Extended __ValidatedData TypedDict to include temp_files_to_cleanup: List[str]

2. Track temp file creation: Modified _preprocess_data to maintain a list of created temporary files and include it in the returned data

3. Centralized cleanup: Added try-finally block in _evaluate that guarantees cleanup of all tracked temporary files, with graceful error handling for edge cases

4. Removed duplicate logic: Simplified _run_callable_evaluators by removing the now-redundant cleanup code

Code Example

Before (problematic):

# In _preprocess_data
temp_file = tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=False)
# ... write data ...
temp_file.close()
batch_run_data = temp_file.name
# If _begin_aoai_evaluation fails here, file is leaked!

After (fixed):

# In _preprocess_data  
temp_file = tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=False)
# ... write data ...
temp_file.close()
batch_run_data = temp_file.name
temp_files_to_cleanup.append(temp_file.name)  # Track for cleanup

# In _evaluate
try:
    # ... all evaluation logic ...
finally:
    # Clean up any temporary files created during evaluation
    for temp_file_path in temp_files_to_cleanup:
        if os.path.exists(temp_file_path):
            try:
                os.unlink(temp_file_path)
            except Exception as e:
                LOGGER.warning(f"Failed to clean up temporary file {temp_file_path}: {e}")

Benefits

• Comprehensive coverage: Temporary files are cleaned up regardless of failure point
• Minimal changes: Only surgical modifications to the existing codebase
• Backward compatible: No breaking changes to existing functionality
• Robust: Handles edge cases like already-deleted files and permission errors gracefully

This fix ensures that temporary files created during evaluation are always cleaned up, preventing file handle leaks and potential disk space issues in long-running applications.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• pypi.org
  • Triggering command: /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/bin/python /home/REDACTED/work/azure-sdk-for-python/azure-sdk-for-python/.venv/lib/python3.9/site-packages/pip/__pip-REDACTED__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-sc3lg2hs/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i REDACTED -- setuptools>=40.8.0 (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.