fix(providers): ensure Python worker writes error response before signaling completion

[mldangelo]

Summary

Fixes ENOENT errors when Python provider fails before writing response file. The Python worker now ensures an error response file is always written before signaling completion to Node.js, preventing race conditions where Node.js tries to read a non-existent file.

Problem

Users on Windows were encountering ENOENT: no such file or directory errors when using Python providers with similarity or llm-rubric assertions:

Error: ENOENT: no such file or directory, open 'C:\Users\...\promptfoo-worker-resp-....json'
    at PythonWorker.executeCall (worker.js:119:49)

The error occurred even after 16 retry attempts with exponential backoff (up to 18 seconds total wait time).

Root Cause

In src/python/persistent_wrapper.py, when an exception occurred before the response file could be written (e.g., non-existent function, failed file I/O, invalid request), the outer exception handler would:

1. Log the error to stderr
2. Send "DONE" signal to Node.js to avoid hanging
3. Not create a response file

This caused Node.js to receive the "DONE" signal and attempt to read a response file that never existed, resulting in ENOENT errors that persisted through all retry attempts.

Affected scenarios:

• get_callable() fails (function doesn't exist in Python module)
• Failed to read request file
• Failed to write response file (permissions, disk space, antivirus blocking)
• Invalid CALL command format

The Fix

Modified handle_call() in persistent_wrapper.py:

1. Declare response_file = None before try block (make it accessible in exception handler)
2. In the outer exception handler, write an error response to response_file before sending "DONE"
3. This ensures Node.js always has a valid response file to read, containing proper error information

Before:

except Exception as e:
    print(f"ERROR handling call: {e}", file=sys.stderr, flush=True)
    print("DONE", flush=True)  # ❌ No response file created

After:

except Exception as e:
    print(f"ERROR handling call: {e}", file=sys.stderr, flush=True)
    
    # Write error response if we have response_file
    if response_file:
        try:
            error_response = {"type": "error", "error": str(e), "traceback": traceback.format_exc()}
            with open(response_file, "w", encoding="utf-8") as f:
                json.dump(error_response, f, ensure_ascii=False)
                f.flush()
                os.fsync(f.fileno())
        except Exception as write_error:
            print(f"ERROR: Failed to write error response: {write_error}", file=sys.stderr, flush=True)
    
    print("DONE", flush=True)  # ✅ Response file now exists

Testing

Created regression test in test/python/worker.test.ts:

• Tests calling a non-existent function (call_nonexistent_api)
• Verifies proper error message instead of ENOENT
• Previously: test timed out after 5 seconds
• Now: test passes in ~460ms with proper AttributeError

Test Results:

✅ All 42 Python worker tests pass (including new test)
✅ All 41 Python provider tests pass
✅ Fix verified on macOS (issue reported on Windows)

Files Changed

• src/python/persistent_wrapper.py - Fixed error handling to write response before DONE
• test/python/fixtures/test_nonexistent_function.py - Test fixture
• test/python/worker.test.ts - Added regression test
• CHANGELOG.md - Documented the fix

Notes

• This fix ensures proper error propagation from Python to Node.js
• Users will now see meaningful error messages (e.g., "AttributeError: module has no attribute 'call_embedding_api'") instead of confusing ENOENT errors
• The fix is backward compatible - all existing tests pass

Fixes #6072

[use-tusk]

⏩ No test execution environment matched (eb0f3a0) View output ↗

---

View check history

Commit	Status	Output	Created (UTC)
fc44afa	⏩ No test execution environment matched	Output	Oct 31, 2025 4:03AM
5926de4	⏩ No test execution environment matched	Output	Oct 31, 2025 4:04AM
b705970	⏩ No test execution environment matched	Output	Oct 31, 2025 4:15AM
8c1726b	⏩ No test execution environment matched	Output	Nov 1, 2025 5:28AM
42e86bb	⏩ No test execution environment matched	Output	Nov 1, 2025 5:38AM
1a189b5	⏩ Skipped due to new commit on branch	Output	Nov 1, 2025 5:42AM
c789521	⏩ Skipped due to new commit on branch	Output	Nov 1, 2025 5:50AM
c60c904	⏩ Skipped due to new commit on branch	Output	Nov 1, 2025 6:08AM
6124d04	⏩ No test execution environment matched	Output	Nov 1, 2025 6:25AM
1eb6612	⏩ No test execution environment matched	Output	Nov 1, 2025 6:35AM
eb0f3a0	⏩ No test execution environment matched	Output	Nov 2, 2025 6:45AM

View output in GitHub ↗

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request fixes an issue where Python worker errors were not being persisted to response files before signaling completion, resulting in ENOENT (file not found) errors. The fix adds error response writing to the persistent_wrapper.py exception handler that serializes error details to the response file before continuing with completion signaling. Supporting changes include a test fixture simulating non-existent function calls and corresponding test cases in the TypeScript worker test suite.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• persistent_wrapper.py: Error response writing logic in exception path requires careful review to ensure proper JSON serialization, file I/O operations (flush/fsync), and that completion signaling is not skipped
• test fixture and test case: Verify the test_nonexistent_function.py fixture correctly simulates the error condition and that the test assertions properly validate error handling behavior
• Cross-language changes: Review spans Python error handling and TypeScript test code, requiring context switching

Pre-merge checks and finishing touches

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The pull request title "fix(providers): ensure Python worker writes error response before signaling completion" directly and clearly summarizes the main change in the changeset. The title is concise, specific, and accurately describes the core fix being implemented in src/python/persistent_wrapper.py, which modifies error handling to write error responses before signaling completion to Node.js. This aligns with the primary objective from the linked issue to prevent ENOENT errors by ensuring response files are always created.
Linked Issues Check	✅ Passed	The code changes fully address the objectives from linked issue #6072. The fix in src/python/persistent_wrapper.py implements the core requirement by ensuring error responses are written to response_file before signaling completion with "DONE", which prevents the race condition where Node.js attempts to read a non-existent file. The added test fixture and regression test (#6073) verify this behavior works correctly for error scenarios like non-existent functions, and the test results confirm the fix eliminates ENOENT errors while providing proper error messages. All changes align with the expected outcome of robust Python provider interactions across platforms.
Out of Scope Changes Check	✅ Passed	All changes in this pull request are in scope and directly related to fixing issue #6072. The modifications to src/python/persistent_wrapper.py implement the core error handling fix, the new test fixture and regression test support validation of the fix, and the CHANGELOG.md update appropriately documents the change. There are no unrelated modifications, dependency updates, refactoring, or other scope creep that would indicate changes outside the defined objectives.
Description Check	✅ Passed	The pull request description is comprehensive and directly related to the changeset. It provides a clear summary, explains the problem and root cause (outer exception handler not creating response files), details the fix (writing error response before sending DONE signal), includes testing results showing 42 Python worker tests pass, and lists all modified files. The description demonstrates a clear understanding of the issue and how the implementation addresses it.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/python-worker-error-response

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

CHANGELOG.md (1)

19-20: Optional wording tweak for consistency.

Consider “Python provider” instead of “Python worker” to match prior entries’ terminology.

-- fix(providers): ensure Python worker writes error response before signaling completion to prevent ENOENT errors when function calls fail (#6073)
+- fix(providers): ensure Python provider writes an error response file before signaling completion to prevent ENOENT errors when function calls fail (#6073)

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c367e25 and 5926de4.

📒 Files selected for processing (4)

• CHANGELOG.md (1 hunks)
• src/python/persistent_wrapper.py (2 hunks)
• test/python/fixtures/test_nonexistent_function.py (1 hunks)
• test/python/worker.test.ts (1 hunks)

🧰 Additional context used

📓 Path-based instructions (11)

**/*.py

📄 CodeRabbit inference engine (.cursor/rules/python.mdc)

**/*.py: Use Python 3.9 or later
Follow the Google Python Style Guide
Use type hints to improve code readability and catch potential errors
Use ruff for linting and formatting
Run ruff check --fix for general linting
Run ruff check --select I --fix for import sorting
Run ruff format for formatting
Keep the Python codebase simple and minimal, without unnecessary external dependencies
When implementing custom providers, prompts, or asserts in Python, follow the promptfoo API patterns

Files:

• test/python/fixtures/test_nonexistent_function.py
• src/python/persistent_wrapper.py

**/test_*.py

📄 CodeRabbit inference engine (.cursor/rules/python.mdc)

Write unit tests for new Python functions using the built-in unittest module

Files:

• test/python/fixtures/test_nonexistent_function.py

test/**

📄 CodeRabbit inference engine (test/CLAUDE.md)

Organize tests to mirror src/ structure (e.g., test/providers → src/providers, test/redteam → src/redteam)

Place tests under test/

Files:

• test/python/fixtures/test_nonexistent_function.py
• test/python/worker.test.ts

test/**/*.{test,spec}.ts

📄 CodeRabbit inference engine (.cursor/rules/jest.mdc)

test/**/*.{test,spec}.ts: Mock as few functions as possible to keep tests realistic
Never increase the function timeout - fix the test instead
Organize tests in descriptive describe and it blocks
Prefer assertions on entire objects rather than individual keys when writing expectations
Clean up after tests to prevent side effects (e.g., use afterEach(() => { jest.resetAllMocks(); }))
Run tests with --randomize flag to ensure your mocks setup and teardown don't affect other tests
Use Jest's mocking utilities rather than complex custom mocks
Prefer shallow mocking over deep mocking
Mock external dependencies but not the code being tested
Reset mocks between tests to prevent test pollution
For database tests, use in-memory instances or proper test fixtures
Test both success and error cases for each provider
Mock API responses to avoid external dependencies in tests
Validate that provider options are properly passed to the underlying service
Test error handling and edge cases (rate limits, timeouts, etc.)
Ensure provider caching behaves as expected
Always include both --coverage and --randomize flags when running tests
Run tests in a single pass (no watch mode for CI)
Ensure all tests are independent and can run in any order
Clean up any test data or mocks after each test

Files:

• test/python/worker.test.ts

test/**/*.test.ts

📄 CodeRabbit inference engine (test/CLAUDE.md)

test/**/*.test.ts: Never increase Jest test timeouts; fix slow tests instead (avoid jest.setTimeout or large timeouts in tests)
Do not use .only() or .skip() in committed tests
Add afterEach(() => { jest.resetAllMocks(); }) to ensure mock cleanup
Prefer asserting entire objects (toEqual on whole result) rather than individual fields
Mock minimally: only external dependencies (APIs, databases), not code under test
Use Jest (not Vitest) APIs in this suite; avoid importing vitest
Import from @jest/globals in tests

Files:

• test/python/worker.test.ts

**/*.{test,spec}.{js,jsx,ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)

Avoid disabling or skipping tests (e.g., .skip, .only) unless absolutely necessary and documented

Files:

• test/python/worker.test.ts

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/gh-cli-workflow.mdc)

Prefer not to introduce new TypeScript types; reuse existing interfaces where possible

**/*.{ts,tsx}: Maintain consistent import order (Biome handles sorting)
Use consistent curly braces for all control statements
Prefer const over let and avoid var
Use object shorthand syntax when possible
Use async/await for asynchronous code
Use consistent error handling with proper type checks

**/*.{ts,tsx}: Use TypeScript with strict type checking enabled
Follow consistent import order (Biome will sort imports)
Use consistent curly braces for all control statements
Prefer const over let; avoid var
Use object property shorthand when possible
Use async/await for asynchronous code instead of raw promises/callbacks
When logging, pass sensitive data via the logger context object so it is auto-sanitized; avoid interpolating secrets into message strings
Manually sanitize sensitive objects with sanitizeObject before storing or emitting outside logging contexts

Files:

• test/python/worker.test.ts

test/**/*.{test,spec}.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

test/**/*.{test,spec}.{ts,tsx}: Follow Jest best practices using describe/it blocks
Test both success and error cases for all functionality

Files:

• test/python/worker.test.ts

test/**/*.{ts,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

Follow Jest best practices with describe/it blocks in tests

Files:

• test/python/worker.test.ts

CHANGELOG.md

📄 CodeRabbit inference engine (.cursor/rules/changelog.mdc)

CHANGELOG.md: Document all user-facing changes in CHANGELOG.md
Every pull request must add or update an entry in CHANGELOG.md under the [Unreleased] section
Follow Keep a Changelog structure under [Unreleased] with sections: Added, Changed, Fixed, Dependencies, Documentation, Tests, Removed
Each changelog entry must include the PR number formatted as (#1234) or temporary placeholder (#XXXX)
Each changelog entry must use a Conventional Commit prefix: feat:, fix:, chore:, docs:, test:, or refactor:
Each changelog entry must be concise and on a single line
Each changelog entry must be user-focused, describing what changed and why it matters to users
Each changelog entry must include a scope in parentheses, e.g., feat(providers): or fix(evaluator):
Use common scopes for consistency: providers, evaluator, webui or app, cli, redteam, core, assertions, config, database
Place all dependency updates under the Dependencies category
Place all test changes under the Tests category
Use categories consistently: Added for new features, Changed for modifications/refactors/CI, Fixed for bug fixes, Removed for removed features
After a PR number is assigned, replace (#XXXX) placeholders with the actual PR number
Be specific, use active voice, include context, and avoid repeating the PR title in changelog entries
Group related changes with multiple bullets in the same category when needed; use one entry per logical change

CHANGELOG.md: All user-facing changes require a CHANGELOG.md entry before creating a PR
Add entries under [Unreleased] in appropriate category (Added, Changed, Fixed, Dependencies, Documentation, Tests)
Each changelog entry must include PR number (#1234) or placeholder (#XXXX)
Use conventional commit prefixes in changelog entries (feat:, fix:, chore:, docs:, test:, refactor:)

CHANGELOG.md: Document all user-facing changes in CHANGELOG.md
Changelog entries must include the PR number in format (#1234)
Use conventional commit prefixes in changelog entries: feat:,...

Files:

• CHANGELOG.md

src/**

📄 CodeRabbit inference engine (AGENTS.md)

Place core application/library logic under src/

Files:

• src/python/persistent_wrapper.py

🧠 Learnings (6)

📚 Learning: 2025-07-18T17:26:23.087Z

Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/jest.mdc:0-0
Timestamp: 2025-07-18T17:26:23.087Z
Learning: Applies to test/**/*.{test,spec}.ts : Test error handling and edge cases (rate limits, timeouts, etc.)

Applied to files:

• test/python/worker.test.ts

📚 Learning: 2025-10-24T22:42:38.674Z

Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-10-24T22:42:38.674Z
Learning: Applies to test/**/*.{test,spec}.{ts,tsx} : Test both success and error cases for all functionality

Applied to files:

• test/python/worker.test.ts

📚 Learning: 2025-10-24T22:41:44.088Z

Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/gh-cli-workflow.mdc:0-0
Timestamp: 2025-10-24T22:41:44.088Z
Learning: Applies to CHANGELOG.md : Add entries under [Unreleased] in appropriate category (Added, Changed, Fixed, Dependencies, Documentation, Tests)

Applied to files:

• CHANGELOG.md

📚 Learning: 2025-10-24T22:42:38.674Z

Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-10-24T22:42:38.674Z
Learning: Applies to CHANGELOG.md : Add new entries under the 'Unreleased' section

Applied to files:

• CHANGELOG.md

📚 Learning: 2025-10-24T22:41:09.485Z

Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/changelog.mdc:0-0
Timestamp: 2025-10-24T22:41:09.485Z
Learning: Applies to CHANGELOG.md : Document all user-facing changes in CHANGELOG.md

Applied to files:

• CHANGELOG.md

📚 Learning: 2025-10-24T22:41:09.485Z

Learnt from: CR
Repo: promptfoo/promptfoo PR: 0
File: .cursor/rules/changelog.mdc:0-0
Timestamp: 2025-10-24T22:41:09.485Z
Learning: Applies to CHANGELOG.md : Follow Keep a Changelog structure under [Unreleased] with sections: Added, Changed, Fixed, Dependencies, Documentation, Tests, Removed

Applied to files:

• CHANGELOG.md

🧬 Code graph analysis (1)

test/python/worker.test.ts (1)

src/python/worker.ts (1)

• PythonWorker (9-275)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: Test on Node 24.x and windows-latest
• GitHub Check: Test on Node 22.x and windows-latest
• GitHub Check: Test on Node 20.x and windows-latest

🔇 Additional comments (4)

test/python/fixtures/test_nonexistent_function.py (1)

1-8: LGTM! Clean test fixture for error scenario.

The fixture appropriately simulates the non-existent function scenario by providing only call_api while omitting call_nonexistent_api. The docstring clearly documents the intent.

test/python/worker.test.ts (1)

165-180: LGTM! Properly tests the error-response fix.

The test correctly verifies that calling a non-existent function now raises a meaningful AttributeError instead of ENOENT. The regex pattern /has no attribute|AttributeError/ appropriately matches the error format from Python's getattr failure.

Based on learnings

src/python/persistent_wrapper.py (2)

111-111: Good defensive initialization.

Declaring response_file = None before the try block ensures it's available in the outer exception handler, even if parsing fails before the file path is determined.

---

179-199: Excellent fix for the ENOENT race condition.

The error-response writing logic correctly addresses the root cause:

• Writes error JSON to response_file before signaling DONE (lines 182-190)
• Uses fsync() to ensure durability before signaling (line 190)
• Handles write failures gracefully without hanging the caller (lines 191-196)
• Always signals DONE to prevent Node.js timeout (line 199)

The error format matches the inner exception handler (lines 139-143), ensuring consistent error structure across all failure paths.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Add a Tests entry under Unreleased for the new regression test.

Per the changelog rules, test changes should be documented under a dedicated “Tests” category. Please add a concise Tests bullet for the regression covering the Python worker failure case.

Apply after Line 20:

+### Tests
+
+- test(providers): add regression ensuring Python worker writes error response file and returns AttributeError instead of ENOENT on nonexistent function calls (#6073)

As per coding guidelines

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	### Fixed
	- fix(providers): ensure Python worker writes error response before signaling completion to prevent ENOENT errors when function calls fail (#6073)
	### Fixed
	- fix(providers): ensure Python worker writes error response before signaling completion to prevent ENOENT errors when function calls fail (#6073)
	### Tests
	- test(providers): add regression ensuring Python worker writes error response file and returns AttributeError instead of ENOENT on nonexistent function calls (#6073)

🤖 Prompt for AI Agents

In CHANGELOG.md around lines 17 to 20, the Unreleased section is missing a Tests
entry for the new regression test; add a new "### Tests" subsection immediately
after line 20 and include a single concise bullet describing the regression
test: "regression(python-worker): add test ensuring worker writes error response
before signaling completion to prevent ENOENT when function calls fail (#6073)".
Ensure formatting matches existing changelog style and place the entry under
Unreleased.