Skip to content

Fix: Prevent memory leak in responses.parse with bounded TypeAdapter … #2676

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix: Prevent memory leak in responses.parse with bounded TypeAdapter … #2676

wants to merge 1 commit into from
+276 −2

Conversation

ishanrajsingh
Copy link

@ishanrajsingh ishanrajsingh commented Oct 3, 2025 •
edited
Loading

Description

Fixes memory leak in responses.parse() caused by unbounded TypeAdapter caching in multi-threaded environments.

Problem

The current implementation uses an unbounded lru_cache keyed by dynamically generated generic types (ParsedResponseOutputMessage[MyClass]). In multi-threaded scenarios, Pydantic regenerates these types with different hashes, preventing cache hits and causing infinite memory growth.

Solution

  1. Bounded Cache: Limit cache size to 128 entries per thread
  2. Stable Keys: Use fully-qualified type names instead of type object hashes
  3. Thread Safety: Implement thread-local caching to prevent conflicts

Changes

  • Added src/openai/lib/_parsing/_type_adapter_cache.py with thread-safe caching
  • Modified src/openai/lib/_parsing/_completions.py to use new cache
  • Added comprehensive tests in tests/test_type_adapter_cache.py

Testing

  • Unit tests pass
  • Memory leak reproduction test passes
  • Multi-threaded scenarios tested
  • No performance regression

Closes

Fixes #2672

Checklist

  • Tests added/updated
  • Documentation updated (if needed)
  • Follows project coding standards

@ishanrajsingh
Fix: Prevent memory leak in responses.parse with bounded TypeAdapter …
9e2914d 
…cache

- Replace unbounded lru_cache with thread-safe bounded cache
- Use stable type names as cache keys instead of type hashes
- Implement thread-local caching to prevent hash conflicts
- Add comprehensive tests for thread safety and memory bounds

Fixes openai#2672
@ishanrajsingh ishanrajsingh requested a review from a team as a code owner October 3, 2025 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Unrestricted caching keyed by generated types causes memory leak in multi-threaded regimes
1 participant
@ishanrajsingh