Skip to content

feat: Add mitmproxy-based HTTP caching for testing (do not merge)#306

Closed
devin-ai-integration[bot] wants to merge 4 commits intomainfrom
devin/1738344527-mitmproxy-cache
Closed

feat: Add mitmproxy-based HTTP caching for testing (do not merge)#306
devin-ai-integration[bot] wants to merge 4 commits intomainfrom
devin/1738344527-mitmproxy-cache

Conversation

@devin-ai-integration
Copy link
Contributor

Description

This PR adds a new mitmproxy-based HTTP caching mechanism for testing, which operates at the OS level by intercepting HTTP requests through environment variables. This approach provides several advantages over the existing requests-cache implementation:

  • Compatible with Python 3.12
  • Works with any HTTP client library, not just requests
  • Can support future Docker-based testing scenarios
  • Provides foundation for Java connector testing support

Implementation Details

  • Added new airbyte_cdk.testing.proxy_cache package with mitmproxy addon
  • Created two pytest fixtures:
    • mitmproxy_auto: Session-scoped, auto-use fixture for global caching
    • mitmproxy_cache: Manual fixture for explicit caching control
  • Added mitmproxy as dev dependency and under new 'testing' extras
  • Includes basic test coverage demonstrating caching functionality

Testing

  • Added unit test verifying cache behavior
  • All type checks and linting pass
  • Manually verified with example HTTP requests

Notes

  • Existing requests-cache implementation remains unchanged
  • This is a new feature that will enable gradual migration path

Link to Devin run: https://app.devin.ai/sessions/1afeef8bcd2c44c5ab4f09600e8ec30f

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

Original prompt from Aaron:

@Devin - Create a PR that implements just the CDK-side of this solution: <https://chatgpt.com/share/679d02a0-0a78-8004-91bc-ddb27c24a1de>
The goal is to create http caching capabilities to significantly speed up connector rests and reduce flakiness.

When this is ready, I'll ask a different Devin to create the connector-side.

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
  • Look at CI failures and help fix them

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions github-actions bot added the enhancement New feature or request label Jan 31, 2025
devin-ai-integration bot and others added 3 commits January 31, 2025 17:47
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>
@aaronsteers Aaron ("AJ") Steers (aaronsteers) marked this pull request as draft February 1, 2025 01:13
@aaronsteers Aaron ("AJ") Steers (aaronsteers) changed the title feat: Add mitmproxy-based HTTP caching for testing feat: Add mitmproxy-based HTTP caching for testing (do not merge) Feb 1, 2025
@devin-ai-integration
Copy link
Contributor Author

Closing due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants