feat(source-harvest): Add HTTPAPIBudget and concurrency_level#75543
Draft
devin-ai-integration[bot] wants to merge 4 commits intomasterfrom
Draft
feat(source-harvest): Add HTTPAPIBudget and concurrency_level#75543devin-ai-integration[bot] wants to merge 4 commits intomasterfrom
devin-ai-integration[bot] wants to merge 4 commits intomasterfrom
Conversation
- Add HTTPAPIBudget with two MovingWindowCallRatePolicy entries: - Reports API: 100 requests per 15 minutes - General API: 100 requests per 15 seconds - Add ConcurrencyLevel with default 2 workers, max 7 - Add configurable num_workers spec parameter (integer, 2-7) - Bump version to 1.3.0-rc.1 (MINOR, progressive rollout) - Add unit tests for api_budget, concurrency, and spec validation Co-Authored-By: bot_apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: bot_apk <apk@cognition.ai>
Contributor
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit e2fd686. |
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/15308:
Adds rate limiting (
HTTPAPIBudget) and configurable concurrency (ConcurrencyLevel) to source-harvest to prevent 429 errors and enable parallel syncing.How
Three additions to
manifest.yaml:api_budget—HTTPAPIBudgetwith twoMovingWindowCallRatePolicyentries:^/reports/): 100 requests per 15 minutesRetry-Afterheader on 429 responsesconcurrency_level—ConcurrencyLevelwith default 2 workers, max 7, driven by a newnum_workersconfig parameter.num_workersspec parameter — Optional integer (2–7, default 2) allowing users to tune concurrency.Version bumped to
1.3.0-rc.1(MINOR, progressive rollout enabled).Review guide
manifest.yaml— Core changes:concurrency_levelblock,api_budgetblock,num_workersin specunit_tests/mock_server/test_api_budget_and_concurrency.py— 6 new tests covering sync behavior, custom num_workers, manifest structure, 429 retry, and spec validationmetadata.yaml— Version bumpdocs/integrations/sources/harvest.md— Changelog entryHuman review checklist:
^/reports/correctly matches all report stream paths in the manifest (streams likeexpenses_categorieshit/reports/expenses/categories)num_workersspec field (with default) won't require action from existing connections{{ config.get('num_workers', 2) }}indefault_concurrencyis valid declarative manifest syntaxUser Impact
num_workers(up to 7) to speed up syncsCan this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/a66874b8b76b4c20bad99c9a45e61016