Disable rate limit for local acceptance tests; increase for cloud#3581
Merged
Disable rate limit for local acceptance tests; increase for cloud#3581
Conversation
denik
requested review from
andrewnester,
anton-107,
pietern and
shreyas-goenka
as code owners
denik
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Respect DATABRICKS_RATE_LIMIT variable and pass it to tests if set. If not set, set to higher defaults depending on local/cloud.
denik
force-pushed
the
denik/rate-limit-acc-tests
branch
from
599d741 to
dc83470
Compare
denik
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
16 failing tests:
|
shreyas-goenka
approved these changes
Sep 10, 2025
denik
added a commit
that referenced
this pull request
Sep 11, 2025
This happens sometimes
```
=== FAIL: acceptance TestAccept/bundle/upload/internal_server_error/DATABRICKS_CLI_DEPLOYMENT=direct-exp (0.05s)
acceptance_test.go:1328: No effective updates from BundleConfig.default_name
server.go:59: Failed to read request body: unexpected EOF
```
Reproducible with:
`go test ./acceptance -run
^TestAccept$/^bundle$/^upload$/^internal_server_error$ -v -count=100
-failfast`
The issue started after disabling rate limit in acc tests
#3581
if I restore the rate limit, it does not fail even without this patch:
`DATABRICKS_RATE_LIMIT=15 go test ./acceptance -run
^TestAccept$/^bundle$/^upload$/^internal_server_error$/direct -v
-count=100 -failfast` succeeds.
denik
added a commit
that referenced
this pull request
Sep 11, 2025
After rate limit was disabled in #3581 this test became flaky. The ApplyParallel lines are already logged in order but the output of those mutator appears as it comes. This PR filters out the SDK request traces + request in question that can change order. ``` === FAIL: acceptance TestAccept/bundle/debuglog/DATABRICKS_CLI_DEPLOYMENT=direct-exp (0.04s) acceptance_test.go:1336: Writing updated bundle config to databricks.yml. BundleConfig sections: default_name acceptance_test.go:803: Diff: --- bundle/debuglog/out.stderr.direct-exp.txt +++ /tmp/TestAcceptbundledebuglogDATABRICKS_CLI_DEPLOYMENT=direct-exp1521238718/001/out.stderr.direct-exp.txt @@ -70,8 +70,6 @@ 10:07:59 Debug: ApplyParallel pid=12345 mutator=validate:folder_permissions 10:07:59 Debug: ApplyParallel pid=12345 mutator=validate:validate_sync_patterns 10:07:59 Debug: ApplyParallel pid=12345 mutator=fast_validate(readonly) mutator=validate:job_cluster_key_defined -10:07:59 Debug: ApplyParallel pid=12345 mutator=fast_validate(readonly) mutator=validate:job_task_cluster_spec -10:07:59 Debug: ApplyParallel pid=12345 mutator=fast_validate(readonly) mutator=validate:artifact_paths 10:07:59 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files < HTTP/1.1 404 Not Found < { @@ -83,6 +81,8 @@ > "path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files" > } < HTTP/1.1 200 OK pid=12345 mutator=validate:files_to_sync sdk=true +10:07:59 Debug: ApplyParallel pid=12345 mutator=fast_validate(readonly) mutator=validate:job_task_cluster_spec +10:07:59 Debug: ApplyParallel pid=12345 mutator=fast_validate(readonly) mutator=validate:artifact_paths 10:07:59 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files < HTTP/1.1 200 OK < { ```
denik
added a commit
that referenced
this pull request
Sep 11, 2025
Fix order issues in the output by sorting the output. Previous attempt to fix this test did not work: #3590 The flakiness started after rate limit was disabled in #3581 For context, this test was originally sorting the output but that was disabled in #2414 because it seemed that output was stable. It was only stable because of delays introduced by rate limiter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Respect DATABRICKS_RATE_LIMIT variable and pass it to tests if set.
If not set, set to higher defaults depending on local/cloud.
Related: #3579
Tests
New acc test with parallel resources that is good to demo the difference. Manually try different rate limits:
Set a very low limit:
Restore SDK default:
New default: