test(amazonq): add performance test for prepareRepoData #5670
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
packages/core/src/test/amazonqFeatureDev/prepareRepoData.test.ts
Outdated
Show resolved
Hide resolved
| } | ||
| /** | ||
| * Tests 250 files w/ 10 bytes each. | ||
| * Running more files can lead to flaky test from timeout. |
Contributor
There was a problem hiding this comment.
would there be any value on seeing how this number scales? e.g. for every 50 files it roughly increases the duration by 2 or is it too unpredictable?
Contributor
Author
There was a problem hiding this comment.
duration in general was very flaky on this one, and also it seemed to not scale linearly. I would be surprised based on what I've seen so far if an increase of 50 files corresponded so some constant duration increase. I could look into it with less files.
| zipFileChecksum: string | ||
| } | ||
|
|
||
| describe('prepareRepoDataPerformanceTest', function () { |
Contributor
There was a problem hiding this comment.
I'd probably just have a top level describe thats prepareRepoData and then a second describe that has performance test in it, that way it can easily seperate perf tests from non perf tests if anyone wants to add it
Contributor
Author
There was a problem hiding this comment.
There are also some existing correctness tests in test/unit/amazonqFeatureDev/util/files.test.ts. I kept the performance tests in core since thats where the startSecurityScan is (src/test/codewhisperer/commands/startSecurityScan.test.ts).
This was referenced Oct 3, 2024
jpinkney-aws
approved these changes
Oct 4, 2024
Hweinstock
added a commit
that referenced
this pull request
Oct 4, 2024
## Problem Last PR: #5670 was not rebased onto a386cc3, so did not include relevant telemetry changes. Therefore tests passed until the merge happened. ## Solution change type signature to reflect telemetry changes. --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
Hweinstock
added a commit
that referenced
this pull request
Oct 14, 2024
## Problem Continuation of #5670 ## Solution same as previous --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
Hweinstock
added a commit
that referenced
this pull request
Oct 14, 2024
## Problem continuation of performance test work: #5670 ## Solution same as previous. CPU threshold set very high because it spikes pretty wildly on this one.  --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
Hweinstock
added a commit
that referenced
this pull request
Oct 14, 2024
…hieve. (#5710) ## Problem Continuation of work here: #5670 ## Solution In this case `downloadExportResultArchieve` is reading a large collection of objects into a buffer, then writing that buffer to a file. We test this with varying collections of objects. We test 1x1KB, 10x100B, 100x10B, and 1000x1B. --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
tverney
pushed a commit
to tverney/aws-toolkit-vscode
that referenced
this pull request
Oct 21, 2024
## Problem Continuation of aws#5670 ## Solution same as previous --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
tverney
pushed a commit
to tverney/aws-toolkit-vscode
that referenced
this pull request
Oct 21, 2024
## Problem continuation of performance test work: aws#5670 ## Solution same as previous. CPU threshold set very high because it spikes pretty wildly on this one.  --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
tverney
pushed a commit
to tverney/aws-toolkit-vscode
that referenced
this pull request
Oct 21, 2024
…hieve. (aws#5710) ## Problem Continuation of work here: aws#5670 ## Solution In this case `downloadExportResultArchieve` is reading a large collection of objects into a buffer, then writing that buffer to a file. We test this with varying collections of objects. We test 1x1KB, 10x100B, 100x10B, and 1000x1B. --- <!--- REMINDER: Ensure that your PR meets the guidelines in CONTRIBUTING.md --> License: I confirm that my contribution is made under the terms of the Apache 2.0 license.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Start of reliability work to enhance performance testing of high risk code paths.
Solution
prepareRepoDatainsrc/amazonqFeatureDev/util/files.tscaptures many potentially high risk paths in a single function, such ascollectFilesand other I/O operations, making it a great place to start performance testing.Two performance tests were added, one for handling a workspace with many small files, and one with few large files. Unsurprisingly, the performance is significantly better on fewer larger files as it reduces # of I/O operations. Thresholds are very lenient to avoid flaky tests.
The "many files" test does 250 files w/ 10 bytes each. The "large files" does 10 files w/ 1000 bytes each. These values were chosen to avoid flakiness in CI.
License: I confirm that my contribution is made under the terms of the Apache 2.0 license.