fix(cli-integ): add retry for iam eventual consistency issue and migration tests for java #788

abidhasan-aws · 2025-08-12T17:32:01Z

Background

The CDK pipelines have been experiencing intermittent failures due to flaky tests that typically pass on retry. This pull request addresses the investigation of the two most frequent failing tests.

AWS IAM Eventual Consistency Issue

Test: docker-credential-cdk-assets can assume role and fetch ECR credentials

Issue: Docker credential fetching fails with AccessDenied errors because newly created IAM roles and policies require time to propagate across AWS regions.

Fix: Implemented a 60-second retry mechanism for fetchDockerLoginCredentials() when encountering AccessDenied errors.

CDK Migration Test Instability

Test: cdk migrate java deploys successfully

Issue: Java CDK migration tests fail sporadically due to Maven Central repository rate limiting errors & dependency resolution failure

Fix: Implemented full test retry logic as these transient network-related issues could not be reproduced in local environments.

Impact

These changes should improve pipeline stability and reduce the need for manual intervention.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

codecov-commenter · 2025-08-12T17:41:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.83%. Comparing base (c6585ad) to head (653d9a2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #788      +/-   ##
==========================================
- Coverage   81.52%   80.83%   -0.69%     
==========================================
  Files          63       63              
  Lines        8611     8611              
  Branches     1038     1032       -6     
==========================================
- Hits         7020     6961      -59     
- Misses       1561     1619      +58     
- Partials       30       31       +1

Flag	Coverage Δ
suite.unit	`80.83% <ø> (-0.69%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Abogical · 2025-08-13T10:00:57Z

packages/cdk-assets/bin/docker-credential-cdk-assets.ts

-  // Write the credentials back to stdout
-  fs.writeFileSync(1, JSON.stringify(credentials));
+
+  const deadline = Date.now() + 60_000; // 60 seconds timeout


Didn't know you could have _ in numbers. nice!

kumvprat

Overall could we have a wrapper method : withBackoffRetry or something like that with a max retry /timeout parameter ?

This would allow us to wrap any such tests in that method and have a centralized place for all retry logic than per test

Could be in future that other tests might come up which need these retries

kumvprat · 2025-08-13T10:18:27Z

packages/cdk-assets/bin/docker-credential-cdk-assets.ts

+
+      // Retry on AccessDenied errors due to eventual consistency in AWS IAM.
+      // Newly created roles and policies may take time to propagate across regions.
+      if (e.name === 'AccessDenied') {


Would it be better to match with error type : Like class/interface type rather than the name ?

mrgrain · 2025-08-13T16:25:58Z

packages/cdk-assets/bin/docker-credential-cdk-assets.ts

This seems like a very heave-handed approach to what appears to be mainly an issue with our internal integration tests (since I am not aware of customer reports).

In any case, this change needs to be a separate PR as it is Cx facing.

cc @rix0rrr

mrgrain · 2025-08-13T16:29:13Z

packages/cdk-assets/bin/docker-credential-cdk-assets.ts

+  const deadline = Date.now() + 60_000; // 60 seconds timeout
+  let lastError: Error | undefined;
+
+  while (Date.now() < deadline) {


This probably can be implemented at the SDK level using retries:
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-smithy-util-retry/
https://github.com/smithy-lang/smithy-typescript/blob/main/packages/types/src/retry.ts

Signed-off-by: github-actions <[email protected]>

kumvprat · 2025-08-14T14:19:51Z

...testing/cli-integ/tests/cli-integ-tests/cdk-assets/cdk-assets-docker-credential.integtest.ts

-    captureStderr: false,
+  let output: string = '';
+
+  await retry(process.stdout, 'Getting docker credentials', retry.forSeconds(60), async () => {


Should this be retry or waitForAssumeRole or even retryOnMatchingErrors ?

It should not be waitForAssumeRole.

We are retrying this because in this call, there is a credential fetching that needs a wait period for IAM eventual consistency. We could use retryOnMatchingErrors, but there is no clean way to catch error and retry based on that. The package just returns exited with code 1 in case of errors.

So we are retrying anyway and not based on errors, thus the generic retry.

Final thought should we add a sleep time to retry function ? Is that something that can be used like the wait and retry we do in retryOnMatchingErrors ? Would that add any value ?

It already has a sleep time of 5 seconds.

…ation tests for java (#788) ### Background The CDK pipelines have been experiencing intermittent failures due to flaky tests that typically pass on retry. This pull request addresses the investigation of the two most frequent failing tests. <img width="1307" height="530" alt="image (1)" src="https://github.com/user-attachments/assets/c03da25a-6921-4358-8a12-81db8722d437" /> ### AWS IAM Eventual Consistency Issue Test: `docker-credential-cdk-assets can assume role and fetch ECR credentials` Issue: Docker credential fetching fails with AccessDenied errors because newly created IAM roles and policies require time to propagate across AWS regions. Fix: Implemented a 60-second retry mechanism for `fetchDockerLoginCredentials()` when encountering AccessDenied errors. ### CDK Migration Test Instability Test: `cdk migrate java deploys successfully` Issue: Java CDK migration tests fail sporadically due to Maven Central repository rate limiting errors & dependency resolution failure Fix: Implemented full test retry logic as these transient network-related issues could not be reproduced in local environments. ### Impact These changes should improve pipeline stability and reduce the need for manual intervention. --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license --------- Signed-off-by: github-actions <[email protected]> Co-authored-by: github-actions <[email protected]>

…ation tests for java (aws#788) ### Background The CDK pipelines have been experiencing intermittent failures due to flaky tests that typically pass on retry. This pull request addresses the investigation of the two most frequent failing tests. <img width="1307" height="530" alt="image (1)" src="https://github.com/user-attachments/assets/c03da25a-6921-4358-8a12-81db8722d437" /> ### AWS IAM Eventual Consistency Issue Test: `docker-credential-cdk-assets can assume role and fetch ECR credentials` Issue: Docker credential fetching fails with AccessDenied errors because newly created IAM roles and policies require time to propagate across AWS regions. Fix: Implemented a 60-second retry mechanism for `fetchDockerLoginCredentials()` when encountering AccessDenied errors. ### CDK Migration Test Instability Test: `cdk migrate java deploys successfully` Issue: Java CDK migration tests fail sporadically due to Maven Central repository rate limiting errors & dependency resolution failure Fix: Implemented full test retry logic as these transient network-related issues could not be reproduced in local environments. ### Impact These changes should improve pipeline stability and reduce the need for manual intervention. --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license --------- Signed-off-by: github-actions <[email protected]> Co-authored-by: github-actions <[email protected]>

add retry for cdk migrate java & fetchDockerLoginCredential

89fd917

abidhasan-aws requested a deployment to integ-approval August 12, 2025 17:32 — with GitHub Actions Waiting

aws-cdk-automation requested a review from a team August 12, 2025 17:32

github-actions bot added the p2 label Aug 12, 2025

abidhasan-aws changed the title ~~add retry for cdk migrate java & fetchDockerLoginCredential~~ fix: mitigate and add retry for flaky tests Aug 12, 2025

abidhasan-aws changed the title ~~fix: mitigate and add retry for flaky tests~~ test: add retry mechanisms for IAM eventual consistency and CDK migration tests for java Aug 13, 2025

abidhasan-aws changed the title ~~test: add retry mechanisms for IAM eventual consistency and CDK migration tests for java~~ test: add retry mechanisms for iam eventual consistency and migration tests for java Aug 13, 2025

abidhasan-aws changed the title ~~test: add retry mechanisms for iam eventual consistency and migration tests for java~~ fix: add retry mechanisms for iam eventual consistency and migration tests for java Aug 13, 2025

abidhasan-aws changed the title ~~fix: add retry mechanisms for iam eventual consistency and migration tests for java~~ fix: add retry for iam eventual consistency issue and migration tests for java Aug 13, 2025

abidhasan-aws marked this pull request as ready for review August 13, 2025 09:52

Abogical approved these changes Aug 13, 2025

View reviewed changes

kumvprat reviewed Aug 13, 2025

View reviewed changes

mrgrain reviewed Aug 13, 2025

View reviewed changes

abidhasan-aws requested a deployment to integ-approval August 14, 2025 08:33 — with GitHub Actions Waiting

remove retry from docker-credential-asset

2b6c048

abidhasan-aws force-pushed the fix-flaky-cli-integ-tests branch from 7b6b015 to 2b6c048 Compare August 14, 2025 08:42

abidhasan-aws requested a deployment to integ-approval August 14, 2025 08:42 — with GitHub Actions Waiting

add retry for async block based on error

897b548

abidhasan-aws requested a deployment to integ-approval August 14, 2025 13:11 — with GitHub Actions Waiting

nit

a14cb43

abidhasan-aws requested a deployment to integ-approval August 14, 2025 13:14 — with GitHub Actions Waiting

chore: self mutation

d6755cd

Signed-off-by: github-actions <[email protected]>

aws-cdk-automation requested a deployment to integ-approval August 14, 2025 13:22 — with GitHub Actions Waiting

Merge branch 'main' into fix-flaky-cli-integ-tests

092f4c9

abidhasan-aws requested a deployment to integ-approval August 14, 2025 13:59 — with GitHub Actions Waiting

abidhasan-aws requested review from Abogical and kumvprat August 14, 2025 14:08

kumvprat reviewed Aug 14, 2025

View reviewed changes

abidhasan-aws enabled auto-merge August 15, 2025 11:18

Merge branch 'main' into fix-flaky-cli-integ-tests

653d9a2

abidhasan-aws temporarily deployed to integ-approval August 15, 2025 11:18 — with GitHub Actions Inactive

abidhasan-aws disabled auto-merge August 15, 2025 11:19

mrgrain enabled auto-merge August 15, 2025 11:41

mrgrain changed the title ~~fix: add retry for iam eventual consistency issue and migration tests for java~~ fix(cli-tests): add retry for iam eventual consistency issue and migration tests for java Aug 15, 2025

mrgrain changed the title ~~fix(cli-tests): add retry for iam eventual consistency issue and migration tests for java~~ fix(cli-integ): add retry for iam eventual consistency issue and migration tests for java Aug 15, 2025

abidhasan-aws temporarily deployed to run-tests August 15, 2025 12:09 — with GitHub Actions Inactive

mrgrain added this pull request to the merge queue Aug 15, 2025

Merged via the queue into main with commit 093e5a6 Aug 15, 2025
36 of 37 checks passed

mrgrain deleted the fix-flaky-cli-integ-tests branch August 15, 2025 12:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(cli-integ): add retry for iam eventual consistency issue and migration tests for java #788

fix(cli-integ): add retry for iam eventual consistency issue and migration tests for java #788

Uh oh!

abidhasan-aws commented Aug 12, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 12, 2025 •

edited

Loading

Uh oh!

Abogical Aug 13, 2025

Uh oh!

kumvprat left a comment

Uh oh!

kumvprat Aug 13, 2025

Uh oh!

mrgrain Aug 13, 2025

Uh oh!

mrgrain Aug 13, 2025

Uh oh!

kumvprat Aug 14, 2025

Uh oh!

abidhasan-aws Aug 14, 2025 •

edited

Loading

Uh oh!

kumvprat Aug 14, 2025

Uh oh!

abidhasan-aws Aug 14, 2025

Uh oh!

Uh oh!

Uh oh!

fix(cli-integ): add retry for iam eventual consistency issue and migration tests for java #788

fix(cli-integ): add retry for iam eventual consistency issue and migration tests for java #788

Uh oh!

Conversation

abidhasan-aws commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

AWS IAM Eventual Consistency Issue

CDK Migration Test Instability

Impact

Uh oh!

codecov-commenter commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Abogical Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

kumvprat left a comment

Choose a reason for hiding this comment

Uh oh!

kumvprat Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

mrgrain Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

mrgrain Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

kumvprat Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

abidhasan-aws Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumvprat Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

abidhasan-aws Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

abidhasan-aws commented Aug 12, 2025 •

edited

Loading

codecov-commenter commented Aug 12, 2025 •

edited

Loading

abidhasan-aws Aug 14, 2025 •

edited

Loading