Skip to content

Conversation

majanjua-amzn
Copy link
Contributor

Issue

K8s OS patching workflow does not succeed some jobs because of throttling when trying to create EC2 instances

Description of changes

  • Added jitter_index to input
  • Added sleep based on jitter index
  • Each job will now sleep 20 seconds for each job before it in the list, ensuring they run somewhat sequentially and not all at the same time

Note: The github scheduler/worker assignment may still lead to some of the jobs doing things at the same time, but the risk of this causing throttling should be low

Testing

https://github.com/aws-observability/aws-application-signals-test-framework/actions/runs/11172296461

Rollback procedure

Revert PR and rerun the workflow

@majanjua-amzn majanjua-amzn added the bug Something isn't working label Oct 4, 2024
@majanjua-amzn majanjua-amzn requested a review from harrryr October 4, 2024 01:21
@majanjua-amzn majanjua-amzn self-assigned this Oct 4, 2024
@majanjua-amzn majanjua-amzn merged commit 927277a into main Oct 4, 2024
17 checks passed
@majanjua-amzn majanjua-amzn deleted the k8s-jitter branch October 4, 2024 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants