Skip to content

Conversation

@harrryr
Copy link
Contributor

@harrryr harrryr commented Oct 23, 2024

… and also add jitter

Issue description:
The Github Action scheduler is frequently not starting up the canary according to schedule, and this is getting worse as we continue to onboard additional canaries. We have decided to reduce the frequency of the canaries from 4 every hour to 2 every hour to reduce the strain.

Description of changes:
Updated the canary scheduler to run 2 times every hour in specific minute of the hour to add some jitter.
Each jitter is 2 minutes apart, and any canaries that only runs in us-east-1 have been assigned the same start period at 0th minute and 25th minute.

Rollback procedure:
Revert

<Can we safely revert this commit if needed? If not, detail what must be done to safely revert and why it is needed.>

Ensure you've run the following tests on your changes and include the link below:

To do so, create a test.yml file with name: Test and workflow description to test your changes, then remove the file for your PR. Link your test run in your PR description. This process is a short term solution while we work on creating a staging environment for testing.

NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL. See the needs keyword to run tests in succession.

  • Run Java EKS on e2e-playground in us-east-1 and eu-central-2
  • Run Python EKS on e2e-playground in us-east-1 and eu-central-2
  • Run metric limiter on EKS cluster e2e-playground in us-east-1 and eu-central-2
  • Run EC2 tests in all regions
  • Run K8s on a separate K8s cluster (check IAD test account for master node endpoints; these will change as we create and destroy clusters for OS patching)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@harrryr harrryr force-pushed the update-canary-scheduler branch from 9aeb2bc to 04be784 Compare October 23, 2024 17:51
@harrryr harrryr force-pushed the update-canary-scheduler branch from 04be784 to 0768872 Compare October 23, 2024 17:52
@mxiamxia mxiamxia merged commit 8803e78 into main Oct 24, 2024
1 check passed
@mxiamxia mxiamxia deleted the update-canary-scheduler branch October 24, 2024 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants