🐛Autoscaling: ensure unstarteable warm buffer are replaced by cold instances if available #8277
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sanderegg
requested review from
YuryHrytsuk,
mrnicegyu11 and
pcrespov
as code owners
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #8277 +/- ##
==========================================
+ Coverage 89.31% 89.79% +0.48%
==========================================
Files 1678 1272 -406
Lines 65431 54789 -10642
Branches 828 225 -603
==========================================
- Hits 58438 49199 -9239
+ Misses 6774 5520 -1254
+ Partials 219 70 -149
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
sanderegg
force-pushed
the
autoscaling/bugfix/8273/ensure-warm-buffer-not-starting-does-not-block-autoscaling
branch
from
0458399 to
40c258a
Compare
sanderegg
requested review from
GitHK,
bisgaard-itis,
Copilot,
matusdrobuliak66 and
mguidon
There was a problem hiding this comment.
Pull Request Overview
This PR fixes a bug in the autoscaling service where warm buffer instances that couldn't be started due to insufficient capacity would prevent new instances from being launched. The solution ensures that when warm buffer instances fail to start, their assigned tasks are de-assigned and can be fulfilled by launching new cold instances.
Key Changes
- Modified the warm buffer starting logic to handle
EC2InsufficientCapacityErrorexceptions gracefully - Added a mechanism to de-assign tasks from warm buffer instances that cannot be started
- Updated the autoscaling flow to retry task assignment with cold instances when warm buffers fail
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
services/autoscaling/src/simcore_service_autoscaling/modules/cluster_scaling/_auto_scaling_core.py |
Core autoscaling logic updated to handle warm buffer start failures and de-assign tasks for retry with cold instances |
services/autoscaling/tests/unit/test_modules_cluster_scaling_dynamic.py |
Removed xfail marker and increased test warm buffer count to properly test the fix |
packages/aws-library/tests/test_ec2_client.py |
Added test coverage for insufficient capacity scenarios and improved code formatting |
...es/autoscaling/src/simcore_service_autoscaling/modules/cluster_scaling/_auto_scaling_core.py
Show resolved
Hide resolved
...es/autoscaling/src/simcore_service_autoscaling/modules/cluster_scaling/_auto_scaling_core.py
Show resolved
Hide resolved
...es/autoscaling/src/simcore_service_autoscaling/modules/cluster_scaling/_auto_scaling_core.py
Show resolved
Hide resolved
GitHK
approved these changes
Sep 1, 2025
...es/autoscaling/src/simcore_service_autoscaling/modules/cluster_scaling/_auto_scaling_core.py
Show resolved
Hide resolved
sanderegg
force-pushed
the
autoscaling/bugfix/8273/ensure-warm-buffer-not-starting-does-not-block-autoscaling
branch
from
7dc87da to
cb8d815
Compare
|
|
@mergify queue |
🛑 Configuration not compatible with a branch protection settingThe branch protection setting |
sanderegg
deleted the
autoscaling/bugfix/8273/ensure-warm-buffer-not-starting-does-not-block-autoscaling
branch
This was referenced Sep 2, 2025
Closed
17 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Related issue/s
How to test
Dev-ops