Error handling scheduling request

[brainbug95]

Describe the bug

I have to schedule ~150 EC2 instances up and down every day in one VPC in one AWS Account. About every 30 days we get an error

ERROR : Error handling scheduling request <...> botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the PutItem operation: Item size has exceeded the maximum allowed size

see scheduler-error.json for a complete log entry.

This stops the scheduler from working for this AWS Account with the effect that the whole environment that should have been started is not resulting in an outage.

To fix this I need to manually delete the Dynamo TableItem for this AWS Account ID in the StateTable of the solution. Then tha scheduler starts working again.

To Reproduce

I have seen this behavior only in two of our AWS accounts where Kubernetes Clusters with a highly volatile number of EC2 instances are deployed. So I am guessing it might have something to do with the constantly changing number of instances and states the scheduler has to track.

Please complete the following information about the solution:

• Version: v3.0.6
• Region: eu-central-1
• Was the solution modified from the version published on this repository? No
• Have you checked your service quotas for
  the sevices this solution uses? Yes, and I know there is a limit on the PutItem size for dynamoDB but the solution should account for that
• Were there any errors in the CloudWatch Logs?
  Error occur in the solutions cloudwatch logs (see attached file)

Appreciate any help on this. We've been seeing this issue with version < 3 of the scheduler solution as well and I hoped it would have been solved with v3 but clearly it is not.

[CrypticCabub]

Hi @brainbug95

Thanks for reporting this issue. I believe that the root cause of this issue is that the kubernetes clusters are constantly creating new EC2 instances rather than starting/stopping existing ones. The state table logic does not currently attempt to detect when a previously scheduled instance no longer exists which results in the constantly changing instance IDs piling up in the state table item until the max size is exceeded.

So you would be correct that the kubernetes clusters are what's causing this issue. Could I ask for a little more detail on how these clusters are setup for scheduling and what you are trying to achieve? This state table scenario is on our backlog to fix, but I am also wondering if the "actual" issue here is that we don't support scheduling kubernetes clusters correctly.

[brainbug95]

Hi @CrypticCabub

Thanks for your insight on this. AWS accounts having this problem host a mix of self-hosted OpenShift kubernetes clusters and other supporting EC2 Instances. We are using the scheduler do stop and start all OCP workload nodes to be available only during business hours instead of leaving them running 24/7. In addition the OCP clusters itself are also capable of creating and terminating EC2 instances.

EKS clusters are also involved but those nodes are not tagged so the scheduler should not "see" them.

I'm happy to provide further info if required.

[brainbug95]

Hi @CrypticCabub

any update on this. This problem is disrupting our services regularly.

[CrypticCabub]

Thank you for your patience. While we've identified the root cause, implementing a comprehensive fix will require additional engineering work and time. We don't have an immediate timeline to share yet. To help dive deep, please open a support ticket through AWS Solutions > Instance Scheduler > Support Request. This will allow our team to better understand your specific use case, particularly regarding the scheduling requirements for OpenShift clusters, share our current solution roadmap, and discuss potential workarounds that might help in the interim. We're especially interested in learning more about your business hours scheduling requirements for OpenShift workload nodes, as this could help inform future product development.

We appreciate your patience as we work on a long-term solution that better accommodates containerized environments like OpenShift and their specific instance management needs.

[mdgm88]

@CrypticCabub Any update on this? It looks like we've run into this as well using EKS.

[CrypticCabub]

Hi @mdgm88

The simple answer for the issue is that Instance Scheduler does not currently support scheduling of instances associated with an EKS cluster. We do currently support scheduling of auto-scaling groups which means you can indirectly schedule services like EKS by applying schedule tags to your ASG, but tagging of the instances underneath is what causes the problem.

My technical understanding of the problem is that behind-the-scenes, when the EC2 instances themselves are being tagged with a schedule via EKS, Instance Scheduler records the ec2 instanceId as part of the regional state record in the DDB state table. EKS then terminates the instance and creates a new one during autoscaling operations, and Instance Scheduler dutifully continues recording state information for each new instance that gets created. Under normal operations this has never been a problem, but now with hundreds or even thousands of extra instance IDs being recorded thanks to EKS getting involved, eventually the max limit of that state object is exceeded and breaks the state management for that region.

The state information in the table is regenerated every 5 minutes, but does not currently include an efficient way to purge the stale instance data (resulting in stale data accumulating in the record). a short-term fix right now is to manually purge the record periodically and allow InstanceScheduler to rebuild its state information on the next execution (this can theoretically cause additional schedulng requests to be issued, so is recommended to be done during off hours). I am currently investigating designs for a more permanent solution to the problem, but it is still too early in the process to provide any ETAs for a release date.