Why do ASG have separate, different scheduling process with other services?

[TranVanDung-Leo]

Is your feature request related to a problem? Please describe.

Hi team, I use and develop instance scheduler but I see that the flow to schedule ASGs is complicated, different than other services, that it uses a separate orchestrator, a lambda to observe change in the dynamodb table stream, and a potential risk of unconvertable schedules into cronjob of ASG scheduling actions.

Describe the feature you'd like

ASG, and other services will be scheduled in the same orchestrator.

Additional context

[CrypticCabub]

Thanks for reaching out! Is there a particular requirement or use-case that would benefit from making all services share the same orchestrator?

ASGs require a fundamentally different scheduling flow due to the requirement to use Schedule Scaling Rules as an intermediary step for schedule operation. As such we set up a longer polling interval than normal (1 hour) for ASG scheduling and then added the dynamodb stream lambda as a short-circuit response to changes in the schedule database.

Since this flow is fundamentally different from the busy-polling approach used by EC2 and RDS, we thought it woould be more appropriate to use a separate ochestrator to manage it rather than overloading the existing orchestrator and needlessly complicating the control flow within.

[TranVanDung-Leo]

@CrypticCabub But I saw the main difference in ASG data and its definition.
Separate workflows cause developers very hard to understand the flow and it contain many lambda, hidden rule (1 days affection, 1 hour cron)

[CrypticCabub]

It is admittedly a confusing difference that we are hoping to be able to simplify in future releases. For regular single-period schedules ASGs should actually be more accurate than EC2/RDS schedules, and we hope to be able to expand the feature support in the future to cover more complex schedules.

[iainelder]

The need for a different scheduler process makes sense: The ASG resource doesn't have start/stop actions, so the original scheduling model doesn't fit. Instead it has a desired capacity and a capacity range ("min-desired-max" in the doc).

---

Is there a technical reason the ASG scheduler runs always and only once per hour? Why does it ignore SchedulingFrequency and SchedulingActiveCondition?

To get faster feedback while exploring, I set set the ASG schedule expression to rate(5 mintues). Will that break something?

[CrypticCabub]

The original reason was that it simply didn't need to be faster than that for the logic to work so we could save some costs by not running the scan as often. This has actually been changed in my current internal dev branch which includes some simplification refactors for the ASG scheduling.

To answer your question. No, changing the rate expression to 5 minutes will not break anything with asg scheduling, but it will also only reduce the delay in detecting newly tagged ASGs -- unless you change the schedule attached to an ASG, we actually only take action once a month to keep the ASG scheduled scaling rules accurate to the schedule. there is a ddb stream embedded in the architecture that automatically updates the schedule whenever you change the DB (this is part of the complicated architecture I am refactoring in the next release), but the discovery process only runs once an hour as you have noticed.

There are several changes to the internal ASG architecture coming, but the most notable one is that the discovery delay will be reduced to essentially 0. once an ASG is tagged with a schedule, the asg should be configured after only a few seconds.

[iainelder]

Thanks for the insight. When I wrote that, I had yet to learn that the Instance Scheduler delegates to ASG scheduled actions!

To check my understanding: The Instance Scheduler recreates the scheduled actions in these cases:

• About every 30 days to remediate possible drift
• When the schedule table changes to apply it ASAP

The is_still_valid function detects these conditions and returns False to trigger recreation.

Every ~30 days

The ASG service recreates the schedule. Here it sets a TTL 30 days from now:

instance-scheduler-on-aws/source/app/instance_scheduler/service/asg.py

Lines 471 to 473 in 7279ee9

	ttl=(
	datetime.now(timezone.utc) + timedelta(days=30)
	).isoformat(),

Here it invalidates the scheduled tag when the TTL is tomorrow or earlier.

instance-scheduler-on-aws/source/app/instance_scheduler/service/asg.py

Lines 136 to 137 in 7279ee9

	if ttl_dt < datetime.now(timezone.utc) + timedelta(days=1):
	return False, "Configuration expiring in less than one day"

On schedule table changes

The DDB stream detects when schedules change. Here it passes a list of changed schedule names to the orchestrator.

instance-scheduler-on-aws/source/app/instance_scheduler/handler/schedule_update.py

Line 51 in 7279ee9

orchestrate_asgs(env, context, schedule_names)

The orchestrator delegates to the ASG handler. Here it sets an override flag when it gets a list of schedules.

instance-scheduler-on-aws/source/app/instance_scheduler/handler/asg.py

Line 121 in 7279ee9

is_schedule_override: Final[bool] = schedule_names is not None

The ASG handler delegates to the ASG service. Here it invalidates the scheduled tag because of the override flag.

instance-scheduler-on-aws/source/app/instance_scheduler/service/asg.py

Lines 121 to 122 in 7279ee9

	if is_schedule_override:
	return False, "Overridden"

---

In this case, running the Orchestrator every hour to check everything seems enough.

Looking forward, 0 discovery delay sounds it will provide a better experience for users.