Stale workflows/cron jobs on worker registration

[KyleKing]

We're in the process of evaluating Hatchet as a more performant alternative to Dagster and one of the features that I miss is how Dagster's orchestrator reflects code changes, but there is an obvious different in architecture (centralized definition vs. decentralized worker registration).

Current behavior:

• When I remove a workflow from code (e.g., delete scheduled-frequent-job), it remains registered in Hatchet
• Cron jobs for removed workflows continue to be scheduled and fail with "no worker available"
• The Dashboard appears to only allow disabling crons, not deleting them

Compared to the Dagster model:

• Workflows not in the source code are automatically pruned

Questions:

1. Is there a recommended pattern for pruning stale workflows on deploy? We may want to rename workflows, delete unused features, etc. particularly early-on. We are likely to use AWS ECS and our initial use case is a single worker.

I would appreciate guidance on whether this is the right approach or if there's built-in functionality we're missing or if I've overlooked some documentation or preexisting discussion! I recently joined the Discord server

[KyleKing]

My current plan is to add a step during deployment that uses the hatchet-sdk to turn off schedules and otherwise sync modifications to our workers in aggregate back to the control plane

[abelanger5]

Hey @KyleKing! This is a common ask, we're still figuring out the right path in this scenario. There are basically two reasons we can't clean up workflows that are removed from a worker:

1. This might be expected behavior, since workers can run only a subset of workflows if they choose. So a worker not registering a workflow doesn't indicate that it should be deleted.
2. Even if the workflow should be deleted, there might be old workers which still have the workflow registered and running. So even if we knew that it should be deleted, we don't know exactly when to delete it.

One pattern which can work is creating a "migration" file of sorts which uses the Hatchet API to remove workflows, crons and schedules (zero-downtime migrations/database schemas are a good reference for these issues).

We also have it on our backlog to set a configurable deletion window if a workflow is stale and unregistered on workers for a certain period of time, but I don't think this addresses the immediate concern here.

Open to other ideas on what we could do here!

[KyleKing]

Thanks for the fast reply!

I finished implementing a ~200 LoC sync script that compares the set of registered workflows with those defined in code (handling one or more workers) and that seems to work fairly well, at least for us to proceed with the implementation! From testing, I like that deleting workflows doesn't delete runs

[KyleKing]

My main concern is that the UI isn't obvious when the available workers and the registered Workflows become out of sync. I think features supporting that edge case would work both in the intended (rename/delete) and unintended (downtime) scenarios.

On "migration" file, I can imagine that a tool like Alembic would be useful and having workflows be marked as Stale would help, but I think maybe going farther to add a large warning or status page to the UI would be really helpful!