Skip to content

feat : Add a scheduling helper for tpu_observability DAGs - part 1#1181

Merged
alfredyu-cienet merged 1 commit intoGoogleCloudPlatform:masterfrom
CIeNET-International:tpu-obs/release/pr-211
Feb 11, 2026
Merged

feat : Add a scheduling helper for tpu_observability DAGs - part 1#1181
alfredyu-cienet merged 1 commit intoGoogleCloudPlatform:masterfrom
CIeNET-International:tpu-obs/release/pr-211

Conversation

@chengpinglin
Copy link
Contributor

Description

This PR introduces the SchedulingHelper utility and the get_dag_timeout function to manage and calculate non-overlapping execution schedules for Airflow DAGs. This implementation ensures resource safety and configuration consistency across TPU clusters through automated time-slot allocation.


1. The Registry: Simplified and Centralized

The implementation uses a centralized REGISTERED_DAGS registry with a refined data structure:
dict[str, dict[str, dt.timedelta]].

  • Structure: It maps a cluster name (key) to a sub-dictionary where keys are dag_id strings and values are dt.timedelta objects.

2. arrange_schedule_time: Discovery and Stacking Logic

The arrange_schedule_time method automates cron string generation using a "Linear Time Stacking" approach with Automatic Cluster Discovery.

Tests

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run one-shot tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

This change introduces the `SchedulingHelper` utility and the `get_dag_timeout` function to manage and calculate non-overlapping execution schedules for Airflow DAGs. This implementation ensures resource safety and configuration consistency across TPU clusters through automated time-slot allocation.
@alfredyu-cienet alfredyu-cienet merged commit e5e5fba into GoogleCloudPlatform:master Feb 11, 2026
7 checks passed
@alfredyu-cienet alfredyu-cienet deleted the tpu-obs/release/pr-211 branch February 11, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants