Fix: jobset based push node_pool_name to nodeselector

[Chrisliao0806]

Description

This PR fixes a critical scheduling issue where JobSet pods were being dispatched to arbitrary node pools instead of the intended ones, and extends the fix to support multi-node-pool environments.

Problem: Previously, the JobSet YAML template had no nodeSelector for the GKE node pool. In environments with multiple node pools, Kubernetes would schedule pods on any available nodes. This led to cases where a Rollback was performed on Node Pool A, but the JobSet pods were actually running on Node Pool B, resulting in inaccurate TTR (Time To Recovery) metrics. Additionally, for DAGs like tpu_info_format_validation_dag that create two node pools for the same workload, pinning to a single node pool name would leave the second pool unused.

Technical Implementation

Added a dynamic $node_pool_selector to the JobSet YAML template that supports scheduling modes:

• Uses a custom label tpu-observability/jobset-group: <jobset_name> applied to all participating node pools via gcloud --node-labels at creation time, allowing the K8s scheduler to dispatch pods across multiple pools.

Test

• GCP Composer name: chris-test (under GCP project: cienet-cmcs)
• GCP Composer version: 2.13.1
• jobset_ttr_rollback.py test result: https://3450723b279f4f47af5e5f22135026a7-dot-us-central1.composer.googleusercontent.com/dags/jobset_rollback_ttr/grid?tab=logs&dag_run_id=manual__2026-02-10T12%3A20%3A29.774345%2B00%3A00&task_id=v6e.create_node_pool
• tpu_info_format_validation_dag.py test result: https://3450723b279f4f47af5e5f22135026a7-dot-us-central1.composer.googleusercontent.com/dags/tpu_info_format_validation_dag/grid?dag_run_id=manual__2026-02-09T14%3A12%3A31.677533%2B00%3A00&tab=graph&task_id=v6e.create_node_pool

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run one-shot tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.