Feat: Allow setting k8s labels and annotations in task decorator #3243

fg91 · 2025-04-27T09:12:23Z

Tracking issue

Why are the changes needed?

See flyteorg/flyte#6421 and flyteorg/flyte#6238.

What changes were proposed in this pull request?

Add labels and annotations arg to @task decorator.

How was this patch tested?

Added unit tests. Ran admin and propeller with changes proposed in flyteorg/flyte#6421 and made sure that labels/annotations end up on Pods and CRD objects.

I updated the documentation accordingly.
All new and existing tests passed. Currently fail until idl is updated.
All commits are signed-off.

Related PRs

flyteorg/flyte#6421

Summary by Bito

This pull request adds Kubernetes metadata support in the task decorator by introducing 'labels' and 'annotations' parameters, enhancing task management in Kubernetes. It also updates the flyteidl dependency for compatibility and includes unit tests to validate these new features.

flyte-bot · 2025-04-27T09:12:34Z

Code Review Agent Run #31ab9e

Actionable Suggestions - 1

flytekit/models/task.py - 1
- Missing import or definition for K8sObjectMetadata type · Line 188-188

Additional Suggestions - 2

flytekit/core/python_auto_container.py - 1
- Missing type validation for labels and annotations · Line 103-106
flytekit/core/task.py - 1
- Use PEP 604 style type annotations · Line 129-171

Review Details

Files reviewed - 4 · Commit Range: d01309b..d01309b
- flytekit/core/base_task.py
- flytekit/core/python_auto_container.py
- flytekit/core/task.py
- flytekit/models/task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

flyte-bot · 2025-04-27T09:19:41Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
New Feature - New Feature: Kubernetes Task Metadata Support	- `base_task.py` - Added 'labels' and 'annotations' parameters to task definitions to enable Kubernetes metadata propagation. - `python_auto_container.py` - Updated container task initialization to handle and pass through labels and annotations. - `task.py` - Integrated labels and annotations into task function signatures and task instantiation for enhanced metadata support. - `task.py` - Enhanced task models with Kubernetes metadata by incorporating labels and annotations, including updates to serialization and property methods. - `test_python_auto_container.py` - Added tests to verify correct transfer of labels and annotations in container tasks. - `test_python_function_task.py` - Validated that function tasks properly propagate Kubernetes metadata using labels and annotations. - `test_tasks.py` - Tested the integration of labels and annotations in task model objects to ensure accurate metadata handling.

flyte-bot · 2025-04-27T09:19:42Z

flytekit/models/task.py

        cache_ignore_input_vars,
        is_eager: bool = False,
        generates_deck: bool = False,
+        metadata: typing.Optional["K8sObjectMetadata"] = None,


Missing import or definition for K8sObjectMetadata type

The K8sObjectMetadata type is referenced but not defined or imported. This will cause a NameError at runtime when the type hint is evaluated.

Code suggestion

Check the AI-generated fix before applying

@@ -1,2 +1,3 @@ import json as _json import typing +from flytekit.models.common import K8sObjectMetadata

Code Review Run #31ab9e

Should Bito avoid suggestions like this for future reviews? (Manage Rules)

Yes, avoid them

codecov · 2025-04-27T10:27:05Z

Codecov Report

Attention: Patch coverage is 63.63636% with 4 lines in your changes missing coverage. Please review.

Project coverage is 52.26%. Comparing base (c95a3df) to head (4c4173d).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
flytekit/core/python_auto_container.py	0.00%	2 Missing and 2 partials ⚠️

❗ There is a different number of reports uploaded between BASE (c95a3df) and HEAD (4c4173d). Click for more details.

HEAD has 24 uploads less than BASE

Flag BASE (c95a3df) HEAD (4c4173d)

26 2

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3243       +/-   ##
===========================================
- Coverage   88.13%   52.26%   -35.87%     
===========================================
  Files         105      215      +110     
  Lines        5165    22471    +17306     
  Branches        0     2950     +2950     
===========================================
+ Hits         4552    11745     +7193     
- Misses        613    10036     +9423     
- Partials        0      690      +690

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

flyte-bot · 2025-04-27T10:37:55Z

Code Review Agent Run #163458

Actionable Suggestions - 0

Additional Suggestions - 1

tests/flytekit/unit/core/test_python_auto_container.py - 1
- Missing task ID verification in serialization test · Line 407-410

Review Details

Files reviewed - 3 · Commit Range: d01309b..19a9abc
- tests/flytekit/unit/models/test_tasks.py
- tests/flytekit/unit/core/test_python_auto_container.py
- tests/flytekit/unit/core/test_python_function_task.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

AI Code Review powered by

Future-Outlier · 2025-05-16T09:51:36Z

flytekit/models/task.py

        cache_ignore_input_vars,
        is_eager: bool = False,
        generates_deck: bool = False,
+        metadata: typing.Optional["K8sObjectMetadata"] = None,


should the naming here be k8s_object_metadata?

wild-endeavor · 2025-05-16T17:32:44Z

@fg91 can you remind me does flyteidl==1.15.4b0 have what this pr needs? if so want to bump the requirement? if not i can cut another idl beta. (just to get tests passing)

fg91 · 2025-05-18T18:07:34Z

@fg91 can you remind me does flyteidl==1.15.4b0 have what this pr needs? if so want to bump the requirement? if not i can cut another idl beta. (just to get tests passing)

Yes, this beta release contains the required change, I bumped the version, thank you.

Some other tests are failing with sth seemingly unrelated though 🤔

Signed-off-by: Fabio Grätz <[email protected]>

…k metadata Signed-off-by: Fabio Grätz <[email protected]>

Signed-off-by: Fabio Grätz <[email protected]>

…changes Signed-off-by: Fabio Grätz <[email protected]>

Signed-off-by: Fabio Grätz <[email protected]>

kumare3 · 2025-06-08T19:36:03Z

I honestly do not love adding labels and annotations to task. This implies that every task can have it, which is not true, only k8s pods can have it. This causes problems when folks are using databricks, memverge, slurm etc etc

fg91 · 2025-06-08T19:42:54Z

I honestly do not love adding labels and annotations to task. This implies that every task can have it, which is not true, only k8s pods can have it. This causes problems when folks are using databricks, memverge, slurm etc etc

Result of this discussion but I'm happy to implement it another way! How would you expose it @kumare3 ?

only k8s pods can have it

Couldn't the same be said about pod template which is a task arg too?

kumare3 · 2025-06-08T21:51:50Z

It can be said about pod_template, by the name pod_template and this does go into the k8spod attribute.
Ok lets keep Labels and annotations, but lets not call is K8sObjectMetadata. Lets keep it as labels and annotations in TaskMetadata and the plugin is responsible to put it into k8s.

Moreover, lets document it saying, labels and annotations are optional parameters, that may not be applied in all cases.

fg91 · 2025-06-09T07:24:00Z

It can be said about pod_template, by the name pod_template and this does go into the k8spod attribute. Ok lets keep Labels and annotations, but lets not call is K8sObjectMetadata. Lets keep it as labels and annotations in TaskMetadata and the plugin is responsible to put it into k8s.

Moreover, lets document it saying, labels and annotations are optional parameters, that may not be applied in all cases.

Ok, will change 👍
I have a clarification question before I modify the implementation:
You mean to remove the added metadata here in flyteidl in favor of directly adding labels/annotations? Or was the change request about the flytekit side only?

Thank you!

kumare3 · 2025-06-25T13:50:52Z

I thought a lot more and let's keep what you have. +1

Sorry to keep you waiting. Please document

…teorg#3243) * Allow settint k8s labels and annotations in task decorator Signed-off-by: Fabio Grätz <[email protected]> * Adapt task models unit test Signed-off-by: Fabio Grätz <[email protected]> * Add python auto container task test Signed-off-by: Fabio Grätz <[email protected]> * Add test that vales passed as label and annotations arg end up in task metadata Signed-off-by: Fabio Grätz <[email protected]> * Cover changes in base_task.py Signed-off-by: Fabio Grätz <[email protected]> * Rename metadata field and pin flyteidl to beta release with metadata changes Signed-off-by: Fabio Grätz <[email protected]> * Pin to > instead of = Signed-off-by: Fabio Grätz <[email protected]> --------- Signed-off-by: Fabio Grätz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Signed-off-by: Atharva <[email protected]>

fg91 requested review from Future-Outlier, cosmicBboy, davidmirror-ops, eapolinario, kumare3, pingsutw, samhita-alla, thomasjpfan and wild-endeavor as code owners April 27, 2025 09:12

fg91 self-assigned this Apr 27, 2025

fg91 added the added For new features label Apr 27, 2025

flyte-bot reviewed Apr 27, 2025

View reviewed changes

fg91 mentioned this pull request Apr 27, 2025

Feat: Apply labels and annotations set via task decorator to pods and CR objects flyteorg/flyte#6421

Merged

3 tasks

fg91 mentioned this pull request May 15, 2025

[Core feature] Allow setting metadata (labels&annotations) individually on all K8s task types flyteorg/flyte#6238

Closed

2 tasks

Future-Outlier reviewed May 16, 2025

View reviewed changes

fg91 force-pushed the fg91/feat/task-k8s-metadata branch from 19a9abc to fa03793 Compare May 18, 2025 16:31

fg91 requested a review from Future-Outlier May 18, 2025 18:08

Fabio Grätz added 7 commits June 8, 2025 15:48

Allow settint k8s labels and annotations in task decorator

c880f2d

Signed-off-by: Fabio Grätz <[email protected]>

Adapt task models unit test

df4c0b6

Signed-off-by: Fabio Grätz <[email protected]>

Add python auto container task test

d2366b4

Signed-off-by: Fabio Grätz <[email protected]>

Add test that vales passed as label and annotations arg end up in tas…

23833e2

…k metadata Signed-off-by: Fabio Grätz <[email protected]>

Cover changes in base_task.py

d9f40cd

Signed-off-by: Fabio Grätz <[email protected]>

Rename metadata field and pin flyteidl to beta release with metadata …

186ab68

…changes Signed-off-by: Fabio Grätz <[email protected]>

Pin to > instead of =

cce6503

Signed-off-by: Fabio Grätz <[email protected]>

fg91 force-pushed the fg91/feat/task-k8s-metadata branch from 4c4173d to cce6503 Compare June 8, 2025 13:48

fg91 requested review from Sovietaced and removed request for thomasjpfan June 8, 2025 15:21

Sovietaced approved these changes Jun 8, 2025

View reviewed changes

davidmirror-ops added the triage/discuss label Jun 10, 2025

kumare3 approved these changes Jun 25, 2025

View reviewed changes

fg91 merged commit 6e5af0a into master Jun 25, 2025
114 of 115 checks passed

Feat: Allow setting k8s labels and annotations in task decorator #3243

Feat: Allow setting k8s labels and annotations in task decorator #3243

Uh oh!

Conversation

fg91 commented Apr 27, 2025 • edited by flyte-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Related PRs

Summary by Bito

Uh oh!

flyte-bot commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #31ab9e

Uh oh!

flyte-bot commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelist by Bito

Uh oh!

flyte-bot Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

flyte-bot commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Agent Run #163458

Uh oh!

Future-Outlier May 16, 2025

Choose a reason for hiding this comment

Uh oh!

fg91 May 18, 2025

Choose a reason for hiding this comment

Uh oh!

wild-endeavor commented May 16, 2025

Uh oh!

fg91 commented May 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumare3 commented Jun 8, 2025

Uh oh!

fg91 commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumare3 commented Jun 8, 2025

Uh oh!

fg91 commented Jun 9, 2025

Uh oh!

kumare3 commented Jun 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

fg91 commented Apr 27, 2025 •

edited by flyte-bot

Loading

flyte-bot commented Apr 27, 2025 •

edited

Loading

flyte-bot commented Apr 27, 2025 •

edited

Loading

codecov bot commented Apr 27, 2025 •

edited

Loading

flyte-bot commented Apr 27, 2025 •

edited

Loading

fg91 commented May 18, 2025 •

edited

Loading

fg91 commented Jun 8, 2025 •

edited

Loading