Skip to content

Flyte 2 WIP#6583

Draft
EngHabu wants to merge 173 commits intomainfrom
v2
Draft

Flyte 2 WIP#6583
EngHabu wants to merge 173 commits intomainfrom
v2

Conversation

@EngHabu
Copy link
Contributor

@EngHabu EngHabu commented Aug 22, 2025

  • Add flyteidl and minimal publishing workflows

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

* Add flyteidl and minimal publishing workflows

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* pretend version for buf python

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* python version fix and use OIDC to authenticate crates

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix python versioning

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Set version before setting up env

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* echo version

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Write to GITHUB_ENV

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Use env:

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf generate

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Mark go package as /v2 to be able to import both at the same time

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Upgrade to go 1.24

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Rename python, rust and npm to flyteidl2

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@flyte-bot
Copy link
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

…6610)

* trying idl2

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update trace def

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor: rename idl2 to flyteidl2

Signed-off-by: machichima <nary12321@gmail.com>

* generate -> buf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor: rename idl2 to flyteidl2

Signed-off-by: machichima <nary12321@gmail.com>

* build: re-gen

Signed-off-by: machichima <nary12321@gmail.com>

* fix: exclude python venv

Signed-off-by: machichima <nary12321@gmail.com>

* Add translator service

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Regen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf gen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* remove node execution service proto

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Breakup task service into a separate package

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Help strings for make file targets

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update TaskLog with read bool

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Trying the updated protovalidate plugin

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* protovalidate with semver

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try claude :shrug

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* add buf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update deps

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* build: remove generate validate for python

Signed-off-by: machichima <nary12321@gmail.com>

* build: use managed mode on buf.gen.yaml

Signed-off-by: machichima <nary12321@gmail.com>

* Update validate go plugin

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update imagebuild

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix oneof buf lint

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Add deps to generated python code

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Be explicit about the index

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* updated idl

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Catch up to main

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Generate mocks for workflow package

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* generate mocks for other packages

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* buf improvements

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* updates

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* feat: generate mock for common

Signed-off-by: machichima <nary12321@gmail.com>

* use make gen in GH Actions

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Do not setup python env in generation CI

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* setup uv

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* set working directory

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Try to reference files by path

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Explicitly move .venv to root

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try exclude in buf.yaml

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* set UV_PROJECT_ENVIRONMENT

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* download tooling

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* add which mockery

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: machichima <nary12321@gmail.com>
Co-authored-by: machichima <nary12321@gmail.com>
@flyte-bot
Copy link
Collaborator

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Add Trigger APIs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* update triggers

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* refactor

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Fix cargo build and add CI check

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* lint lib.rs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update Readme and contributing guides

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* signed?

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build-crate target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix included ts files in npm

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Add device class to `GpuAccelerator`

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* make gen

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add HABANA_GAUDI as supported device class

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Clean up triggers api

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

* make gen

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

---------

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
@bito-code-review
Copy link

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

pingsutw and others added 3 commits March 22, 2026 21:39
…eamed updates (#7070)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
* feat: add dummy trigger service

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add dummy app service

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
@github-actions github-actions bot mentioned this pull request Mar 23, 2026
3 tasks
generated files and lint fixes

Signed-off-by: Adam Brock <adam@union.ai>
Co-authored-by: Adam Brock <adam@union.ai>
machichima and others added 6 commits March 23, 2026 16:34
* feat: add attempts and cache status to CR

Signed-off-by: machichima <nary12321@gmail.com>

* feat: get attempts/cache status from object

Signed-off-by: machichima <nary12321@gmail.com>

* feat: store attempts/cache status to CR and send through action event

Signed-off-by: machichima <nary12321@gmail.com>

* feat: consider retry count in generated name

Signed-off-by: machichima <nary12321@gmail.com>

* feat: get max attempt from task template

Signed-off-by: machichima <nary12321@gmail.com>

* feat: run service save attempts/cache status to DB

Signed-off-by: machichima <nary12321@gmail.com>

* build: make sandbox-build

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
* feat: add CacheKey to TaskAction spec

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add cache service skeleton

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add cache manager and repository

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add cache service client

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add max cache age

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add cache to task action reconcile loop

Signed-off-by: machichima <nary12321@gmail.com>

* fix: better reservation cleanup

Signed-off-by: machichima <nary12321@gmail.com>

* fix: calculate hash key always go through hash formula

Signed-off-by: machichima <nary12321@gmail.com>

* feat: start cache service in manager/

Signed-off-by: machichima <nary12321@gmail.com>

* feat: copy cache service in root Dockerfile

Signed-off-by: machichima <nary12321@gmail.com>

* fix: use promutils.NewScope

Signed-off-by: machichima <nary12321@gmail.com>

* fix: remove getStringValue()

Signed-off-by: machichima <nary12321@gmail.com>

* fix: correcct serialize logic

Signed-off-by: machichima <nary12321@gmail.com>

* refactor: update function/variable name

Signed-off-by: machichima <nary12321@gmail.com>

* fix: set metricKey only once

Signed-off-by: machichima <nary12321@gmail.com>

* fix: heartbeatGracePeriodMultiplier to int type

Signed-off-by: machichima <nary12321@gmail.com>

* docs: update comment to make them more clear

Signed-off-by: machichima <nary12321@gmail.com>

* fix: get newest data again before return

Signed-off-by: machichima <nary12321@gmail.com>

* fix: construct new struct rather than deep copy

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
…7038)

* feat: Add the field that GetRunDetails works expected

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>

* fix: fixup after merge v2

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add get run details test script

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add the cache status for run details

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>

* fix: GetRunDetails call buildActionDetails

Signed-off-by: machichima <nary12321@gmail.com>

* fix: fallback to action table spec & use event as source of truth

Signed-off-by: machichima <nary12321@gmail.com>

* test: set Attempts to 1

Signed-off-by: machichima <nary12321@gmail.com>

* fix: get cache status from DB

Signed-off-by: machichima <nary12321@gmail.com>

* test: set default org to localhost

Signed-off-by: machichima <nary12321@gmail.com>

* test: mock match error

Signed-off-by: machichima <nary12321@gmail.com>

* refactor: specJSON to specProto

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: SZL741023 <shengzhe.lin@icloud.com>
Signed-off-by: machichima <nary12321@gmail.com>
Co-authored-by: machichima <nary12321@gmail.com>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
machichima and others added 3 commits March 24, 2026 02:43
Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
@github-actions github-actions bot mentioned this pull request Mar 25, 2026
3 tasks
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
popojk and others added 2 commits March 24, 2026 22:49
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: machichima <nary12321@gmail.com>
* abort reconciller

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* mocs added

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix: workers run before startup scan

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* review fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.