Skip to content
This repository was archived by the owner on Aug 20, 2025. It is now read-only.

Move Airflow to TFXA #186

Open
murthyvs wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Move Airflow to TFXA #186

Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8cb7c4e
Create 20220915airfloworchestration
murthyvs Sep 15, 2022
a994a00
Rename 20220915airfloworchestration to 20220915airflow_orchestration
murthyvs Sep 15, 2022
58f7f2c
Create 20220919-airflow_orchestration
murthyvs Sep 19, 2022
16bad93
Delete 20220915airflow_orchestration
murthyvs Sep 19, 2022
aaf70b3
Rename 20220919-airflow_orchestration to 20220919-airflow_orchestrati…
murthyvs Sep 19, 2022
5a0d181
Update 20220919-airflow_orchestration.md
murthyvs Sep 19, 2022
fdfa5f1
Update 20220919-airflow_orchestration.md
murthyvs Sep 19, 2022
d9a2374
Update 20220919-airflow_orchestration.md
murthyvs Sep 21, 2022
2a389b9
Update 20220919-airflow_orchestration.md
murthyvs Sep 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions proposals/20220919-airflow_orchestration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#### SIG TFX-Addons
# Project Proposal

**Your name:** Varun Murthy

**Your email:** murthyvs@google.com

**Your company/organization:** Google Core ML TFX Team (MTV + Seoul)

**Project name:** Airflow Orchestration

## Project Description
Moving Airflow to TFX add-ons due to decreasing native support.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please describe what the results of the project will be, and a short introduction to what Airflow is and how it's used. You can also discuss the transition from having Airflow support included in TFX directly, versus having it as a separate install.

Copy link
Author

@murthyvs murthyvs Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can we installed and used as a separate package under tfxa/.


## Project Category
Orchestrator

## Project Use-Case(s)
We are moving Airflow from tfx/orchestration to tfx-addons. Native support for Airflow won't be provided in the near future.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A use-case describes how the project will be used by developers. In this case it should be something more like:

Apache Airflow is a widely used orchestrator for ML workflows. Developers can use Airflow with TFX to orchestrate TFX pipelines. This project continues support for Airflow, which includes the web console and CLI tooling which developers can use to monitor and control their TFX pipelines.

Copy link
Contributor

@rcrowe-google rcrowe-google Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine that you're a developer who is new to TFX. What do you need to know about Airflow, and how it fits with TFX? Why should you consider installing this module?


## Project Implementation
Move Airflow from tfx/orchestration to tfx-addons.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs some more detail about how the move will be accomplished.


## Project Dependencies
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't dependencies. For comparison, see:

https://github.com/tensorflow/tfx-addons/blob/main/proposals/20210525-feast_example_gen.md#project-dependencies

https://github.com/tensorflow/tfx-addons/blob/main/proposals/20210707-xgboost_evaluator.md#project-dependencies

https://github.com/tensorflow/tfx-addons/blob/main/proposals/20220117-exit-handler-slack.md#project-dependencies

etc.

Same as tfx/orchestration/airflow.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These need to be listed out.


## Project Team
Google Core ML TFX Team (MTV + Seoul)
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The project team members need to be listed, including Github IDs and email addresses. The team members themselves will be responsible for maintaining the project, so they need to be aware of their responsibilities.


# Note
Please be aware of the processes and requirements which are outlined here:

* [SIG-TFX-Addons](https://github.com/tensorflow/tfx-addons)
* [Contributing Guidelines](https://github.com/tensorflow/tfx-addons/blob/main/CONTRIBUTING.md)
* [TensorFlow Code of Conduct](https://github.com/tensorflow/tfx-addons/blob/main/CODE_OF_CONDUCT.md)