Skip to content
This repository was archived by the owner on Aug 20, 2025. It is now read-only.

Move Airflow to TFXA #186

Open
murthyvs wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Move Airflow to TFXA #186

Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8cb7c4e
Create 20220915airfloworchestration
murthyvs Sep 15, 2022
a994a00
Rename 20220915airfloworchestration to 20220915airflow_orchestration
murthyvs Sep 15, 2022
58f7f2c
Create 20220919-airflow_orchestration
murthyvs Sep 19, 2022
16bad93
Delete 20220915airflow_orchestration
murthyvs Sep 19, 2022
aaf70b3
Rename 20220919-airflow_orchestration to 20220919-airflow_orchestrati…
murthyvs Sep 19, 2022
5a0d181
Update 20220919-airflow_orchestration.md
murthyvs Sep 19, 2022
fdfa5f1
Update 20220919-airflow_orchestration.md
murthyvs Sep 19, 2022
d9a2374
Update 20220919-airflow_orchestration.md
murthyvs Sep 21, 2022
2a389b9
Update 20220919-airflow_orchestration.md
murthyvs Sep 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions proposals/20220919-airflow_orchestration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#### SIG TFX-Addons
# Project Proposal

**Your name:** Varun Murthy

**Your email:** murthyvs@google.com

**Your company/organization:** Google Core ML TFX Team (MTV + Seoul)

**Project name:** Airflow Orchestration

## Project Description
Moving Airflow to TFX add-ons due to decreasing native support.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please describe what the results of the project will be, and a short introduction to what Airflow is and how it's used. You can also discuss the transition from having Airflow support included in TFX directly, versus having it as a separate install.

Copy link
Author

@murthyvs murthyvs Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can we installed and used as a separate package under tfxa/.


## Project Category
Orchestrator

## Project Use-Case(s)
We are moving Airflow from tfx/orchestration to tfx-addons. Native support for Airflow won't be provided in the near future.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A use-case describes how the project will be used by developers. In this case it should be something more like:

Apache Airflow is a widely used orchestrator for ML workflows. Developers can use Airflow with TFX to orchestrate TFX pipelines. This project continues support for Airflow, which includes the web console and CLI tooling which developers can use to monitor and control their TFX pipelines.

Copy link
Contributor

@rcrowe-google rcrowe-google Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imagine that you're a developer who is new to TFX. What do you need to know about Airflow, and how it fits with TFX? Why should you consider installing this module?


## Project Implementation
1. Copy tfx/orchestration/airflow to tfx-addons/airflow.
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Please describe how the install will work
  2. Please discuss any issues with testing in CI, given that this code will be independent of TFX
  3. Please describe any restructuring of the code, such as adding a setup script and tests

Copy link
Author

@murthyvs murthyvs Sep 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So ... this is where it gets a bit interesting because the tutorials on the TF website need to reflect the deprecation too. It's a multi-part effort in which we need to first move the orchestrator to TFXA and THEN update the documentation, delete/remove dependencies from tfx/. We can't provide an accurate, line-by-line description as of today ... but we need to create a project in TFXA to get started.

Copy link
Contributor

@rcrowe-google rcrowe-google Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok, just describe the issues and some options for dealing with them, and note that we haven't settled on a particular option yet. At some point however, we will need a solid plan to do this so that we can be successful.

2. Mark Airflow as deprecated in tfx/ and indicate that support will be dropped in 1-2 releases.
3. Update TFX tutorials on www.tensorflow.org to indicate deprecated and moving to TFXA

## Project Dependencies
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't dependencies. For comparison, see:

https://github.com/tensorflow/tfx-addons/blob/main/proposals/20210525-feast_example_gen.md#project-dependencies

https://github.com/tensorflow/tfx-addons/blob/main/proposals/20210707-xgboost_evaluator.md#project-dependencies

https://github.com/tensorflow/tfx-addons/blob/main/proposals/20220117-exit-handler-slack.md#project-dependencies

etc.

1. tfx/dsl/components/base:base_node
2. tfx/dsl/components/base:base_component
3. tfx/dsl/components/base:base_executor
4. tfx/dsl/components/base:executor_spec
5. tfx/orchestration:data_types
6. tfx/orchestration:metadata
7. tfx/orchestration/config:base_component_config
8. tfx/orchestration/launcher:base_component_launcher
9. tfx/orchestration:pipeline
10. tfx/orchestration:tfx_runner
11. tfx/orchestration/config:config_utils
12. tfx/orchestration/config:pipeline_config
13. tfx/utils:json_utils
14. tfx/utils:telemetry_utils

## Project Team
Varun Murthy (murthyvs@google.com)
Copy link
Contributor

@rcrowe-google rcrowe-google Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, and we also need your Github ID

Google Core ML TFX Team (tfx-team@google.com)

# Note
Please be aware of the processes and requirements which are outlined here:

* [SIG-TFX-Addons](https://github.com/tensorflow/tfx-addons)
* [Contributing Guidelines](https://github.com/tensorflow/tfx-addons/blob/main/CONTRIBUTING.md)
* [TensorFlow Code of Conduct](https://github.com/tensorflow/tfx-addons/blob/main/CODE_OF_CONDUCT.md)