Skip to content
This repository was archived by the owner on Jul 10, 2025. It is now read-only.

Commit a87167c

Browse files
author
Zhitao Li
committed
Some more explanation related to Beam and Airflow.
1 parent 13a1e0d commit a87167c

File tree

1 file changed

+8
-5
lines changed

1 file changed

+8
-5
lines changed

rfcs/20210307-tfx-api-artifact-compat.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,12 @@ Therefore we propose the following for all TFX 1.x versions:
203203
to test tfx with `apache-beam` RC so such breakages should be
204204
exceedingly rare in the future.
205205

206+
For users who wants to use older versions of `apache-beam` and `tfx` together,
207+
while TFX team cannot directly support them in our released packages, we plan to
208+
refactor our build scripts so that beam dependency can be overridden by
209+
themselves, so they can build Python packages and container images in a
210+
self-service fashion.
211+
206212
#### PyArrow {#pyarrow}
207213

208214
TFX’s dependency on PyArrow comes from
@@ -360,11 +366,8 @@ compatibility:
360366
The following Dag runners will remain experimental:
361367

362368
* AirflowDagRunner
363-
* Honestly speaking, support of this one is on the fence. Due to lack of
364-
proper configuration for a distributed backend support, it’s not much
365-
more powerful than the LocalDagRunner when used out-of-box. However,
366-
some partner teams managed to connect this to a distributed backend.
367-
Until that is supported in TFX, we don’t think this one is stable.
369+
* Due to low utilization of the AirflowDagRunner, we want to avoid calling
370+
it stable. Support will continue to graduate it into production as utilization evolves.
368371

369372
The following dag runners are slated to be deprecated and therefore won’t
370373
comply:

0 commit comments

Comments
 (0)