(torchx/pipelines) Retire support for OSS pipelines; kfp and airflow (#1119)

Summary:
Pull Request resolved: #1119

Virtually no usages on github (https://github.com/search?q=torchx.pipelines.kfp&type=code) other than references from forks of torchx.

Retiring support to reduce maintainer burden and quicker/lighter CI.

Reviewed By: ethanbwaite

Differential Revision: D82579476

Author: kiukchung

.github/workflows/kfp-integration-tests.yaml

@@ -39,11 +39,6 @@ lintrunner init
 lintrunner -a
 ```
 
-## Integration Tests
-
-See the [KFP integration test](scripts/kfpint.py) file for more details on setup
-and running them.
-
 ## License
 By contributing to TorchX, you agree that your contributions will be licensed
 under the LICENSE file in the root directory of this source tree.

CONTRIBUTING.md

@@ -55,9 +55,6 @@ pip install torchx
 # install torchx sdk and CLI -- all dependencies
 pip install "torchx[dev]"
 
-# install torchx kubeflow pipelines (kfp) support
-pip install "torchx[kfp]"
-
 # install torchx Kubernetes / Volcano support
 pip install "torchx[kubernetes]"
 

README.md

@@ -12,9 +12,6 @@ google-cloud-logging==3.10.0
 google-cloud-runtimeconfig==0.34.0
 hydra-core
 ipython
-kfp==1.8.22
-# pin protobuf to the version that is required by kfp
-protobuf==3.20.3
 mlflow-skinny
 moto~=5.0.8
 pyre-extensions
@@ -45,4 +42,3 @@ grpcio==1.62.1
 grpcio-status==1.48.1
 googleapis-common-protos==1.63.0
 google-api-core==2.18.0
-protobuf==3.20.3 # kfp==1.8.22 needs protobuf < 4

dev-requirements.txt

@@ -141,29 +141,6 @@ There are two ways to access runners in TorchX:
 See :ref:`Schedulers` for a list of schedulers that the runner can
 launch apps to.
 
-Pipeline Adapters
-~~~~~~~~~~~~~~~~~~~~~~
-While runners launch components as standalone jobs, ``torchx.pipelines``
-makes it possible to plug components into an ML pipeline/workflow. For a
-specific target pipeline platform (e.g. kubeflow pipelines), TorchX
-defines an adapter that converts a TorchX app spec to whatever the
-"stage" representation is in the target platform. For instance,
-``torchx.pipelines.kfp`` adapter for kubeflow pipelines converts an
-app spec to a ``kfp.ContainerOp`` (or more accurately, a kfp "component spec" yaml).
-
-
-In most cases an app spec would map to a "stage" (or node) in a pipeline.
-However advanced components, especially those that have a mini control flow
-of its own (e.g. HPO), may map to a "sub-pipeline" or an "inline-pipeline".
-The exact semantics of how these advanced components map to the pipeline
-is dependent on the target pipeline platform. For example, if the
-pipeline DSL allows dynamically adding stages to a pipeline from an upstream
-stage, then TorchX may take advantage of such feature to "inline" the
-sub-pipeline to the main pipeline. TorchX generally tries its best to adapt
-app specs to the **most canonical** representation in the target pipeline platform.
-
-See :ref:`Pipelines` for a list of supported pipeline platforms.
-
 Runtime
 ~~~~~~~~
 .. important:: ``torchx.runtime`` is by no means is a requirement to use TorchX.

docs/source/basics.rst

@@ -341,7 +341,7 @@ def handle_item(fieldarg, content):
     code_url = f"https://github.com/pytorch/torchx/archive/refs/heads/{notebook_version}.tar.gz"
 
 first_notebook_cell = f"""
-!pip install torchx[kfp]
+!pip install torchx
 !wget --no-clobber {code_url}
 !tar xf {notebook_version}.tar.gz --strip-components=1
 
@@ -351,7 +351,6 @@ def handle_item(fieldarg, content):
 sphinx_gallery_conf = {
     "examples_dirs": [
         "../../torchx/examples/apps",
-        "../../torchx/examples/pipelines",
     ],
     "gallery_dirs": [
         "examples_apps",

docs/source/conf.py

@@ -91,14 +91,6 @@ Works With
 
       schedulers/fb/*
 
-.. _Pipelines:
-.. toctree::
-   :maxdepth: 1
-   :caption: Pipelines
-
-   pipelines/kfp
-   pipelines/airflow.md
-
 .. fbcode::
 
    .. toctree::
@@ -116,7 +108,6 @@ Examples
    :caption: Examples
 
    examples_apps/index
-   examples_pipelines/index
 
 
 Components Library