Aggressively deprecate Channels and AdHocProvider (#3569)

This is described in issue #3515.

Issue #3515 has not received any comments arguing in favour of retaining
channels and the AdHocprovider, and several in support of removing them,
 and so this PR takes a heavy handed approach that is well on the way to
the end goal of #3515 of deleting channels and the AdHocProvider entirely:

Channels except LocalChannel are renamed, so that any users of
other channels will have to make a change to their code and
actively observe the word "Deprecated" in the name.

The AdHocProvider is renamed in the same way.

Most documentation (but not docstrings) about channels and the
ad-hoc provider is removed or replaced with a link to issue

Tests which much be manually run, and so in effect are never run and
shouldn't be expected to work now - parsl/tests/manual_tests and
parsl/tests/integration/ - are deleted rather than fixed to follow
the above naming change.

The tests for SSH channels and the AdHocProvider that run in CI are modified
to continue passing.

Exposure of the deprecated components via top level parsl.providers and
parsl.channels re-export is removed. To use this components, the deprecated
modules must be imported directly.

Author: benclifford

docs/historical/changelog.rst

@@ -334,7 +334,7 @@ New Functionality
 
 * New launcher: `parsl.launchers.WrappedLauncher` for launching tasks inside containers.
 
-* `parsl.channels.SSHChannel` now supports a ``key_filename`` kwarg `issue#1639 <https://github.com/Parsl/parsl/issues/1639>`_
+* ``parsl.channels.SSHChannel`` now supports a ``key_filename`` kwarg `issue#1639 <https://github.com/Parsl/parsl/issues/1639>`_
 
 * Newly added Makefile wraps several frequent developer operations such as:
 
@@ -442,7 +442,7 @@ New Functionality
   module, parsl.data_provider.globus
 
 * `parsl.executors.WorkQueueExecutor`: a new executor that integrates functionality from `Work Queue <http://ccl.cse.nd.edu/software/workqueue/>`_ is now available.
-* New provider to support for Ad-Hoc clusters `parsl.providers.AdHocProvider`
+* New provider to support for Ad-Hoc clusters ``parsl.providers.AdHocProvider``
 * New provider added to support LSF on Summit `parsl.providers.LSFProvider`
 * Support for CPU and Memory resource hints to providers `(github) <https://github.com/Parsl/parsl/issues/942>`_.
 * The ``logging_level=logging.INFO`` in `parsl.monitoring.MonitoringHub` is replaced with ``monitoring_debug=False``:
@@ -468,7 +468,7 @@ New Functionality
 
 * Several test-suite improvements that have dramatically reduced test duration.
 * Several improvements to the Monitoring interface.
-* Configurable port on `parsl.channels.SSHChannel`.
+* Configurable port on ``parsl.channels.SSHChannel``.
 * ``suppress_failure`` now defaults to True.
 * `parsl.executors.HighThroughputExecutor` is the recommended executor, and ``IPyParallelExecutor`` is deprecated.
 * `parsl.executors.HighThroughputExecutor` will expose worker information via environment variables: ``PARSL_WORKER_RANK`` and ``PARSL_WORKER_COUNT``
@@ -532,7 +532,7 @@ New Functionality
 
 * Cleaner user app file log management.
 * Updated configurations using `parsl.executors.HighThroughputExecutor` in the configuration section of the userguide.
-* Support for OAuth based SSH with `parsl.channels.OAuthSSHChannel`.
+* Support for OAuth based SSH with ``parsl.channels.OAuthSSHChannel``.
 
 Bug Fixes
 ^^^^^^^^^

docs/reference.rst

@@ -38,15 +38,9 @@ Configuration
 Channels
 ========
 
-.. autosummary::
-    :toctree: stubs
-    :nosignatures:
-
-    parsl.channels.base.Channel
-    parsl.channels.LocalChannel
-    parsl.channels.SSHChannel
-    parsl.channels.OAuthSSHChannel
-    parsl.channels.SSHInteractiveLoginChannel
+Channels are deprecated in Parsl. See
+`issue 3515 <https://github.com/Parsl/parsl/issues/3515>`_
+for further discussion.
 
 Data management
 ===============
@@ -109,7 +103,6 @@ Providers
     :toctree: stubs
     :nosignatures:
 
-    parsl.providers.AdHocProvider
     parsl.providers.AWSProvider
     parsl.providers.CobaltProvider
     parsl.providers.CondorProvider

docs/userguide/configuring.rst

@@ -15,15 +15,14 @@ queues, durations, and data management options.
 The following example shows a basic configuration object (:class:`~parsl.config.Config`) for the Frontera
 supercomputer at TACC.
 This config uses the `parsl.executors.HighThroughputExecutor` to submit
-tasks from a login node (`parsl.channels.LocalChannel`). It requests an allocation of
+tasks from a login node. It requests an allocation of
 128 nodes, deploying 1 worker for each of the 56 cores per node, from the normal partition.
 To limit network connections to just the internal network the config specifies the address
 used by the infiniband interface with ``address_by_interface('ib0')``
 
 .. code-block:: python
 
     from parsl.config import Config
-    from parsl.channels import LocalChannel
     from parsl.providers import SlurmProvider
     from parsl.executors import HighThroughputExecutor
     from parsl.launchers import SrunLauncher
@@ -36,7 +35,6 @@ used by the infiniband interface with ``address_by_interface('ib0')``
                 address=address_by_interface('ib0'),
                 max_workers_per_node=56,
                 provider=SlurmProvider(
-                    channel=LocalChannel(),
                     nodes_per_block=128,
                     init_blocks=1,
                     partition='normal',                                 
@@ -197,22 +195,6 @@ Stepping through the following question should help formulate a suitable configu
           are on a **native Slurm** system like :ref:`configuring_nersc_cori`
 
 
-4) Where will the main Parsl program run and how will it communicate with the apps?
-
-+------------------------+--------------------------+---------------------------------------------------+
-| Parsl program location | App execution target     | Suitable channel                                  |
-+========================+==========================+===================================================+
-| Laptop/Workstation     | Laptop/Workstation       | `parsl.channels.LocalChannel`                     |
-+------------------------+--------------------------+---------------------------------------------------+
-| Laptop/Workstation     | Cloud Resources          | No channel is needed                              |
-+------------------------+--------------------------+---------------------------------------------------+
-| Laptop/Workstation     | Clusters with no 2FA     | `parsl.channels.SSHChannel`                       |
-+------------------------+--------------------------+---------------------------------------------------+
-| Laptop/Workstation     | Clusters with 2FA        | `parsl.channels.SSHInteractiveLoginChannel`       |
-+------------------------+--------------------------+---------------------------------------------------+
-| Login node             | Cluster/Supercomputer    | `parsl.channels.LocalChannel`                     |
-+------------------------+--------------------------+---------------------------------------------------+
-
 Heterogeneous Resources
 -----------------------
 
@@ -337,7 +319,6 @@ Provide either the number of executors (Parsl will assume they are named in inte
                 worker_debug=True,
                 available_accelerators=2,
                 provider=LocalProvider(
-                    channel=LocalChannel(),
                     init_blocks=1,
                     max_blocks=1,
                 ),
@@ -372,7 +353,6 @@ Select the best blocking strategy for processor's cache hierarchy (choose ``alte
                 worker_debug=True,
                 cpu_affinity='alternating',
                 provider=LocalProvider(
-                    channel=LocalChannel(),
                     init_blocks=1,
                     max_blocks=1,
                 ),
@@ -412,18 +392,12 @@ These include ``OMP_NUM_THREADS``, ``GOMP_COMP_AFFINITY``, and ``KMP_THREAD_AFFI
 Ad-Hoc Clusters
 ---------------
 
-Any collection of compute nodes without a scheduler can be considered an
-ad-hoc cluster. Often these machines have a shared file system such as NFS or Lustre.
-In order to use these resources with Parsl, they need to set-up for password-less SSH access.
-
-To use these ssh-accessible collection of nodes as an ad-hoc cluster, we use
-the `parsl.providers.AdHocProvider` with an `parsl.channels.SSHChannel` to each node. An example
-configuration follows.
+Parsl's support of ad-hoc clusters of compute nodes without a scheduler
+is deprecated.
 
-.. literalinclude:: ../../parsl/configs/ad_hoc.py
-
-.. note::
-   Multiple blocks should not be assigned to each node when using the `parsl.executors.HighThroughputExecutor`
+See
+`issue #3515 <https://github.com/Parsl/parsl/issues/3515>`_
+for further discussion.
 
 Amazon Web Services
 -------------------

docs/userguide/examples/config.py

@@ -1,4 +1,3 @@
-from parsl.channels import LocalChannel
 from parsl.config import Config
 from parsl.executors import HighThroughputExecutor
 from parsl.providers import LocalProvider
@@ -8,9 +7,7 @@
         HighThroughputExecutor(
             label="htex_local",
             cores_per_worker=1,
-            provider=LocalProvider(
-                channel=LocalChannel(),
-            ),
+            provider=LocalProvider(),
         )
     ],
 )

docs/userguide/execution.rst

@@ -47,8 +47,7 @@ Parsl currently supports the following providers:
 7. `parsl.providers.AWSProvider`: This provider allows you to provision and manage cloud nodes from Amazon Web Services.
 8. `parsl.providers.GoogleCloudProvider`: This provider allows you to provision and manage cloud nodes from Google Cloud.
 9. `parsl.providers.KubernetesProvider`: This provider allows you to provision and manage containers on a Kubernetes cluster.
-10. `parsl.providers.AdHocProvider`: This provider allows you manage execution over a collection of nodes to form an ad-hoc cluster.
-11. `parsl.providers.LSFProvider`: This provider allows you to schedule resources via IBM's LSF scheduler.
+10. `parsl.providers.LSFProvider`: This provider allows you to schedule resources via IBM's LSF scheduler.
 
 
 

docs/userguide/plugins.rst

@@ -16,8 +16,8 @@ executor to run code on the local submitting host, while another executor can
 run the same code on a large supercomputer.
 
 
-Providers, Launchers and Channels
----------------------------------
+Providers and Launchers
+-----------------------
 Some executors are based on blocks of workers (for example the
 `parsl.executors.HighThroughputExecutor`: the submit side requires a
 batch system (eg slurm, kubernetes) to start worker processes, which then
@@ -34,10 +34,9 @@ add on any wrappers that are needed to launch the command (eg srun inside
 slurm). Providers and launchers are usually paired together for a particular
 system type.
 
-A `Channel` allows the commands used to interact with an `ExecutionProvider` to be
-executed on a remote system. The default channel executes commands on the
-local system, but a few variants of an `parsl.channels.SSHChannel` are provided.
-
+Parsl also has a deprecated ``Channel`` abstraction.  See
+`issue 3515 <https://github.com/Parsl/parsl/issues/3515>`_
+for further discussion.
 
 File staging
 ------------

parsl/channels/__init__.py

@@ -1,7 +1,4 @@
 from parsl.channels.base import Channel
 from parsl.channels.local.local import LocalChannel
-from parsl.channels.oauth_ssh.oauth_ssh import OAuthSSHChannel
-from parsl.channels.ssh.ssh import SSHChannel
-from parsl.channels.ssh_il.ssh_il import SSHInteractiveLoginChannel
 
-__all__ = ['Channel', 'SSHChannel', 'LocalChannel', 'SSHInteractiveLoginChannel', 'OAuthSSHChannel']
+__all__ = ['Channel', 'LocalChannel']

parsl/channels/oauth_ssh/oauth_ssh.py

@@ -3,7 +3,7 @@
 
 import paramiko
 
-from parsl.channels.ssh.ssh import SSHChannel
+from parsl.channels.ssh.ssh import DeprecatedSSHChannel
 from parsl.errors import OptionalModuleMissing
 
 try:
@@ -17,7 +17,7 @@
 logger = logging.getLogger(__name__)
 
 
-class OAuthSSHChannel(SSHChannel):
+class DeprecatedOAuthSSHChannel(DeprecatedSSHChannel):
     """SSH persistent channel. This enables remote execution on sites
     accessible via ssh. This channel uses Globus based OAuth tokens for authentication.
     """

parsl/channels/ssh/ssh.py

@@ -24,7 +24,7 @@ def _auth(self, username, *args):
         return
 
 
-class SSHChannel(Channel, RepresentationMixin):
+class DeprecatedSSHChannel(Channel, RepresentationMixin):
     ''' SSH persistent channel. This enables remote execution on sites
     accessible via ssh. It is assumed that the user has setup host keys
     so as to ssh to the remote host. Which goes to say that the following

parsl/channels/ssh_il/ssh_il.py

@@ -3,12 +3,12 @@
 
 import paramiko
 
-from parsl.channels.ssh.ssh import SSHChannel
+from parsl.channels.ssh.ssh import DeprecatedSSHChannel
 
 logger = logging.getLogger(__name__)
 
 
-class SSHInteractiveLoginChannel(SSHChannel):
+class DeprecatedSSHInteractiveLoginChannel(DeprecatedSSHChannel):
     """SSH persistent channel. This enables remote execution on sites
     accessible via ssh. This channel supports interactive login and is appropriate when
     keys are not set up.