Register plugins from entry points (#1991)

* Add test data to load plugins from entry points

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Load REZ plugins from setuptools entry points

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Fix flake8 report

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Fix import of importlib metadata in python 3.8, 3.9

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Remove dupplicated plugin definition

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Install test plugin to test data to speed up test

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Add try except on load plugin from entry points

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Update plugin install doc

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* use modern build backend, use dist-info, fix tests and simplify file structure.

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Update src/rez/tests/util.py

Signed-off-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>

* Remove vendored code for safety reason.

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Re-add importlib-metadata (this time with the right version to support py 3.7) and add missing dependencies (zipp, typing_extensions)

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Make it compatible with python 3.8 and 3.9

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Fix python compat again...

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Last fix for real

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Fix flake8

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Add new plugin install methods

Signed-off-by: Nig3l <nig3lpro@gmail.com>

* Organize entrypoints by plugin type to avoid having plugins being registered for every type

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Use entrypoint name for plugin name instead of module name

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Add debug logs to match the other plugin loading modes

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Fix tests

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* First pass at improving the plugins documentation

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Fix test due to missing commit

Signed-off-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>

* Improve layout of plugin guide and add a warning about the fact that it's under construction

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Re-order things in the plugins doc to make it more user friendly

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Clarify the docs a bit more

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

* Expose the plugins key in the configuration page

Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

---------

Signed-off-by: Nig3l <nig3lpro@gmail.com>
Signed-off-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>
Signed-off-by: Maxime Cots <42718803+Ni-g-3l@users.noreply.github.com>
Co-authored-by: Jean-Christophe Morin <jean_christophe_morin@hotmail.com>

Author: Ni-g-3l

docs/source/guides/developing_your_own_plugin.rst

@@ -0,0 +1,133 @@
+==========================
+Developing your own plugin
+==========================
+
+This guide will walk you through writing a rez plugin.
+
+.. warning::
+   This section is under constructions. The instructions provided might not be
+   accurate or up-to-date. We welcome contributions!
+
+Structure
+=========
+
+If you decide to register your plugin using the :ref:`entry-points method <plugin-entry-points>`, you are free
+to structure your plugin however you like.
+
+If you decide to implement your plugins using the ``rezplugins`` namespace package, please
+refer to :ref:`rezplugins-structure` to learn about the file structure that you will need to follow.
+
+Registering subcommands
+=======================
+
+Optionally, plugins can provide new ``rez`` subcommands.
+
+To register a plugin and expose a new subcommand, the plugin module:
+
+- **MUST** have a module-level docstring (used as the command help)
+- **MUST** provide a ``setup_parser()`` function
+- **MUST** provide a ``command()`` function
+- **MUST** provide a ``register_plugin()`` function
+- **SHOULD** have a module-level attribute ``command_behavior``
+
+For example, a plugin named ``foo`` and this is the ``foo.py`` in the plugin type
+root directory:
+
+.. code-block:: python
+   :caption: foo.py
+
+   """The docstring for command help, this is required."""
+   import argparse
+
+   command_behavior = {
+       "hidden": False,   # optional: bool
+       "arg_mode": None,  # optional: None, "passthrough", "grouped"
+   }
+
+
+   def setup_parser(parser: argparse.ArgumentParser):
+       parser.add_argument("--hello", action="store_true")
+
+
+   def command(
+       opts: argparse.Namespace,
+       parser: argparse.ArgumentParser,
+       extra_arg_groups: list[list[str]],
+   ):
+       if opts.hello:
+           print("world")
+
+
+   def register_plugin():
+       """This function is your plugin entry point. Rez will call this function."""
+
+       # import here to avoid circular imports.
+       from rez.command import Command
+
+       class CommandFoo(Command):
+           # This is where you declare the settings the plugin accepts.
+           schema_dict = {
+               "str_option": str,
+               "int_option": int,
+           }
+
+           @classmethod
+           def name(cls):
+               return "foo"
+
+       return CommandFoo
+
+Install plugins
+===============
+
+1. Copy directly to rez install folder
+
+   To make your plugin available to rez, you can install it directly under
+   ``src/rezplugins`` (that's called a namespace package).
+
+2. Add the source path to :envvar:`REZ_PLUGIN_PATH`
+
+   Add the source path to the ``REZ_PLUGIN_PATH`` environment variable in order to make your plugin available to rez.
+
+3. Add entry points to pyproject.toml
+
+   To make your plugin available to rez, you can also create an entry points section in your
+   ``pyproject.toml`` file, that will allow you to install your plugin with ``pip install`` command.
+
+   .. code-block:: toml
+      :caption: pyproject.toml
+
+       [build-system]
+       requires = ["hatchling"]
+       build-backend = "hatchling.build"
+
+       [project]
+       name = "foo"
+       version = "0.1.0"
+
+       [project.entry-points."rez.plugins"]
+       foo_cmd = "foo"
+
+4. Create a setup.py
+
+   To make your plugin available to rez, you can also create a ``setup.py`` file,
+   that will allow you to install your plugin with ``pip install`` command.
+
+   .. code-block:: python
+      :caption: setup.py
+
+       from setuptools import setup, find_packages
+
+       setup(
+           name="foo",
+           version="0.1.0",
+           package_dir={
+               "foo": "foo"
+           },
+           packages=find_packages(where="."),
+           entry_points={
+               'rez.plugins': [
+                   'foo_cmd = foo',
+               ]
+           }
+       )

docs/source/guides/index.rst

@@ -8,3 +8,4 @@ This section contains various user guides.
    :maxdepth: 1
 
    update_to_3
+   developing_your_own_plugin

docs/source/plugins.rst

@@ -6,6 +6,53 @@ Rez is designed around the concept of plugins. Plugins can be used to extend rez
 
 Rez comes with built-in plugins that are located at :gh-rez:`src/rezplugins`. New plugins are encouraged to be developed out-of-tree (outside rez).
 
+This page documents what plugins are available, the plugin types and how plugins are discovered.
+If you want to learn how to develop a plugin, please refer to :doc:`guides/developing_your_own_plugin`.
+
+.. _plugin-types:
+
+Existing plugin types
+=====================
+
+.. table::
+   :align: left
+
+   ====================== =========================================================== ==================
+   Type                   Base class(es)                                              Top level settings [1]_
+   ====================== =========================================================== ==================
+   ``build_process``      | :class:`rez.build_process.BuildProcess`
+                          | :class:`rez.build_process.BuildProcessHelper` [2]_        No
+   ``build_system``       :class:`rez.build_system.BuildSystem`                       No
+   ``command``            :class:`rez.command.Command`                                Yes
+   ``package_repository`` | :class:`rez.package_repository.PackageRepository`         No
+                          | :class:`rez.package_resources.PackageFamilyResource`
+                          | :class:`rez.package_resources.PackageResourceHelper`
+                          | :class:`rez.package_resources.VariantResourceHelper` [3]_
+   ``release_hook``       :class:`rez.release_hook.ReleaseHook`                       Yes
+   ``release_vcs``        :class:`rez.release_vcs.ReleaseVCS`                         Yes
+   ``shell``              :class:`rez.shells.Shell`                                   No
+   ====================== =========================================================== ==================
+
+.. [1] Top level settings: The concept of top level settings is documented in :ref:`default-settings`.
+.. [2] build_process: You have to choose between on of the two classes.
+.. [3] package_repository: All 4 classes have to be implemented.
+
+.. _configuring-plugins:
+
+Configuring plugins
+===================
+
+Plugins can be configured by adding a ``plugins`` key to your ``rezconfig.py``
+like this:
+
+.. code-block:: python
+
+   plugins = {
+       "package_repository": {
+           "filesystem": {}
+       }
+   }
+
 List installed plugins
 ======================
 
@@ -38,103 +85,104 @@ Currently installed plugins can be queried by running :option:`rez -i`
    shell               powershell  Windows PowerShell 5                               loaded
    shell               pwsh        PowerShell Core 6+                                 loaded
 
+Discovery mechanisms
+====================
 
-Configuring plugins
-===================
+There are three different discovery mechanisms for external/out-of-tree plugins:
 
-Plugins can be configured by adding a ``plugins`` key to your ``rezconfig.py``
-like this:
+#. :ref:`rezplugins-structure`
+#. :ref:`plugin-entry-points`
 
-.. code-block:: python
-
-   plugins = {
-       "filesystem": {}
-   }
+Each of these mechanisms can be used independently or in combination. It is up to you to
+decide which discovery mechanism is best for your use case. Each option has pros and cons.
 
-Existing plugin types
-=====================
+.. _rezplugins-structure:
 
-- :gh-rez:`build_process <src/rezplugins/build_process>`
-- :gh-rez:`build_system <src/rezplugins/build_system>`
-- :gh-rez:`command <src/rezplugins/command>`
-- :gh-rez:`package_repository <src/rezplugins/package_repository>`
-- :gh-rez:`release_hook <src/rezplugins/release_hook>`
-- :gh-rez:`release_vcs <src/rezplugins/release_vcs>`
-- :gh-rez:`shell <src/rezplugins/shell>`
+``rezplugins`` structure
+------------------------
 
-Developing your own plugin
-==========================
+This method relies on the ``rezplugins`` namespace package. Use the :data:`plugin_path` setting or
+the :envvar:`REZ_PLUGIN_PATH` environment variable to tell rez where to find your plugin(s).
 
-Rez plugins require a specific folder structure as follows:
+You need to follow the following file structure:
 
 .. code-block:: text
 
-    /plugin_type
-        /__init__.py (adds plugin path to rez)
-        /rezconfig.py (defines configuration settings for your plugin)
-        /plugin_file1.py (your plugin file)
-        /plugin_file2.py (your plugin file)
-        etc.
+   rezplugins/
+   ├── __init__.py
+   └── <plugin_type>/
+       ├── __init__.py
+       └── <plugin name>.py
 
-To make your plugin available to rez, you can install them directly under
-``src/rezplugins`` (that's called a namespace package) or you can add
-the path to :envvar:`REZ_PLUGIN_PATH`.
+``<plugin_type>`` refers to types defined in the :ref:`plugin types <plugin-types>` section. ``<plugin_name>`` is the name of your plugin.
+The ``rezplugins`` directory is not optional.
 
-Registering subcommands
------------------------
+.. note::
+    The path(s) MUST point to the directory **above** your ``rezplugins`` directory.
 
-Optionally, plugins can provide new ``rez`` subcommands.
+.. note::
+   Even though ``rezplugins`` is a python package, your sparse copy of
+   it should  not be on the :envvar:`PYTHONPATH`, just the :envvar:`REZ_PLUGIN_PATH`.
+   This is important  because it ensures that rez's copy of
+   ``rezplugins`` is always found first.
 
-To register a plugin and expose a new subcommand, the plugin module:
+.. _plugin-entry-points:
 
-- MUST have a module-level docstring (used as the command help)
-- MUST provide a `setup_parser()` function
-- MUST provide a `command()` function
-- MUST provide a `register_plugin()` function
-- SHOULD have a module-level attribute `command_behavior`
+Entry-points
+------------
 
-For example, a plugin named 'foo' and this is the ``foo.py`` in the plugin type
-root directory:
+.. versionadded:: 3.3.0
 
-.. code-block:: python
-   :caption: foo.py
+Plugins can be discovered by using `Python's entry-points <https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-package-metadata>`_.
 
-   """The docstring for command help, this is required."""
-   from rez.command import Command
+There is one entry-point per :ref:`plugin type <plugin-types>`:
 
-   command_behavior = {
-       "hidden": False,   # optional: bool
-       "arg_mode": None,  # optional: None, "passthrough", "grouped"
-   }
+* ``rez.plugins.build_process``
+* ``rez.plugins.build_system``
+* ``rez.plugins.command``
+* ``rez.plugins.package_repository``
+* ``rez.plugins.release_hook``
+* ``rez.plugins.release_vcs``
+* ``rez.plugins.shell``
 
-   def setup_parser(parser, completions=False):
-       parser.add_argument("--hello", ...)
+This allows a package to define multiple plugins. In fact, a package can contain multiple plugins of the same type and plugins for multiple types.
 
-   def command(opts, parser=None, extra_arg_groups=None):
-       if opts.hello:
-           print("world")
+.. note::
+   Unlike the other discovery mechanisms, this method doesn't require any special file structure. It is thus more flexible, less restricting
+   and easier to use.
 
-   class CommandFoo(Command):
-       # This is where you declare the settings the plugin accepts.
-       schema_dict = {
-           "str_option": str,
-           "int_option": int,
-           ...
-       }
-       @classmethod
-       def name(cls):
-           return "foo"
+.. _default-settings:
+
+Default settings
+----------------
+
+You can define default settings for the plugins you write by adding a ``rezconfig.py`` or ``rezconfig.yml``
+beside your plugin module. Rez will automatically load these settings.
 
-   def register_plugin():
-       return CommandFoo
+This is valid both all the discovery mechanisms.
 
-All new plugin types must define an ``__init__.py`` in their root directory
-so that the plugin manager will find them.
+Note that the format of that ``rezconfig.py`` or ``rezconfig.yml`` file for plugins is as follows:
 
 .. code-block:: python
-   :caption: __init__.py
 
-    from rez.plugin_managers import extend_path
-    __path__ = extend_path(__path__, __name__)
+   top_level_setting = "value"
+
+   plugin_name = {
+       "setting_1": "value1"
+   }
+
+In this case, the settings for ``plugin_name`` would be available in your plugin as ``self.settings``
+and ``top_level_setting`` would be available as ``self.type_settings.top_level_setting``.
+
+.. note::
+
+   Not all plugin types support top level settings. Please refer to the table in :ref:`plugin-types` to
+   see which types support them.
+
+Overriding built-in plugins
+===========================
 
+Built-in plugins can be overridden by installing a plugin with the same name and type.
+When rez sees this, it will prioritie your plugin over its built-in plugin.
 
+This is useful if you want to modify a built-in plugin without having to modify rez's source code.

src/rez/data/tests/extensions/baz/baz-0.1.0.dist-info/METADATA

@@ -0,0 +1,3 @@
+Metadata-Version: 2.4
+Name: baz
+Version: 0.1.0

src/rez/data/tests/extensions/baz/baz-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[rez.plugins.command]
+baz_cmd = baz