Change entry-points to be toml files (#28)

Changes entry-points to be toml files

This also changes the auto-generation from decorator version
to create the entry-point files to use:
```
python -m spatch update-entrypoints file [file ...]
```
if you add the special:
```
[functions.auto-generation]
backend = "spatch._spatch_example.backend:backend1"
modules = ["spatch._spatch_example.backend"]
```
section to the entry-point file itself.

I really like this, plus the old abuse of black to format things nicely is fun.

TIL: `tomlkit` is really cool about editing toml files!

Note that we assume that `module` is the main `module` and any `.submodule`
is just a subdirectory within the same package.
I am not 100% sure if this is right, but since some tooling rejects
``/`` in entry-point values (to the extend that they break ALL
entry-point loading, not just ours!), we really can't rely on ``/``.

I am not 100% sure that this is always correct (i.e. I feel subpackages
are a thing now in principle, and those might not follow a directory
structure always?).
But, it seems practical enough, unless the entry-point validation is
changed.
(Or unless the packaging people explain why this is horrifying.)

* Apply suggestions from code review
* More fixes from review

Co-authored-by: Erik Welch <[email protected]>

---------

Signed-off-by: Sebastian Berg <[email protected]>
Co-authored-by: Erik Welch <[email protected]>

Author: seberg

docs/source/api/for_backends.rst

@@ -3,14 +3,19 @@ Backend API
 
 Backends have to do the heavy lifting of using spatch.
 At the moment we suggest to check the
-`example <https://github.com/scientific-python/spatch/tree/main/spatch/_spatch_example>`_.
+`example <https://github.com/scientific-python/spatch/tree/main/src/spatch/_spatch_example>`_.
 
 Entry point definition
 ----------------------
-To extend an existing library with a backend, you need to define a `Python entry-point <https://packaging.python.org/en/latest/specifications/entry-points/>`_.
+To extend an existing library with a backend, you need to define a
+`Python entry-point <https://packaging.python.org/en/latest/specifications/entry-points/>`_.
 This entry point includes the necessary information for spatch to find and
 dispatch to your backend.
 
+``spatch`` entry-points are TOML files and *not* Python objects.
+The entry-point value must point to the file with
+``module.submodule:filename.toml``. [#ep-value-structure]_
+
 Before writing a backend, you need to think about a few things:
 
 * Which types do you accept?  This could be NumPy, dask, jax, etc. arrays.
@@ -25,11 +30,11 @@ Before writing a backend, you need to think about a few things:
   In that case, your backend should likely only be used if prioritized
   by the user.
 
-Please check the example linked above.  These example entry-points include
-code that means running them modifies them in-place if the ``@implements``
-decorator is used (see next section).
+Please check the example linked above.  ``spatch`` can automatically update
+the functions entries in these entry-points if the ``@implements`` decorator
+is used (see next section).
 
-Some of the most important things are:
+Some of the most important fields are:
 
 ``name``
 ^^^^^^^^
@@ -55,13 +60,14 @@ However, we do support the following, e.g.:
 - ``"~numpy:ndarray"`` to match any subclass of NumPy arrays
 - ``"@module:qualname"`` to match any subclass of an abstract base class
 
-If you use an abstract base class, note that you must take a lot of care:
+.. warning::
+  If you use an abstract base class, note that you must take additional care:
 
-- The abstract base class must be cheap to import, because we cannot avoid
-  importing it.
-- Since we can't import all classes, ``spatch`` has no ability to order abstract
-  classes correctly (but we order them last if a primary type, which is typically right).
-- ``spatch`` will not guarantee correct behavior if an ABC is mutated at runtime.
+  - The abstract base class must be *cheap to import*, because we cannot avoid
+    importing it.
+  - Since we can't import all classes, ``spatch`` has no ability to order abstract
+    classes correctly (but we order them last if a primary type, which is typically correct).
+  - ``spatch`` will not guarantee correct behavior if an ABC is mutated at runtime.
 
 ``requires_opt_in``
 ^^^^^^^^^^^^^^^^^^^
@@ -93,19 +99,36 @@ functions
 ^^^^^^^^^
 
 A mapping of library functions to your implementations.  All fields use
-the ``__module__:__qualname__`` identifiers to avoid immediate import.
+the ``__module__:__qualname__`` identifiers.
 The following fields are supported for each function:
 
 - ``function``: The implementation to dispatch to.
 - ``should_run`` (optional): A function that gets all inputs (and context)
   and can decide to defer.  Unless you know things will error, try to make sure
   that this function is light-weight.
-- ``uses_context``: Whether the implementation needs a ``DispatchContext``.
+- ``uses_context`` (optional): Whether the implementation needs a ``DispatchContext``.
 - ``additional_docs`` (optional): Brief text to add to the documentation
   of the original function. We suggest keeping this short but including a
   link, but the library guidance should be followed.
 
-``spatch`` provides tooling to help create this mapping.
+A typical part of the entry-point TOML will look like this::
+
+  [functions."skimage.filters:gaussian"]
+  function = "cucim.skimage.filters:gaussian"
+  uses_context = true
+  additional_docs = "CUDA enabled version..."
+
+An additional ``[functions.defaults]`` key can be added to set defaults for all
+functions and avoid repeating e.g. ``uses_context``.
+
+``spatch`` provides tooling to help create this mapping.  This tooling uses the
+additional fields::
+
+  [functions.auto-generation]
+  # where to find the BackendImplementation:
+  backend = "module.submodule:backend_name"
+  # Additional modules to be imported (to ensure all functions are found):
+  modules = ["spatch._spatch_example.backend"]
 
 Manual backend prioritization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -127,6 +150,13 @@ More?
   maybe a version?  (I don't think we generally need it, but it may be
   interesting.)
 
+.. [#ep-value-structure]
+   ``spatch`` currently assumes submodules follow directory structure so that
+   the file is located relative to the main ``module`` at
+   ``module/submodule/filename.toml``.  As of 2025, do *not* use ``/`` in
+   the filename, since the entry-point value may be checked for being a
+   valid Python object ``__qualname__``, which a ``/`` is not.
+
 Implementations for dispatchable functions
 ------------------------------------------
 

docs/source/conf.py

@@ -1,11 +1,11 @@
 from __future__ import annotations
 
-import importlib_metadata
+import importlib.metadata
 
 project = "spatch"
 copyright = "2025, Spatch authors"
 author = "Spatch authors"
-version = release = importlib_metadata.version("spatch")
+version = release = importlib.metadata.version("spatch")
 
 extensions = [
     "myst_parser",

pyproject.toml

@@ -10,7 +10,6 @@ description = "Coming soon: Python library for enabling dispatching to backends"
 readme = "README.md"
 license = "BSD-3-Clause"
 requires-python = ">=3.10"
-dependencies = ["importlib_metadata"]
 classifiers = [
   "Development Status :: 3 - Alpha",
   "Environment :: Console",
@@ -19,7 +18,6 @@ classifiers = [
   "Operating System :: OS Independent",
   "Programming Language :: Python",
   "Programming Language :: Python :: 3",
-  "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: 3.13",
@@ -28,6 +26,7 @@ classifiers = [
   "Topic :: Software Development :: Libraries :: Python Modules",
 ]
 keywords = ["dispatching"]
+dependencies = ['tomli ; python_version < "3.11"']
 
 [project.urls]
 homepage = "https://github.com/scientific-python/spatch"
@@ -36,8 +35,8 @@ source = "https://github.com/scientific-python/spatch"
 changelog = "https://github.com/scientific-python/spatch/releases"
 
 [dependency-groups]
-# black is used to format entry-point files
-backend_utils = ["black"]
+# tomlkit is used to format entry-point files
+backend_utils = ["tomlkit"]
 test = [
   "pytest >=6",
   "pytest-cov >=3",
@@ -66,8 +65,12 @@ content-type = "text/markdown"
 path = "README.md"
 
 [project.entry-points._spatch_example_backends]
-backend1 = 'spatch._spatch_example.entry_point'
-backend2 = 'spatch._spatch_example.entry_point2'
+backend1 = 'spatch._spatch_example:entry_point.toml'
+backend2 = 'spatch._spatch_example:entry_point2.toml'
+
+[[tool.spatch.update_functions]]
+backend1 = "spatch._spatch_example.backend"
+backend2 = "spatch._spatch_example.backend"
 
 [tool.pixi.workspace]
 channels = ["https://prefix.dev/conda-forge"]

src/spatch/__main__.py

@@ -0,0 +1,27 @@
+import argparse
+
+from .backend_utils import update_entrypoint
+
+
+def main():
+    parser = argparse.ArgumentParser(prog="python -m spatch")
+    subparsers = parser.add_subparsers(help="subcommand help", required=True, dest="subcommand")
+
+    update_entrypoint_cmd = subparsers.add_parser(
+        "update-entrypoints", help="update the entrypoint toml file"
+    )
+    update_entrypoint_cmd.add_argument(
+        "paths", type=str, nargs="+", help="paths to the entrypoint toml files to update"
+    )
+
+    args = parser.parse_args()
+
+    if args.subcommand == "update-entrypoints":
+        for path in args.paths:
+            update_entrypoint(path)
+    else:
+        raise RuntimeError("unreachable: subcommand not known.")
+
+
+if __name__ == "__main__":
+    main()

src/spatch/_spatch_example/README.md

@@ -14,8 +14,9 @@ The "library" contains only:
   designed for only `int` inputs.
 
 We then have two backends with their corresponding definitions in `backend.py`.
-The entry-points are `entry_point.py` and `entry_point2.py` and these files
-can be run to generate their `functions` context (i.e. if you add more functions).
+The entry-points are `entry_point.toml` and `entry_point2.toml`. When code changes,
+these can be updated via `python -m spin update-entrypoints *.toml`
+(the necessary info is in the file itself).
 
 For users we have the following basic capabilities. Starting with normal
 type dispatching.

src/spatch/_spatch_example/entry_point.py

@@ -0,0 +1,22 @@
+name = "backend1"
+primary_types = ["builtins:float", "builtins:int"]
+secondary_types = []
+requires_opt_in = false
+
+# higher_priority_than = ["default"]
+
+[functions.auto-generation]
+# `backend = BackendImplementation` object used for `@backend.implements()`
+backend = "spatch._spatch_example.backend:backend1"
+# Modules to load (including submodules) to ensure all functions are imported.
+# `spatch` will also import all submodules.
+modules = ["spatch._spatch_example.backend"]
+
+[functions.defaults]
+# Options here will be defaults for all functions.
+uses_context = true
+
+[functions."spatch._spatch_example.library:divide"]
+function = "spatch._spatch_example.backend:divide"
+should_run = "spatch._spatch_example.backend:divide._should_run"
+additional_docs = """This implementation works well on floats."""

src/spatch/_spatch_example/entry_point.toml

@@ -0,0 +1,14 @@
+name = "backend2"
+primary_types = ["builtins:float", "builtins:complex"]
+secondary_types = ["builtins:int"]
+requires_opt_in = false
+
+[functions.auto-generation]
+backend = "spatch._spatch_example.backend:backend2"
+# `modules` not needed (backend import loads all functions)
+
+[functions."spatch._spatch_example.library:divide"]
+function = "spatch._spatch_example.backend:divide2"
+should_run = "spatch._spatch_example.backend:divide2._should_run"
+additional_docs = """This is a test backend!
+and it has a multi-line docstring which makes this longer than normal."""

src/spatch/_spatch_example/entry_point2.py

@@ -1,20 +1,26 @@
 import contextvars
 import dataclasses
 import functools
+import importlib.metadata
+import importlib.util
 import os
+import pathlib
 import sys
 import textwrap
 import warnings
 from collections.abc import Callable
 from dataclasses import dataclass
-from types import MethodType
+from types import MethodType, SimpleNamespace
 from typing import Any
 
-import importlib_metadata
-
 from spatch import from_identifier, get_identifier
 from spatch.utils import EMPTY_TYPE_IDENTIFIER, TypeIdentifier, valid_backend_name
 
+try:
+    import tomllib
+except ImportError:
+    import tomli as tomllib  # for Python 3.10 support
+
 __doctest_skip__ = ["BackendOpts.__init__"]
 
 
@@ -583,12 +589,34 @@ def _get_entry_points(group, blocked):
             return []
 
         backends = []
-        eps = importlib_metadata.entry_points(group=group)
+        eps = importlib.metadata.entry_points(group=group)
         for ep in eps:
             if ep.name in blocked:
                 continue
             try:
-                namespace = ep.load()
+                mod, _, filename = ep.value.partition(":")
+                # As of writing, entry-points are assumed to be valid Python
+                # names (e.g. no `/`).  But we can presumably assume that the
+                # submodules follow a directory structure (at least in practice).
+                # See also https://github.com/python/importlib_metadata/issues/523
+                mod, *path = mod.split(".")
+
+                spec = importlib.util.find_spec(mod)
+                reader = spec.loader.get_resource_reader(spec.name)
+                with reader.open_resource(pathlib.Path(*path, filename)) as f:
+                    backend_info = tomllib.load(f)
+
+                # We allow a `functions.defaults` field, apply them here to all functions
+                # and clean up the special fields (defaults and auto-generation).
+                backend_info["functions"].pop("auto-generation", None)  # no need to propagate
+                defaults = backend_info["functions"].pop("defaults", None)
+                if defaults:
+                    functions = backend_info["functions"]
+                    for fun in functions:
+                        functions[fun] = {**defaults, **functions[fun]}
+
+                # We use a namespace internally right now (convenient for in-code backends/tests)
+                namespace = SimpleNamespace(**backend_info)
                 if ep.name != namespace.name:
                     raise RuntimeError(
                         f"Entrypoint name {ep.name!r} and actual name {namespace.name!r} mismatch."