EHN: add tool.meson-python.wheel.{exclude,include} settings

Author: dnicolodi

docs/how-to-guides/shared-libraries.rst

@@ -187,38 +187,27 @@ strategies for folding a library built in a subproject into a wheel built with
    to be within the Python package's tree, or rely on ``meson-python`` to fold
    it into the wheel when it'd otherwise be installed to ``libdir``.
 
-Option (1) tends to be easier, so unless the library of interest cannot be
-built as a static library or it would inflate the wheel size too much because
-it's needed by multiple Python extension modules, we recommend trying option
-(1) first.
-
-A typical C or C++ project providing a library to link against tends to provide
-(a) one or more ``library()`` targets, which can be built as shared, static, or both,
-and (b) headers, pkg-config files, tests and perhaps other development targets
-that are needed to use the ``library()`` target(s). One of the challenges to use
-such projects as a subproject is that the headers and other installable targets
-are targeting system locations (e.g., ``<prefix>/include/``) which isn't supported
-by wheels and hence ``meson-python`` errors out when it encounters such an install
-target. This is perhaps the main issue one encounters with subproject usage,
-and the following two sections discuss how options (1) and (2) can work around
-that.
+Static linking tends to be easier, and it is the recommended solution, unless
+the library of interest cannot be built as a static library or it would
+inflate the wheel size too much because it's needed by multiple Python
+extension modules.
 
 Static library from subproject
 ------------------------------
 
-The major advantage of building a library target as static and folding it directly
-into an extension module is that no targets from the subproject need to be installed.
-To configure the subproject for this use case, add the following to the
-``pyproject.toml`` file of your package:
+The major advantage of building a library target as static and folding it
+directly into an extension module is that the RPATH or the DLL search path do
+not need to be adjusted and no targets from the subproject need to be
+installed. To ensures that ``library()`` targets are built as static, and that
+no parts of the subprojects are installed, the following configuration can be
+added in ``pyproject.toml`` to ensure the relevant options are passed to Meson:
 
 .. code-block:: toml
 
     [tool.meson-python.args]
     setup = ['--default-library=static']
     install = ['--skip-subprojects']
 
-This ensures that ``library`` targets are built as static, and nothing gets installed.
-
 To then link against the static library in the subproject, say for a subproject
 named ``bar`` with the main library target contained in a ``bar_dep`` dependency,
 add this to your ``meson.build`` file:
@@ -235,30 +224,37 @@ add this to your ``meson.build`` file:
         install: true,
     )
 
-That is all!
-
 Shared library from subproject
 ------------------------------
 
-If we can't use the static library approach from the section above and we need
-a shared library, then we must have ``install: true`` for that shared library
-target. This can only work if we can pass some build option to the subproject
-that tells it to *only* install the shared library and not headers or other
-targets that we don't need. Install tags don't work per subproject, so
-this will look something like:
+Sometimes it may be necessary or preferable to use dynamic linking to a shared
+library provided in a subproject, for example to avoid inflating the wheel
+size having multiple copies of the same object code in different extension
+modules using the same library. In this case, the subproject needs to install
+the shared library in the usual location in ``libdir``.  ``meson-python``
+will automatically include it into the wheel in
+``.<project-name>.mesonpy.libs`` just like an internal shared library.
+
+Most projects, however, install more than the shared library and the extra
+components, such as header files or documentation, should not be included in
+the Python wheel. Projects may have configuration options to disable building
+and installing additional components, in this case, these options can be
+passed to the ``subproject()`` call:
 
 .. code-block:: meson
 
     foo_subproj = subproject('foo',
         default_options: {
-            # This is a custom option - if it doesn't exist, can you add it
-            # upstream or in WrapDB?
-            'only_install_main_lib': true,
+            'docs': 'disabled',
         })
     foo_dep = foo_subproj.get_variable('foo_dep')
 
-Now we can use ``foo_dep`` like a normal dependency, ``meson-python`` will
-include it into the wheel in ``.<project-name>.mesonpy.libs`` just like an
-internal shared library that targets ``libdir`` (see
-:ref:`internal-shared-libraries`).
-*Remember: this method doesn't support Windows (yet)!*
+Install tags do not work per subproject, therefore to exclude other parts of
+the subproject form being included in the wheel, we need to resort to
+``meson-python`` install location filters using the
+:option:`tool.meson-python.wheel.exclude` build option:
+
+.. code-block:: toml
+
+    [tool.meson-python.wheel]
+    exclude = ['{prefix}/include/*']

docs/reference/pyproject-settings.rst

@@ -63,6 +63,29 @@ use them and examples.
 
    Extra arguments to be passed to the ``meson install`` command.
 
+.. option:: tool.meson-python.wheel.exclude
+
+   List of glob patterns matching paths of files that must be excluded from
+   the Python wheel. The accepted glob patterns are the ones implemented by
+   the Python :mod:`fnmatch` with case sensitive matching. The paths to be
+   matched are as they appear in the Meson introspection data, namely they are
+   rooted in one of the Meson install locations: ``{bindir}``, ``{datadir}``,
+   ``{includedir}``, ``{libdir_shared}``, ``{libdir_static}``, et cetera.
+
+   This configuration setting is measure of last resort to exclude installed
+   files from a Python wheel. It is to be used when the project includes
+   subprojects that do not allow fine control on the installed files. Better
+   solutions include the use of Meson install tags and excluding subprojects
+   to be installed via :option:`tool.meson-python.args.install`.
+
+.. option:: tool.meson-python.wheel.include
+
+   List of glob patterns matching paths of files that must not be excluded
+   from the Python wheel. All files recorded for installation in the Meson
+   project are included in the Python wheel unless matching an exclude glob
+   pattern specified in :option:`tool.meson-python.wheel.exclude`. An include
+   glob pattern is useful exclusively to limit the effect of an exclude
+   pattern that matches too many files.
 
 __ https://docs.python.org/3/c-api/stable.html?highlight=limited%20api#stable-application-binary-interface
 __ https://mesonbuild.com/Python-module.html#extension_module

mesonpy/__init__.py

@@ -16,6 +16,7 @@
 import contextlib
 import copy
 import difflib
+import fnmatch
 import functools
 import importlib.machinery
 import io
@@ -111,14 +112,32 @@ class InvalidLicenseExpression(Exception):  # type: ignore[no-redef]
 }
 
 
-def _map_to_wheel(sources: Dict[str, Dict[str, Any]]) -> DefaultDict[str, List[Tuple[pathlib.Path, str]]]:
+def _compile_patterns(patterns: List[str]) -> Callable[[str], bool]:
+    if not patterns:
+        return lambda x: False
+    func = re.compile('|'.join(fnmatch.translate(os.path.normpath(p)) for p in patterns)).match
+    return typing.cast('Callable[[str], bool]', func)
+
+
+def _map_to_wheel(
+    sources: Dict[str, Dict[str, Any]],
+    exclude: List[str],
+    include: List[str],
+) -> DefaultDict[str, List[Tuple[pathlib.Path, str]]]:
     """Map files to the wheel, organized by wheel installation directory."""
     wheel_files: DefaultDict[str, List[Tuple[pathlib.Path, str]]] = collections.defaultdict(list)
     packages: Dict[str, str] = {}
+    excluded = _compile_patterns(exclude)
+    included = _compile_patterns(include)
 
     for key, group in sources.items():
         for src, target in group.items():
-            destination = pathlib.Path(target['destination'])
+            target_destination = os.path.normpath(target['destination'])
+
+            if excluded(target_destination) and not included(target_destination):
+                continue
+
+            destination = pathlib.Path(target_destination)
             anchor = destination.parts[0]
             dst = pathlib.Path(*destination.parts[1:])
 
@@ -581,6 +600,10 @@ def _string_or_path(value: Any, name: str) -> str:
         'args': _table({
             name: _strings for name in _MESON_ARGS_KEYS
         }),
+        'wheel': _table({
+            'exclude': _strings,
+            'include': _strings,
+        }),
     })
 
     table = pyproject.get('tool', {}).get('meson-python', {})
@@ -829,6 +852,10 @@ def __init__(
         # from the package, make sure the developers acknowledge this.
         self._allow_windows_shared_libs = pyproject_config.get('allow-windows-internal-shared-libs', False)
 
+        # Files to be excluded from the wheel
+        self._excluded_files = pyproject_config.get('wheel', {}).get('exclude', [])
+        self._included_files = pyproject_config.get('wheel', {}).get('include', [])
+
     def _run(self, cmd: Sequence[str]) -> None:
         """Invoke a subprocess."""
         # Flush the line to ensure that the log line with the executed
@@ -912,7 +939,7 @@ def _manifest(self) -> DefaultDict[str, List[Tuple[pathlib.Path, str]]]:
                 sources[key][target] = details
 
         # Map Meson installation locations to wheel paths.
-        return _map_to_wheel(sources)
+        return _map_to_wheel(sources, self._excluded_files, self._included_files)
 
     @property
     def _meson_name(self) -> str:

tests/packages/subproject/pyproject.toml

@@ -5,3 +5,15 @@
 [build-system]
 build-backend = 'mesonpy'
 requires = ['meson-python']
+
+[tool.meson-python.wheel]
+exclude = [
+    # Meson before version 1.3.0 install data files in
+    # ``{datadir}/{project name}/``, later versions install
+    # in the more correct ``{datadir}/{subproject name}/``.
+    '{datadir}/*/data.txt',
+    '{py_purelib}/dep.*',
+]
+include = [
+    '{py_purelib}/dep.py',
+]

tests/packages/subproject/subprojects/dep/data.txt

@@ -0,0 +1 @@
+excluded via tool.meson-python.wheel.exclude

tests/packages/subproject/subprojects/dep/meson.build

@@ -7,3 +7,5 @@ project('dep')
 py = import('python').find_installation()
 
 py.install_sources('dep.py')
+
+install_data('data.txt')