Skip to content

Commit 122049a

Browse files
rgommersrgommers
committed
DOC: add documentation about using shared libraries
1 parent 3eddddb commit 122049a

File tree

2 files changed

+254
-0
lines changed

2 files changed

+254
-0
lines changed

‎docs/how-to-guides/shared-libraries.rst‎

Lines changed: 253 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,253 @@
1+
.. SPDX-FileCopyrightText: 2024 The meson-python developers
2+
..
3+
.. SPDX-License-Identifier: MIT
4+
5+
.. _shared-libraries:
6+
7+
**********************
8+
Using shared libraries
9+
**********************
10+
11+
Python projects may build shared libraries as part of their project, or link
12+
with shared libraries from a dependency. This tends to be a common source of
13+
issues, hence this page aims to explain how to include shared libraries in
14+
wheels, any limitations and gotchas, and how support is implemented in
15+
``meson-python`` under the hood.
16+
17+
We distinguish between *internal* shared libraries, i.e. they're built as part
18+
of the build executed by ``meson-python``, and *external* shared libraries that
19+
are only linked against from targets (usually Python extension modules) built
20+
by ``meson-python``. For internal shared libraries, we also distinguish whether
21+
the shared library is being installed to its default location (i.e. ``libdir``,
22+
usually something like ``<prefix>/lib/``) or to a location in ``site-packages``
23+
within the Python package install tree. All these scenarios are (or will be)
24+
supported, with some caveats:
25+
26+
+-----------------------+------------------+---------+-------+-------+
27+
| shared library source | install location | Windows | macOS | Linux |
28+
+=======================+==================+=========+=======+=======+
29+
| internal | libdir | no (1) | ✓ | ✓ |
30+
+-----------------------+------------------+---------+-------+-------+
31+
| internal | site-packages | ✓ | ✓ | ✓ |
32+
+-----------------------+------------------+---------+-------+-------+
33+
| external | n/a | ✓ (2) | ✓ | ✓ |
34+
+-----------------------+------------------+---------+-------+-------+
35+
36+
.. TODO: add subproject as a source
37+
38+
1: Internal shared libraries on Windows cannot be automatically handled
39+
correctly, and currently ``meson-python`` therefore raises an error for them.
40+
`PR meson-python#551 <https://github.com/mesonbuild/meson-python/pull/551>`__
41+
may improve that situation in the near future.
42+
43+
2: External shared libraries require ``delvewheel`` usage on Windows (or
44+
some equivalent way, like amending the DLL search path to include the directory
45+
in which the external shared library is located). Due to the lack of RPATH
46+
support on Windows, there is no good way around this.
47+
48+
.. _internal-shared-libraries:
49+
50+
Internal shared libraries
51+
=========================
52+
53+
A shared library produced by ``library()`` or ``shared_library()`` built like this
54+
55+
.. code-block:: meson
56+
57+
example_lib = shared_library(
58+
'example',
59+
'examplelib.c',
60+
install: true,
61+
)
62+
63+
is installed to ``libdir`` by default. If the only reason the shared library exists
64+
is to be used inside the Python package being built, then it is best to modify
65+
the install location to be within the Python package itself:
66+
67+
.. code-block:: python
68+
69+
install_path: py.get_install_dir() / 'mypkg/subdir'
70+
71+
Then an extension module in the same install directory can link against the
72+
shared library in a portable manner by using ``install_rpath``:
73+
74+
.. code-block:: meson
75+
76+
py3.extension_module('_extmodule',
77+
'_extmodule.c',
78+
link_with: example_lib,
79+
install: true,
80+
subdir: 'mypkg/subdir',
81+
install_rpath: '$ORIGIN'
82+
)
83+
84+
The above method will work as advertised on macOS and Linux; ``meson-python`` does
85+
nothing special for this case. Windows needs some special handling though, due to
86+
the lack of RPATH support:
87+
88+
.. literalinclude:: ../../tests/packages/sharedlib-in-package/mypkg/__init__.py
89+
:lines: 17-51
90+
91+
If an internal shared library is not only used as part of a Python package, but
92+
for example also as a regular shared library in a C/C++ project or as a
93+
standalone library, then the method shown above won't work - the library has to
94+
be installed to the default ``libdir`` location. In that case, ``meson-python``
95+
will detect that the library is going to be installed to ``libdir`` - which is
96+
not a recommended install location for wheels, and not supported by
97+
``meson-python``. Instead, ``meson-python`` will do the following *on platforms
98+
other than Windows*:
99+
100+
1. Install the shared library to ``<project-name>.mesonpy.libs`` (i.e., a
101+
top-level directory in the wheel, which on install will end up in
102+
``site-packages``).
103+
2. Rewrite RPATH entries for install targets that depend on the shared library
104+
to point to that new install location instead.
105+
106+
This will make the shared library work automatically, with no other action needed
107+
from the package author. *However*, currently an error is raised for this situation
108+
on Windows. This is documented also in :ref:`reference-limitations`.
109+
110+
111+
External shared libraries
112+
=========================
113+
114+
External shared libraries are installed somewhere on the build machine, and
115+
usually detected by a ``dependency()`` or ``compiler.find_library()`` call in a
116+
``meson.build`` file. When a Python extension module or executable uses the
117+
dependency, the shared library will be linked against at build time. On
118+
platforms other than Windows, an RPATH entry is then added to the built
119+
extension module or executable, which allows the shared library to be loaded at
120+
runtime.
121+
122+
.. note::
123+
124+
An RPATH entry alone is not always enough - if the directory that the shared
125+
library is located in is not on the loader search path, then the shared
126+
library may go missing at runtime. See, e.g., `meson#2121
127+
<https://github.com/mesonbuild/meson/issues/2121>`__ and
128+
`meson#13046 <https://github.com/mesonbuild/meson/issues/13046>`__ for
129+
issues this can cause.
130+
131+
If this happens, workarounds include adding the directory that the shared
132+
library is located in to the search path by amending the ``LD_LIBRARY_PATH``
133+
environment variable, or by using a compile flag ``-Wl,-rpath,/path/to/sharedlib/dir``.
134+
135+
On Windows, the shared library can either be preloaded, or vendored with
136+
``delvewheel`` in order to make the built Python package usable locally.
137+
138+
139+
Publishing wheels which depend on external shared libraries
140+
-----------------------------------------------------------
141+
142+
On all platforms, wheels which depend on external shared libraries usually need
143+
post-processing to make them usable on machines other than the one on which
144+
they were built. This is because the RPATH entry for an external shared library
145+
contains a path specific to the build machine. This post-processing is done by
146+
tools like ``auditwheel`` (Linux), ``delvewheel`` (Windows), ``delocate``
147+
(macOS) or ``repair-wheel`` (any platform, wraps the other tools).
148+
149+
Running any of those tools on a wheel produced by ``meson-python`` will vendor
150+
the external shared library into the wheel and rewrite the RPATH entries (it
151+
may also do some other things, like symbol mangling).
152+
153+
On Windows, the package author may also have to add the preloading like shown
154+
above with ``_enable_sharedlib_loading()`` to the main ``__init__.py`` of the
155+
package, ``delvewheel`` may or may not take care of this (please check its
156+
documentation if your shared library goes missing at runtime).
157+
158+
Note that we don't cover using shared libraries contained in another wheel
159+
and depending on such a wheel at runtime in this guide. This is inherently
160+
complex and not recommended (you need to be in control of both packages, or
161+
upgrades may be impossible/breaking).
162+
163+
164+
Using libraries from a Meson subproject
165+
=======================================
166+
167+
It can often be useful to build a shared library in a
168+
`Meson subproject <https://mesonbuild.com/Subprojects.html>`__, for example as
169+
a fallback in case an external dependency isn't detected. There are two main
170+
strategies for folding a library built in a subproject into a wheel built with
171+
``meson-python``:
172+
173+
1. Build the library as a static library instead of a shared library, and
174+
link it into a Python extension module that needs it.
175+
2. Build the library as a shared library, and either change its install path
176+
to be within the Python package's tree, or rely on ``meson-python`` to fold
177+
it into the wheel when it'd otherwise be installed to ``libdir``.
178+
179+
Option (1) tends to be easier, so unless the library of interest cannot be
180+
built as a static library or it would inflate the wheel size too much because
181+
it's needed by multiple Python extension modules, we recommend trying option
182+
(1) first.
183+
184+
A typical C or C++ project providing a library to link against tends to provide
185+
(a) one or more ``library()`` targets, which can be built as shared, static, or both,
186+
and (b) headers, pkg-config files, tests and perhaps other development targets
187+
that are needed to use the ``library()`` target(s). One of the challenges to use
188+
such projects as a subproject is that the headers and other installable targets
189+
are targeting system locations (e.g., ``<prefix>/include/``) which isn't supported
190+
by wheels and hence ``meson-python`` errors out when it encounters such an install
191+
target. This is perhaps the main issue one encounters with subproject usage,
192+
and the following two sections discuss how options (1) and (2) can work around
193+
that.
194+
195+
Static library from subproject
196+
------------------------------
197+
198+
The major advantage of building a library target as static and folding it directly
199+
into an extension module is that no targets from the subproject need to be installed.
200+
To configure the subproject for this use case, add the following to the
201+
``pyproject.toml`` file of your package:
202+
203+
.. code-block:: toml
204+
205+
[tool.meson-python.args]
206+
setup = ['--default-library=static']
207+
install = ['--skip-subprojects']
208+
209+
This ensures that ``library`` targets are built as static, and nothing gets installed.
210+
211+
To then link against the static library in the subproject, say for a subproject
212+
named ``bar`` with the main library target contained in a ``bar_dep`` dependency,
213+
add this to your ``meson.build`` file:
214+
215+
.. code-block:: meson
216+
217+
bar_proj = subproject('bar')
218+
bar_dep = bar_proj.get_variable('bar_dep')
219+
220+
py.extension_module(
221+
'_example',
222+
'_examplemod.c',
223+
dependencies: bar_dep,
224+
install: true,
225+
)
226+
227+
That is all!
228+
229+
Shared library from subproject
230+
------------------------------
231+
232+
If we can't use the static library approach from the section above and we need
233+
a shared library, then we must have ``install: true`` for that shared library
234+
target. This can only work if we can pass some build option to the subproject
235+
that tells it to *only* install the shared library and not headers or other
236+
targets that we don't need. Install tags don't work per subproject, so
237+
this will look something like:
238+
239+
.. code-block:: meson
240+
241+
foo_subproj = subproject('foo',
242+
default_options: {
243+
# This is a custom option - if it doesn't exist, can you add it
244+
# upstream or in WrapDB?
245+
'only_install_main_lib': true,
246+
})
247+
foo_dep = foo_subproj.get_variable('foo_dep')
248+
249+
Now we can use ``foo_dep`` like a normal dependency, ``meson-python`` will
250+
include it into the wheel in ``<project-name>.mesonpy.libs`` just like an
251+
internal shared library that targets ``libdir`` (see
252+
:ref:`internal-shared-libraries`).
253+
*Remember: this method doesn't support Windows (yet)!*

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ the use of ``meson-python`` and Meson for Python packaging.
8282
how-to-guides/config-settings
8383
how-to-guides/meson-args
8484
how-to-guides/debug-builds
85+
how-to-guides/shared-libraries
8586
reference/limitations
8687
projects-using-meson-python
8788

0 commit comments

Comments
 (0)