Skip to content

Commit a07a392

Browse files
committed
docs: document sysconfig quirks
1 parent 268fe2b commit a07a392

File tree

1 file changed

+61
-0
lines changed

1 file changed

+61
-0
lines changed

docs/quirks.rst

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -384,3 +384,64 @@ to provide the missing ``libcrypt.so.1`` library. Modern versions of these
384384
distros should install this package automatically when installing
385385
``redhat-lsb-core`` (or a similarly named) package. This package should be
386386
present in the base OS install.
387+
388+
.. _quirk_references_to_build_paths:
389+
390+
References to Build-Time Paths
391+
==============================
392+
393+
The built Python distribution captures some absolute paths and other
394+
build-time configuration in a handful of files:
395+
396+
* In a ``_sysconfigdata_*.py`` file in the standard library. e.g.
397+
``lib/python3.10/_sysconfigdata__linux_x86_64-linux-gnu.py``.
398+
* In a ``Makefile`` under a ``config-*`` directory in the standard library.
399+
e.g. ``lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile``.
400+
* In ``pkgconfig`` files. e.g. ``lib/pkgconfig/python3.pc``.
401+
* In ``python*-config`` files. e.g. ``bin/python3.10-config``.
402+
* In ``PYTHON.json`` (mostly reflected values from ``_sysconfigdata_*.py``.
403+
404+
Each of these serves a different use case. But the general theme is various
405+
aspects of the Python distribution attempt to capture how Python was built.
406+
The most common use of these values is to facilitate compiling or linking
407+
other software against this Python build. For example, the ``_sysconfigdata*``
408+
module is loaded by the `sysconfig <https://docs.python.org/3/library/sysconfig.html>`_
409+
module. ``sysconfig`` in turn is used by packaging tools like ``setuptools``
410+
and ``pip`` to figure out how to invoke a compiler for e.g. compiling C
411+
extensions from source.
412+
413+
On Linux, our distributions are built in containers. The container has a
414+
custom build of Clang in a custom filesystem location. And Python is
415+
installed to the prefix ``/install``. So you may see references to
416+
``/install`` in Linux distributions.
417+
418+
On macOS, most distributions are built from GitHub Actions runners. They
419+
use a specific macOS SDK. So you may see references to SDK paths that don't
420+
exist on your machine. e.g.
421+
``/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk``.
422+
423+
On Windows, builds are performed from a temporary directory. So you may
424+
see references to temporary directories in Windows distributions.
425+
426+
**The existence of hard-coded paths in our produced distributions can confuse
427+
consumers of these values and break common workflows, like compiling C
428+
extensions.**
429+
430+
We don't currently have a great idea for how to solve this problem. We
431+
can't hardcode values that will work on every machine because every machine
432+
has different filesystem layouts. For example, if we hardcode ``gcc`` as
433+
the compiler, someone with only ``clang`` installed will complain. And
434+
we certainly don't know where end-users will extract their Python
435+
distribution to!
436+
437+
To solve this problem requires executing dynamic code after extracting
438+
our custom distributions in order to patch these hardcoded values into
439+
conformance with the new machine. We're unsure how to actually do this
440+
because figuring out what values to set is essentially equivalent to
441+
reinventing autoconf / configure! Perhaps we could implement something
442+
that works in common system layouts (e.g. hardcoded defaults for common
443+
distros like Debian/Ubuntu and RedHat).
444+
445+
Until we have a better solution here, just understand that anything looking
446+
at ``sysconfig`` could resolve non-existent paths or names of binaries that
447+
don't exist on the current machine.

0 commit comments

Comments
 (0)