How to install data

[peekxc]

It is mentioned here that one limitation of meson-python is no data--via meson's install_data function---is not available, though it hints that you can perhaps still include data inside a Python module.

How? It is entirely opaque how to proceed.

For example, if I have the layout:

├── pyproject.toml
├── src
│   └── my_package
│       ├── __init__.py
│       ├── data
│       │   └── __init__.py
|       |   ├── meson.build
│       └── meson.build

What do I need to put in either of the meson.build files to use importlib.resources here?

[dnicolodi]

There are just Python source files in the package layout in your example. Which data files do you want to install?

[dnicolodi]

I'll be happy to do it, but I don't have access to the project on readthedocs... @rgommers can you please give me access?

[peekxc]

That's the documentation for a very old version of meson-python.

Ahh, thank you, I didn't realize I was reading from an older doc site.

The error message explains clearly what's wrong. Just follow the guidance in the error message on how to fix it.

Not quite. The error message recommends setting pure: false, but this makes no sense as this is not an option in install_data; I have already set this in the find_installation call, i.e.:

py = py_mod.find_installation(pure: true, required: true)

Semantically, the pure parameter is apparently there to allow for the placement of platform-dependent files (e.g. *.so) alongside source *.py files. But I need data files, not just python source files and platform binaries.

I was able to finally get the data included in the wheel by treating the data files as python source files and using py.install_sources, i.e.

py.install_sources(
	meson.project_source_root() / 'src' / 'my_package' / 'data' / 'aggregation.npz',
	subdir: pkg_name / 'data',
	pure: false       # Will be installed next to binaries
)

This keeps it in the wheel and lets me import with importlib, though I haven't tested it on all platforms. The docs should prob be changed to mention this, as install_data seems for me to be a red herring.

[dnicolodi]

The pure: false is an option for py.get_install_dir().

py = py_mod.find_installation(pure: true, required: true)

This is wrong if you are installing extension modules.

Your solution is semantically wrong.

[dnicolodi]

What you need is:

py = import('python').find_installation(pure: false)
install_data(
    'src/my_package/data/foo.npz',
    install_dir: py.get_install_dir() / 'my_package/data',
)

or the equivalent

py = import('python').find_installation()
install_data(
    'src/my_package/data/foo.npz',
    install_dir: py.get_install_dir(pure: false) / 'my_package/data',
)

[eli-schwartz]

Semantically, the pure parameter is apparently there to allow for the placement of platform-dependent files (e.g. *.so) alongside source *.py files. But I need data files, not just python source files and platform binaries.

It is semantically there to semantically delineate the wheel (*.whl) file format's concept of purelib and platlib data, where some of that data may be files named *.py and possibly even files named *.so.

Python imports operate at the level of directory entries and you cannot have two directories named the same thing, one in the purelib part of sys.path and one in the platlib part of sys.path, without juggling the intricacies of namespace packages. This is a problem regardless of whether you are installing .py files or .npz files or .txt files.

The split between purelib and platlib is an artifact of a handful of distributors who want to put compiled C extensions in /usr/lib/python3.12/site-packages but put pure python modules in /use/share/python3.12/site-packages in order to conform to their own policies about "multilib" installations (32-bit software running on a 64-bit OS) where architecture-independent files are only installed once but compiled C code is installed once for 32-bit and once for 64-bit. It's kind of a weird appendage but everyone is forced to know about it -- in general you want your entire package to be either one or the other but if you have multiple top-level imports it is technically possible to split them across both installation schemes.

[peekxc]

This is wrong if you are installing extension modules.

Ahh, I guess I misunderstood the purpose of the pure flag, which is what caused this issue (I am installing extension modules). Changing all values to pure: false seems to have fixed the issue.

Sorry for wasting your time

[peekxc]

The split between purelib and platlib is an artifact of a handful of distributors who want to put compiled C extensions in /usr/lib/python3.12/site-packages but put pure python modules in /use/share/python3.12/site-packages in order to conform to their own policies about "multilib" installations (32-bit software running on a 64-bit OS) where architecture-independent files are only installed once but compiled C code is installed once for 32-bit and once for 64-bit. It's kind of a weird appendage but everyone is forced to know about it -- in general you want your entire package to be either one or the other but if you have multiple top-level imports it is technically possible to split them across both installation schemes.

Ahh I see. I assume the former usecase for placing something in /usr/lib/../site-packages is for larger often shared binaries (like blas?). In my case I have pythranized modules so keeping them alongside the original python source files is preferable.

[eli-schwartz]

No, it's literally because C extensions, even if they are only 15kb, semantically belong in the "libdir" for FHS reasons, and files such as ASCII / unicode interpreted script files, which do not care which bitness your processor is nor which operating system syscall ABI is used, are permitted in "datadir".

• binaries are architecture-dependent, the FHS says: put it in libdir.
• .py files, and .npz files, are architecture-agnostic, the FHS says: you may feel free to put them in datadir if you please, though if they have to maintain directory structure with binaries then you may prefer to put them in libdir.

Everyone must put python "platlib" (architecture-dependent) in /usr/lib/python3.12/site-packages, because that is the only valid location.

Most distributors also put "purelib" (architecture-agnostic) in /usr/lib/python3.12/site-packages, because it is allowed and has the advantage of being straightforward and simple.

Some put "purelib" (architecture-agnostic) in /usr/share/python3.12/site-packages, because... they like doing it? They have the right to do it according to the spec and I guess it allows them to save installation disk space when you have a 32-bit and a 64-bit python installed side by side, since you can install 3 packages:

• 32-bit binaries
• 64-bit binaries
• architecture-agnostic .py files

and have the first and second each depend on the third.

If they put .py files alongside the binaries then they'd have to install the .py files twice, once for 32-bit and once for 64-bit.

They are not using pip as a package manager, that is for sure.