Packaging, related, non-runtime files with Setuptools without deprecated data_files

[Demiguise]

Hello,

I'm currently trying to get a python package via setuptools to correctly create a .data folder to be distributed in the wheel as it's specified in https://peps.python.org/pep-0427/#the-data-directory. I couldn't see a PEP or anything deprecating the usage of this .data folder, and it seems to be the one needed for usage in bazel rules.

For context, I'm trying to ship a single file with the python package that must be used to generate some code in users' environment, but it's not actually tied to the runtime of the python package at all. Bazel does provide a method to get access to these data files that are shipped in the .data folder.

I've seen that usage of data_files is deprecated, but that seems the only method to actually get that .data folder to be created is through this keyword. So far I've tried the MANIFEST.in, which seems to find the file and inserts it into the Egg's SOURCES.txt but then nothing is inserted into the actual wheel. include_package_data is also turned on, and I have other non-python files which are depended on at runtime that are correctly pulled into the wheel.

Is there some documentation/guide I'm missing? I'd rather not rely on deprecated methods and suddenly have things drop from under me when I upgrade setuptools next and it's just "Gone".

[mdklatt]

I also need a solution to the deprecation of data_files. I use the *nix convention of an etc/ directory for storing user-facing runtime files, such as config files, assets, etc. I have to support multiple languages, and having a Python application that looks and works like every other application is a requirement for my users.

My current project directories look this:

AppName/
    etc/
        config.toml
    src/
        AppName/
            __init__.py
    setup.py

I use data_files and entrypoints["console_scripts"] in setup.py to give me an install directory that looks like this (the install root is actually a virtualenv directory):

/opt/AppName/
    etc/
        config.toml
    bin/
        AppName
        python
    lib/python/site-packages
        AppName
        

This structure give me a one-to-one mapping between an editable install in my dev environment and a production install because the etc/ file lookup is done based on the current working directory.

Is there a way to achieve this with setuptools and a pyproject.toml file?

[abravalheri]

data_files can still be used as means of populating the *.data directory inside the generate wheel (.whl file).

However, this does not mean that the contents of the *.data directory inside the wheel will be copied to a specific location in the final user's machine.

Depending on how the user install the wheel and the specific characteristics of the OS, the contents of the *.data directory will end up in a different place. Therefore, using data_files to distribute OS-specific files (like desktop launchers, manpages, etc...) to a specific disk location is no longer supported when using PyPA tools (I am not sure this was ever supported 100%).

Users can still use data_files if they are happy with the way pip install them, considering all the variety that comes with that approach (e.g. virtual environments, pip install --user, pip install --target, pip on Windows, pip on BSD, etc).

@Demiguise, if you would like to propose any documentation changes please feel free to open a PR. I think it is important to make it clear for users that they cannot rely on data_files to distribute OS-specific files or for files that need to be placed at a specific fixed location at the end-user's machine¹.

@mdklatt I am afraid the PyPA tools cannot offer any guarantee that data_files will always be installed to /etc or /opt. If you need something like that you probably will need to have a look on OS-specific packaging tools.

Footnotes

1. To a certain extent, this assumption kind of alleviates the need for data_files, as package developers can place files inside the package directory itself and use importlib-resources to retrieve them. Even when multiple packages are involved, devs can always create a namespace package for the data files. ↩