4.1.1: `build_sphinx` fails when module sourvce code is not in `<module_name/>` directory

[kloczek]

Describe the bug

Example agronholm/anyio#354
Currently many modules relocates its own source code to src/ directory. In most of the cases setup.py build_sphinx fails because looks like build_sphinx assumes that source code will be in `<module_name/> in project root directory.

Q: what should be kind of generic solution in such cases?

How to Reproduce

git clone https://github.com/agronholm/anyio/
cd anyio
python3 setup.py build_sphinx --build-dir build/sphinx

Expected behavior

sphinx-build should build documentation.

Your project

N/A

Screenshots

+ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx
running build_sphinx
Running Sphinx v4.1.1

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/sphinx/config.py", line 327, in eval_config_file
    exec(code, namespace)
  File "/home/tkloczko/rpmbuild/BUILD/anyio-3.3.0/docs/conf.py", line 17, in <module>
    v = pkg_resources.get_distribution('anyio').parsed_version
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 482, in get_distribution
    dist = get_provider(dist)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 358, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'anyio' distribution was not found and is required by the application

OS

Linux/x86_64

Python version

3.8.11

Sphinx version

4.1.1

Sphinx extensions

N/A

Extra tools

N/A

Additional context

I think that this is partially related to more generic case described in #9378
adding build/lib and build/lib-<os>.<arch> to sys.path could IMO solve both cases when module consist of DSOs are build and actual source code of the module is not <module_name/> directory.

[tk0miya]

because looks like build_sphinx assumes that source code will be in `<module_name/> in project root directory.

No. Sphinx and build_sphinx don't do anything for import-paths. It simply tries import module_name. So a project having src/ directory can't import its library on building document. How about modifying sys.path at your conf.py?

# conf.py
sys.path.append(os.path.abspath('../src'))

BTW, this case tries to load the target distribution via pkg_resources module. So installation the package would be needed.

  File "/home/tkloczko/rpmbuild/BUILD/anyio-3.3.0/docs/conf.py", line 17, in <module>
    v = pkg_resources.get_distribution('anyio').parsed_version

[kloczek]

BTW, this case tries to load the target distribution via pkg_resources module. So installation the package would be needed.

First I've been thinking that this is about sphinx_build was not able to locate module. Than when I've tries to run build_sphinx with PYTHONPATH=$PWD/build/lib env variable I've relegalised that this is not about that python was not able to locate module but message is about loading module resources:

pkg_resources.DistributionNotFound: The 'anyio' distribution was not found and is required by the application

I have no idea how to solve that exact case. Can someone point at least some direction in which solution of that issue could be? Is is correct what that module is trying to so?

[mgeier]

IMHO adding the source directory to sys.path is more of a hack, which should only be used in exceptional cases.

The "proper" way to handle this is to install the module before running Sphinx.

This is one of the main reasons why putting the source files into a separate src/ directory is nowadays generally recommended. By requiring the module to be installed, you make sure installation actually works.

Here's some links regarding src/ (not specific to Sphinx):

https://hynek.me/articles/testing-packaging/
https://blog.ionelmc.ro/2014/05/25/python-packaging/

[astrojuanlu]

+1 to installing the package before running Sphinx.

Besides, I don't get the excitement about setup.py build_sphinx: the ecosystem is moving away from setup.py files altogether, setup.cfg and pyproject.toml with standardized static metadata are gaining traction, backends different than setuptools are available, tox provides a Pythonic interface for project commands, and python -m sphinx, despite its flaws, just works.

[kloczek]

IMHO adding the source directory to sys.path is more of a hack, which should only be used in exceptional cases.

This is why when I've realised that build actually prepares all resources in build/lib* I've started thinking about use that fixed location to solve issue on which I've stumped.
I've already found few modules which are fiddling in sys.path in copy.py code so looks like that issues was spotted by at least few modules developers. Last example of such hack I found here https://github.com/zopefoundation/zope.interface/blob/master/docs/conf.py#L18
Sad only is that none of those people have been trying to find some generic solution of that kind of issues :(

[kloczek]

Besides, I don't get the excitement about setup.py build_sphinx: the ecosystem is moving away from setup.py files altogether, setup.cfg and pyproject.toml with standardized static metadata are gaining traction, backends different than setuptools are available, tox provides a Pythonic interface for project commands, and python -m sphinx, despite its flaws, just works.

IMO problem with all those standardisation is that none of them are trying to take care of building documentation :(
build module only assembles sdist or .whl file (or both) which cannot be used directly to build documentation and to use what build module generates pip or installer module must be used to deploy .whl file in exact location (please correct me if I'm wrong but that is my current state of the knowledge about new approach).

For some reasons looking on the code of those modules which already migrated to project.toml and setup.cfg I found that none of those modules test suites and documentation are added to .whl or sdist tar balls, and module maintainers suggests to use git trees which is IMO against standardisation which is like going back to stone age instead shiny future.
Testing and packaging documentation is incredibly important part of the packaging processes. Looks like new python standard is not even trying to provide some scaffolding for those two important steps/packaging stages.
IMO it would be good if build module will take care of building documentation as well (for now at least in form files used by sphinx).

And about using tox ..
Using tox to build documentation IMO is not a solution as well.
Why? Something like tox -e doc does not offer building documentation in exact format. For example in all my distribution python modules packages I'm building and packaging sphinx documentation as man pages (because this basic U*nix documentation format [1]). in tox.ini is hardcoded which one output format will be generated. sphinx-builder offer many possibilist of building documentation and tox procedure does not offer anything more than raw exact command execution in venv which is waaay not enough to provide kind of socket to plug custom build documentation procedure.

[1] just short demonstration what that approach offers with my already +550 python modules packages with documentation as man pages):

[tkloczko@barrel SRPMS]$ man python-
Display all 232 possibilities? (y or n)
python-amqp                               python-GitPython.tex                      python-paste                              python-sphinxcontrib-asyncio
python-anytree                            python-greenlet                           python-path                               python-sphinxcontrib-autoprogram
python-argcomplete                        python-h2                                 python-pillow                             python-sphinxcontrib-bibtex
python-arrow                              python-hacking                            python-platformdirs                       python-sphinxcontrib-httpdomain
python-"asgi                              python-hpack                              python-pluggy                             python-sphinxcontrib-programoutput
python-aspectlib                          python-httplib2                           python-polib                              python-sphinxcontrib-trio
python-astor                              python-hyperframe                         python-prb                                python-sphinxcopybutton
python-astroid                            python-hyperlink                          python-priority                           python-sphinxext-opengraph
python-async_generator                    python-hypothesis                         python-productmd-compose                  python-sphinx-inline-tabs
python-atomicwrites                       python-ifaddr                             python-productmd-composeinfo              python-sphinx-removed-in
python-attrs                              python-importlib-metadata                 python-productmd-discinfo                 python-sphinx_rtd_theme
python-augeas                             python-inflect                            python-productmd-images                   python-sphinx-tabs
python-autodocsumm                        python-ipykernel                          python-productmd-rpms                     python-sphinx-typlog-theme
python-babel                              python-ipythonparallel                    python-productmd-terminology              python-sphobjinv
python-backcall                           python-itsdangerous                       python-productmd-treeinfo                 python-sqlparse
python-backports.entry-points-selectable  python-jaraco-classes                     python-prompt_toolkit                     python-stdlib-list
python-benchmark                          python-jaraco-collections                 python-ptyprocess                         python-stem
python-betamax                            python-jaraco-envs                        python-purl                               python-sure
python-billiard                           python-jaraco-functools                   python-py                                 python-sybil
python-bleach                             python-jaraco.itertools                   python-pyasn1                             python-systemd
python-blinker                            python-jaraco-packaging                   python-pybtex-docutils                    python-terminado
python-boto3                              python-jaraco-path                        python-pycodestyle                        python-testpath
python-botocore                           python-jaraco-text                        python-pyfakefs                           python-testrepository
python-bottle                             python-jedi                               python-pyftpdlib                          python-test_server
python-breathe                            python-Jinja                              python-pygal                              python-testtools
python-build                              python-jinja2_pluralize                   python-pygments                           python-tidy
python-cachetools                         python-jmespath                           python-pyhamcrest                         python-tinycss2
python-case                               python-jupyter_client                     python-pylama                             python-tornado
python-cffi                               python-jupyter_core                       python-pylint                             python-traitlets
python-characteristic                     python-kiwi                               python-pymeeus                            python-transaction
python-chardet                            python-kombu                              python-pynacl                             python-trio
python-click                              python-lark                               python-pyopenssl                          python-trustme
python-click-log                          python-latexcodec                         python-pyrad                              python-twisted
python-contextlib2                        python-lazy-object-proxy                  python-pyrsistent                         python-uritemplate
python-convertdate                        python-libevdev                           python-pyscss                             python-urllib3
python-coveragepy                         python-linkify-it-py                      python-pytest                             python-validators
python-coveralls                          python-lockfile                           python-pytest-checkdocs                   python-vine
python-cppy                               python-lxml                               python-pytest-cov                         python-virtualenv
python-cssselect2                         python-mako                               python-pytest-regressions                 python-waitress
python-cython                             python-markupsafe                         python-pytest-runner                      python-wcwidth
python-dateutil                           python-mdit-py-plugins                    python-pytest-xprocess                    python-webcolors
python-dictdiffer                         python-mistune                            python-python-sphinx-contribspelling      python-webencodings
python-dpkt                               python-more-itertools                     python-pyudev                             python-webob
python-dulwich                            python-msgpack                            python-pyxattr                            python-websocket-client
python-elementpath                        python-multidict                          python-pyxdg                              python-websockets
python-evdev                              python-mypy                               python-rdflib                             python-webtest
python-eventlet                           python-myst-parser                        python-requests-mock                      python-werkzeug
python-execnet                            python-nbclient                           python-requests_toolbelt                  python-wheel
python-faker                              python-nbformat                           python-rich                               python-Whoosh
python-fields                             python-netaddr                            python-rsa                                python-wrapt
python-flake8                             python-nose2                              python-rst.linker                         python-WSGIProxy2
python-flask                              python-notebook                           python-scons                              python-wsproto
python-flask-sqlalchemy                   python-olefile                            python-scripttest                         python-xmlschema
python-flit                               python-openstackdocstheme                 python-semantic-version                   python-yamlloader
python-fonttools                          python-outcome                            python-service-identity                   python-yarl
python-gcovr                              python-paramiko                           python-smartypants                        python-zeroconf
python-gidocgen                           python-parso                              python-smmap.tex                          python-zipp
python-gitdb                              python-parver                             python-sniffio                            python-zope-event

In other words I've already reached more than only POC stage of that kind of approach and I'm now more and more worry that more modules will be dropping setuptools support :(

[tk0miya]

I have no idea how to solve that exact case. Can someone point at least some direction in which solution of that issue could be? Is is correct what that module is trying to so?

The only way to resolve this is installing anyio package. The conf.py of anyio requires to install itself on building documentation to obtain its version from the package metadata. If you'd not like to do, please ask them to modify the conf.py not to require installation.

AFAIK, fetching the version from the package metadata is a well-known technique to define the version of the package only on setup.py. In other words, it can avoid that the version appears in the multiple files.

I've already found few modules which are fiddling in sys.path in copy.py code so looks like that issues was spotted by at least few modules developers. Last example of such hack I found here https://github.com/zopefoundation/zope.interface/blob/master/docs/conf.py#L18
Sad only is that none of those people have been trying to find some generic solution of that kind of issues :(

IMO, the users of Sphinx can choose how they build their documentation. I guess zopefundation chooses sphinx-build command to build their documentation instead of setup.py build_sphinx command. The hack will still be needed if we'll improve build_sphinx command (refs: #9520) if somebody also uses sphinx-build command. I don't think this is so wrong.

[astrojuanlu]

There are several things in @kloczek comment:

• Maintainers not including tests or docs in their sdists: I agree there should be better practices around this (I'm not even sure I do it correctly with my own open source projects). Sadly, Python packaging sometimes makes it difficult.
• Maintainers not including tests or docs in their wheels: as it should be! Wheels are meant to be uncompressed somewhere and that's it.
• tox not being flexible enough: I think you're missing the fact that you can pass command line parameters to tox commands, like tox -e docs -- html. Then tox.ini needs to properly use posargs as explained in https://tox.readthedocs.io/en/latest/example/general.html

My only concern is that, by making the setuptools integration too sophisticated, Sphinx also has a significant maintenance burden, for a functionality that will be used by a decreasing number of projects (since setup.py is fading away, especially for simple, small ones).

[kloczek]

• Maintainers not including tests or docs in their wheels: as it should be! Wheels are meant to be uncompressed somewhere and that's it.

To be honest recently I've been thinking about kind of scenario in which setuptools still would be possible to use.
What about if:

• project.toml and setup.cfg are present in project root:
  • If yes: prepare build source tree with generated setup.py actual contest which will be used to perform build, generate documentation, install and tests. Optionally (or by default as well) could be generated sdist and .whl file(s).
    In this scenario should be generated as well files like tox.ini (I'm not sure if tox is able to use directly project.toml and setup.cfg). Provide as well any necessity build, install, generate documentation or test in virtual env if it is specified in project description)
  • if not: continue classic way with setuptools if setup.py is present

Another missing bit IMO is related to other data file. Some projects uses some additional json, text or gettext .mo files. So far looks like none of the PEP specifications takes care of how to deal with those files (please correct me if I'm wrong).
Classic examples of packages with gettext files are many sphinx extensions which are using i18n to provide translations of some generated documentations. Pure python seems is able to handle gettext .mo files however none of the build python frameworks are able to provide eny descriptions to handle those files (and maintain those resources like analog of make -C po update-po in case of GNU autotools).

Next part is documentation. IMO building documentation or provide non-empty list of test units should be optional.
Typical packaging process (like building deb, rpm or publishing IPS package) is divides in to few stages:

• prep: unpackage and prepare source tree to be used (apply patches, add/remove some files etc.)
• build: configure source tree check python modules build dependencies, other tools (like C compiler) and build binaries, build documentation etc in separated subtree which on next stages will be used by install and to execute test suite(s)
• install: install file in <install/prefix>
• check: perform additional checks like performing test suites if it is only possible those test should be performed on actual files deployed by install in install stage in <install/prefix> instead on subtree generated in build.
  If test suite needs to be executed in venv that should be part of the project.toml/setup.cfg description and checking availability of virtualenv or tox should be part of the initial checks performed in build stage.
• assembly: perform additional operations like separate debug info files from ELF binaries, check consistency of the generated by all pervious stages (for example do eny of the python module accidentally have shebangs like #!/bin/foobar and warn/error/automatically strip those lines before actual packaging)

As the person who is building every day tenths to sometimes even thousands packages exact day (sometimes only to confirm that still build is OK in constantly evolving build env) I would expect that that python build framework will take total control of build, install and check stages allowing to fire exact of those stages separately (today I've noticed that recently tox definitions which up to now was used to mainly test project as well by default calls build documentation (example: just released new version of build module).

If pip wold be used it should automatically hook into build to take care of automatic installation of missing modules otherwise error message should be printed to allow react packager (actual person) or packaging automation.

Additionally pair of project.toml/setup.cfg should possible to query to see actual set of build dependencies. This would allow automatically updated packaging description dependencies or perform some automatic actions if build dependencies would change between versions. Output in for example json format would be really welcome and could be as well useful to plug easy any other possible automation (pip could use that as well).

I think that this would allow provide enough interface used automatically by pip and other automations used for example generate OS distribution packages.

Comments?

[astrojuanlu]

Another missing bit IMO is related to other data file. Some projects uses some additional json, text or gettext .mo files. So far looks like none of the PEP specifications takes care of how to deal with those files (please correct me if I'm wrong).

I have no idea if this is specified in PEPs. In any case, setuptools has package_data and include_package_data, whereas flit includes all data files, and I understand that poetry does a similar thing. Even if they are not specified, all relevant backends have mechanisms in place to include data files, including (but not limited to) setuptools.

To be honest recently I've been thinking about kind of scenario in which setuptools still would be possible to use. What about if:
...Additionally pair of project.toml/setup.cfg should possible to query to see actual set of build dependencies.

With modern packaging tools, one can do all these things without depending on a specific backend:

• One can install a package using pip install ., regardless of the build backend (setuptools, flit, poetry, other)
• One can build a sdist and a wheel using python -m build https://pypi.org/project/build/, regardless of the build backend (setuptools, flit, poetry, other)
• One can programmatically query the dependencies of a project using the pep517 package https://pypi.org/project/pep517/, regardless of the build backend (setuptools, flit, poetry, other) as pip-tools started doing thanks to this recommendation:

>>> from pep517 import meta
>>> print(meta.load('.').requires[:5])
['sphinxcontrib-applehelp', 'sphinxcontrib-devhelp', 'sphinxcontrib-jsmath', 'sphinxcontrib-htmlhelp (>=2.0.0)', 'sphinxcontrib-serializinghtml (>=1.1.5)']

And then, these tools have existed for a while:

• One can usually run the tests calling pytest or tox. There is less agreement there, but at least these commands do not depend on a specific build backend.
• One can usually build the documentation using python -m sphinx -b html docs docs/build/html, or cd docs && make html, or similar. Again, there is some leeway here, proposals to make a new sphinx command Provide sphinx command and integrate sphinx commands #5618 and so forth, but it doesn't depend on the build backend.

So, in my opinion treating setuptools in a special way is not in line with current developments in modern Python packaging. Quite the opposite.

As the person who is building every day tenths to sometimes even thousands packages exact day (sometimes only to confirm that still build is OK in constantly evolving build env) I would expect that that python build framework will take total control of build, install and check stages allowing to fire exact of those stages separately

That's a sane aspiration! To achieve it, in my opinion we need to depend less on setuptools, not more.

[astrojuanlu]

In any case, I doubt this is the right place to have this conversation. I just wanted to chime in because I've been seeing a lot of issues being filed for the setuptools integration of Sphinx. Ultimately it's the maintainers decision to prioritize them or not, and on the other hand I am not here to convince anyone to change their workflow :) Feel free to mark these comments as off-topic, since they are only tangentially related with the issue at hand.