4.0.2: problem with setuptools `build_sphinx` command and DSO modules

[kloczek]

Describe the bug
I'm trying to build documentation argon2-cffi module (https://github.com/hynek/argon2-cffi) using setuptools build_sphinx command.

$ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx

Problem is that this module in build builds DSO modules and to generate documentation build_sphinx needs the access to those mopdule.
python3 setup.py build stores all resources in build/lib-<os>.<arch> and in such case IMO build_sphinx should automatically add that path to python syst.path

[tkloczko@barrel argon2-cffi-20.1.0]$ strace -fe trace=file /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx 2>&1 | grep -w build
execve("/usr/bin/python3", ["/usr/bin/python3", "setup.py", "build_sphinx", "-b", "man", "--build-dir", "build/sphinx"], 0x7ffc5c1c5b30 /* 33 vars */) = 0
newfstatat(AT_FDCWD, "/usr/lib64/python3.8/distutils/command/build.py", {st_mode=S_IFREG|0644, st_size=5767, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib64/python3.8/distutils/command/build.py", {st_mode=S_IFREG|0644, st_size=5767, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib64/python3.8/distutils/command/__pycache__/build.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/PKG-INFO", {st_mode=S_IFREG|0644, st_size=2935, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/PKG-INFO", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/namespace_packages.txt", 0x7ffcf814d200, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info", {st_mode=S_IFDIR|0755, st_size=160, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/PKG-INFO", {st_mode=S_IFREG|0644, st_size=2935, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/PKG-INFO", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/entry_points.txt", {st_mode=S_IFREG|0644, st_size=109, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/python3.8/site-packages/build-0.4.0-py3.8.egg-info/entry_points.txt", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/usr/lib64/python3.8/distutils/command/build.py", {st_mode=S_IFREG|0644, st_size=5767, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib64/python3.8/distutils/command/build.py", {st_mode=S_IFREG|0644, st_size=5767, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/sphinx/cmd/build.py", {st_mode=S_IFREG|0644, st_size=12497, ...}, 0) = 0
newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/sphinx/cmd/build.py", {st_mode=S_IFREG|0644, st_size=12497, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/python3.8/site-packages/sphinx/cmd/__pycache__/build.cpython-38.pyc", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/man", {st_mode=S_IFDIR|0755, st_size=26, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/man", {st_mode=S_IFDIR|0755, st_size=26, ...}, 0) = 0
newfstatat(AT_FDCWD, "build/sphinx/man", {st_mode=S_IFDIR|0755, st_size=26, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx", {st_mode=S_IFDIR|0755, st_size=22, ...}, 0) = 0
mkdir("/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees", 0777) = -1 EEXIST (File exists)
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees", {st_mode=S_IFDIR|0755, st_size=396, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/environment.pickle", {st_mode=S_IFREG|0644, st_size=1719965, ...}, 0) = 0
loading pickled environment... openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/environment.pickle", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/contributing.doctree", {st_mode=S_IFREG|0644, st_size=42630, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/faq.doctree", {st_mode=S_IFREG|0644, st_size=9797, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/index.doctree", {st_mode=S_IFREG|0644, st_size=9348, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/installation.doctree", {st_mode=S_IFREG|0644, st_size=16951, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/parameters.doctree", {st_mode=S_IFREG|0644, st_size=17323, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/license.doctree", {st_mode=S_IFREG|0644, st_size=15666, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/api.doctree", {st_mode=S_IFREG|0644, st_size=136822, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/cli.doctree", {st_mode=S_IFREG|0644, st_size=4324, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/changelog.doctree", {st_mode=S_IFREG|0644, st_size=64037, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/backward-compatibility.doctree", {st_mode=S_IFREG|0644, st_size=5968, ...}, 0) = 0
newfstatat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/argon2.doctree", {st_mode=S_IFREG|0644, st_size=16390, ...}, 0) = 0
argon2-cffi.3 { openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/index.doctree", O_RDONLY|O_CLOEXEC) = 3
argon2 openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/argon2.doctree", O_RDONLY|O_CLOEXEC) = 3
installation openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/installation.doctree", O_RDONLY|O_CLOEXEC) = 3
api openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/api.doctree", O_RDONLY|O_CLOEXEC) = 3
parameters openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/parameters.doctree", O_RDONLY|O_CLOEXEC) = 3
cli openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/cli.doctree", O_RDONLY|O_CLOEXEC) = 3
faq openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/faq.doctree", O_RDONLY|O_CLOEXEC) = 3
backward-compatibility openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/backward-compatibility.doctree", O_RDONLY|O_CLOEXEC) = 3
contributing openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/contributing.doctree", O_RDONLY|O_CLOEXEC) = 3
changelog openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/changelog.doctree", O_RDONLY|O_CLOEXEC) = 3
license openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/doctrees/license.doctree", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/home/tkloczko/rpmbuild/BUILD/argon2-cffi-20.1.0/build/sphinx/man/argon2-cffi.3", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3

This output shows that build_sphinx is not trying to access to build/lib-<os>.<arch>

To Reproduce

• Download https://github.com/hynek/argon2-cffi//archive/20.1.0/argon2-cffi-20.1.0.tar.gz
• uplack it
• execute

cd argon2-cffi
python3 setup.py build
python3 setup.py build_sphinx -b man --build-dir build/sphinx

Expected behavior
Woould be good if build_sphinx will expect that

• some resources will be available in build/lib or build/lib-<os>.<arch> (if none are avaliable print error messabge about execute build first
• add build/lib and/or build/lib-<os>.<arch> to sys.path before shinx-build execution.

Your project
https://github.com/hynek/argon2-cffi//archive/20.1.0/argon2-cffi-20.1.0.tar.gz

Screenshots
In case of argon2-cffi prints warnings about missing functions which are provided by DSO.

+ /usr/bin/python3 setup.py build_sphinx -b man --build-dir build/sphinx
running build_sphinx
Running Sphinx v4.0.2
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [man]: all manpages
updating environment: [new config] 11 added, 0 changed, 0 removed
reading sources... [100%] parameters
WARNING: autodoc: failed to import class 'PasswordHasher' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import exception 'exceptions.VerificationError' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import exception 'exceptions.VerifyMismatchError' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import exception 'exceptions.HashingError' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import exception 'exceptions.InvalidHash' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'extract_parameters' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import class 'Parameters' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import module 'low_level' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import class 'Type' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import data 'ARGON2_VERSION' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'hash_secret' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'verify_secret' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'hash_secret_raw' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'core' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'error_to_str' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'hash_password' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'hash_password_raw' from module 'argon2'; the following exception was raised:
No module named 'argon2'
WARNING: autodoc: failed to import function 'verify_password' from module 'argon2'; the following exception was raised:
No module named 'argon2'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
writing... argon2-cffi.3 { argon2 installation api parameters cli faq backward-compatibility contributing changelog license } done
build succeeded, 18 warnings.

Environment info

• OS: Linux x86_64
• Python version: 3.8.9
• Sphinx version: 4.0.2
• Sphinx extensions: not relevant in this case
• Extra tools: N/A

Additional context
N/A

• [e.g. URL or Ticket]
  20.1.0: sphinx warnings hynek/argon2-cffi#93

[tk0miya]

I'm not good at the rule of search paths. So I don't know why Sphinx should refer libraries on build/lib or build/lib-<os>.<arch> automatically. Could you let me know the pointer of the rule (like PEP).
Thanks,

[kloczek]

I'no sure is it anywhere in any PEP mentioned that (probably not).

What I'm trying to propose goes probably beyond what up to now was proposed.
However it is only logical rule which could be applied when build_sphinx would need access to those DSOa like it is in hynek/argon2-cffi#93.
If build_sphinx wold requre build/lib-<os>.<arch> logical is that it should use build/lib as well.

If you are thinking that any currently implemented functionalities of build_sphinx could fbe used to fix build documentation of modules which produces DSOs pleas let me know.

[tk0miya]

I'd like to know who (or what library) determines the rule for the name of build directories.

If you are thinking that any currently implemented functionalities of build_sphinx could fbe used to fix build documentation of modules which produces DSOs pleas let me know.

How about using python setup.py develop or python setup.py install before building the document? Alternatively, you can add the build directory into your conf.py

sys.path.insert(0, os.path.abspath('build/lib-<os>.<arch>'))

[kloczek]

I'd like to know who (or what library) determines the rule for the name of build directories.

Thant is another question. IMO use sometimes `build/lib-.' or 'build/lib' is a bit strange :)

If you are thinking that any currently implemented functionalities of build_sphinx could fbe used to fix build documentation of modules which produces DSOs pleas let me know.

How about using python setup.py develop or python setup.py install before building the document? Alternatively, you can add the build directory into your conf.py

That would impact any existing packages build procedures like building rpm or publishing Solaris IPS packages.
Typical order of doing some stages are:

• build
• install
• execute test suite
• final package assembly/publish to the repo

From that point of view it would be necessary to build documentation in install. Which would look like dirty hack.
IMO the best would be to have integrated build_sphinx as part of the build and have build_sphinx to execute separatelly. However it may be to big change of current seruptools semantica. From my point of fiew (someone who is packaginf suff) enough would be if build_sphinx would only be trying blindly use resources from build/lib-<os>.<arch> or build/lib.

sys.path.insert(0, os.path.abspath('build/lib-<os>.<arch>'))

I have no idea where exactly it is but setuptools know is it used build/lib-<os>.<arch> or build/lib and exactly that path IMO should be added upfront to the sys.path of the build_sphinx command.

[shimizukawa]

I think it is relatively easy to create a setup.py command and bundle multiple subcommands to run it.
So I guess you can prepare this command in your project, do the build before the sphinx_build command, and adjust the sys.path to achieve this.

Please refer to the following source: https://github.com/pypa/distutils/blob/main/distutils/command/build.py

[kloczek]

I think it is relatively easy to create a setup.py command and bundle multiple subcommands to run it.
So I guess you can prepare this command in your project, do the build before the sphinx_build command, and adjust the sys.path to achieve this.

Issue is that that fixing that centrally in setuptools sphinx extension would allow to not spread such changes to every possible module which produces DSOs.
IMO adjusting in setuptools sphinx extension centrally could solve as well the case when for example module code is relocated to for example src/ directory like it is done now in case of many modules.
In other words altering sys.path could solve as well #9517.

In other words it is now more than one reason alter sys.path by setuptools sphinx extension ad assume that build_sphinx will be allways executed after build (build_sphinx could warn that there is no build/lib* or even instead print such warning automatically execute build).

[tk0miya]

I overlooked @shimizukawa's comment. Thank you for your wisdom.

Please refer to the following source: https://github.com/pypa/distutils/blob/main/distutils/command/build.py

This is what I want. It seems distutils.build::build.build_lib is the directory that contains the final form of the distribution (library). So it would be nice if we can obtain the value from the command, and insert it to the sys.path.

I think it's reasonable change.