Sphinx 8.2.0 uses significantly more memory than 8.1.x

[djhoese]

Describe the bug

Builds are now using much more memory than with 8.1.x and my readthedocs builds are failing. From what I can tell I'm going from about a maximum RSS of ~2.6GB to ~8.6GB.

How to Reproduce

1. On any linux-like system with GNU time available (ex. /usr/bin/time on ubuntu, different than builtin bash time command) with 8.1.3 installed with theoretically any source files, run /usr/bin/time -v sphinx-build -b html source build/html. The output should include a "Maximum resident set size".
2. Clean the build dir and run again with sphinx 8.2.0 and comnpare the Maximum RSS.

A full comparison with my project:

git clone https://github.com/pytroll/satpy`
cd satpy/doc

conda env create -n satpy_docs -f rtd_environment.yaml
conda activate satpy_docs

touch source/area_def_list.rst
make clean
/usr/bin/time -v make html

8.1.3 results

Build finished. The HTML pages are in build/html.
	Command being timed: "make html"
	User time (seconds): 75.01
	System time (seconds): 1.38
	Percent of CPU this job got: 100%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:15.76
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 2579220
	Average resident set size (kbytes): 0

8.2.0 results

Build finished. The HTML pages are in build/html.
	Command being timed: "make html"
	User time (seconds): 84.49
	System time (seconds): 4.46
	Percent of CPU this job got: 100%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:28.61
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8653184
	Average resident set size (kbytes): 0

Environment Information

Platform:              linux; (Linux-6.9.3-76060903-generic-x86_64-with-glibc2.35)
Python version:        3.12.8 | packaged by conda-forge | (main, Dec  5 2024, 14:24:40) [GCC 13.3.0])
Python implementation: CPython
Sphinx version:        8.2.0
Docutils version:      0.21.2
Jinja2 version:        3.1.5
Pygments version:      2.18.0

Sphinx extensions

extensions = ["sphinx.ext.autodoc", "sphinx.ext.intersphinx", "sphinx.ext.todo", "sphinx.ext.coverage",
              "sphinx.ext.doctest", "sphinx.ext.napoleon", "sphinx.ext.autosummary", "sphinx.ext.autosectionlabel",
              "doi_role", "sphinx.ext.viewcode", "sphinxcontrib.apidoc",
              "sphinx.ext.mathjax"]

I'll work on reducing this and update.

Additional context

No response

[djhoese]

I removed the sphinxcontrib.apidoc and generic the files manually with sphinx-apidoc in the same manner and the memory difference remains. I have a feeling the problem is my use of --separate in the sphinx-apidoc options which ends up creating almost 400 files. I'll try a few things and see if it changes the memory usage.

Edit: Nope, down to ~26 api files then and it went to 8.85GB of memory.

[AA-Turner]

Are you able to git bisect Sphinx? That would be very helpful.

[djhoese]

Yeah. It's running now and I'll check it in the morning. My check is naively just on the max RSS reported by /usr/bin/time being over 3GB or not and does not handle the build failing in any way so there could be mistakes. From the output I've seen so far there are some in-between numbers (~7.8GB instead of 8.6GB) so it may not be a single PR/commit that changed things.

[djhoese]

@AA-Turner It's this one: fec4d7c

I have to get through some emails and messages but can try to understand the changes in that commit later unless you have an idea what's going wrong.

[AA-Turner]

A very suprising commit... A first thought though is that it seems to load and execute the module, but not save it into sys.modules, so perhaps it is a module getting repeatedly executed but its objects never freed? I don't have any better guesses without looking properly (and I don't have a linux computer to hand).

A

[djhoese]

I'm not sure I mentioned it above, but even without all my API docs I saw 8.1.3 peak at about 288MB and 8.2.0 peak at 299MB. So not as huge as with the API docs, but definitely surprising for just updating sphinx on a relatively simple project. I don't know if this ~10MB increase was due to this same commit though.

[djhoese]

What OS are you on @AA-Turner? I would expect this memory change on any OS as long as you can monitor it.

[djhoese]

I've confirmed, on master, if I disable sphinx.ext.viewcode in my conf.py that memory peak goes from ~8.9GB to ~2.6GB.

[djhoese]

Ok I think I'm on to something. In my project we document our test modules. I think in the old version of sphinx it was not finding the test module source code and so was never importing it. Now it is working with the fix in the mentioned commit/PR and this is running the test code. If I hack the sphinx code to say if "satpy.tests" is in the module name skip getting the fullname then memory usage goes back down to normal.

So, there must be some import side effect of our tests that is blowing up memory usage. This amount of memory is not unexpected since one of the package's main features is reading satellite data formats. Granted it shouldn't be doing much on import time but still. I'd like to confirm this hypothesis before this issue gets closed.

@AA-Turner could you give me more information on what the viewcode_follow_imported_members option is? Based on the documentation, if nothing is listening to the "viewcode-follow-imported" event then this config option doesn't do anything, right? But I also see that _get_full_modname is only called when this option is True.

[djhoese]

Ok, our tests definitely aren't helping. I commented out some poorly designed test modules creating large numpy arrays at import time and it dropped memory usage a lot. This still doesn't really explain the ~10MB increase I saw when excluding my API docs altogether though.

[djhoese]

Small update: I compared the modules that were being passed to the problematic _get_full_modname and there was no difference between v8.1.3 and v8.2.0 except the v8.2.0 fails to document modules that have missing dependencies. The old sphinx used to do that from what I can tell.

So although my modules are wasting memory on import time, this new _get_full_modname does seem to be importing and then not releasing what it has imported. Even so, the memory usage I see from sphinx generation is almost 3x higher than what I see from memory_profiler of just importing those modules alone.