Consider using content hashes for change detection instead of modification time

[dbitouze]

Continuous integration with Sphinx-doc, as described here, works well.

Unfortunately, on gitlab.com, it is Docker based and clones each repository fresh when it starts running continuous integration. So even if a single source file is modified, all the corresponding HTML pages of all the .rst source files are rebuilt (although the cache claims to be restored and, indeed, so is the doctree directory). This isn't a problem if there are only a few source files, but it becomes unusable if there are a lot (more than 1,200 in my real-life use case: the rebuild takes more than 15 minutes and a lot of resources are consumed unnecessarily).

This problem may be due to the fact that Git, unlike other version control systems, does not preserve the original timestamp of committed files. So relying on git-restore-mtime should be a solution. But this is not the case, as you can see with the following sandbox repository:

https://gitlab.com/denisbitouze/minimal-sphinx-minimal/

where the commit changes (only) the source test.rst file but triggers also the rebuild of the index.html file corresponding to the index.rst source file that hasn't been changed.

I've had a look at the code but can't work out how sphinx change detection works. Would it be possible to document this? It would be very useful, especially nowadays when CI/CD is becoming more and more popular and useful.

[picnixz]

The reason why the HTML was rebuilt is because the HTML file did not exist anymore. The _build/html is the output HTML directory but the latter is only put as an artifact and not cached (at least that's how I see it, I haven't checked that this is really the case).

Concerning your documentation request, to my understanding, this is what happens:

• The environment decides which .rst files are using BuildEnvironment.get_outdated_files. More precisely, it determines which .doctree file to write. It is based on whether the .rst original file is modified or not (using mtime values) or if one of its dependencies was modified.
• In an incremental build approach (Builder.build_update), the following is done:
  • Determine which output (e.g., HTML) files are outdated using Builder.get_outdated_docs.
  • Determine which source (RSt) files are outdated using BuildEnvironment.get_outdated_files.
  • Write the output (HTML) files corresponding to the outdated source or output files.

Let's illustrate this by an example. Assume that we have the following RST files:

.. index.rst 

.. toctree::
   :maxdepth: 1

   bar.rst

.. foo.rst

The Foo
-------
foo

.. bar.rst

The Bar
-------
and the foo:

.. include:: foo.rst

• Run Sphinx first. The three documents will generate _build/doctrees containing the serialized docutils tree. The first time you run Sphinx, all three documents are marked as "added" (they are fresh).
• If you change bar.rst, the latter is marked as changed. As such _build/doctrees/bar.doctree will be updated. Since bar.rst was changed, _build/html/bar.html is marked as updated by Builder.get_outdated_docs. Which HTML file to rebuild is decided by whether on the timestamp of the original source (i.e., bar.rst) and compared with the timestamp of the HTML file.
• If you change foo.rst, all the files depending on foo.rst are to be rebuilt as well. In particular, bar.rst must be rebuilt. Here, note that Builder.get_outdated_docs will not include bar.rst because bar.rst was not modified as a file. However, BuildEnvironment.get_outdated_files will include bar.rst as a document that was "virtually" changed.

[dbitouze]

The reason why the HTML was rebuilt is because the HTML file did not exist anymore. The _build/html is the output HTML directory but the latter is only put as an artifact and not cached (at least that's how I see it, I haven't checked that this is really the case).

Do you see how this could be worked around? I tried to replace _build/doctrees by _build/html in:

https://gitlab.com/denisbitouze/minimal-sphinx-minimal/-/blob/main/.gitlab-ci.yml#L4-L6

but with no success. Do you advise for the _build/html content to be :

1. before sphinx build, downloaded to,
2. and after the sphinx build, upload from,

somewhere on the Cloud?

Concerning your documentation request, to my understanding, this is what happens:

[...]

Many thanks for this detailed explanation! Unfortunately I don't see how I could use in the context of the CI/CD.

[picnixz]

Actually, I think you can just add _build/html as a cached path as well:

  cache:
    paths:
    - _build/doctrees
    - _build/html

Unfortunately I don't see how I could use in the context of the CI/CD.

One way to do it is to have a custom extension and an event handler for the env-get-outdated event if you want to hack into things. But for your specific CI/CD, I think you simply need to cache the _build directory (you could actually cache the entire directory).

[dbitouze]

Actually, I think you can just add _build/html as a cached path as well:

  cache:
    paths:
    - _build/doctrees
    - _build/html

Unfortunately, it doesn't work. After this addition and a single change of (only) test.rst, the index.html file is rebuilt as well.

Unfortunately I don't see how I could use in the context of the CI/CD.

One way to do it is to have a custom extension and an event handler for the env-get-outdated event if you want to hack into things.

Well, I'm afraid this is far beyond my scope :$

But for your specific CI/CD, I think you simply need to cache the _build directory (you could actually cache the entire directory).

Unfortunately, it doesn't work either.

[picnixz]

Ok, I've looked a bit more. The reason why the index is rebuilt is because there is a toctree directive. Actually, files (maybe not all) containing such directive will be automatically rebuilt.

sphinx/sphinx/builders/__init__.py

Lines 550 to 555 in 8a990db

	# add all toctree-containing files that may have changed
	for docname in list(docnames):
	for tocdocname in self.env.files_to_rebuild.get(docname, set()):
	if tocdocname in self.env.found_docs:
	docnames.add(tocdocname)
	docnames.add(self.config.root_doc)

When you are "included" as a file in a toctree, you are marked as a dependency of that file (I agree that this is not clearly stated). So, you won't be able to escape from the fact the index is rebuilt everytime.

[dbitouze]

The reason why the index is rebuilt is because there is a toctree directive. Actually, files (maybe not all) containing such directive will be automatically rebuilt.

The problem concerns other files as well: I (only) have added another test-bis.rst source file but the test.html file is rebuilt as well.

[picnixz]

I cannot reproduce this locally. I'd advise you to check it locally by the way. By looking at the traceback, it appears:

[build target] targetname '/builds/denisbitouze/minimal-sphinx-minimal/_build/html/test.html'(2023-08-07 09:45:27+00:00), template(2023-08-07 10:27:49.543082+00:00), docname 'test'(2023-08-07 09:28:33+00:00)

Here (up to a timestamp shift):

• the target is test.html and has been modified at 09:45.
• the template timestamp indicates the latest HTML file time of modification due to a change in the HTML template. Here it appears that it's at 10:27.
• the source is test and was modified at 09:28.

The "real" source time is actually set to the template timestamp because it's the largest. Since 10:27 was after 09:45, it causes test to be rebuilt. And I think I know the cause of it:

• When you install Sphinx, you also install jinja templates. We have a bunch of bundled templates as well. But because you install them, you end up having different timestamps for those library files (I think). This is the reason why the template seems to be always up-to-date. So you need to actually cache the pip dependencies as well (or at least some of them). I think if you look at the timestamps of the files ending with .html in the /path/to/site-packages/alabaster and /path/to/site-packages/sphinx/sphinx/themes/basic folders, you'll find those template timestamps.

[dbitouze]

I cannot reproduce this locally. I'd advise you to check it locally by the way.

Indeed, locally, everything works as expected. That's the point: I worked a lot on the migration of our FAQ from DokuWiki to Sphinx-doc precisely because the latter is very nice and works pretty well locally: if a single (.rst or .md) source file (among the more than 1,200 ones) is modified, only the corresponding .html file is rebuilt. But, AFAIU, because GitLab relies on Docker (which relies on a git fetch), that's not the case when CI/CD is involved.

I'll have a look at the other part of your answer later. Many thanks!

[dbitouze]

Here (up to a timestamp shift):

• the target is test.html and has been modified at 09:45.

• the template timestamp indicates the latest HTML file time of modification due to a change in the HTML template. Here it appears that it's at 10:27.

• the source is test and was modified at 09:28.

The "real" source time is actually set to the template timestamp because it's the largest. Since 10:27 was after 09:45, it causes test to be rebuilt.

Looks very interesting!

And I think I know the cause of it:

• When you install Sphinx, you also install jinja templates. We have a bunch of bundled templates as well. But because you install them, you end up having different timestamps for those library files (I think). This is the reason why the template seems to be always up-to-date. So you need to actually cache the pip dependencies as well (or at least some of them). I think if you look at the timestamps of the files ending with .html in the /path/to/site-packages/alabaster and /path/to/site-packages/sphinx/sphinx/themes/basic folders, you'll find those template timestamps.

I don't understand where I'm supposed to have a look at these files ending with .html. I found them locally:

$ ls /home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/*html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/defindex.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/domainindex.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/genindex.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/genindex-single.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/genindex-split.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/globaltoc.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/layout.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/localtoc.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/page.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/relations.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/searchbox.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/searchfield.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/search.html
/home/bitouze/.local/lib/python3.8/site-packages/sphinx/themes/basic/sourcelink.html
$ ls /home/bitouze/.local/lib/python3.8/site-packages/alabaster/*.html
/home/bitouze/.local/lib/python3.8/site-packages/alabaster/about.html
/home/bitouze/.local/lib/python3.8/site-packages/alabaster/donate.html
/home/bitouze/.local/lib/python3.8/site-packages/alabaster/layout.html
/home/bitouze/.local/lib/python3.8/site-packages/alabaster/navigation.html
/home/bitouze/.local/lib/python3.8/site-packages/alabaster/relations.html

Moreover, I must admit I don't know how to “cache the pip dependencies as well”.

And, in my real use case, I use other extensions and theme:

extensions = [
    'sphinx_comments',
    'sphinx.ext.todo',
    'sphinx.ext.mathjax',
    'sphinx.ext.extlinks',
    'sphinx_design',
    'sphinxext.opengraph',
    'sphinx.ext.intersphinx',
    'myst_parser',
]

html_theme = 'furo'

[picnixz]

In the gitlab configuration, you have pip install sphinx. Since pip dependencies are not cached, you end up with a fresh sphinx package and in particular, the files that you found locally are newer (they are always recreated). For caching pip dependencies, you need to play with PIP_CACHE_DIR as well and gitlab cache system. You should look at the official documentation for pip/gitlab for that or ask for help on Stackoverflow because this is no more an issue on our side I think.

Also, technically speaking, a CI/CD job should actually run the whole flow and not in an incremental manner by default. As such, I don't think we need to change our workflow example (or it would be a low priority task).

[dbitouze]

In the gitlab configuration, you have pip install sphinx. Since pip dependencies are not cached, you end up with a fresh sphinx package and in particular, the files that you found locally are newer (they are always recreated). For caching pip dependencies, you need to play with PIP_CACHE_DIR as well and gitlab cache system. You should look at the official documentation for pip/gitlab for that or ask for help on Stackoverflow because this is no more an issue on our side I think.

OK, I'll try this: many thanks again!

Also, technically speaking, a CI/CD job should actually run the whole flow and not in an incremental manner by default.

Why if the incremental way (as can be seen locally) does all and only what is needed? And, as I said, running the whole flow unnecessarily consumes time and resources.

As such, I don't think we need to change our workflow example (or it would be a low priority task).

Would be very, very nice for the tutorial to expose both the whole flow and the incremental manners.

[dbitouze]

A suggestion has been made to me outside of here concerning the fact that the date which seems to justify the "out of date" is not that of the source file (the .rst) but the date of the template (no doubt the default one installed with sphinx).

This involves creating a Python virtual environment in '_build', activating it, installing 'sphinx' and adding this virtual environment to the cache (like _build/html). In practical terms, this would mean replacing the lines:

  - pip3 install -U pip
  - pip3 install -U sphinx
  - apt-get update
  - apt-get install git-restore-mtime -y

with:

  - apt-get update
  - apt-get install git-restore-mtime -y
  - python3 -m venv _build/venv
  - source _build/venv/bin/activate
  - pip install sphinx

and add the _build/venv directory to the cache.

With these modifications, the trigger for rebuild is:

writing output...
[build target] did not in env: 'test-bis'
[build target] did not in env: 'test'
[build target] did not in env: 'index'
building [html]: targets for 3 source files that are out of date

We admit that we don't really understand these three messages (with “did not in env”).

According to sphinx's sources, it's in https://www.sphinx-doc.org/en/master/_modules/sphinx/builders/html.html:

for docname in self.env.found_docs:
            if docname not in self.env.all_docs:
                logger.debug('[build target] did not in env: %r', docname)

But we confess we don't know what self.env.found_docs and self.env.all_docs...

In any case, this message isn't explicit enough to give us a clue...

[picnixz]

Why if the incremental way (as can be seen locally) does all and only what is needed? And, as I said, running the whole flow unnecessarily consumes time and resources.

Ok, I shouldn't have phrased it like that. What I meant is that you are running a "fresh environment" (Docker) everytime so it is correct to assume that everything should be generated as if it was the first time. In order to make it incremental, the environment itself must be configured differently (which is what we are trying to achieve now).

We admit that we don't really understand these three messages (with “did not in env”).

Thank you for making me remember this. Actually, it should be "did not exist" and I forgot to fix the typo. Now,

• found_docs: all documents that were found (on the filesystem) for this build
• all_docs: mapping from documents that were read to their time of reading.

So if you find a document that was never read once, this means you need to generate the corresponding file accordingly.

and add the _build/venv directory to the cache.

I wouldn't cache the whole venv directory. It's better to only cache the pip packages. Also, I would cache them in another directory and add it to the exclude_pattern configuration variable. You should definitely read this section.