Expandable sidebar sections with huge sphinx sites (eg large API reference) results in a very slow build

[jorisvandenbossche]

Now that the collapsible sections in the sidebar are in master, I am noticing that this gives some challenges in the pandas docs, and especially the API reference part.

If you have a huge site with many pages, the change in #349 has the consequence that each page is now included in the toctree (we use collapse=False now), which means that 1) the build is much slower and 2) the size of the html build artefact is much bigger.

Specifically for the pandas API docs (https://pandas.pydata.org/docs/dev/reference/index.html), we have more than 1000 API pages, and since the docs are built with #349 being merged, the build time went from ~15min to ~55min and the build artifact from 180Mb to 1.1GB. The html of a single page now contains 15,000 lines for the sidebar ..

(aside: we have too many API pages in pandas, as some are identical (duplicated) pages for shared methods accross subclasses, this is something we can improve and deduplicate. But even with less API pages, you will still notice the difference)

Brainstorming some ways this issue could be addressed (in general, and for pandas specifically):

• Speed-wise: could we cache some results? Since each page shares almost the same html, but not exactly (the current/active page is different), I am not sure this is possible
• The fact that autosummary automatically adds those pages to the toctree, so they end up in the sidebar, is IMO not really necessary (eg see https://pandas.pydata.org/docs/dev/reference/io.html, all functions are already listed on the page, so that they are also included in the sidebar is a bit duplicative, and it makes the sidebar also very long). But I am not sure how avoid this using sphinx.
• We could set collapse=True specifically in the API reference section of the pandas docs. Pandas could do this by overriding the template and depending on the pagename set collapse to True or False. Or would there be another way to configure this?

[choldgraf]

Could you make the api pages orphans, and then manually link to them in the toctree?

[cheekyshibe]

Ref: #321

Users have could gain clear API information from the summary table in the main part. The automatically generated subsections in the side-navbar are a bit needless. Now any :toctree: in the .rst file can be captured. It would be a disaster if autosummary and the default :toctree: are used on the same page.

[jorisvandenbossche]

I opened #386 to make this configurable, so at least it's easy to turn off expandable navigation in the sidebar in case you don't want it.

In pandas, I solved it for now by turning it off only specifically for the API section of the docs with a small template override: pandas-dev/pandas#40761

But I think the :orphan: approach is also interesting to look into (that would specifically be a solution for #321)

[jorisvandenbossche]

Thinking a bit more about the :orphan: idea to not include generated API stub pages in the toctree: based on my current understanding of sphinx, I think that wouldn't work by itself, because it's still the .. autosummary:: directive that includes a toctree node for those items. So updating the autosummary templates to include :orphan: won't help I think.

In addition, we would need to instruct autosummary not to include such a toctree. That's probably only possible upstream in sphinx itself.

[choldgraf]

I was just playing around with this since the Dask folks just ran into this as well (👋 @jacobtomlinson)

EDIT: I just realized that I was encountering an artifact from previous builds, and my nifty suggestion wasn't so good after all 😬

[choldgraf]

I think that I have a fix for this here: #492