Skip to content

Conversation

@bsipocz
Copy link
Member

@bsipocz bsipocz commented Jul 25, 2025

Opening this PR so we only iterate on the TOC structure once.

TODOs blocking merge:

  • restructure TOC based on the google docs we have
  • enable execution in JB2
    • conditional skips of notebooks (depending on OS, CI, etc.)
    • install python dependencies
      • create a universal make file to make sure we use the same in all CI (and then easily reproduce locally) -- added to tox for now
    • review rendered pages, collect and report missing features upstream
      • scrolling for outputs
      • add launch button
      • Move author and date metadata into notebook header -- needs some experimenting as not sure how it will look like in jupyterlab

jira: IRSA-7092

resolves #98 closes #123 resolves #124 resolves #140

Checklist from review:

  • Spitzer doesn't need "Spitzer Enhanced Imaging Products" with just one notebook, as in, we can remove the second layer under Spitzer and just have a single notebook for Spitzer.
  • In the upper right are a few symbols that don't work, can we either make them work, or remove them until they do work?
    • jupyterlab -- **my understanding that the unclickable logo just shows if it's a notebook, but I'll be enabling the binder launch in the next follow-up, that may resolve this one issue, too, and if not I'll open an issue upstream
    • edit link -- This is a known bug (BUG: wrong link in github edit link jupyter-book/mystmd#2234); TL;DR: we do tunnel through a proxy for the preview and use forks for PRs and that is just not supported. In the deployed version I expect the button to work (it does for fornax docs); though it doesn't work for The Turing Way. So we'll see once we merge this.
  • Interesting find; the reason was to have text before the top level heading. I moved it around and now it's just one title WISE: NEOWISE: "Make light curves from ..." and "Strategies to Efficiently..." have two titles in the rendered versions.
  • I don't think so. I think we need a better landing page, but having another TOC doesn't make much sense to me Does it make sense to list the notebooks on the main index page, like we do for Fornax?
  • Comments in ignore_*_testing files about why each notebook is excluded would be really helpful.
  • IMO, the heading titles in the notebook you linked below, as well as in a handful others should be reworked, but now one fewer heading layer is included The notebook TOCs on the right: The Fornax ones only show h1 and h2 until you scroll to a section that has lower-level headings and then that section in the TOC expands to show them. On these IRSA ones, all heading levels are shown. Is that an easy setting to change? Personally, I like the Fornax behavior better.
  • Reported issue upstream: BUG: scrolling a hover pop-up scrolls the entire page along with the pop-up jupyter-book/mystmd#2231 (comment) -- This notebook’s TOC is so long that I can’t even scroll to the bottom of it. It cuts off at some point and I can’t see the last few sections. Seems like a bug in the rendering, but changing to Fornax style (above) should also fix it. https://circle.scientific-python.dev/output/job/ad753bcf-9ff8-48ca-a683-6c9888e57bfb/artifacts/0/_build/html/openuniverse2024preview-firefly
  • expected issue, we should port Jaladh's fix to the new theme, too: Add scroll_outputs configuration executablebooks/MyST-NB#683 Some of the cell output is really long and not very informative. Is there a way to limit output cell height and make them scrollable? I think we’ve talked about this before but I don’t remember the answer.

@bsipocz bsipocz added the html rendering / skip testing Rendering related issues/PRs. Skips tests in PRs. label Jul 25, 2025
@bsipocz bsipocz force-pushed the ENH_switch_to_JB2 branch from 1aba9ba to 85238de Compare July 25, 2025 23:27
@bsipocz
Copy link
Member Author

bsipocz commented Jul 26, 2025

Currently there is one build error that produces an incorrect linking, reported it upstream, but we can also just work around it in the meantime by not linking to the figure anchor: jupyter-book/mystmd#2209

@bsipocz bsipocz force-pushed the ENH_switch_to_JB2 branch from c50edd5 to a898c26 Compare July 26, 2025 05:20
@bsipocz bsipocz force-pushed the ENH_switch_to_JB2 branch from 89a4ed7 to 2e4a1f4 Compare August 9, 2025 00:58
@bsipocz bsipocz force-pushed the ENH_switch_to_JB2 branch 8 times, most recently from a75e5b3 to 2ebddd0 Compare August 29, 2025 22:53
@troyraen
Copy link
Contributor

@bsipocz if the JB2 stuff is still going to take awhile, can the TOC reorg go through separately? It would be really nice to get that published.

@bsipocz
Copy link
Member Author

bsipocz commented Sep 10, 2025

The bad news is that while toc files can work with JB1 their content/syntax is a bit different. But I'll get back to this and try to get the build going on GHA to narrow down if it's a circle issue or not.

@bsipocz bsipocz added the GHA buildhtml Enable extra buildhtml job on GHA label Sep 12, 2025
@bsipocz
Copy link
Member Author

bsipocz commented Sep 12, 2025

OK, so the cryptic build/kernel issue doesn't seem to effect GHA. It's really puzzling of what's going wrong on circleCI as why it is so undeterministic

@bsipocz bsipocz force-pushed the ENH_switch_to_JB2 branch 2 times, most recently from 870e20d to 33cab0f Compare September 26, 2025 03:54
@bsipocz
Copy link
Member Author

bsipocz commented Sep 26, 2025

I have hunted down the issue for the CI troubles: Our build is hovering at the total memory limit. Currently I have no control over the parallelism (turning it off is expected to help) nor have solutions to potentially run the new stack more efficiently.

I do propose the current workaround:

  • short term: ignore a bunch of notebooks for rendering in circleCI. This is an experimental list, I can imagine that maybe 1 or 2 can be removed, but I couldn't yet find such a combination. I don't expect this to be a deal breaker as for the actual hosting we use GHA and there is only the parallel convolution that needed to be excluded from execution there. -- I feel that is an acceptable compromise as that notebook doesn't in fact demonstrate any IRSA functionality.

  • longer term:

    • properly enable caching -- that may very well help between sessions
    • only execute for rendering preview the notebooks that are changed in a given PR, I suspect more in the currently excluded list of 10 can be run individually, but I haven't tested. (2 of the 10 are known to not work with any of our CI providers
  • very longer term: build the problematic notebooks at a 3rd party site (e.g. Fornax, or even locally on a laptop) and transfer the artifacts into the cache/to the site.

So with all this being said, I think this PR is very close to be being ready, I'll fix up some more of the comments that you have given above and leave the remaining checkbox items as well as the longer-term items for follow-ups.

@bsipocz bsipocz marked this pull request as ready for review September 26, 2025 06:15
Copy link
Contributor

@troyraen troyraen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bsipocz this will be great! Please open an issue with three lists of notebooks:

  1. Excluded from rendering
  2. Excluded from testing in PRs
  3. Excluded from cron job testing

@troyraen troyraen merged commit 8901b1e into Caltech-IPAC:main Sep 26, 2025
5 of 6 checks passed
@bsipocz bsipocz deleted the ENH_switch_to_JB2 branch September 26, 2025 18:01
@bsipocz
Copy link
Member Author

bsipocz commented Sep 26, 2025

See #145

github-actions bot pushed a commit that referenced this pull request Sep 26, 2025
ENH: switching to JB2 infrastructure 8901b1e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

GHA buildhtml Enable extra buildhtml job on GHA html rendering / skip testing Rendering related issues/PRs. Skips tests in PRs.

Projects

None yet

2 participants