Add notebooks to docs download #652

AoifeHughes · 2025-09-29T20:32:10Z

This was to try and fix #642 as I was afraid of a caching issue, please see that PR for previous discussions

Co-authored-by: Penelope Yong <[email protected]>

…into add_notebook_dl

…_notebook_dl

github-actions · 2025-09-30T12:39:29Z

Preview the changes: https://turinglang.org/docs/pr-previews/652
Please avoid using the search feature and navigation bar in PR previews!

Copilot

Pull Request Overview

This PR adds notebook generation and download functionality to the documentation site by creating Jupyter notebooks from Quarto .qmd files and providing download links in the rendered HTML pages.

Introduces a Python script to convert .qmd files to .ipynb format with proper cell structure
Adds shell scripts to generate notebooks and inject download links into HTML pages
Integrates notebook generation into both publish and preview GitHub workflows

Reviewed Changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
assets/scripts/qmd_to_ipynb.py	Python converter that parses .qmd files and creates Jupyter notebooks with appropriate cell types
assets/scripts/generate_notebooks.sh	Shell script that finds .qmd files and converts them to notebooks in the _site directory
assets/scripts/add_notebook_links.sh	Shell script that adds download links to HTML pages using Perl regex replacement
.github/workflows/publish.yml	Adds notebook generation and link injection steps to the publish workflow
.github/workflows/preview.yml	Adds notebook generation and link injection steps to the preview workflow

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-01T08:27:09Z

assets/scripts/qmd_to_ipynb.py

+                    "display_name": "Julia 1.11",
+                    "language": "julia",
+                    "name": self.kernel_name


The hardcoded display name 'Julia 1.11' doesn't match the dynamic kernel name when using Python. Consider making the display name dynamic based on the kernel.

Copilot · 2025-10-01T08:27:09Z

assets/scripts/qmd_to_ipynb.py

+                "language_info": {
+                    "file_extension": ".jl",
+                    "mimetype": "application/julia",
+                    "name": "julia",
+                    "version": "1.11.0"
+                }


The language_info is hardcoded for Julia even when the kernel is Python. This should be conditional based on the detected engine/kernel type.

assets/scripts/add_notebook_links.sh

Copilot · 2025-10-01T08:27:10Z

assets/scripts/generate_notebooks.sh

+echo "Generating Jupyter notebooks from .qmd files..."
+
+# Find all .qmd files in tutorials, usage, and developers directories
+find tutorials usage developers -name "index.qmd" | while read qmd_file; do


Using 'while read' with 'find' can fail with filenames containing spaces or special characters. Use 'find ... -print0 | while IFS= read -r -d "" qmd_file; do' for safer file handling.

Suggested change

find tutorials usage developers -name "index.qmd" | while read qmd_file; do

find tutorials usage developers -name "index.qmd" -print0 | while IFS= read -r -d '' qmd_file; do

AoifeHughes · 2025-10-01T08:32:52Z

This PR adds a download button to get jupyter notebooks from the docs.

Some notes:

quarto doesn't like either using convert or the _quarto.yml config options to make notebooks as well as html
To avoid running each file 2x in the deployment I created a few scripts, the main of which is https://github.com/TuringLang/docs/pull/652/files#diff-fa790f7c53fe8322e42e5baf383fa2350e43f4f279c3ee25e4aeac1d4680f3a4 which handles the conversion and builds the notebooks
The CI has been updated to perform this
I would have much rather used quarto's built in solutions, but they're just not fit for purpose in this instance (please see the dozens of commits in which I tried every solution I could find before raising them as possible solutions @mhauru can attest to the numerous attempts I made in the last week).

@shravanngoswamii if you'd have some time to check these out, I'd appreciate it.

AoifeHughes · 2025-10-01T08:33:54Z

I should also mention, I had a few issues with the CI versions of bash/posix vs deploying on my mac, so some of the scripting is done to appease github actions moreso than how users may run locally.

AoifeHughes · 2025-10-01T08:36:18Z

I think this partially satisfies some of the points from #641 ~ it might be worth adding some note about these being available in the docs / have a quick way to export into online systems, as discussed in that issue.

shravanngoswamii · 2025-10-02T05:35:00Z

assets/scripts/qmd_to_ipynb.py

+    def __init__(self, qmd_path: str):
+        self.qmd_path = Path(qmd_path)
+        self.cells: List[Dict[str, Any]] = []
+        self.kernel_name = "julia-1.11"  # Default kernel


Generated notebooks don't work in Google Colab, or am I doing something wrong? I think the defined kernel might be the issue since Colab only supports the lts version of Julia by default. Also, we'll need to change this hardcoded version every time we update the docs to a new version. Maybe we should add a note in the README or change it to use the version from an env file or some variable that we can use everywhere.

I think latest push should fix this. Lemme verify and confirm once it builds

shravanngoswamii · 2025-10-02T05:36:00Z

Notebooks work fine with IJulia

penelopeysm

Overall thoughts:

Looks very nice! I like the little icon and it fits in thematically with the rest of the website very nicely.
When you download the notebook, the first line is using Pkg; Pkg.instantiate(). I attempted to run this in a VSCode and this loads the global Julia environment. So if you subsequently run using Turing that will also add Turing to the global environment, which is probably not ideal for users.

Is it possible to modify the notebook generation such that the very top of the notebook contains a new code block saying using Pkg; Pkg.activate(; temp=true), before the call to instantiate()?

AoifeHughes · 2025-10-06T11:35:19Z

Overall thoughts:

Looks very nice! I like the little icon and it fits in thematically with the rest of the website very nicely.

When you download the notebook, the first line is using Pkg; Pkg.instantiate(). I attempted to run this in a VSCode and this loads the global Julia environment. So if you subsequently run using Turing that will also add Turing to the global environment, which is probably not ideal for users.

Is it possible to modify the notebook generation such that the very top of the notebook contains a new code block saying using Pkg; Pkg.activate(; temp=true), before the call to instantiate()?

Yes! I'll need to modify the new file added assets/scripts/qmd_to_ipynb.py but that should be straight-forward. I'm also still pondering some of the other things about making it run in collab, so will make changes for that at the same time.

penelopeysm · 2025-10-06T11:38:23Z

assets/scripts/generate_notebooks.sh

My main other question is: if the qmd-to-ipynb script is already in Python, why not write the Bash scripts in Python as well?

I don't mind the use of Python (it's way better for quick scripts than Julia is), but I think reducing the number of moving parts that need to work together will make for easier maintenance in the future. Especially because all of these scripts serve a common purpose and you're unlikely to want to run one without running the others.

Eh, personally I find any bash stuff gets harder to read the more complex it gets, I can do that though, not a problem. Let me do the current changes and verify it works then I can explore this fully.

oh yes bash is definitely hard to read but I meant moving the bash to python - not the other way round!

aha, gotchu. tbh, I originally wrote the bash as a quick fix with the CI, it was only because the default convert didn't work that I did the python too.

penelopeysm · 2025-10-06T11:39:07Z

assets/scripts/add_notebook_links.sh

+        if ! grep -q 'Download notebook' "$html_file"; then
+            # Insert the notebook link AFTER the "Report an issue" link
+            # This ensures it goes in the right place in the sidebar toc-actions
+            # The download="index.ipynb" attribute forces browser to download instead of navigate
+            perl -i -pe 's/(<a href="[^"]*issues\/new"[^>]*><i class="bi[^"]*"><\/i>Report an issue<\/a><\/li>)/$1<li><a href="index.ipynb" class="toc-action" download="index.ipynb"><i class="bi bi-journal-code"><\/i>Download notebook<\/a><\/li>/g' "$html_file"


Could you make the string 'Download notebook' a variable?

penelopeysm · 2025-10-06T11:40:00Z

Overall very excited about this, very happy that I clicked the download link and it just worked (tm) :)

yebai · 2025-10-06T23:07:26Z

Looks good -- it is very helpful to add an "Open in Colab" button too. URLs like

https://colab.research.google.com/github/TuringLang/docs/blob/gh-pages/pr-previews/652/usage/automatic-differentiation/index.ipynb

will automatically open a notebook from the given Github repo.

Aoife and others added 29 commits September 18, 2025 13:31

Add download button for Jupyter notebooks in tutorial pages

54f23e3

Refactor Quarto rendering and remove download notebook script

ff55d5a

Merge branch 'main' into add_notebook_dl

8dd8fbf

Update to Turing 0.40

091e1a1

Update version in _quarto.yml

fc4df94

Fix DPPL 0.37 run_ad change

df1f9cd

Fix VI tutorial

14973ad

Fix model-manual's use of contexts

06bb33d

Fix references to __context__

e7519c9

Fix use of addlogprob for log prior

4530247

Fix typo

57c3162

Regenerate manifest

2065a14

Remove version pin of DelayDiffEq and update Manifest

2babb2d

Fix call to evaluate

b41a4cb

Add note about contexts tutorial being out of date

425539d

Apply suggestions from code review

dde0ed3

Co-authored-by: Penelope Yong <[email protected]>

Add ipynb format support

a32ec1b

maybe added links to dl?

1e7f351

Merge branch 'add_notebook_dl' of https://github.com/TuringLang/docs …

e455881

…into add_notebook_dl

reset mainifest

d1bacac

dont execute ipynb

a78fac4

bump local changes

98bf999

Merge branch 'main' into add_notebook_dl

60a07a8

add post-render script for converting .quarto_ipynb files to .ipynb

af7ceb4

Merge branch 'add_notebook_dl' of https://github.com/TuringLang/docs …

4459016

…into add_notebook_dl

Refactor notebook generation process and clean up tutorial metadata

7f00453

Merge branch 'main' into add_notebook_dl

708d22b

please work :(

64c62a4

Merge branch 'add_notebook_dl' of github.com:TuringLang/docs into add…

1c6fad9

…_notebook_dl

AoifeHughes self-assigned this Sep 29, 2025

Aoife added 4 commits September 30, 2025 09:51

freeze?

26cdf37

last try

01e6954

forgot to add links...

f55de5c

grep...

848dc87

Aoife and others added 5 commits September 30, 2025 13:43

fixed edit loc

fdbee40

placement

2a56d63

added a script to do notebooks justice

9ed76f1

posix

27b9d8b

Merge branch 'main' into add_notebook_dl

e06a333

AoifeHughes marked this pull request as ready for review October 1, 2025 08:25

AoifeHughes requested review from shravanngoswamii and Copilot October 1, 2025 08:26

Copilot AI reviewed Oct 1, 2025

View reviewed changes

AoifeHughes requested a review from penelopeysm October 1, 2025 08:34

shravanngoswamii reviewed Oct 2, 2025

View reviewed changes

penelopeysm reviewed Oct 6, 2025

View reviewed changes

AoifeHughes and others added 4 commits October 6, 2025 12:45

patched to insert pkg local usage over global.

2a396e4

removed specifying versions

3a1691e

Merge branch 'main' into add_notebook_dl

e7979d9

made variable thingy in script.

d07ab29

Add environment variable for Colab path prefix in notebook link scripts

c00b84d

	find tutorials usage developers -name "index.qmd" \| while read qmd_file; do
	find tutorials usage developers -name "index.qmd" -print0 \| while IFS= read -r -d '' qmd_file; do

Add notebooks to docs download #652

Are you sure you want to change the base?

Add notebooks to docs download #652

Uh oh!

Conversation

AoifeHughes commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

AoifeHughes commented Oct 1, 2025

Uh oh!

shravanngoswamii Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

AoifeHughes Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

shravanngoswamii commented Oct 2, 2025

Uh oh!

penelopeysm left a comment

Choose a reason for hiding this comment

Uh oh!

AoifeHughes commented Oct 6, 2025

Uh oh!

penelopeysm Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

AoifeHughes Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

penelopeysm Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

AoifeHughes Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

penelopeysm Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

penelopeysm commented Oct 6, 2025

Uh oh!

yebai commented Oct 6, 2025

Uh oh!

Uh oh!

AoifeHughes commented Sep 29, 2025 •

edited

Loading