MAINT/DOC: Designing notebook execution workflows

[asoplata]

@dylansdaniels This is partially related to #96 , but also related to how we manage the "process" of adding new notebooks or editing them according to new changes in development which are not yet on master (see here jonescompneurolab/hnn-core#1064 (comment) ). I think we need to have a discussion of our approach for how we handle development on tutorials that use code (e.g. notebooks) in its entirety.

Currently

We have two places where we are deploying notebooks (or "notebook-like-content"):

1. The distinct Jupyter notebooks at textbook, which are only built against the latest stable version.
2. The notebook-like "scripts" in our examples at hnn-core, which are deployed and built for every version, including new development versions as of each PR.

There is also currently no way for a script/notebook to automatically "go between" these two repos; currently we must manually copy Jupyter notebooks, which is problematic for a number of reasons (e.g. the notebooks that hnn-core builds are malformed).

Proposal

Essentially, I propose the following:

1. We change hnn-core's doc system to use and execute true Jupyter notebooks, instead of scripts that are currently deployed as both 1. empty (no output and un-executed) Jupyter notebooks and 2. webpages that are built using sphinx out of script code and output. There are multiple sphinx extensions that support this, including myst-nb and nbsphinx. Here is some good documentation. Instead of the scripts getting tested via CircleCI for every PR (and release), the Jupyter notebooks would be tested. Honestly, even if we don't proceed with this proposal, this is a good idea for hnn-core regardless.
2. We remove the "execution" part of Jupyter notebook processing in the textbook repo, but we still retain the code to take a Jupyter notebook, extract it into the JSON files you've made, and use them to assemble webpages that look good on the textbook website.
3. Instead, the textbook repo will be changed to directly use the Jupyter notebooks from the hnn-core repo itself. There are at least 2 ways to do this:
   1. on deployment, textbook grabs notebook files directly from raw.githubusercontent and updates the hashes and its JSON files, or
   2. on deployment, textbook depends on and downloads hnn-core as a git submodule, then processes the newly-downloaded versions of the notebooks (on first glance I prefer this, and it has the added benefit that the only hash we need to track is that of the hnn-core submodule commit itself, rather than the hash of multiple files).
   3. There are probably other ways to do this.

Pros:

• This means that, if desired, we can actually use textbook and point to BOTH stable and development notebook versions, if we want! For example, most of our notebooks would presumably be pointing to the stable version of each notebook (which will be located on the hnn-core/gh-pages branch), but if we want to add a page for a new feature that is still in development, that page by itself could point to the new notebook off of the hnn-core/master branch.
• textbook could access development versions of textbooks, even including those of unfinished PRs. This is the issue I ran into here feat(network): Add spike_train_drive for explicit inter-network inputs hnn-core#1064 (comment) which made me come up with this solution.
• There would no longer be any ambiguity about which version a notebook was successfully run on, or between websites. textbook and hnn-core would never have alternate versions of a notebook that the other repo could not access. There is "one and only one" version of each notebook per hnn-core commit.
• Notebooks would only need to be executed once for each version, rather than twice (both hnn-core and separately textbook).
  • Similarly, textbook would also never have to double-check that each notebook actually works.
• Similar to our code website, if we wanted to provide separate "stable" and "development" webpage versions of our textbook website, it's just a matter of pointing to different versions of the same notebooks.
• The current method of manually copying Jupyter notebooks from hnn-core (which need to be heavily changed still, and do not necessarily work out-of-the-box!) is prone to issues, including 1. it's manual, 2. the current notebooks on textbook are not the same as the notebooks on hnn-core, 3. updates to scripts (or new ones) on hnn-core need to be tested on hnn-core, then converted to notebooks, then those notebooks need to have some changes made, then the updated notebooks need to be manually copied over to textbook, and this process needs to be repeated every time. If we are planning to add major rewrites or new additions of notebooks, which we are, then this is not a good approach.
• This would prevent us from having to start dealing with multiple kinds of execution styles like from [WIP] Remove automatic forced-re-execution for versions #96.

Cons:

• This is obviously a non-trivial amount of work. However, I think the current method is unsustainable, and we need to streamline (and connect the two repos) somehow.
• Authors would have to edit notebooks on hnn-core rather than the textbook repo, even though they want the notebook content to show up on textbook. However, I think this is inevitable if we want textbook and hnn-core execution of every notebook to be synchronized.
• Much of the execution code here would no longer be needed (but the rest of the code would still be).
• ???

[asoplata]

@dylansdaniels
In discussion at last HNN Dev meeting on 2025-07-21, Dylan noted that his idea was to transfer all example/notebook execution to the Textbook repository, and remove both the files and their execution from the Code repository.

However, in order to develop new notebooks and update existing notebooks, I think its is best if we reinvent some of the version control offered by the Code website ( https://jonescompneurolab.github.io/hnn-core/stable/index.htmlin the upper right dropdown) to the textbook. Many new/updated notebooks that depend on changes that are in development but not stable releases need to be executed differently. I'll suggest some options in order of increasing complexity.

1. Option 1: For new/updated notebooks that depend on post-stable changes, we give up. We add them to the Textbook repo, but mark them to be ignored. Then, when there's the next stable release, we remove them from the ignored list and start executing them as normal. I think this is a very bad idea for a number of reasons, including that we're not testing notebook execution with the current hnn-core/master development version, etc.

   • In this case there would only be 1 version of the Textbook site.

2. Option 2: For new/updated notebooks that depend on post-stable changes, we come up with a way for the Textbook repo to execute them off of hnn-core/master instead of the latest stable version. This would probably involve doing some processing and making some Policy decisions on version metadata for each notebook. This could get messy. This also means that we would not be executing the same notebooks on both development AND stable, which is a problem.

   • In this case there would only be 1 version of the Textbook site.

3. Option 3: We split the Textbook website into separate development and stable branches, each of which are deployed as their own version of the website (available via a dropdown).

   • stable is always built on every commit as of the latest stable version.
   • dev is always built using the latest hnn-core/master commit, and redeployed on every new commit.
   • All new changes go into the dev branch first.
   • Changes to "existing content" such as "text edits" or "existing notebook changes that don't use new code" can happen on dev and can have their commits (but not their parent commits) merged immediately onto stable.
   • New notebooks or changes to notebooks that use development code remain on the dev branch locally, until their code changes are supported in a new HNN-Core stable release.

4. Option 4: ??? Any ideas

[asoplata]

In Dev mtg 2025-07-28, we decided on Option 2, and next task is to figure out the details. Some notes:

• Decided that we do want to execute all notebooks against both stable and hnn-core/master
• probably going to add more metadata to the top level of the IPYNBs
• New, or updated notebooks that use code not available in stable will NOT be shown on the website, until the next stable release
  • after which they will THEN be added to the user accessible parts of the site

[asoplata]

Version-aware notebook execution workflow proposal (version Austin2.0)

Summary

The main thing to read is the below "Expected workflows" section, where I've tried to think through every practical case.

Before deployment, we test all new notebook code changes against a build.py --execute-notebooks run using the current HNN stable version; this is done using a new Github Action that will run on PRs. We also test new code changes against a build.py --force-execute-all run that uses the current HNN master (aka development) version; this is also done using a new Github Action that runs on both PRs and whenever there is a new push to hnn-core master. We add some notebook exclusion handling to account for these two versions. The version of HNN that the notebook is run against are always determined by the environment, not metadata. When there is a new HNN stable release, we make a new PR that includes a complete re-execution of all notebooks (but executed locally, not using a Github Action). (If we go with this or something similar, then I will definitely re-write the workflow instructions to be more user-friendly.)

Definitions:

• Dev: Developer person
• MD: Markdown files (*.md)
• NB: Jupyter notebook files (*.ipynb)
• stable: The current stable "release" of the hnn-core package, as used by pip from https://pypi.org/project/hnn-core/ . Not the Textbook or its repo.
• env-stable: A Python environment used by the Textbook repo that includes a full installation of hnn-core's stable version as defined above.
• GA-env-stable-deploy: The existing Textbook Github Action, using the env-stable environment, that runs build.py with some arguments and then deploys the website.
• GA-env-stable-test: A new Textbook Github Action using the env-stable environment, see below.
• master: The current master (aka development) branch of the hnn-core source code at https://github.com/jonescompneurolab/hnn-core . Not the Textbook or its repo.
• env-master: A Python environment used by the Textbook repo that includes a full installation of hnn-core's master branch as defined above.
• GA-env-master-test: A new Textbook Github Action using the env-master environment, see below.

Expected workflows

1. Dev wants to change only Markdown content (NOT code), either in MD files or Markdown cells inside NB files. Suggested workflow:

   • They make a PR, add their changes, and push. They only need to push the MD and NB files. No NB execution is necessary. (Currently, this is broken, see [WIP]: Bug: Fix Markdown changes not being applied #105 , but this is how it should work in the future.)

2. Dev wants to change code in an existing or new NB file, and that code is compatible with the current stable version of hnn-core. Suggested workflow:

   • If updating an existing NB file, they only need to push that file.
   • If adding a new NB file, then they should push the new NB file and add an accompanying MD file.
   • GA-env-stable-test is run to test that the website builds and will be deployed properly.
   • GA-env-master-test is run to test that the code change works against the latest master.

3. Dev wants to change code in an existing or new NB file, and that code is NOT compatible with the current stable hnn-core version; it instead depends on new code in the master version of hnn-core. Suggested workflow:

   • They push both their updated or new NB file, and they add it to the new notebooks_master_only list (see below).
   • If adding a new NB file, then they should NOT add an accompanying MD file for the NB. This means there will be no NB "webpage" until the next stable version release (see below), but the new NB file will still be accessible on the site directly via its link.
   • GA-env-stable-test is run to test that the website builds and will be deployed properly.
   • GA-env-master-test is run to test that the code change works against the latest master.

4. If a new stable HNN version is released, then either Dylan or Austin make a PR and do the following:

   • Notebooks are removed from notebooks_master_only (see below) as needed.
   • A corresponding MD file for each of these newly-stable notebooks is created.
   • A full build is run using the new --execute-super-omega-all-notebooks-but-seriously-this-time flag (see below), including execution of normally skipped notebooks. All the content of this run is pushed.
   • The programmed version to use as the "stable" version is updated, which will change the version used in any Github Actions using env-stable. This is not currently programmed into the code.

5. If there are new, breaking changes to the public API of hnn-core's master that cause existing stable notebooks to break (aka master breaks API backwards-compatibility):

   • Those notebooks are then added to notebooks_stable_only, and will be excluded from GA-env-master-test.

Updates to Github Actions:

1. GA-env-stable-deploy: [WIP] Check version against latest and force execute nbs #95 is reverted and --force-execute-all is no longer used in this Action, since most notebooks will now be run against multiple versions in a different way. Its use of build.py will be restricted to --execute-notebooks. Its purpose is deployment of the website, not testing.

2. GA-env-stable-test: We create a new Github Action that runs using env-stable, and also uses build.py --execute-notebooks. However, this runs on PRs and is intended for testing that the PR works using the stable version of HNN before it is deployed.

3. GA-env-master-test: We create a new Github Action that is very different from before. This runs using env-master and performs all unskipped execution using build.py --force-execute-all. This also tests a new category of notebooks called "master-only" which use new features (i.e. the notebooks included in notebooks_master_only explained below). This is run both on PRs and whenever there is a new push to hnn-core's master branch.

Updates to Code:

• Data structures:

  • Existing notebooks_to_skip becomes notebooks_all_skip to indicate that both stable and master should skip execution of them.
  • New list notebooks_master_only is what it sounds like. Intended for new or updated notebooks that should be run under master, but are using new features not yet available in stable.
  • New list notebooks_stable_only is what it sounds like. Intended for existing notebooks that use the API in ways that master has changed in non-backwards-compatible ways.

• build.py argument changes:

  1. We add two NEW arguments: --stable and --master. These arguments tell build.py whether it should be including notebooks_master_only or notebooks_stable_only (see above). One of these is always required.
  2. --execute-notebooks works the same, but maybe we change the name to --execute-only-updated-notebooks or similar.
  3. --force-execute-all works the same, but maybe we change the name to --execute-all-unskipped-notebooks or similar.
  4. We add a third type of execution that does not skip ANY notebooks, but still obeys the --stable/--master distinction. This is only used for when we need to manually push a full rebuild, such as when there's a new stable version (including the Optimization notebooks) or for personal testing. Maybe we give it a name like --execute-super-omega-all-notebooks-but-seriously-this-time.

[ntolley]

@asoplata thanks for writing this up! The steps/rationale are super clear

I have a higher level question that would help me evaluate this: why do we want to be able to run development version notebooks on stable HNN code? Is the main advantage to push updates to the textbook that are decoupled from HNN releases? It seems like there's a lot of administrative overhead to allow this flexibility, and I wonder if it's actually worth the trouble

[asoplata]

Good question. Some reasons:

1. If a notebook has version-difference issues, then we can stop it from being run on master by adding it to notebooks_master_only.

2. Last meeting we discussed how, by default, we want the Textbook to only show notebooks that work on the stable version -- in other words, development-only notebooks will not be clearly shown on or linked to by existing webpages. This is so users don't mistakenly think this notebook will work with stable, when it will not.

3. However, to balance point 2, we also want to have as few version-specific notebooks as possible for maintenance burden. If our notebooks are run on both code versions, then we only need to have one notebook version, instead of two (one of which will not be publicly visible).

4. This therefore allows us to still update existing notebooks if we want, and the updates can be shown, if the updates are compatible with stable. Most notebooks and updates are expected to work with both versions, IMO.

[asoplata]

The current main work handling this effort is in #114 .

This comment (and that PR) deprecates all previous attempts at this issue, including jonescompneurolab/hnn-core#1093 , and jonescompneurolab/hnn-core#1092 , #96 .