[MRG] Austin workflow refactor part 2 (#134)

* ref: minor of check_nb, replace os with pathlib

* ref: build arg str var, use "repo-owner"

* ref: minor log output improvement

* ref: add unused args+docs for new version args

* ref: debug comments, and pathlib replacement

* ref: add CLI args to get_commit and ruff

* ref: major overhaul of indexing, part 1 of 2-3

This is the first part of a major refactoring overhaul of how we handle
indexing, input-markdown and output-HTML pathing, and sidebar
generation. This is still a work in progress, and most functionality has
been commented out for debugging/testing.

Major changes:

- `update_page_index.py` has been renamed to `create_indices.py`. This
new file now creates two indices:

1. "hier_index", which is the same as the old "index.json". Its content
has not changed. Instead of being saved to file every time and then
loaded later during sidebar creation, it creates and returns the
dictionary instead, and that is passed to sidebar for usage. I'm not
sure if we need to save it as a file at all, since its value can always
be easily debugged live, and we were not doing any kind of checking to
skip its loading before (i.e. if we have to reconstruct it every time,
then there's no gain from saving a copy to disk).

2. "flat_index". This will eventually replace "ordered_links" and is
greatly expanded from what "ordered_links" does. This is treated like
"hier_index" as an explicit "generate_page_html" function, but will replace
much functionality/data of the rest of the code. This index includes the
ordering of legitimate "pages" (i.e. not "sections"), and for every page
contains a dictionary containing:

A. title of the page
B. absolute path of the "input" markdown file
C. absolute path of the "output" HTML file (including processing
dev_build, and including creation of any parent directories to this
not-yet-created file)
D. "website-root"-relative path of the "output" HTML file (including processing
dev_build)

The next major step is using "flat_index" to replace ordered_links and
output path/filename lookups throughout the rest of generate_page_html.

Minor changes:
- Location of the saved index files has been moved to inside "scripts"
from the root
- Many functions and some script files have been renamed

* ref: overhaul of index/pathing, part 2 of 3

This continues the use of "flat_index", including in every case of the
main loop, except for the "ordered_links" which are next.

This includes many minor changes as well for formatting (including a
ruff run), documentation, and other small changes.

* ref: indexing overhaul part 3 of 3

Among other minor changes, including adding a new optional
"save_indices" CLI arg that does what it sounds like, this finishes
applying the usage of the new "flat_index" to the remaining code, which
was mostly the "ordered" links used in the per-page footers.

Everything appears to be working correctly, including that the sidebar
and footer navigation uses "dev" or "content" properly.

Next up is general cleaning and documenting.

* ref: restore future _structure_json function

* ref: rename execution-filter to execution-type

* propagate rename to CI

* ref: major overhaul of dev CLI args, part 1 of 2/3

This begins a multi-step transition from the CLI "--build-on-dev"
argument to the combination of "--code-version" (formerly the unused
"--build-type" argument) and "--custom-owner-commit".

The CLI args have already been created, and this commit focuses on
applying their use inside `get_commit_hash.py`.

This also includes a great deal of minor changes `get_commit_hash`,
including better error messages, more explicit variable names, etc.

There is also incomplete support for a new "--code-version=no-check"
option that skips the version checking in case one is debugging or if
one wants to run 'build.py' a lot, but doesn't want to spam HTTP
requests. This use-case is useful in itself, but also related to a weak
hypothesis I had: that the fact that version mismatches in the previous
"default" case of 'build.py' usage provided only *warnings*, instead of
RuntimeErrors, like in all other mismatch cases (see
https://github.com/dylansdaniels/textbook/blob/workflow_refactor/build.py#L610
)

* ref: finish overhaul of dev CLI args, part 2 of 2

`dev_build` has finished being replaced by a combination of
`code_version` and (either `custom_owner_commit` or `commit_hash`)
depending on the function. Documentation has not yet been updated, but
it was incomplete to begin with.

* ref: ruff run

* ref: incr recorded JSON flags verbosity

* ref: attn: remove non-exec hash json recording

This comments out what I think is something that may have been added by
accident. Compare this line to
https://github.com/dylansdaniels/textbook/blob/31dbaf09d6d854a4e8527024fa99f2b415820192/scripts/convert_notebooks.py#L923
.

What was originally going on at the time of the above commit is the
following:
`notebook_was_run` was used as whether or not the notebook was attempted
to be run at all, regardless of if it succeeded (I have changed this
variable to `execution_initiated`). `notebook_executed` was used as
whether or not the notebook successfully finished its execution (I have
changed this variable to `execution_successful`).

In the control flow, which has not changed at all, in this particular
case, we are in the case where `notebook_was_run`/`execution_initiated`
is False, meaning no execution was attempted at all. Also in the
original version, `dev_build` would have been activated, meaning we're
in a dev state. However, even if we were in a dev state, but if the
notebook execution was never initiated/attempted, then in this case, I
do not think it makes sense to record the commit hash of the current HNN
version into the notebook. If we did, it would imply that the last dev
execution was using this commit, but that is not the case.

* ref: 99% formatting and tons of COMMENTS

* doc: Add large synopsis of entire building program

This adds a large description of the building program as a whole to the
CLI for 'build.py' such that it is output when calling

python build.py --help

This includes description of the assumed file structure (which has many,
many assumptions), the overall code path, some examples, and the
important options along the way.

Minor: this also adds a docstring for 'get_hnn_commit_hash', renames the
'write_standalone_html' variable to 'save_standalone_nb_html', and makes
a new CLI argument of a similar name. This documentation was first
generated with Claude AI before undergoing *heavy* modification of every
word.

* doc: docs and comments for get-hash

* ref: ref all post-val vers checks to is_dev_build

This partially reverts our post-hnn_commit_hash-validation code's checks
of the "code-version" to use a new var like in the original, called
"is_dev_build". The distinction between a "dev" and non dev build is now
fully documented, apparent, and set in 'build.py'. "is_dev_build" is
also a proper boolean flag (true/false only) and the actual commit hash
to use (when necessary) is handled separately.

More refactoring: this also renames the hnn-core commit hash module and
splits its functions, so that the 'get_hnn_commit_hash' function is ONLY
getting the installed version, and the rest of the code is validation of
the installed version against various other versions.

* ref: move nb_html->json up in call tree

* doc: (mostly) incomplete updating of execute docs

There are also some minor variable name changes

* ref: change calculate_nb_hash to copy not load nb

Previously, we were loading the same notebook from disk twice, once to
calculate the hash, then once to actually use it. It's more efficient to
load from disk once.

* doc: more docs

This includes a special comment about how
`_read_nb_json_output_metadata` looks in different places for the JSON
output files depending on if you're doing a stable or dev build

* ref: minor var name change

* fix: proper nb JSON output loading

This fixes what I believe is a bug introduced in the original
"dev_build" changes.

Here
https://github.com/dylansdaniels/textbook/blob/31dbaf09d6d854a4e8527024fa99f2b415820192/build.py#L114
there is a bug at the later step when the final HTML page is being
built, and when the HTML content of the notebook (from inside its JSON
output file) is being loaded. In this case, because the JSON output file
was being loaded using unique code (different from the JSON output
loading code used at an earlier step inside
`scripts/convert_notebooks.py` see here
https://github.com/dylansdaniels/textbook/blob/31dbaf09d6d854a4e8527024fa99f2b415820192/scripts/convert_notebooks.py#L459
), only the "content" version of the JSON output file was being loaded,
and never the "dev" version. This meant that for "dev" final-page HTML
output containing a notebook's contents, the wrong JSON output was being
used to generate the page.

This has been fixed in that both instances of loading the JSON output
files have been changed to use the same function. Also, the later step
has extra configuration added in order to reconstruct the path of the
desired JSON output file location (which differs in the "dev") case.

More generally, this is another instance of where the complexity
introduced by trying to accomodate the different "dev" vs "stable" file
output locations is a lot, and maybe too much. Perhaps in the future we
can consider changing how he handle a separate "dev" build on the
filesystem on a fundamental level. For example, for a "dev" build, we
could make a full copy in the filesystem of the "content" directory into
the "dev" directory directly, and then replace our "content_path"
variable at the highest level with the "dev" directory.

* doc and minor ref: docstrings for remaining funcs

This also moves the output-directory creation step outside of
`create_flat_index` and overtly into the main loop of the HTML page
building code. (If you have to write a ton of comments explaining that a
step is done in a particular location, then maybe it shouldn't be in
that location! I'm dumb and it never should have gone in the index
creation func in the first place).

As is typical, these docstrings were *initially* created with Claude but
then inspected and edited substantially.

* ref: change code_version=no_check to CLI arg

This removes the '--code-version' option 'no-check', and instead adds a
standalone CLI argument called '--no-version-validation' that does the
same thing. This allows both "stable" and "dev" builds to EITHER be run
without code-version being checked. (Before, no-check was always a "dev"
build, but this is an improvement since not validating is useful when
inspecting or debugging a local stable build).

This also replaces the confusing argparse not-recommended use of bool in
favor of `action="store_true"`. This is the recommended way to treat CLI
arguments as "presence flags" so to speak, albeit the naming is a little
confusing (the default is False for "store_true").

* doc: docstring, comment, help, and ruff changes

SHOULD be no code changes...

* DELETE dev folder entirely, since bug is fixed

* ref: rename sidebar-header to sidebar-section

You can probably guess where this is going...

Note: this does not affect the Workshop page, since it's raw HTML.

* doc: remove Claude hallucination

My Claude's filesystem scan was out-of-date, and lightly "hallulcinated"
a `templates/navbar.html` that no longer exists. This removes that.

* ref: finish renaming navbar to sidebar

Appears to work just as well.

* maint: remove deprecated ordered_links comments

* maint and doc: sidebar code docs, and rename funcs

* doc: add tiny note about "dev" sidebar

* conflict resolution: re-add then delete HTML

* conf res: re-delete

* lazy file revert to remove conflicts

* delete files that no longer exist

* re-add final scripts changes

* ugh re-add deleted files again

* accidentally moved HTML file to wrong folder

* add misssing no-check argument to --code-version

* fix: add no-check to dev builds, and document it

* ref: remove --no-vers-val, add --build-dir

This removes the CLI arg `--no-version-validation` (so that only
`--code-version=no-check` remains), and adds a new CLI arg named
`--build-directory`. This new arg accepts values of `auto` (the default,
which follows our `is_dev_build` "algorithm"), `content` (user forces
website to be built into content dir), and `dev` (user forces website to
be built into `dev`). This also upgrades all the documentation to
reflect these changes, along with some minor other docstring
improvements.

* ref: decr verbosity of some CLI args

---------

Co-authored-by: dylansdaniels <dylansdaniels@gmail.com>

Author: asoplata

.github/workflows/deploy.yml

@@ -40,9 +40,9 @@ jobs:
       - name: Determine Build Method
         run: |
           if conda run -n textbook-stable-build python scripts/check_nb_versions.py; then
-            echo "BUILD_FLAG=--execution-filter execute-updated-unskipped-notebooks" >> $GITHUB_ENV
+            echo "BUILD_FLAG=--execution-type updated-unskipped-notebooks" >> $GITHUB_ENV
           else
-            echo "BUILD_FLAG=--execution-filter execute-updated-unskipped-notebooks" >> $GITHUB_ENV
+            echo "BUILD_FLAG=--execution-type updated-unskipped-notebooks" >> $GITHUB_ENV
             echo "Not all notebooks were executed on the latest version. Forcing re-execution."
           fi
 

Makefile

@@ -45,25 +45,25 @@ endef
 all: build
 
 build:
-	@#"This option is now equivalent to running build with '--execution-filter no-execution'."
+	@#"This option is now equivalent to running build with '--execution-type no-execution'."
 	python build.py
 
 force-execute-all-notebooks:
-	@echo "This option has been replaced with '--execution-filter execute-all-unskipped-notebooks'"
-	@echo "You can run 'make execute-all-unskipped-notebooks' for the same command."
+	@echo "This option has been replaced with '--execution-type all-unskipped-notebooks'"
+	@echo "You can run 'make all-unskipped-notebooks' for the same command."
 
 execute-notebooks:
-	@echo "This option has been replaced with '--execution-filter execute-updated-unskipped-notebooks'"
-	@echo "You can run 'make execute-updated-unskipped-notebooks' for the same command."
+	@echo "This option has been replaced with '--execution-type updated-unskipped-notebooks'"
+	@echo "You can run 'make updated-unskipped-notebooks' for the same command."
 
-execute-absolutely-all-notebooks:
-	python build.py --execution-filter execute-absolutely-all-notebooks
+absolutely-all-notebooks:
+	python build.py --execution-type absolutely-all-notebooks
 
-execute-all-unskipped-notebooks:
-	python build.py --execution-filter execute-all-unskipped-notebooks
+all-unskipped-notebooks:
+	python build.py --execution-type all-unskipped-notebooks
 
-execute-updated-unskipped-notebooks:
-	python build.py --execution-filter execute-updated-unskipped-notebooks
+updated-unskipped-notebooks:
+	python build.py --execution-type updated-unskipped-notebooks
 
 clean:
 	rm -rf content/*.html

build.py

@@ -505,10 +505,10 @@ body.is-mobile .sidebar-close {
 }
 
 /*
-    Element: .navbar-header
+    Element: .sidebar-header
     Description: Contain top-level and non-indexed sidebar elements
 */
-.navbar-header {
+.sidebar-header {
     display: flex;
     padding: 10px 0px 20px 0px;
     align-items: center;
@@ -521,13 +521,13 @@ body.is-mobile .sidebar-close {
     Element: .title-row
     Description: style 'title' text in navbar header
 */
-.navbar-header .title-row {
+.sidebar-header .title-row {
     display: flex;
     align-items: center;
     width: 100%;
 }
 
-.navbar-header a {
+.sidebar-header a {
     margin: 0px;
 }
 
@@ -850,7 +850,7 @@ body.dark-mode .topbar {
 
 
 /*
-# Collapse navbar
+# Collapse sidebar
 # ------------------------------------------------------------ 
 */