Code-cell output, myst and co

[renefritze]

Coming from here (after almost a year..) I'm somewhat confused about the current and planned state of the ecosystem and hope somebody can clarify this for me.

I have notebooks in myst markdown in a sphinx project. Currently the html builder processes those, I get html output with all the code-cells' output in it. Myst-nb also outputs .ipynbs for me. However those have unprocessed sphinx roles/directives in the markdown cells.
To fix that I would use sphinx-tojupyter, run its jupyter builder and get nice markdown cells in the output notebooks that would be (mostly?) equivalent to the text produced from the html builder. However code-cell currently do not get executed/their outputs included. That would be the goal of #5. After that I would need some more glue code in my deployment that puts the sphinx-tojupyter .ipynb into the html structure so they can be downloaded from there when I upload my final page.

Is that about right?

I've also stumbled upon this in #3:

Eventually myst will be supported in jupyter and we can deprecate this package.

This I don't understand at all. Would jupyter somehow be rendering sphinx directives itself?

I do have some resources to work on #5, if that's mostly what's needed for me to get where I want to be. However if that's not a long-term solution anyway, maybe the work would be better spent elsewhere?

[mmcky]

Hi @renefritze -- We have limited developer time so this this package is currently in a maintenance mode (at the moment) as we figure out the "best" way to support ipynb as an output for the jupyter-book project. Having said that we (QuantEcon) use this package daily to build notebooks that support our download notebook feature for our lecture sites such as: https://python.quantecon.org/intro.html

So I understand your workflow request pretty well I think as we have a very similar need.

The technical downside of this package is that it takes the input content (md:myst, ipynb) and then passes it through the sphinx abstract syntax tree before it arrives at the writer in this package to extract the information from that tree and write ipynb again which is a lossy process. We have worked hard to make that as lossless as possible but as sphinx internals change this writer/translator package will have to be maintained etc.

There are some side projects in the works in the executable book community exploring rendering of myst in juypyter using javascript which might be able to render the directives and roles introduce by md:myst. As you say you can use jupytext to convert md:myst to ipynb but the roles and directives markup will remain unprocessed in the jupyter notebook. If these could render we wouldn't necessarily need to transit the sphinx ast

However code-cell currently do not get executed/their outputs included

This is a feature we had in the original version of this package sphinxcontrib-jupyter but we removed it as it used nbclient (after writing ipynb) and not the jupyter-cache so it doubles up on computation runs.

Regardless of where we end up with more direct translation routes (i.e. sphinx agnostic), there is still a use case for this package for rst based projects in sphinx.

Does that give you a bit more information about this package? Happy to follow up and discuss ideas etc.

[renefritze]

Thank you very much for all that background info. Yes, the current situation is now much clearer.

Executing the produced notebooks "again" was an idea I also had, and discarded just the same.

Since you see utility for this package, even after a potential "frontend myst" thingy becomes available, I guess I'd consider working on #5 worthwhile.
I might put in a little research time over the next couple of weeks of how the interaction with the jupyter cache works in myst-nb and to see how it could be used here. I'm not super familiar with sphinx/docutils internals however. I've written tiny extensions, but nothing major.

[mmcky]

thanks @renefritze -- your help is appreciated -- let me know how you get on.

I'm also not familiar with jupyter-cache but I know who wrote it (great programmer -- so should be entry points we can use etc.) and we can try and get some pointers from them after we learn a bit more about it. I can also try and answer any questions.

The myst-parser and myst-nb layers have just gone though a massive re-write recently, with myst-nb in the final development phases.

executablebooks/MyST-NB#380

The team is still looking at how to support the current glue behaviour (across pages as the last hurdle) but I bring this up as there will be some internal changes on how the sphinx.ast looks with regards to images (as outputs from cells) etc. as there will be no CellOutputNode node type. So you might want to wait a week to see if that PR is merged first -- we will then be updating jupyter-book to leverage the new myst-nb version and I'll have better insight into this integration with jupyter-cache also.

[renefritze]

Alright, thanks for the heads up. I'll be watching executablebooks/MyST-NB#380