Persistent failure to find jupyter-cache with Julia

[tobydriscoll]

Description

I have a Quarto project running Julia 1.9 blocks that renders fine with --no-cache, but fails with "jupyter-cache package is required for cached execution" when cacheing.

I have tried:

• add jupyter-cache to base conda environment
• or, in a clean new conda environment, install jupyter-cache
• Conda.add("jupyter-cache") from within Julia
• quarto check runs all clear
• jcache runs at the command line and run(`jcache`) works at the Julia REPL
• IJulia launches a notebook, and run(`jcache`) also works within the notebook

I am stumped. What else can I try?

[mcanouil]

Could you share a repository where you have the "faulty" setup? Or the steps to reproduce?
Could you double you followed https://quarto.org/docs/computations/julia.html#caching?

PS: I do not use Conda for many reasons and prefer virtualenv.

[tobydriscoll]

The repo is at https://github.com/tobydriscoll/udmath351/tree/master/course.

I did look at those docs. pip did not behave any differently than installing via conda. If the answer is in there, I've missed it.

P.S. I use mamba rather than conda in practice. Just what I grew up with, so to speak.

[tobydriscoll]

I did create the "Plots demo" document by "Norah Jones" in a temp directory. When I run

quarto render test.qmd --to html

all is well. When I run

quarto render test.qmd --to html --cache

I get the error about jupyter-cache. I just meant that at first I forgot to change Julia-1.8 to 1.9 in the test file, and I get the error message either way.

[mcanouil]

And do you get the error message when you enable cache from the yaml frontmatter?

[tobydriscoll]

Confirmed.

[mcanouil]

So definitely something wrong in your setup, as in a fresh Julia install, with Jupyter from a virtual env, I have no issue using cache.
https://quarto.org/docs/gallery/#books

Could you share the output of quarto check (especially the part about Jupyter, which can be checked independently using quarto check jupyter)?

Below is mine for reference:

[tobydriscoll]

[cderv]

Can you check somehow that the python version used with your kernelspec is indeed the one you have installed the jupyter-cache package into ?

I know that I already had issues with Julia and Jupyter because Julia was not using the Jupyter installed with Conda, but another Jupyter installed in my usual python setup. As explained at https://quarto.org/docs/computations/julia.html#ijulia it could happen that when you install IJulia in Julia, it will use an existing Jupyter and not install conda or anything. It would mean to me that Conda.add("jupyter-cache") would not install in the right python setup.

Though you said

IJulia launches a notebook, and run(jcache) also works within the notebook

So it seems that IJulia is configured with a Jupyter that knows about Jupyter-cache. Unless it was not tested with same kernel Quarto loads from YAML spec.

I don't know how mamba works - so maybe there is a specificity and Quarto miss something. THough I believe we rely on jupyter kernelspec list to get the correct kernel to run.

[tobydriscoll]

Here is the kernel.json for the python kernel:

{
 "argv": [
  "/Users/driscoll/mambaforge/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

The first arg is the base python, which does have jcache installed.

What's maddening is that I've now installed the same toolchain on my laptop and quarto works fine with the cache (both are darwin). I can't pin down the crucial delta between the two machines.

[tobydriscoll]

I still don't know "why," but I do have "how."

According to the IJulia docs, unsetting ENV["JUPYTER"]="" and rebuilding IJulia is supposed to force it to use its own miniconda environment. That wasn't happening, no matter what traces of IJulia I tried removing and rebuilding, so I finally just nuked the .julia folder and installed packages from scratch. Now it uses the julia miniconda, and the quarto cacheing works again.

Thanks for looking into it!

[cscheid]

I'm glad that solved the problem for you! I'm wondering if there's a way we could have documented the environment better in quarto check so that your debugging life would have been easier. If you have suggestions, I'd be happy to try and incorporate them.

[tobydriscoll]

In a sense it was IJulia's or my bug, because it did not perform as documented about which python env to use. But I still don't understand why the system's environment wasn't good enough in the first place, so I don't know what change to advise.

[cscheid]

Sorry, I should have been clearer. quarto check prints out a information about the user's environment, hoping that this information will help you diagnose downstream issues. If there was information that you needed to know about the Jupyter/IJulia environment which we could have printed and didn't, then that's a candidate for improvement on our part.

(Truly no pressure either way! I just like to ask folks this question right after they fixed their problems since the memory would still be fresh.)

[cderv]

If there was information that you needed to know about the Jupyter/IJulia environment which we could have printed and didn't, then that's a candidate for improvement on our part.

I got bitten by this internal Julia Conda too, and ENV["JUPYTER"]="" (but the other way around as I wanted to use the external Jupyter).
I wonder if we could try something like:

• If we detect a Julia kernel
• We run a .qmd document that will print the JUPYTER version used with this Julia kernel
• We report that information in quarto check

This could help solve this issue on which Jupyter and Python is really used when using Julia