`jupyter kernelspec install` copies way too much data

[dimpase]

in the Python library sagemath we have 10-year old code which installs our Jupyter kernel. We would like to replace it with jupyter kernelspec install

(sagemath can be programmed in an extension of Python which needs on the fly pre-parsing to convert it into Python, hence we can't just use ipython kernel).

However, while jupyter kernelspec install --user works, it results in gigabytes of data being copied to ~/, most of it being .so files coming from the compiled Cython and other Python extensions. Is there a way around this, by setting appropriate paths?

Sagemath typically is installed in a venv, so we hoped that replacing --user with --sys-prefix might be the more logical choice, but this does not work, as jupyter tries to install the kernel into /usr (where the Python used to create the venv lives) rather than into the the venv itself. Is this a bug? Why does it do this kind of de-referencing, breaking the very idea of what a Python venv is for?

This is observed on a reasonably up to date jupyter:

jupyter --version
Selected Jupyter core packages...
IPython          : 9.4.0
ipykernel        : 6.29.5
ipywidgets       : 8.1.7
jupyter_client   : 8.6.3
jupyter_core     : 5.8.1
jupyter_server   : 2.16.0
jupyterlab       : 4.4.6
nbclient         : 0.10.2
nbconvert        : 7.16.6
nbformat         : 5.10.4
notebook         : 7.4.5
qtconsole        : 5.6.1
traitlets        : 5.14.3

[dimpase]

@jasongrout - not sure if this is the right repo for this.

The --sys-prefix issue goes away gets less bad (less disk is consumed, but it works (!) - still .so files are copied)
if, instead of Sage's venv, a standard uv venv is created and sagelib is built there (yes, we can do this now :-))
I don't know why this happens @tobiasdiez

Probably files being copied is a consequence of install not being designed with editable installs in mind?

[tobiasdiez]

From a more general vantage point, what is the current advice for packages to install a jupyter kernel? I read somewhere that it should be enough to manually install the kernel.json in the right folder in the venv, but that is not working for me.

[dimpase]

That's all a bit of a rabbit hole it seems, modern jupyter(lab): 🥲
I spent some time getting rid of IPP Server errors in my jupyterlab install, and it turns out one needs
a lot of extra python modules (IPP is actually IPyParallel): ipyparallel google-api-python-client google-generativeai dotenv (and their deps, and then jupyter server extension enable ipyparallel)

[minrk]

I'm not sure I understand what you're doing that is pulling in so much, or where IPython Parallel is getting involved at all (it certainly isn't needed and also doesn't need any of the other packages listed there, I'm not sure where they would be coming from).

jupyter kernelspec install takes a kernelspec directory (typically only contains a kernel.json file and maybe one or more logo files), and then copies the kernelspec directory to the Jupyter kernelspec discovery path. The whole kernelspec directory is copied, so if you give it a directory with lots of stuff in it, lots of stuff will be copied. But it is unlikely that a directory with lots of stuff in it is a kernelspec directory. It should not be given the path to a full environment. The kernel.json should contain the references to the environment, in paths, env variables, or both.

what is the current advice for packages to install a jupyter kernel?

This is pretty severely under-documented, I will try to take a stab at updating the docs for this. But it should be:

1. create a kernelspec directory, containing a kernel.json file, and optionally one or more logo files. The main thing is that the argv should resolve to your env via absolute paths
2. run jupyter kernelspec install /path/to/directory/containing # kernel.json, i.e. if you have ./sage-kernel/kernel.json you would jupyter kernelspec install ./sage-kernel

A kernel.json for a venv would probably look something like (this example is for ipykernel):

{
 "argv": [
  "/path/to/my/env/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (example kernel)",
 "language": "python",
 "env": {
  "PATH": "/path/to/my/env/bin:$PATH"
 },
 "metadata": {
  "debugger": true
 }
}

Key points:

• argv[0] uses an absolute path to make sure invocation of the executable is correct
• env.PATH extends PATH to ensure the env is on PATH for the kernel and its subprocesses (maybe more env vars are appropriate)

most often jupyter kernelspec install --sys-prefix is what you want (installs to the prefix of jupyter itself), though depending on the invocation you might specify the prefix explicitly via --prefix or do a global (all envs) per-user install with --user.

[dimpase]

Sage's kernel.json is embarrassingly short, and the 1st argument does not specify the the full path to Python.

{
  "argv": [
    "python",
    "-m",
    "sage.repl.ipython_kernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "SageMath 10.8.beta1",
  "language": "sage"
}

we certainly need to try setting PATH:

[dimpase]

@minrk - how does one modify Jupyter kernelspec discovery path? It seems that we're stuck on this point.
Is it something which can be done in kernel.json ?

Indeed, with the system-wide jupyter(lab) install, the problem is that somehow Sage's custom venv does not like --sys-prefix option in jupyter kernelspec install, and insists on installing in /usr/share/jupyter/kernels/sagemath (even if I provide the correct full path to the venv Python (which is a symlink to a Python in /usr/).

If I provide --prefixin the venv for the installation, I get a warning The kernelspec may not be found..

$ jupyter kernelspec install --name=sagemath --prefix /home/dima/software/sage/venv/ build/sage-distro/src/sage
[InstallKernelSpec] WARNING | Installing to /home/dima/software/sage/venv/share/jupyter/kernels, which is not in ['/home/dima/.local/share/jupyter/kernels', '/usr/local/share/jupyter/kernels', '/usr/share/jupyter/kernels', '/home/dima/.ipython/kernels']. The kernelspec may not be found.
[InstallKernelSpec] Installed kernelspec sagemath in /home/dima/software/sage/venv/share/jupyter/kernels/sagemath

Using --user works, but copies 2.4Gb of data (a lot of it .so files). And that's not the right location if you want to use several venvs, anyway.

$ du -h /home/dima/.local/share/jupyter/kernels/sagemath | tail -n1
2.4G	/home/dima/.local/share/jupyter/kernels/sagemath

Omitting --prefix/--sys-prefix/--user leads to an attempt to install the kernel in /usr/local/share/jupyter - again, wrong.

(PS. using uv venv instead of Sage's on does not lead to this problem, `--, but we can't at this point just ditch the Sage's venv, yet...)

[minrk]

how does one modify Jupyter kernelspec discovery path? It seems that we're stuck on this point.
Is it something which can be done in kernel.json ?

You can add locations by adding prefixes to JUPYTER_PATH. It can't be done in kernel.json because it has to take effect before looking for kernel.json. But I'm not sure this is really what you want (I don't know what you want, though, so maybe it is).

The root issue of "copies too much data" appears to be because, as I tried to explain before, jupyter kernelspec install should be given a kernelspec directory (contains kernel.json and little else), which it always copies in its entirety to the requested location. So if you give it a directory containing way more than a kernelspec, it will copy that. The root problem appears to be that you are not giving it a kernelspec directory. How are you creating the kernelspec directory, what is in it, and how are you passing that to kernelspec install? It should be a directory containing

• kernel.json
• (optionally) logo files
• that's it

I'm inferring from context that build/sage-distro/src/sage contains more than a kernel.json. If so, it's the wrong path to give kernelspec install, it should probably look something like

jupyter kernelspec install --name=sagemath --prefix /home/dima/software/sage/venv/ build/sage-distro/src/sage/kernelspec

As for the install paths, can you clarify more what you want to happen? It's hard to answer questions without knowing what you want. jupyter kernelspec install really means "make this kernelspec directory available to one or more envs." How to invoke it depends on which env(s) you want the kernel to be available in. Is it one specific env? All envs for one user? All envs for all users?

If you have a specific env in mind, and you want to make the kernel available in that env, use:

/path/to/that/env/bin/jupyter-kernelspec install --sys-prefix ./path/to/kernelspec

should work, unless it is a very strange env where sys.prefix doesn't resolve in the env.

--sys-prefix means into the prefix of the Python installation of the invoking jupyter process, so it is $PATH-sensitive if you don't invoke via absolute path. You can check which -a jupyter and/or jupyter --paths to get more info.

Alternatively, you can get the same result with:

jupyter kernelspec install --prefix /path/to/that/env ./path/to/kernelspec

which installs to that env, regardless of which env $(which jupyter) resolves to. If the invoking env is not the destination env, you'll get that The kernelspec may not be found warning, which only means that the invoking env doesn't include that path, which is correct and fine if you're trying to install a kernel for a different env, but may indicate a typo if you mean to be installing to the current env.

If you want the kernel to be available to all envs for a single user, that's what --user will do.

If you want to make it available to all users and all envs on the system, use --prefix /usr/local (or leave it out, since this is the default behavior).

Omitting --prefix/--sys-prefix/--user leads to an attempt to install the kernel in /usr/local/share/jupyter - again, wrong.

Not wrong, this is a "system-wide install" and correct if you want the kernel to be available to all envs for all users. All Jupyter envs will look here. Maybe not what you want, but definitely not wrong.

[dimpase]

Oh wow, sorry, thanks for the patience in explaining.

Now I realised where this 2.4Gb are coming from - it's from kernel.json being prepared for installation in more or less the root of the meson build tree.
@tobiasdiez

[tobiasdiez]

Thanks also from side @minrk. These are very helpful explanations. As Dima said, the big install size comes from our wrong setup and that's relatively easy to fix.

Two more follow-up questions to what you said above:

1. How do you normally get the path to the env when generating the kernel.json? Say when building a wheel, I don't know yet in which python env it will be installed later. If I understand the example project https://github.com/jupyter/echo_kernel/blob/6e880d796fe5c86b65b651cfff050f506a65539e/hatch_build.py#L16 correctly, then it embeds the path to the python executable that build the wheel, not where the kernel will be installed in. This looks strange to me.

2. So an explicit jupyter kernelspec install is always necessary, even if you only want to make the kernel available in the env where it is installed? Again from the example repo https://github.com/jupyter/echo_kernel, I got the impression that somehow a pip install of the kernel package is already sufficient to also register the kernel. Perhaps this comes from https://github.com/jupyter/echo_kernel/blob/6e880d796fe5c86b65b651cfff050f506a65539e/hatch_build.py#L41? Should a package on install have such a call, or is it recommended to just have the user invoke jupyter kernelspec install later manually?

[dimpase]

2. an explicit jupyter kernelspec install is always necessary, even if you only want to make the kernel available in the env where it is installed?

sage has old, 10-years old, code to install Jupyter kernel.
https://github.com/sagemath/sage/blob/develop/src/sage/repl/ipython_kernel/install.py
It gets invoked as a part of meson-driven installation of sagelib. Then the external (for the venv) jupyter lists the result in the call jupyter kernelspec list. However, external jupyter notebook doesn't work with this kernel
(the internal one, installed in the same venv, seems to work).

What does work is explicitly calling jupyter kernelspec install --name=sagemath --user src/sage/ext_data/notebook-ipython/. Then everything (jupyter notebook with the kernel in question) works, if kernel.json is modified as Min sets out above, with full path to python, and
env.PATH added.

{
  "argv": [
    "/home/dima/software/sage-src/.venv/bin/python3",
    "-m",
    "sage.repl.ipython_kernel",
    "-f",
    "{connection_file}"
  ],
  "env": {
  "PATH": "/home/dima/software/sage-src/.venv/bin:$PATH"
         },
  "display_name": "SageMath 10.8.beta2",
  "language": "sage"
}

the original json, as in #1070 (comment), does not work.

Also, if I replace --user with --sys-prefix, so that the kernel is installed in the venv, not in ~/.local, then I see
Kernel does not exist errors while calling jupyter notebook (which does not work then, it can't connect to the kernel).

[minrk]

How do you normally get the path to the env when generating the kernel.json? Say when building a wheel, I don't know yet in which python env it will be installed later.

This is a shortcoming in wheels that conda solves. 'data files' can't reference absolute paths and be rewritten at install time. Conda does this. To deal with that for the simple case (wheel install), we use the python placeholder. When jupyter-client sees python (or python3 or python3.14 matching the current process) as argv[0], it explicitly invokes sys.executable instead. This is right for in-env kernels such as you get from a wheel-installed kernelspec. It is not right if you want to copy this kernelspec to a different env and have it always invoke in your env.

As a result, ipykernel does two things:

1. creates kernel.json in data_files with python as the executable as part of the wheel. That way, pip install ipykernel results in a kernel being found. This works for the vast majority of installs that are wheels in a single env.
2. the command python3 -m ipykernel install ... generates a kernelspec with the current value of sys.executable, so this always points to the invoking ipykernel env, no matter what env you are installing the kernelspec into.

So if you are preparing this as part of a wheel, use python as the executable. If you are installing from a known, existing env, write sys.executable evaluated at runtime. If you're building conda packages, this doesn't come up and you can use sys.executable all the time.

So an explicit jupyter kernelspec install is always necessary, even if you only want to make the kernel available in the env where it is installed?

No, placing share/jupyter/kernels/sagemath/kernel.json in data_files with argv[0] = 'python' should work the vast majority of the time. Exceptions include some system Pythons.

Should a package on install have such a call, or is it recommended to just have the user invoke jupyter kernelspec install later manually?

Because the data_files case doesn't work 100% of the time, having an explicit kernel install option is good to have. This addresses the case of wanting to make your kernel available to other envs (just copying the kernel.json with argv[0]=python will not work, but the same json after replacing argv[0] = sys.executable will), or the weird envs where data_files don't actually go in a place they can be found (mainly System Pythons that are increasingly discouraged as direct pip-install targets)..

then I see Kernel does not exist errors while calling jupyter notebook (which does not work then, it can't connect to the kernel).

This suggests that the installing jupyter kernelspec install and later jupyter notebook have different values for sys.prefix.