change from using a build script to auto installing kernels and providing a jupyter path update function #1215

KristofferC · 2025-11-05T10:30:15Z

The Pkg build functionality was made for a different era in Julia package management. It gets run whenever a package is downloaded (which is a weird thing to trigger on where packages are considered immutable and multiple environments share the same package downloads), it requires the package to recompile when build options are changed etc. There are now better options based on e.g. scratch spaces.

Instead of using Pkg.build to install the default kernel, this instead auto installs it if needed.
It provides a separate function to update the jupyter path because tying kernel installation and updating of jupyter path together felt wrong.

It moves to using the Preference system over using DEPOT_PATH[1]/prefs (but reads the old file as a fallback)

It changes some things that previously had to be done at build time (like setting the debug mode) to now just be a runtime thing.

Using IJulia from scratch (by using an empty depot and clearing out all kernels) looks like:

JULIA_DEPOT_PATH="$(mktemp -d):" julia +1.12 --project -q

(IJulia) pkg> instantiate 
...
  Installing artifacts ━━━━━━━━━━ 2/2
    Building Conda → `/var/folders/2c/rk2fkhgn6c35lr6qp7cll4t00000gn/T/tmp.8kRd4JRUqk/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/8f06b0cfa4c514c7b9546756dbae91fcfbc92dc9/build.log`

julia> using IJulia
Precompiling IJulia finished.
  13 dependencies successfully precompiled in 11 seconds. 30 already precompiled.

julia> notebook()
┌ Info: No default Julia kernel found for Julia 1.12.
│ Installing kernel automatically. You can reinstall or update the kernel
│ anytime by running: IJulia.installkernel()
│ 
│ To disable this auto-installation, set the environment variable:
└     ENV["IJULIA_NODEFAULTKERNEL"] = "true"
[ Info: Installing 'Julia 1.12' kernelspec in /Users/kc/Library/Jupyter/kernels/julia-1.12
...
install Jupyter via Conda, y/n? [y]: y
[ Info: Downloading miniconda installer ...
[ Info: Installing miniconda ...
PREFIX=/var/folders/2c/rk2fkhgn6c35lr6qp7cll4t00000gn/T/tmp.8kRd4JRUqk/conda/3/aarch64
Unpacking bootstrapper...
Unpacking payload...
...
...
[ Info: running setenv(`/var/folders/2c/rk2fkhgn6c35lr6qp7cll4t00000gn/T/tmp.8kRd4JRUqk/conda/3/aarch64/bin/jupyter notebook`)
...

…ding a jupyter path update function The Pkg build functionality is not good. It gets run whenever a package is *downloaded* (which is a weird thing to trigger on where packages are considered immutable and multiple environments share the same package downloads), it requires the package to recompile when build options are changed etc. This instead auto installs the kernel if needed and also provides a separate function to update the jupyter path. It also moves to using the Preference system over using `DEPOT_PATH[1]/prefs` (but reads the old file as a fallback) It also changes some things that had to be done at build time (like setting the debug mode) to now just be a runtime thing

codecov · 2025-11-05T10:33:46Z

Codecov Report

❌ Patch coverage is 10.60606% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.84%. Comparing base (656ed86) to head (e23228b).

Files with missing lines	Patch %	Lines
src/config.jl	7.14%	39 Missing ⚠️
src/jupyter.jl	0.00%	15 Missing ⚠️
src/IJulia.jl	20.00%	4 Missing ⚠️
src/kspec.jl	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1215      +/-   ##
==========================================
- Coverage   70.21%   68.84%   -1.38%     
==========================================
  Files          17       19       +2     
  Lines        1276     1409     +133     
==========================================
+ Hits          896      970      +74     
- Misses        380      439      +59

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

JamesWrigley

Thanks for making it so quickly ❤️

Also CC @goerz and @stevengj, in case you have any suggestions.

JamesWrigley · 2025-11-05T10:51:46Z

docs/src/manual/installation.md


 !!! important
-    `Pkg.build("IJulia")` **must** be run at the Julia command line.
+    `IJulia.installkernel()` **must** be run in the Julia REPL.


I think the whole admonition can be deleted now right?

JamesWrigley · 2025-11-05T10:55:06Z

docs/src/_changelog.md

  patch release of Julia, but it does mean that IJulia will only create kernels
  for each Julia minor release instead of each patch release.

+### Added


These should be in a new 1.33.0 section.

JamesWrigley · 2025-11-05T10:56:08Z

src/config.jl

@@ -0,0 +1,112 @@
+using Scratch


Am I correct in thinking that we want to use Scratch here instead of Preferences because we don't want these settings to be tied to precompilation? Also, let's change this to using Scratch: ... to avoid implicit imports.

Suggested change

using Scratch

using Scratch: @get_scratch!

JamesWrigley · 2025-11-05T11:03:48Z

src/config.jl

+# Returns the stored Jupyter path, or empty string if not set.
+function load_jupyter_preference()
+    # Check new Scratch.jl location first
+    prefsfile = joinpath(@get_scratch!("prefs"), "jupyter")


Could this just be @get_scratch!("jupyter")? IIUC we don't need the extra prefs directory with Scratch.

JamesWrigley · 2025-11-05T11:06:41Z

src/IJulia.jl

 """
 module IJulia
-export notebook, jupyterlab, installkernel
+export notebook, jupyterlab, installkernel, update_jupyter_path


I would not export update_jupyter_path, I expect it's less used than the other functions.

JamesWrigley · 2025-11-05T11:13:59Z

src/config.jl

+# Use Conda Jupyter (empty string triggers auto-detection)
+IJulia.update_jupyter_path("")


I would delete this in favour of having just one way of triggering auto-detection.

You mean only ENV["JUPYTER"] = "" for auto?

stevengj · 2025-11-05T11:58:58Z

docs/src/_changelog.md

+  or update kernels, and [`update_jupyter_path()`](@ref) to configure the
+  Jupyter executable path.
+- IJulia now uses [Scratch.jl](https://github.com/JuliaPackaging/Scratch.jl) to
+  store configuration preferences instead of writing to `DEPOT_PATH/prefs/`.


Why not Preferences.jl?

To be honest, the existence of Preferences.jl completely slipped my mind while I was doing this. It does seem better, I will rework this PR to use it and see how it feels.

Wouldn't that make changing these settings force re-precompilation? I don't think we need that.

They only force re-precompilation if the preferences are read during precompile time. I don't think they are.

Actually... would it be too much of a pain to go back to Scratch.jl? The reason is because I want to add some internal instrumentation (basically @trace_compile) that can be turned on or off, and environment variables don't really work for that since I don't always control the environment the kernel process is launched in. So I'd like to have some persistent setting that doesn't hook into precompilation, which sounds like Scratch.jl.

stevengj · 2025-11-05T12:00:22Z

docs/src/_changelog.md


 ### Changed
+- Removed `Pkg.build("IJulia")` support. IJulia no longer uses a
+  build step for kernel installation. The build-time configuration system has


Can't we still call installkernel() in a build.jl script, so that an external jupyter (or similar) front-end will immediately work after running add IJulia?

We could but if add IJulia runs the build script or not is subject to the state of the file system. If the IJulia version is found in some existing DEPOT it won't build. So it feels kind of brittle to rely on this.

We don't need to re-run installkernel() if it was already run for that Julia version … what about running it during precompilation, which gets triggered on Pkg.add if you add to a new Julia version, even if the IJulia version was already present?

what about running it during precompilation, which gets triggered on Pkg.add if you add to a new Julia version, even if the IJulia version was already present?

My point is sort of that if you have set up IJulia at some point in the past, and then say removed all kernels and reinstalled conda etc, completely botched your existing IJulia setup, IJulia will not necessarily precompile nor build when you run add IJulia.

stevengj · 2025-11-05T12:05:01Z

(This still won't fix external jupyter programs from breaking every time juliaup does a patch update unless the user manually re-installs the kernel, unfortunately. The root of this problem is that juliaup keeps moving the path underneath us, and gives us no official way to tell the actual command that was used to launch julia. JuliaLang/juliaup#857 … but at least we can fix running IJulia.jupyter(), I guess.)

KristofferC · 2025-11-05T12:26:11Z

Just FYI, while doing this I was not aware of the juliaup issue being referenced (and this PR was not meant to address that) but from the comment I can sort of imagine what the problem is. Pkg Apps have the same problem in that we store the absolute path to the Julia executable that installed the path. When juliaup does GC that exe, it's sad times.

My plan to help with the Pkg App problem is by using JuliaLang/juliaup#1315 in combination with a Pkg App specific env variable to override the executable that launch the app to be just julia. The automatic julia version selection will then pick the correct julia version by looking at the manifest, if needed download the needed julia version and launch the app

KristofferC · 2025-11-05T14:27:32Z

Changed to using Preferences and did some updates based on review and own reading of code. I think the update_jupyter_path is quite important to look at now.

KristofferC requested a review from JamesWrigley November 5, 2025 10:30

JamesWrigley requested changes Nov 5, 2025

View reviewed changes

JamesWrigley added the enhancement label Nov 5, 2025

stevengj reviewed Nov 5, 2025

View reviewed changes

KristofferC added 3 commits November 5, 2025 14:46

run jupyterlab and notebook in verbose when ijulia debug is set

051723f

use Preferences.jl instead of Scratch

7fb5ebe

fixes from reivews

e23228b

JamesWrigley mentioned this pull request Nov 5, 2025

Error starting kernel #1213

Closed

		# Use Conda Jupyter (empty string triggers auto-detection)
		IJulia.update_jupyter_path("")

Uh oh!

change from using a build script to auto installing kernels and providing a jupyter path update function #1215

Are you sure you want to change the base?

change from using a build script to auto installing kernels and providing a jupyter path update function #1215

Conversation

KristofferC commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JamesWrigley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KristofferC Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevengj commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KristofferC commented Nov 5, 2025 •

edited

Loading

codecov bot commented Nov 5, 2025 •

edited

Loading

KristofferC Nov 5, 2025 •

edited

Loading

stevengj commented Nov 5, 2025 •

edited

Loading

KristofferC commented Nov 5, 2025 •

edited

Loading