TOML parser error, generation of backup files, and notebook execution hangs

[haakon-e]

The issue with TOML parser error from #25 persists. Starting the server is fine. When I next, click "execute all cells" in the notebook viewer, I get the following messages:

show_logs errors

stack trace

I get ~36k lines of this error, which apparently is too long to create a github gist to share! So this is just the first of many of these:

┌ Error: Failed to parse embedded TOML content
│   cell_id = Base.UUID("d051b6ea-af20-491b-8428-bac6f93da7d5")
│   exception =
│    TOML Parser error:
│    none:3:11 error: key already has a value
│      show_logs = false
│                ^       
│    Stacktrace:
│      [1] parse
│        @ ./toml_parser.jl:445 [inlined]
│      [2] parse
│        @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/TOML/src/TOML.jl:86 [inlined]
│      [3] _read_notebook_collected_cells!(io::IO)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:236
│      [4] load_notebook_nobackup(io::IO, path::AbstractString; skip_nbpkg::Bool)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:336
│      [5] load_notebook_nobackup
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:334 [inlined]
│      [6] #63
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:363 [inlined]
│      [7] open(::Pluto.var"#63#64"{@Kwargs{}, String}, ::String, ::Vararg{String}; kwargs::@Kwargs{})
│        @ Base ./io.jl:410
│      [8] open
│        @ ./io.jl:407 [inlined]
│      [9] #load_notebook_nobackup#61
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:362 [inlined]
│     [10] load_notebook_nobackup
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:361 [inlined]
│     [11] load_notebook(path::String; disable_writing_notebook_files::Bool)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:380
│     [12] load_notebook
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:370 [inlined]
│     [13] (::Pluto.SessionActions.var"#4#5"{Base.UUID, Pluto.ServerSession})()
│        @ Pluto.SessionActions ~/.julia/packages/Pluto/FKi1T/src/webserver/SessionActions.jl:76
│     [14] lock(f::Pluto.SessionActions.var"#4#5"{Base.UUID, Pluto.ServerSession}, l::ReentrantLock)
│        @ Base ./lock.jl:335
│     [15] open(session::Pluto.ServerSession, path::String; execution_allowed::Bool, run_async::Bool, compiler_options::Nothing, as_sample::Bool, risky_file_source::Nothing, clear_frontmatter::Bool, notebook_id::Base.UUID)
│        @ Pluto.SessionActions ~/.julia/packages/Pluto/FKi1T/src/webserver/SessionActions.jl:67
│     [16] open
│        @ ~/.julia/packages/Pluto/FKi1T/src/webserver/SessionActions.jl:49 [inlined]
│     [17] (::Pluto.var"#try_launch_notebook_response#http_router_for##9"{Pluto.var"#try_launch_notebook_response#352#http_router_for##10"{Pluto.ServerSession}})(action::typeof(Pluto.SessionActions.open), path_or_url::String; as_redirect::Bool, title::String, advice::String, home_url::String, action_kwargs::@Kwargs{as_sample::Bool, execution_allowed::Bool, clear_frontmatter::Bool})
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/Router.jl:24
│     [18] (::Pluto.var"#serve_notebookupload#http_router_for##23")(request::HTTP.Messages.Request)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/Router.jl:201
│     [19] (::HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing})(req::HTTP.Messages.Request)
│        @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:439
│     [20] (::Pluto.var"#auth_middleware##0#auth_middleware##1"{HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing}})(request::HTTP.Messages.Request)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/Authentication.jl:102
│     [21] #create_session_context_middleware##1
│        @ ~/.julia/packages/Pluto/FKi1T/src/webserver/Authentication.jl:54 [inlined]
│     [22] (::Pluto.var"#run!##11#run!##12"{Pluto.ServerSession, Pluto.var"#create_session_context_middleware##1#create_session_context_middleware##2"{Pluto.var"#auth_middleware##0#auth_middleware##1"{HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing}}, Pluto.ServerSession}})(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}})
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/WebServer.jl:296
│     [23] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Int64)
│        @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469
│     [24] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{Pluto.var"#run!##11#run!##12"{Pluto.ServerSession, Pluto.var"#create_session_context_middleware##1#create_session_context_middleware##2"{Pluto.var"#auth_middleware##0#auth_middleware##1"{HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing}}, Pluto.ServerSession}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Int64, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})()
│        @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:401
└ @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:238

---

Notebook backups

Interdispersed with these errors, I get a bunch of warnings of this kind:

┌ Warning: Old Pluto notebook might not have loaded correctly. Backup saved to: 
│   backup_path = "/Users/haakon/.julia/pluto_notebooks/Exciting creation backup 1.jl"
└ @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:394
# [...]
┌ Warning: Old Pluto notebook might not have loaded correctly. Backup saved to: 
│   backup_path = "/Users/haakon/.julia/pluto_notebooks/Wild discovery backup 1.jl"
└ @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:394
# [...]
[Worker 5493]: Precompiling packages...
[Worker 5493]: Info Given PlutoRunner was explicitly requested, output will be shown live 
[Worker 5493]: ┌ Warning: Module Markdown with build ID fafbfcfd-9cd6-1538-974a-adb09cbe1f5f is missing from the cache.
[Worker 5493]: │ This may mean Markdown [d6f4376e-aef5-505a-96c1-9c027394607a] does not support precompilation but is imported by a module that does.
[Worker 5493]: └ @ Base loading.jl:2613
[Worker 5493]:     123.8 ms  ? PlutoRunner
┌ Warning: Old Pluto notebook might not have loaded correctly. Backup saved to: 
│   backup_path = "/Users/haakon/.julia/pluto_notebooks/Small conjecture backup 1.jl"
└ @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:394
# [...]
[Worker 5493]: ┌ Warning: Module Markdown with build ID fafbfcfd-9cd6-1538-974a-adb09cbe1f5f is missing from the cache.
[Worker 5493]: │ This may mean Markdown [d6f4376e-aef5-505a-96c1-9c027394607a] does not support precompilation but is imported by a module that does.
[Worker 5493]: └ @ Base loading.jl:2613
┌ Warning: Old Pluto notebook might not have loaded correctly. Backup saved to: 
│   backup_path = "/Users/haakon/.julia/pluto_notebooks/Cute story 1 backup 1.jl"
└ @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:394
# [...]

I can confirm that my ~/.julia/pluto_notebooks/ folder now has ~180 .jl files, with many "duplicates" and "backups:

list of files

"Cute analysis.jl"
 "Cute blueprint 1 backup 1.jl"
 "Cute blueprint 1.jl"
 "Cute blueprint backup 1.jl"
 "Cute blueprint.jl"
 "Cute creation.jl"
 "Cute discovery backup 1.jl"
 "Cute discovery.jl"
 "Cute experiment backup 1.jl"
 "Cute experiment.jl"
 "Cute journal 1 backup 1.jl"
 "Cute journal 1.jl"
 "Cute journal backup 1.jl"
 "Cute journal.jl"
 "Cute report backup 1.jl"
 "Cute report.jl"
 "Cute story 1 backup 1.jl"
 "Cute story 1.jl"
 "Cute story 2 backup 1.jl"
 "Cute story 2.jl"
 "Cute story.jl"
 "Exciting blueprint backup 1.jl"
 "Exciting blueprint.jl"
 "Exciting creation backup 1.jl"
 ⋮
 "Surprising theory.jl"
 "Tiny computation.jl"
 "Tiny conjecture 1 backup 1.jl"
 "Tiny conjecture 1.jl"
 "Tiny conjecture backup 1.jl"
 "Tiny conjecture.jl"
 "Tiny program.jl"
 "Tiny proof backup 1.jl"
 "Tiny proof.jl"
 "Wild blueprint backup 1.jl"
 "Wild blueprint.jl"
 "Wild discovery backup 1.jl"
 "Wild discovery.jl"
 "Wild lecture backup 1.jl"
 "Wild lecture.jl"
 "Wonderful blueprint.jl"
 "Wonderful creation backup 1.jl"
 "Wonderful creation.jl"
 "Wonderful discovery backup 1.jl"
 "Wonderful discovery.jl"
 "Wonderful magic backup 1.jl"
 "Wonderful magic.jl"
 "Wonderful proof backup 1.jl"
 "Wonderful proof.jl"

The notebook I'm running has about 60 cells, not sure if that has any impact on the generation of these files.

---

State of the notebook itself

It also appears that in the time it took to write this issue (about 13 minutes), the notebook has successfully executed one cell, which:

begin
	import Pkg
	dir = expanduser("~/MyProject")
	Pkg.activate(dir)
	using Revise

	# Pluto packages
	import PlutoLinks
	import PlutoHooks
	import PlutoUI
end

and is still running the next cell:

begin
	# Local modules
	_Plt = PlutoLinks.@ingredients(joinpath(dir, "Plotting.jl"))
	Plt = _Plt.Plotting

	# Main package
	using MyPackage

	# Other dependencies
	import TOML, LinearAlgebra, Dates, FFTW, DimensionalData, CairoMakie, Printf
end

[haakon-e]

Another issue related to TOML parsing:

I notice that my notebook now has the cell id as a metadata tag:

# ╔═╡ 9d9a6f05-e39d-431e-ac01-ebaaf048d206
# ╠═╡ show_logs = false
# ╠═╡ pluto_cell_id = 9d9a6f05-e39d-431e-ac01-ebaaf048d206
b=1

This fails to parse, with the error below.

The issue is essentially that:

using TOML
str_fail = """
show_logs = false
pluto_cell_id = 9d9a6f05-e39d-431e-ac01-ebaaf048d206
"""
TOML.parse(str_fail)  # doesn't understand that the id is a string.

# alternatively,
str_success = """
show_logs = false
pluto_cell_id = "9d9a6f05-e39d-431e-ac01-ebaaf048d206"
"""
TOML.parse(str_success) 
# output:
Dict{String, Any} with 2 entries:
  "show_logs"     => false
  "pluto_cell_id" => "9d9a6f05-e39d-431e-ac01-ebaaf048d206"

Full stacktrace:

┌ Error: Failed to parse embedded TOML content
│   cell_id = Base.UUID("7142201c-ed1f-4edf-af77-ef612d31e6ca")
│   exception =
│    TOML Parser error:
│    none:1:24 error: failed to parse value
│      pluto_cell_id = 7142201c-ed1f-4edf-af77-ef612d31e6ca
│                             ^                             
│    Stacktrace:
│      [1] parse
│        @ ./toml_parser.jl:445 [inlined]
│      [2] parse
│        @ ~/.julia/juliaup/julia-1.12.1+0.aarch64.apple.darwin14/share/julia/stdlib/v1.12/TOML/src/TOML.jl:86 [inlined]
│      [3] _read_notebook_collected_cells!(io::IO)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:236
│      [4] load_notebook_nobackup(io::IO, path::AbstractString; skip_nbpkg::Bool)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:336
│      [5] load_notebook_nobackup
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:334 [inlined]
│      [6] #63
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:363 [inlined]
│      [7] open(::Pluto.var"#63#64"{@Kwargs{}, String}, ::String, ::Vararg{String}; kwargs::@Kwargs{})
│        @ Base ./io.jl:410
│      [8] open
│        @ ./io.jl:407 [inlined]
│      [9] #load_notebook_nobackup#61
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:362 [inlined]
│     [10] load_notebook_nobackup
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:361 [inlined]
│     [11] load_notebook(path::String; disable_writing_notebook_files::Bool)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:380
│     [12] load_notebook
│        @ ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:370 [inlined]
│     [13] (::Pluto.SessionActions.var"#4#5"{Base.UUID, Pluto.ServerSession})()
│        @ Pluto.SessionActions ~/.julia/packages/Pluto/FKi1T/src/webserver/SessionActions.jl:76
│     [14] lock(f::Pluto.SessionActions.var"#4#5"{Base.UUID, Pluto.ServerSession}, l::ReentrantLock)
│        @ Base ./lock.jl:335
│     [15] open(session::Pluto.ServerSession, path::String; execution_allowed::Bool, run_async::Bool, compiler_options::Nothing, as_sample::Bool, risky_file_source::Nothing, clear_frontmatter::Bool, notebook_id::Base.UUID)
│        @ Pluto.SessionActions ~/.julia/packages/Pluto/FKi1T/src/webserver/SessionActions.jl:67
│     [16] open
│        @ ~/.julia/packages/Pluto/FKi1T/src/webserver/SessionActions.jl:49 [inlined]
│     [17] (::Pluto.var"#try_launch_notebook_response#http_router_for##9"{Pluto.var"#try_launch_notebook_response#352#http_router_for##10"{Pluto.ServerSession}})(action::typeof(Pluto.SessionActions.open), path_or_url::String; as_redirect::Bool, title::String, advice::String, home_url::String, action_kwargs::@Kwargs{as_sample::Bool, execution_allowed::Bool, clear_frontmatter::Bool})
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/Router.jl:24
│     [18] (::Pluto.var"#serve_notebookupload#http_router_for##23")(request::HTTP.Messages.Request)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/Router.jl:201
│     [19] (::HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing})(req::HTTP.Messages.Request)
│        @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:439
│     [20] (::Pluto.var"#auth_middleware##0#auth_middleware##1"{HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing}})(request::HTTP.Messages.Request)
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/Authentication.jl:102
│     [21] #create_session_context_middleware##1
│        @ ~/.julia/packages/Pluto/FKi1T/src/webserver/Authentication.jl:54 [inlined]
│     [22] (::Pluto.var"#run!##11#run!##12"{Pluto.ServerSession, Pluto.var"#create_session_context_middleware##1#create_session_context_middleware##2"{Pluto.var"#auth_middleware##0#auth_middleware##1"{HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing}}, Pluto.ServerSession}})(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}})
│        @ Pluto ~/.julia/packages/Pluto/FKi1T/src/webserver/WebServer.jl:296
│     [23] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Int64)
│        @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469
│     [24] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{Pluto.var"#run!##11#run!##12"{Pluto.ServerSession, Pluto.var"#create_session_context_middleware##1#create_session_context_middleware##2"{Pluto.var"#auth_middleware##0#auth_middleware##1"{HTTP.Handlers.Router{typeof(Pluto.default_404_response), typeof(HTTP.Handlers.default405), Nothing}}, Pluto.ServerSession}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Int64, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})()
│        @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:401
└ @ Pluto ~/.julia/packages/Pluto/FKi1T/src/notebook/saving and loading.jl:238

[pankgeorg]

Oh no, thanks for catching these!

I just addressed the TOML serialization here. I'm not sure why it hangs, but I proactively removed the JULIA_CPU_TARGET=generic that the extension used to have. Should make it at least a little bit faster.

The issue with TOML parser error from #25 persists.

The patch will not fix the current notebooks unfortunately; both show_logs = false and pluto_cell_id entries need to disappear manually 😭.

I noticed Markdown is not precompiled; are you in 1.12 by any chance? I'm not sure what the behaviour is there, I'll try to find some time to check.

[haakon-e]

I just addressed the TOML serialization here. I'm not sure why it hangs, but I proactively removed the JULIA_CPU_TARGET=generic that the extension used to have. Should make it at least a little bit faster.

Thanks! Is the actual source code hosted somewhere else, by chance? just curious :)

The issue with TOML parser error from #25 persists.

The patch will not fix the current notebooks unfortunately; both show_logs = false and pluto_cell_id entries need to disappear manually 😭.

No problem! I just removed those tags manually (which is very easy with multiple cursors in vscode!)

I noticed Markdown is not precompiled; are you in 1.12 by any chance? I'm not sure what the behaviour is there, I'll try to find some time to check.

Yes -- that's correct. Do you expect this extension to be more stable on 1.11? I'm happy to downgrade until 1.12 has been tested more extensively.