Segfaults when loading packages concurrently

[JamesWrigley]

EDIT: the JLL stuff was a red herring, see #59219 (comment).

I'm hitting some segfaults on 1.12 RC1 and latest nightly when loading pkgimages for packages that use a JLL. Here's an MWE:

module Foo

import ZeroMQ_jll

end # module Foo

If I put that in a project with ZeroMQ_jll and try importing it I get:

$ julia +1.12 --project -E 'import Foo'
Precompiling Foo finished.
  1 dependency successfully precompiled in 1 seconds. 7 already precompiled.
nothing

Which is expected. But afterwards I consistently get a segfault:

$ julia +1.12 --project -E 'import Foo'

[980142] signal 11 (1): Segmentation fault
in expression starting at none:1
jl_to_typeof at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:974 [inlined]
must_be_new_dt at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata_utils.c:18
must_be_new_dt at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata_utils.c:64
jl_restore_system_image_from_stream_ at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4048
jl_restore_package_image_from_stream at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4418
jl_restore_incremental_from_buf at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4487
ijl_restore_package_image_from_file at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4554
#_include_from_serialized#863 at ./loading.jl:1270
_include_from_serialized at ./loading.jl:1246 [inlined]
#_require_search_from_serialized#879 at ./loading.jl:2070
_require_search_from_serialized at ./loading.jl:1977
jfptr__require_search_from_serialized_89198.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
__require_prelocked at ./loading.jl:2595
_require_prelocked at ./loading.jl:2461
macro expansion at ./loading.jl:2389 [inlined]
macro expansion at ./lock.jl:376 [inlined]
__require at ./loading.jl:2354
jfptr___require_109206.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:2374 [inlined]
jl_f_invoke_in_world at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/builtins.c:894
require at ./loading.jl:2330
jfptr_require_109941.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:2374 [inlined]
call_require at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:494 [inlined]
eval_import_path at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:532
jl_toplevel_eval_flex at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:886
jl_toplevel_eval_flex at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:975
ijl_toplevel_eval at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:1047
ijl_toplevel_eval_in at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:1092
eval at ./boot.jl:489
exec_options at ./client.jl:289
_start at ./client.jl:554
jfptr__start_76350.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:2374 [inlined]
true_main at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/jlapi.c:971
jl_repl_entrypoint at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/jlapi.c:1139
main at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/cli/loader_exe.c:58
unknown function (ip: 0x7ff313e3feaf) at /lib64/libc.so.6
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S
Allocations: 1 (Pool: 1; Big: 0); GC: 0
[1]    980142 segmentation fault (core dumped)  julia +1.12 --project -E 'import Foo'

However importing the package from a REPL session works fine, which is quite confusing 🤔 As far as I can tell this only happens with JLL dependencies. I also tried CImGuiPack_jll and perfetto_jll in the MWE to make sure it wasn't some weird problem with ZeroMQ_jll. Discovered by running IJulia, which launches kernels like this.

[giordano]

I don't think many people ran into this, otherwise we would have heard complaints about this issue much earlier, so you'll have to provide much more information to move this forward.

Do you get more details with --check-bounds=yes? I wouldn't expect so (error seems to come from the runtime?), but this is quick and easy to check. Also, can you do an assert build and see if the segfault is caught by an assertion? In any case running a debug build under gdb and reporting your discoveries may be helpful.

[JamesWrigley]

Ok, but does the MWE not reproduce the issue for you?

[JamesWrigley]

The rabbit hole deepens 🐙 --check-bounds=yes gives a slightly different error message:

➜  Foo git:(xfel) julia +1.12 --check-bounds=yes --project -E 'import Foo'                                                                                                                                                                     
double free or corruption (!prev)                                                                                                                                                                                                              
                                                                                                                                                                                                                                               
[3728760] signal 6 (-6): Aborted                                                                                                                                                                                                               
in expression starting at none:1                                                                                                                                                                                                               
unknown function (ip: 0x7f25cd2a156c) at /lib64/libc.so.6                                                                                                                                                                                      
raise at /lib64/libc.so.6 (unknown line)                                                                                                                                                                                                       
abort at /lib64/libc.so.6 (unknown line)                                                                                                                                                                                                       
unknown function (ip: 0x7f25cd22912f) at /lib64/libc.so.6                                                                                                                                                                                      
unknown function (ip: 0x7f25cd2ab636) at /lib64/libc.so.6                                                                                                                                                                                      
unknown function (ip: 0x7f25cd2ad32b) at /lib64/libc.so.6                                                                                                                                                                                      
unknown function (ip: 0x7f25cd2aeefa) at /lib64/libc.so.6                                                                                                                                                                                      
realloc at /lib64/libc.so.6 (unknown line)                                                                                                                                                                                                     
arraylist_grow at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/support/arraylist.c:58                                                                                                                                    
arraylist_push at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/support/arraylist.c:69                                                                                                                                    
jl_read_symbols at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:2046 [inlined]                                                                                                                              
jl_restore_system_image_from_stream_ at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:3949                                                                                                                   
jl_restore_package_image_from_stream at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4418                                                                                                                   
jl_restore_incremental_from_buf at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4487                                                                                                                        
ijl_restore_package_image_from_file at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/staticdata.c:4554                                                                                                                    
#_include_from_serialized#863 at ./loading.jl:1270                                                                                                                                                                                             
_include_from_serialized at ./loading.jl:1246 [inlined]                                                                                                                                                                                        
#_require_search_from_serialized#879 at ./loading.jl:2070                                                                                                                                                                                      
_require_search_from_serialized at ./loading.jl:1977                                                                                                                                                                                           
jfptr__require_search_from_serialized_89198.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)                                                                                                   
__require_prelocked at ./loading.jl:2595                                                                                                                                                                                                       
_require_prelocked at ./loading.jl:2461                                                                                                                                                                                                        
macro expansion at ./loading.jl:2389 [inlined]                                                                                                                                                                                                 
macro expansion at ./lock.jl:376 [inlined]                                                                                                                                                                                                     
__require at ./loading.jl:2354                                                                                                                                                                                                                 
jfptr___require_109206.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)                                                                                                                        
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:2374 [inlined]                                                                                                                                          
jl_f_invoke_in_world at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/builtins.c:894                                                                                                                                      
require at ./loading.jl:2330                                                                                                                                                                                                                   
jfptr_require_109941.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)                                                                                                                          
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:2374 [inlined]                                                                                                                                          
call_require at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:494 [inlined]                                                                                                                                    
eval_import_path at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:532                                                                                                                                          
jl_toplevel_eval_flex at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:886                                                                                                                                     
jl_toplevel_eval_flex at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:975                                                                                                                                     
ijl_toplevel_eval at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:1047                                                                                                                                        
ijl_toplevel_eval_in at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/toplevel.c:1092                                                                                                                                     
eval at ./boot.jl:489                                                                                                                                                                                                                          
exec_options at ./client.jl:289                                                                                                                                                                                                                
_start at ./client.jl:554                                                                                                                                                                                                                      
jfptr__start_76350.1 at /home/james/.julia/juliaup/julia-1.12.0-rc1+0.x64.linux.gnu/lib/julia/sys.so (unknown line)                                                                                                                            
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/julia.h:2374 [inlined]
true_main at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/jlapi.c:971
jl_repl_entrypoint at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/src/jlapi.c:1139
main at /cache/build/tester-amdci4-11/julialang/julia-release-1-dot-12/cli/loader_exe.c:58
unknown function (ip: 0x7f25cd23feaf) at /lib64/libc.so.6
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S
Allocations: 1 (Pool: 1; Big: 0); GC: 0
[1]    3728760 IOT instruction (core dumped)  julia +1.12 --check-bounds=yes --project -E 'import Foo'

But when playing around with the arguments I found that --startup-file=no fixes it, and this line in my startup.jl is the real culprit: Threads.@spawn @eval using Revise
If I change it to using Revise or Threads.@spawn @eval 1 everything works. So I guess it's some kind of race condition when loading packages? Will try with a debug build.

[JamesWrigley]

With an assert build and Threads.@spawn @eval using Distributed (Revise didn't trigger it on this particular node):

$ ~/src/julia/julia --project -E 'import Foo'
julia: /home/wrigleyj/src/julia/src/staticdata.c:2138: get_item_for_reloc: Assertion `offset < deser_sym.len && deser_sym.items[offset] && "corrupt relocation item id"' failed.

[235742] signal 6 (-6): Aborted
in expression starting at none:1
__pthread_kill_implementation at /lib64/libc.so.6 (unknown line)
raise at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__assert_fail_base.cold at /lib64/libc.so.6 (unknown line)
__assert_fail at /lib64/libc.so.6 (unknown line)
get_item_for_reloc at /home/wrigleyj/src/julia/src/staticdata.c:2138
jl_read_reloclist at /home/wrigleyj/src/julia/src/staticdata.c:2287
jl_restore_system_image_from_stream_ at /home/wrigleyj/src/julia/src/staticdata.c:3970
jl_restore_package_image_from_stream at /home/wrigleyj/src/julia/src/staticdata.c:4429
jl_restore_incremental_from_buf at /home/wrigleyj/src/julia/src/staticdata.c:4498
ijl_restore_package_image_from_file at /home/wrigleyj/src/julia/src/staticdata.c:4565
#_include_from_serialized#882 at ./loading.jl:1271
_include_from_serialized at ./loading.jl:1247 [inlined]
#_require_search_from_serialized#898 at ./loading.jl:2076
_require_search_from_serialized at ./loading.jl:1983
jfptr__require_search_from_serialized_50681.5 at /home/wrigleyj/src/julia/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /home/wrigleyj/src/julia/src/gf.c:3999 [inlined]
ijl_apply_generic at /home/wrigleyj/src/julia/src/gf.c:4212
__require_prelocked at ./loading.jl:2602
_require_prelocked at ./loading.jl:2468
macro expansion at ./loading.jl:2396 [inlined]
macro expansion at ./lock.jl:376 [inlined]
__require at ./loading.jl:2360
jfptr___require_49842.5 at /home/wrigleyj/src/julia/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /home/wrigleyj/src/julia/src/gf.c:3999 [inlined]
ijl_apply_generic at /home/wrigleyj/src/julia/src/gf.c:4212
jl_apply at /home/wrigleyj/src/julia/src/julia.h:2375 [inlined]
jl_f_invoke_in_world at /home/wrigleyj/src/julia/src/builtins.c:896
require at ./loading.jl:2336 [inlined]
eval_import_path at ./module.jl:36 [inlined]
_eval_import at ./module.jl:111
jfptr__eval_import_30845.5 at /home/wrigleyj/src/julia/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /home/wrigleyj/src/julia/src/gf.c:3999 [inlined]
ijl_apply_generic at /home/wrigleyj/src/julia/src/gf.c:4212
jl_apply at /home/wrigleyj/src/julia/src/julia.h:2375 [inlined]
do_call at /home/wrigleyj/src/julia/src/interpreter.c:123
eval_value at /home/wrigleyj/src/julia/src/interpreter.c:243
eval_stmt_value at /home/wrigleyj/src/julia/src/interpreter.c:194 [inlined]
eval_body at /home/wrigleyj/src/julia/src/interpreter.c:708
jl_interpret_toplevel_thunk at /home/wrigleyj/src/julia/src/interpreter.c:899
jl_toplevel_eval_flex at /home/wrigleyj/src/julia/src/toplevel.c:773
jl_toplevel_eval_flex at /home/wrigleyj/src/julia/src/toplevel.c:713
ijl_toplevel_eval at /home/wrigleyj/src/julia/src/toplevel.c:785
ijl_toplevel_eval_in at /home/wrigleyj/src/julia/src/toplevel.c:830
eval at ./boot.jl:489
exec_options at ./client.jl:288
_start at ./client.jl:553
jfptr__start_55948.5 at /home/wrigleyj/src/julia/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /home/wrigleyj/src/julia/src/gf.c:3999 [inlined]
ijl_apply_generic at /home/wrigleyj/src/julia/src/gf.c:4212
jl_apply at /home/wrigleyj/src/julia/src/julia.h:2375 [inlined]
true_main at /home/wrigleyj/src/julia/src/jlapi.c:971
jl_repl_entrypoint at /home/wrigleyj/src/julia/src/jlapi.c:1138
main at /home/wrigleyj/src/julia/cli/loader_exe.c:58
__libc_start_call_main at /lib64/libc.so.6 (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /home/wrigleyj/src/julia/julia (unknown line)
Allocations: 1 (Pool: 1; Big: 0); GC: 0
zsh: IOT instruction (core dumped)  ~/src/julia/julia --project -E 'import Foo'

[giordano]

but does the MWE not reproduce the issue for you?

No.

But I do get a segfault by adding

Threads.@spawn @eval using Distributed

to my config/startup.jl file and starting the REPL, without having to load modules or such.

[JamesWrigley]

Hmm interesting, that doesn't do it for me.

[DilumAluthge]

[disregard this comment]

[nsajko]

But I do get a segfault by adding

OK, so I guess this can be closed and the moral of the story is, don't do Threads.@spawn @eval using SomePackage? Seems like an obvious race condition that would cause crashes or worse.

[nsajko]

Even if I'm wrong and concurrent package loading is supposed to be supported, that seems like it should go into a new issue here on Github.

[JamesWrigley]

OK, so I guess this can be closed and the moral of the story is, don't do Threads.@spawn @eval using SomePackage? Seems like an obvious race condition that would cause crashes or worse.

It doesn't seem obvious at all to me (it's worked fine on previous Julia versions), and if it is meant to be unsupported then I'd say it should be documented or a lock put around loading. A segfault is impossible to recover from as a user.

Even if I'm wrong and concurrent package loading is supposed to be supported, that seems like it should go into a new issue here on Github.

...or just rename the current issue? 🙃

[KristofferC]

I think this can be opened a bit more