TensorOperations does not precompile

[johannesnauta]

I am not sure what the cause of this is, hence the rather vague title. But, in short, I have found that TensorOperations.jl simply does not precompile on some of my remotes. I have encountered this issue when updating everything to julia version 1.11.4. This does not seem to depend on other dependencies, as even in a fresh project with only TensorOperations.jl, the precompilation step simply fails. I probably has to do with some architectural things, as it precompiles just fine on my own pc, and it also fails precompilation on julia version 1.10.9. Finally, it also appears to use up most available memory when inspecting with htop, so perhaps it is a memory leak somewhere during precompilation?

To reproduce

julia> import Pkg; Pkg.generate("tensor")
shell> cd tensor
pkg> activate .
(tensor) pkg> activate .
  Activating project at `~/test/tensor`
pkg> add TensorOperations
(tensor) pkg> add TensorOperations
   Resolving package versions...
      Compat entries added for TensorOperations
    Updating `~/test/tensor/Project.toml`
  [6aa20fa7] + TensorOperations v5.2.0
    Updating `~/test/tensor/Manifest.toml`
  [8ac3fa9e] + LRUCache v1.6.2
  [65ce6f38] + PackageExtensionCompat v1.0.2
  [43287f4e] + PtrArrays v1.3.0
  [5e0ebb24] + Strided v2.3.0
  [4db3bf67] + StridedViews v0.4.1
  [6aa20fa7] + TensorOperations v5.2.0
  [9d95972d] + TupleTools v1.6.0
  [409d34a3] + VectorInterface v0.5.0
Precompiling project...
  ✗ TensorOperations
  ✗ TensorOperations → TensorOperationscuTENSORExt
  2 dependencies successfully precompiled in 249 seconds. 111 already precompiled.
  2 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

julia> err
PkgPrecompileError: The following 2 direct dependencies failed to precompile:

TensorOperations

Failed to precompile TensorOperations [6aa20fa7-93e2-5fca-9bc0-fbd0db3c71a2] to "/home/ubuntu/.julia/compiled/v1.11/TensorOperations/jl_mmpeGV".

TensorOperationscuTENSORExt

Failed to precompile TensorOperationscuTENSORExt [5a347f6a-6589-5393-bbed-383ac4730ab8] to "/home/ubuntu/.julia/compiled/v1.11/TensorOperationscuTENSORExt/jl_KP0QDK".
ERROR: LoadError: Failed to precompile TensorOperations [6aa20fa7-93e2-5fca-9bc0-fbd0db3c71a2] to "/home/ubuntu/.julia/compiled/v1.11/TensorOperations/jl_n7tHiL".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, loadable_exts::Nothing)
    @ Base ./loading.jl:3174
  [3] (::Base.var"#1110#1111"{Base.PkgId})()
    @ Base ./loading.jl:2579
  [4] mkpidlock(f::Base.var"#1110#1111"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.4+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:95
  [5] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.11.4+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.11.4+0.x64.linux.gnu/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:116
  [7] #invokelatest#2
    @ ./essentials.jl:1057 [inlined]
  [8] invokelatest
    @ ./essentials.jl:1052 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1110#1111"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:3698
 [10] maybe_cachefile_lock
    @ ./loading.jl:3695 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2565
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2388
 [13] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2375
 [16] macro expansion
    @ ./loading.jl:2314 [inlined]
 [17] macro expansion
    @ ./lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2271
 [19] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [20] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2260
 [22] include
    @ ./Base.jl:557 [inlined]
 [23] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2881
 [24] top-level scope
    @ stdin:6
in expression starting at /home/ubuntu/.julia/packages/TensorOperations/jVddt/ext/TensorOperationscuTENSORExt.jl:1
in expression starting at stdin:6

And some other information that may be helpful:

$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          48 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   8
  On-line CPU(s) list:    0-7
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 7413 24-Core Processor
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy
...

Please let me know any other information that may be necessary to resolve this issue.

[lkdvos]

Thanks for reporting this!

This is indeed somewhat mysterious, and I can't say I immediately know what's going on.
I'm a bit confused that the cuTENSOR extension is being triggered for precompilation though, since that shouldn't automatically happen.
Is it possible that you have cuTENSOR in your global environment (or any other shared environment)?
This might also explain why it is happening on some machines and not on others.

Unfortunately for me the precompilation seems to run fine, both without and with CUDA and cuTENSOR, so I'm just guessing a bit about what could be happening and can only provide some things to possibly check, for which I apologize.
The latest release does have some precompilation support, but there are no precompile statements in the cuTENSOR extension so I'm a bit confused by that too.

• Could you verify that my assumption is correct and the errors are happening when you have cuTENSOR in the global environment?
• Does removing cuTENSOR make everything compile? Ie, does TensorOperations compile on its own, without that extension?
• Do cuTENSOR and CUDA compile on their own, ie without TensorOperations?
• Is this something that recently changed? Is there a version of TensorOperations that does compile? You can verify this by for example pkg> add TensorOperations@5.0.0 or pkg> add TensorOperations@5.1.0 to fix the versions.

[johannesnauta]

Thank you for the swift reply;

• I do not have cuTENSOR in the global environment. In fact, I just now did a fresh install of Julia, and then the above steps still lead to the same results (i.e., TensorOperations.jl not compiling).
• Removing cuTENSOR does not make TensorOperations precompile
• cuTENSOR and CUDA both precompile.
• This is indeed something that recently changed (but I only noticed after a ] update). It appears that TensorOperations@5.1.4 does precompile, as do older version I tested, yet 5.2.0 does not.

I knew I had forgotten something, but I indeed did not try other versions of TensorOperations.jl. Yet, it seems that something did happen in the update to 5.2.0.

[lkdvos]

Do the error messages in these cases still mention the cuTENSOR extension? I might be mistaken by this, but I somehow assumed package extensions wouldn't be precompiled if the project cannot include their dependencies.

This being said, it sounds like it might not even be the cuTENSOR extension, and simply the main package.

I think the following snippet can be used to globally disable precompilation:

using PrecompileTools, Preferences
set_preferences!(PrecompileTools, "precompile_workloads" => false; force=true)

This should be respected by TensorOperations as well, so this might make it so you can get past the precompile stage, and try to see if the code actually runs.
I'm hoping that if this would error, the messages might give more indications as to what is going on.

It might be useful to try and run the code that is in the precompile.jl file manually, i.e. replace the if PrecompileTools.workload_enabled(@__MODULE__) with a let block and see if that throws any errors.

[Jutho]

Is the error spawned quickly or only after a long time trying to compile? I am currently seeing also that precompilation for TensorOperations has been running for over 30 minutes on a remote linux machine, and I don't know if it will complete or fail, but clearly this should not happen either way.

[lkdvos]

The precompilation takes that long when StridedView is not updated, since the type "normalization" happens there. In principle we can disable precompilation here if that version is not found?

[Jutho]

I've just checked on my remote machine, and StridedViews was the latest version, so I don't know what caused the long/stalled compilation (I killed the process).

[dfelmanl]

I just had the same issue and solved it my creating a LocalPreferences.toml file in the main directory with the following text:

[TensorOperations]
precompile_workload = false

[lkdvos]

@Jutho , since I still don't have any clue what is going on, how do you feel about disabling the precompilation by default, and making this an opt-in feature instead?