Datadep problem

[andrew-saydjari]

On a clean install, I have now repeatedly had issues unlinking during the MIST_DECam download. I don't think this is my fault... seems to be some issue with the specific way this datadep is being downloaded. (Fixed on the development version, which might suggest a new release soon would be helpful)

This program has requested access to the data dependency MIST_DECam.
which is not currently installed. It can be installed automatically, and you will not see this message again.

MIST bolometric corrections for the Dark Energy Camera (DECam)


Do you want to download the dataset from https://waps.cfa.harvard.edu/MIST/BC_tables/DECam.txz to "/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/scratchspaces/124859b0-ceae-595e-8997-d05f6a7a8dfe/datadeps/MIST_DECam"?
[y/n]
stdin>  y
┌ Info: Downloading
│   source = "https://waps.cfa.harvard.edu/MIST/BC_tables/DECam.txz"
│   dest = "/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/scratchspaces/124859b0-ceae-595e-8997-d05f6a7a8dfe/datadeps/MIST_DECam/DECam.txz"
│   progress = 1.0
│   time_taken = "0.6 s"
│   time_remaining = "0.0 s"
│   average_speed = "3.210 MiB/s"
│   downloaded = "1.935 MiB"
│   remaining = "0 bytes"
└   total = "1.935 MiB"

IOError: unlink("/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/scratchspaces/124859b0-ceae-595e-8997-d05f6a7a8dfe/datadeps/MIST_DECam/files/.nfs0000008708961c62000510dc"): resource busy or locked (EBUSY)

Stacktrace:
  [1] uv_error
    @ ./libuv.jl:106 [inlined]
  [2] unlink(p::String)
    @ Base.Filesystem ./file.jl:1105
  [3] rm(path::String; force::Bool, recursive::Bool)
    @ Base.Filesystem ./file.jl:283
  [4] rm(path::String; force::Bool, recursive::Bool) (repeats 2 times)
    @ Base.Filesystem ./file.jl:294
  [5] rm
    @ ./file.jl:273 [inlined]
  [6] download(datadep::DataDeps.DataDep, localdir::String; remotepath::String, i_accept_the_terms_of_use::Nothing, skip_checksum::Bool)
    @ DataDeps ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution_automatic.jl:86
  [7] download(datadep::DataDeps.DataDep, localdir::String)
    @ DataDeps ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution_automatic.jl:63
  [8] handle_missing
    @ ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution_automatic.jl:10 [inlined]
  [9] _resolve
    @ ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution.jl:83 [inlined]
 [10] resolve(datadep::DataDeps.DataDep, inner_filepath::String, calling_filepath::String)
    @ DataDeps ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution.jl:29
 [11] resolve(datadep_name::String, inner_filepath::String, calling_filepath::String)
    @ DataDeps ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution.jl:54
 [12] resolve
    @ ~/scratch/.julia/packages/DataDeps/Y2lje/src/resolution.jl:73 [inlined]
 [13] MISTBCGrid(grid::String)
    @ BolometricCorrections.MIST ~/scratch/.julia/packages/BolometricCorrections/Tef8p/src/MIST/MIST.jl:224
 [14] top-level scope
    @ In[6]:1

[andrew-saydjari]

Also, on the development version, just trying to do using BolometricCorrections leads to an error in downloading the YBC table.

[ Info: Precompiling BolometricCorrections [37ee4594-8306-40da-a4c2-d7ed95560e0b] (cache misses: wrong dep version loaded (2), wrong source (2))
[ Info: Initializing YBC repository in /uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/scratchspaces/37ee4594-8306-40da-a4c2-d7ed95560e0b/YBC
Cloning into 'ybc_tables'...
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
	refs

InitError: failed process: Process(setenv(`/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/artifacts/ee9b2c2f64d0c71226fea0a244d017d15da0447c/bin/git -C /uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/scratchspaces/37ee4594-8306-40da-a4c2-d7ed95560e0b/YBC/ybc_tables lfs install --local`,["__LMOD_REF_COUNT_PATH=/uufs/chpc.utah.edu/sys/bin:1;/usr/share/Modules/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/uufs/chpc.utah.edu/common/home/u6039752/.dotnet/tools:1;/uufs/notchpeak.peaks/sys/installdir/slurm/std/bin:1", "__LMOD_SET_FPATH=1", "_CE_M=", "PATH=/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/artifacts/ee9b2c2f64d0c71226fea0a244d017d15da0447c/bin:/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/artifacts/7b46720b034a5501393d9cbb503824d984b6c3d0/bin:/uufs/chpc.utah.edu/common/home/u6039752/software/pkg/miniconda/3.9.7/bin:/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.9/condabin:/uufs/chpc.utah.edu/common/home/u6039752/.local/bin:/uufs/chpc.utah.edu/common/home/u6039752/software:/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.9/bin:/uufs/chpc.utah.edu/sys/installdir/git/2.10.0-c7/bin:/uufs/chpc.utah.edu/common/home/u6039752/scratch/software/.juliaup/bin:/uufs/chpc.utah.edu/sys/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/uufs/chpc.utah.edu/common/home/u6039752/.dotnet/tools:/uufs/notchpeak.peaks/sys/installdir/slurm/std/bin:/uufs/chpc.utah.edu/common/home/u6039752/bin", "LMOD_ROOT=/uufs/chpc.utah.edu/sys/installdir/lmod", "OSVER=r8.10", "LD_LIBRARY_PATH=/uufs/chpc.utah.edu/common/home/sdss50/sdsswork/users/u6039752/.julia/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia:/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/artifacts/96bc7b9ddb319d1a051535c85184aa2bf42dbd8b/lib:/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/artifacts/1c2814ca8577e0356a6ecf8447364c19e97b86a8/lib:/uufs/chpc.utah.edu/common/home/u6039752/scratch/.julia/artifacts/5acd766faaca59c3c1f3cfa67e2bf6dcf1e3e883/lib:/uufs/notchpeak.peaks/sys/installdir/slurm/std/lib", "OSREL=Rocky", "TMUX=/tmp/tmux-1547799/default,2827895,0", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1547799/bus"  …  "LMOD_SETTARG_FULL_SUPPORT=no", "CONDA_PREFIX=/uufs/chpc.utah.edu/common/home/u6039752/software/pkg/miniconda/3.9.7", "XDG_SESSION_ID=2946", "KDEDIRS=/usr", "TMOUT=0", "GIT_SSL_CAINFO=/uufs/chpc.utah.edu/common/home/sdss50/sdsswork/users/u6039752/.julia/juliaup/julia-1.11.3+0.x64.linux.gnu/share/julia/cert.pem", "MODULES_INIT=\$MODULESHOME/init/bash", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:", "BASH_FUNC__module_raw%%=() {  unset _mlshdbg;\n if [ \"\${MODULES_SILENT_SHELL_DEBUG:-0}\" = '1' ]; then\n case \"\$-\" in \n *v*x*)\n set +vx;\n _mlshdbg='vx'\n ;;\n *v*)\n set +v;\n _mlshdbg='v'\n ;;\n *x*)\n set +x;\n _mlshdbg='x'\n ;;\n *)\n _mlshdbg=''\n ;;\n esac;\n fi;\n unset _mlre _mlIFS;\n if [ -n \"\${IFS+x}\" ]; then\n _mlIFS=\$IFS;\n fi;\n IFS=' ';\n for _mlv in \${MODULES_RUN_QUARANTINE:-};\n do\n if [ \"\${_mlv}\" = \"\${_mlv##*[!A-Za-z0-9_]}\" -a \"\${_mlv}\" = \"\${_mlv#[0-9]}\" ]; then\n if [ -n \"`eval 'echo \${'\$_mlv'+x}'`\" ]; then\n _mlre=\"\${_mlre:-}\${_mlv}_modquar='`eval 'echo \${'\$_mlv'}'`' \";\n fi;\n _mlrv=\"MODULES_RUNENV_\${_mlv}\";\n _mlre=\"\${_mlre:-}\${_mlv}='`eval 'echo \${'\$_mlrv':-}'`' \";\n fi;\n done;\n if [ -n \"\${_mlre:-}\" ]; then\n eval `eval \${_mlre} /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '\"\$@\"'`;\n else\n eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash \"\$@\"`;\n fi;\n _mlstatus=\$?;\n if [ -n \"\${_mlIFS+x}\" ]; then\n IFS=\$_mlIFS;\n else\n unset IFS;\n fi;\n unset _mlre _mlv _mlrv _mlIFS;\n if [ -n \"\${_mlshdbg:-}\" ]; then\n set -\$_mlshdbg;\n fi;\n unset _mlshdbg;\n return \$_mlstatus\n}", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]

during initialization of module YBC

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:598 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:513
  [3] run
    @ ./process.jl:510 [inlined]
  [4] _ybc_path(path::String)
    @ BolometricCorrections.YBC ~/scratch/.julia/packages/BolometricCorrections/Y902t/src/YBC/init.jl:59
  [5] __init__()
    @ BolometricCorrections.YBC ~/scratch/.julia/packages/BolometricCorrections/Y902t/src/YBC/init.jl:17
  [6] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1378
  [7] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1366
  [8] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Bool; register::Bool)
    @ Base ./loading.jl:1254
  [9] _include_from_serialized
    @ ./loading.jl:1210 [inlined]
 [10] _tryrequire_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String)
    @ Base ./loading.jl:1963
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2594
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2388
 [13] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2375
 [16] macro expansion
    @ ./loading.jl:2314 [inlined]
 [17] macro expansion
    @ ./lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2271
 [19] #invoke_in_world#3
    @ ./essentials.jl:1089 [inlined]
 [20] invoke_in_world
    @ ./essentials.jl:1086 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2260

[andrew-saydjari]

MIST_LSST works without error for me... (Fixed on the development version)

[cgarling]

The new YBC implementation relies on this PR for git-lfs support which is not merged yet. It should work if you have a system git-lfs install (i.e., if git-lfs is on your PATH). I am planning to release after merging the YBC PR.

Just to clarify, are you saying MIST_LSST works for you but MIST_DECam does not on the development version?

[andrew-saydjari]

Sorry, that was a confusing train of comments. I had lots of issues with the download on the registry version. When I switched to the development version, both MIST_LSST and MIST_DECam worked and my only remaining issue was the lfs error you said is being addressed.

[cgarling]

Closing with #14, if still an issue please reopen