Trivial Changes to `spack.yaml` Require Rebuild of Some Packages

[CodeGat]

Currently, doing trivial modifications to the spack.yaml (seen most prominently in #5) require rebuilding parallelio, esmf and (therefore) access-om3-nuopc. This is a bit of a time sink since esmf can take upwards of 1/2 an hour.

See the following runs, in which nothing was changed relating to the above packages: https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/10987685371/job/30502929361?pr=5#step:8:767, https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/10985399681/job/30497188155#step:8:773.

Determine if the current spack.packages.*.require statements are not restrictive enough to prevent a rebuild - https://github.com/ACCESS-NRI/ACCESS-OM3/pull/5/files#diff-e8582e74fa156f4e5729a850e52b24f2fde2d815c2c9c360f88c4cf90db851abR10-R20

Or maybe it's another problem?

[harshula]

Double check if the Spack hash is the same or if it has changed.

[CodeGat]

parallelio, esmf, access-om3-nuopc do change between runs.

[anton-seaice]

Both esmf and access-om3-nuopc have dependencies on parallelio (e.g. https://github.com/spack/spack/blob/c118c7733b9e20f079bb19b84b8ad60cacd2a673/var/spack/repos/builtin/packages/esmf/package.py#L91), so i'd guess its related to parallelio or one of its deps

[CodeGat]

Another avenue is the spack gc garbage collection that happens on removal of old Prerelease environments as part of on.pull_request.types.closed, see https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/11396341321/job/31710012054#step:6:18

[CodeGat]

Due to chats with spack devs, it might be useful to check the difference between the hashes with spack diff /HASH1 /HASH2

[anton-seaice]

I don't know what this tells us :-)

spack diff /qcd4giqfkytonx3kmqkjwrfat3kiogai /pj4c37ym76pt3eaqqypzgtxl5rqkve7l
==> Warning: This interface is subject to change.

--- parallelio@2.6.2/qcd4giqfkytonx3kmqkjwrfat3kiogai
+++ parallelio@2.6.2/pj4c37ym76pt3eaqqypzgtxl5rqkve7l
@@ hash @@
-  parallelio qcd4giqfkytonx3kmqkjwrfat3kiogai
+  parallelio pj4c37ym76pt3eaqqypzgtxl5rqkve7l

[anton-seaice]

Lots of noise ...

But I have parallelio installed, do a concretise and it decides I need a new one (note - next to parallelio) :

[as2285@gadi-login-03 spack0.22]$ spack find parallelio
==> In environment cice-spack-package
==> 1 root specs
 -  access-om3@git.main=0-git.14

==> Installed packages
-- linux-rocky8-x86_64 / intel@2021.10.0 ------------------------
parallelio@2.6.2
==> 1 installed package
[as2285@gadi-login-03 spack0.22]$ spack concretise -f --fresh
==> Warning: using "cmake@3.24.2" which is a deprecated version
==> Concretized access-om3@git.main=0-git.14
 -   b26hvt5  access-om3@git.main=0-git.14%intel@2021.10.0 build_system=bundle arch=linux-rocky8-x86_64
 -   apv4p3o      ^access3-exe@git.main=0-git.171%intel@2021.10.0~ipo~openmp build_system=cmake build_type=Release configurations=CICE6,MOM6-CICE6 dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/access3-share generator=make arch=linux-rocky8-x86_64
 -   gtx27bo          ^access-cice@git.5-port_cmake_build=6.2.0-git.203%intel@2021.10.0+access3+cesmcoupled~ipo build_system=cmake build_type=Release dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/access-cice generator=make io_type=PIO arch=linux-rocky8-x86_64
 -   rje7cyo              ^access3-share@git.ca8a1ba8ca2716d2ae57846faf5c5ddc8bc5d35f=0-git.171%intel@2021.10.0~ipo build_system=cmake build_type=Release dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/access3-share generator=make arch=linux-rocky8-x86_64
 -   gvv2k6y                  ^esmf@git.v8.7.0=8.7.0%intel@2021.10.0 cflags='-fp-model precise' fflags='-traceback -fp-model precise' ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+shared~xerces build_system=makefile esmf_comm=auto esmf_os=auto esmf_pio=auto patches=f63d405 snapshot=none arch=linux-rocky8-x86_64
[+]  bcbruwv                  ^fortranxml@4.1.2%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
 -   pm7t3uw              ^parallelio@2.6.2%intel@2021.10.0 cflags='-std=gnu99 -qno-opt-dynamic-align -fp-model precise' fflags='-convert big_endian -ftz -assume realloc_lhs -assume byterecl -qno-opt-dynamic-align -fp-model source -traceback' +fortran~ipo~logging+mpi~ncint~pnetcdf+shared~timing build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  2c4wu5t                  ^netcdf-c@4.9.2%intel@2021.10.0+blosc~byterange~dap~fsync~hdf4~ipo~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared+szip+zstd build_system=cmake build_type=RelWithDebInfo generator=make patches=0161eb8 arch=linux-rocky8-x86_64
[+]  357vng5                      ^bzip2@1.0.8%intel@2021.10.0~debug~pic+shared build_system=generic arch=linux-rocky8-x86_64
[+]  y7n7vkn                          ^diffutils@3.10%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  55ipnye                              ^libiconv@1.17%intel@2021.10.0 build_system=autotools libs=shared,static arch=linux-rocky8-x86_64
[+]  jdlivnc                      ^c-blosc@1.21.5%intel@2021.10.0+avx2~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  7a5olrr                          ^lz4@1.9.4%intel@2021.10.0+pic build_system=makefile libs=shared,static arch=linux-rocky8-x86_64
[+]  ju23cym                          ^snappy@1.1.10%intel@2021.10.0~ipo+pic+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  ozfo7fd                      ^hdf5@1.14.3%intel@2021.10.0~cxx~fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo generator=make patches=82088c8 arch=linux-rocky8-x86_64
[+]  vrupasu                          ^pkgconf@2.2.0%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  675weom                      ^libaec@1.0.6%intel@2021.10.0~ipo+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  rglvnt5                      ^m4@1.4.19%intel@2021.10.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rocky8-x86_64
[+]  ou7jkjx                          ^libsigsegv@2.14%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  wa2x7rh                      ^zlib-ng@2.1.6%intel@2021.10.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64
[+]  kzc7pcv                      ^zstd@1.5.6%intel@2021.10.0~programs build_system=makefile libs=shared,static arch=linux-rocky8-x86_64
 -   xo3zqcq          ^access-mom6@git.cmake_build=0-git.11338%intel@2021.10.0+access3+cesmcoupled~ipo+mom_symmetric~openmp build_system=cmake build_type=Release dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/MOM6 generator=make arch=linux-rocky8-x86_64
[+]  on4aolu              ^fms@git.2024.03=2024.01.02-git.26%intel@2021.10.0~deprecated_io~gfs_phys+internal_file_nml~ipo+large_file~openmp~pic~quad_precision~yaml build_system=cmake build_type=RelWithDebInfo constants=GFDL generator=make precision=32,64 arch=linux-rocky8-x86_64
[+]  bky2d7s              ^netcdf-fortran@4.6.1%intel@2021.10.0~doc+pic+shared build_system=autotools arch=linux-rocky8-x86_64
[e]  tkmozao          ^cmake@3.24.2%intel@2021.10.0~doc+ncurses+ownlibs build_system=generic build_type=RelWithDebInfo arch=linux-rocky8-x86_64
[+]  doeoclg          ^gmake@4.4.1%intel@2021.10.0~guile build_system=generic arch=linux-rocky8-x86_64
[e]  5kn4pgy          ^openmpi@4.1.7%intel@2021.10.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64
[e]  5elnsoi      ^glibc@2.28%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64

==> Updating view at /g/data/tm70/as2285/spack0.22/environments/cice-spack-package/.spack-env/view

[harshula]

Hi @anton-seaice , Can you please try spack concretise -f --reuse-deps?

[harshula]

https://github.com/spack/spack/releases/tag/v0.22.0:

With the install command you now have three options:

    --reuse (default): reuse as many existing installations as possible.
    --reuse-deps / --fresh-roots: upgrade (freshen) roots but reuse dependencies if possible.
    --fresh: install fresh versions of requested packages (roots) and their dependencies.

We've also introduced --fresh-roots as an alias for --reuse-deps to make it more clear
that it may give you fresh versions. More details in #41302 and #43988.

[anton-seaice]

Thanks @harshula - that looks better:

[as2285@gadi-cpu-clx-2101 as2285]$ spack concretise -f --reuse-deps
==> Concretized access-om3@git.main=0-git.14
 -   u6sovce  access-om3@git.main=0-git.14%intel@2021.10.0 build_system=bundle arch=linux-rocky8-x86_64
 -   kqd3bkh      ^access3-exe@git.main=0-git.171%intel@2021.10.0~ipo~openmp build_system=cmake build_type=Release configurations=CICE6,MOM6-CICE6 dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/access3-share generator=make arch=linux-rocky8-x86_64
[+]  hpiqyr6          ^access-cice@git.5-port_cmake_build=6.2.0-git.203%intel@2021.10.0+access3+cesmcoupled~ipo build_system=cmake build_type=Release dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/access-cice generator=make io_type=PIO arch=linux-rocky8-x86_64
[+]  wje7ves              ^access3-share@git.ca8a1ba8ca2716d2ae57846faf5c5ddc8bc5d35f=0-git.171%intel@2021.10.0~ipo build_system=cmake build_type=Release dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/access3-share generator=make arch=linux-rocky8-x86_64
[+]  rgw4y5k                  ^esmf@git.v8.7.0=8.7.0%intel@2021.10.0 cflags='-fp-model precise' fflags='-traceback -fp-model precise' ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+shared~xerces build_system=makefile esmf_comm=auto esmf_os=auto esmf_pio=auto patches=f63d405 snapshot=none arch=linux-rocky8-x86_64
[+]  bcbruwv                  ^fortranxml@4.1.2%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  v7eqbrl              ^parallelio@2.6.2%intel@2021.10.0 cflags='-qno-opt-dynamic-align -fp-model precise -std=gnu99' fflags='-assume byterecl -traceback -assume realloc_lhs -convert big_endian -fp-model source -qno-opt-dynamic-align -ftz' +fortran~ipo~logging+mpi~ncint~pnetcdf+shared~timing build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  2c4wu5t                  ^netcdf-c@4.9.2%intel@2021.10.0+blosc~byterange~dap~fsync~hdf4~ipo~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared+szip+zstd build_system=cmake build_type=RelWithDebInfo generator=make patches=0161eb8 arch=linux-rocky8-x86_64
[+]  357vng5                      ^bzip2@1.0.8%intel@2021.10.0~debug~pic+shared build_system=generic arch=linux-rocky8-x86_64
[+]  y7n7vkn                          ^diffutils@3.10%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  55ipnye                              ^libiconv@1.17%intel@2021.10.0 build_system=autotools libs=shared,static arch=linux-rocky8-x86_64
[+]  jdlivnc                      ^c-blosc@1.21.5%intel@2021.10.0+avx2~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  7a5olrr                          ^lz4@1.9.4%intel@2021.10.0+pic build_system=makefile libs=shared,static arch=linux-rocky8-x86_64
[+]  ju23cym                          ^snappy@1.1.10%intel@2021.10.0~ipo+pic+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  ozfo7fd                      ^hdf5@1.14.3%intel@2021.10.0~cxx~fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo generator=make patches=82088c8 arch=linux-rocky8-x86_64
[+]  vrupasu                          ^pkgconf@2.2.0%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  675weom                      ^libaec@1.0.6%intel@2021.10.0~ipo+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-rocky8-x86_64
[+]  rglvnt5                      ^m4@1.4.19%intel@2021.10.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rocky8-x86_64
[+]  ou7jkjx                          ^libsigsegv@2.14%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64
[+]  wa2x7rh                      ^zlib-ng@2.1.6%intel@2021.10.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64
[+]  kzc7pcv                      ^zstd@1.5.6%intel@2021.10.0~programs build_system=makefile libs=shared,static arch=linux-rocky8-x86_64
[+]  vygcg6d          ^access-mom6@git.cmake_build=0-git.11338%intel@2021.10.0+access3+cesmcoupled~ipo+mom_symmetric~openmp build_system=cmake build_type=Release dev_path=/g/data/tm70/as2285/spack0.22/environments/cice-spack-package/MOM6 generator=make arch=linux-rocky8-x86_64
[+]  on4aolu              ^fms@git.2024.03=2024.01.02-git.26%intel@2021.10.0~deprecated_io~gfs_phys+internal_file_nml~ipo+large_file~openmp~pic~quad_precision~yaml build_system=cmake build_type=RelWithDebInfo constants=GFDL generator=make precision=32,64 arch=linux-rocky8-x86_64
[+]  bky2d7s              ^netcdf-fortran@4.6.1%intel@2021.10.0~doc+pic+shared build_system=autotools arch=linux-rocky8-x86_64
[e]  tkmozao          ^cmake@3.24.2%intel@2021.10.0~doc+ncurses+ownlibs build_system=generic build_type=RelWithDebInfo arch=linux-rocky8-x86_64
[+]  doeoclg          ^gmake@4.4.1%intel@2021.10.0~guile build_system=generic arch=linux-rocky8-x86_64
[e]  5kn4pgy          ^openmpi@4.1.7%intel@2021.10.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64
[e]  5elnsoi      ^glibc@2.28%intel@2021.10.0 build_system=autotools arch=linux-rocky8-x86_64

No promises its robust against compiler changes etc ?

[harshula]

Hi @anton-seaice , I think we've been misusing --fresh because it solved a problem that we were experiencing with the default. Most of the time we don't want its functionality. i.e. " --fresh: install fresh versions of requested packages (roots) and their dependencies."

What kind compiler changes are you concerned about? Spack is meant to identify compiler incompatibilities.

[anton-seaice]

My memory is that without --fresh spack will try and re-use dependencies from different compiler versions?

e.g. it might try and reuse a netcdf using intel 2021.06.0 even if the top level spec is using intel 2021.10.0

For example, I deactivate and try to install netcdf using intel@2021.06.0 and it uses dependencies from 2021.10.0:

[as2285@gadi-cpu-clx-2101 as2285]$ spack env deactivate
[as2285@gadi-cpu-clx-2101 as2285]$ spack install netcdf-c@4.9.2%intel@2021.06.0
[+] /usr (external glibc-2.28-5elnsoiqgcg5k5zmmwsp33bmnmaa3g5p)
[+] /usr (external glibc-2.28-csotjiltnkhc4qq55pnhueyamwf6uh5q)
==> openmpi@4.1.7 : has external module in ['openmpi/4.1.7']
[+] /apps/openmpi/4.1.7 (external openmpi-4.1.7-5kn4pgyj7aa5e2wcqjgxoybzn5l75i64)
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaec-1.0.6-675weomq5zwi6c53fnhntqodgqlm6nqz
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/snappy-1.1.10-ju23cym4rijzildlxsvhql7pv4av7eg5
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/bzip2-1.0.8-357vng5dpd7w7s7lletycxccjbl45ngt
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/gmake-4.4.1-doeoclgsn63cvh7zfx3trjmyk7rlcqn4
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/zstd-1.5.6-kzc7pcve7csxlonb2uaxzgyyuqfx6cz4
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/lz4-1.9.4-7a5olrrnewy7kmlh5x4bstziuheiqkz4
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/pkgconf-2.2.0-vrupasu7smpgcbarzpdwap45fcvjbjoa
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/zlib-ng-2.1.6-wa2x7rho3km6qpiki56dpjlpsce4c5n6
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/c-blosc-1.21.5-jdlivncntjhhznjhl62q7yxdt7mwcaj6
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/hdf5-1.14.3-ozfo7fd2a4ow6ivjte5lfehn5qjhacpp
==> Installing netcdf-c-4.9.2-ulskek34jvensle5yriejgho7vjonctv [14/14]
==> ...

spack install --reuse-deps netcdf-c@4.9.2%intel@2021.06.0 behaves the same

(And for completeness, using different compiler might work in that circumstance), so repeat with fortran code + deps:

e.g. parallelio with intel 2021.06.0 tries to use netcdf-fortran from intel 2021.10.0 :

spack install --reuse-deps parallelio@2.6.2%intel@2021.06.0
[+] /apps/cmake/3.24.2 (external cmake-3.24.2-tkmozaoejrs2hj3hbjh3dt3qdvqy2rya)
[+] /usr (external glibc-2.28-csotjiltnkhc4qq55pnhueyamwf6uh5q)
[+] /usr (external glibc-2.28-5elnsoiqgcg5k5zmmwsp33bmnmaa3g5p)
==> openmpi@4.1.7 : has external module in ['openmpi/4.1.7']
[+] /apps/openmpi/4.1.7 (external openmpi-4.1.7-5kn4pgyj7aa5e2wcqjgxoybzn5l75i64)
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/gmake-4.4.1-doeoclgsn63cvh7zfx3trjmyk7rlcqn4
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/zstd-1.5.6-kzc7pcve7csxlonb2uaxzgyyuqfx6cz4
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/zlib-ng-2.1.6-wa2x7rho3km6qpiki56dpjlpsce4c5n6
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/lz4-1.9.4-7a5olrrnewy7kmlh5x4bstziuheiqkz4
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/bzip2-1.0.8-357vng5dpd7w7s7lletycxccjbl45ngt
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/snappy-1.1.10-ju23cym4rijzildlxsvhql7pv4av7eg5
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/pkgconf-2.2.0-vrupasu7smpgcbarzpdwap45fcvjbjoa
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaec-1.0.6-675weomq5zwi6c53fnhntqodgqlm6nqz
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/c-blosc-1.21.5-jdlivncntjhhznjhl62q7yxdt7mwcaj6
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/hdf5-1.14.3-ozfo7fd2a4ow6ivjte5lfehn5qjhacpp
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/netcdf-c-4.9.2-2c4wu5th6gxpm3hsohdqwq43nii5uuuu
[+] /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0/netcdf-fortran-4.6.1-bky2d7sqvg4kfbiwveqnvr634hj2tdot
==> Installing parallelio-2.6.2-22bnphc3jgvru7f5ttu2kpgwjvctvnk7 [17/17]
==> ...

[harshula]

it might try and reuse a netcdf using intel 2021.06.0 even if the top level spec is using intel 2021.10.0

If Spack considers intel 2021.06.0 and intel 2021.10.0 to be compatible, do you think that is not "robust"?