Thoughts on system externals

Basically, due to the Linux distribution habit of having package fred and fred-devel, but having the possibility of many pre-built packages being installed from the distribution, we have several categories:

1. packages whose runtime is guaranteed to be installed (i.e. on a grid/batch node) but whose -devel package is probably not.
2. packages which are availaible but might not be installed, (and have plain and -devel variants).

We have several options for dealing with this for Spack, esp. for broadly distributed packages:

1. build externals in spack anyway, and ship with installs. This creates longer build times, longer install times, etc. but does in general work.
2. build all externals in spack separately. For this, we build a whole slew of packages (i.e x11 libraries, etc), install them in a cvs spack area, and provide a packages.yaml file that refers to those /cvmfs externals for all our builds. People who don't have access to cvmfs would have to install an equivalent area on their system, (we could make a Bundle package for it) and setup the packages.yaml entries for it.
3. Have a script that builds an externals area from the system RPMS (takes a list of packages, downloads them, runs them through rpm2cpio, and unpacks the files, optionally collapsing things like usr/lib64 vs usr/lib) and provides a packages.yaml file for those packages. We could place one such externals area in cvmfs, and build generic grid software against that.
4. Require all build systems to have a specific list of -devel rpm's installed, and put a smaller list of runtime libraries in /cvmfs for packages that are available for the system but are not installed on grid worker nodes.

Marc's thoughts: Option 1 is more painful than it seems to be; actually building all the externals in spack tends to find issues people have overlooked, so it will involve some amount of support load, but would overall improve the support level of spack if we maintained it. And installing all of those dependencies via a spack buildcache install will double or triple install times. In a perfect Spack world, I expect we would want to go with probably option 2; it is a "pure spack" solution, but it doesn't require us to redistribute the external packages very often, thus keeping our buildcache install times down; but the same issues of some of the external packages recipes needing some maintenance occur here. This would put us in the position of helping maintain those external packages.. So I suspect in actuality option 3, using the RPMS to make an externals area, is the least effort to maintain. Other folks are already doing the builds for us; we need only maintain a script that builds/updates the area. Option 3 also gives us the best odds of being compatible with the runtime libraries in /usr/lib64, by using the corresponding -devel rpm libraries and headers in the externals area. It is also the most similar to what many experiments have done in the past to deal with some of these libraries, by having a libs area in cvmfs. Option 4 is theoretically the least work for us, since other people have to install the build rpms, etc. but hardest for our users, who often dont have admin priveleges. We could perhaps provide container images for option 4 for people to do builds in, but we still need a way to get the runtime libraries we want on grid worker nodes, etc. I suppose we could talk with the glideinwms folks who maintain the container images, but again, adding all that to the grid containers slows them down as far as install times, etc; and many of them aren't really used in the grid environment, they're just dynamic libraries we could call...