Add Rivet

[matthewfeickert]

• Source code: https://gitlab.com/hepcedar/rivet/
• License: GPLv3
• Requirements: (Going off of what is listed in the rivet-bootstrap at https://gitlab.com/hepcedar/rivetbootstrap/-/blob/d2068bd027483333b82265487adecd454f8aa49e/rivet-bootstrap#L44-49 ),
  • hepmc3: https://github.com/conda-forge/hepmc3-feedstock
  • fastjet: https://github.com/conda-forge/fastjet-cxx-feedstock
  • fastjet-contrib: https://github.com/conda-forge/fastjet-contrib-feedstock
  • hdf5: https://github.com/conda-forge/hdf5-feedstock
  • HighFive: https://github.com/conda-forge/highfive-feedstock
  • cython: https://github.com/conda-forge/cython-feedstock
  • lhapdf: https://github.com/conda-forge/lhapdf-feedstock
  • yoda: https://github.com/conda-forge/yoda-feedstock
    • Picks up root_base.
  • python: https://github.com/conda-forge/python-feedstock
• Requester: @matthewfeickert
• Relevant build experts: @agbuckley, @20DM

[matthewfeickert]

Relevant build scripts:

• https://gitlab.com/hepcedar/hepstore/-/blob/c1a6e38cb9c933ad3960eac71ab34b6911b08ded/rivet/Dockerfile#L29-41
• https://gitlab.com/hepcedar/rivetbootstrap/-/blob/d4e9d6bde6f536fed95935e052aca11a1cc9643b/rivet-bootstrap#L276-L368

Not exactly clear how to enable ONNX support in https://gitlab.com/hepcedar/rivet/-/tags/rivet-4.1.0.

[matthewfeickert]

@agbuckley, @20DM in the Rivet ./configure --help output there's the option

  --disable-analyses      don't try to build or install standard analyses

Is a Rivet install without all these analyses considered a full/valid install? Or is it considered incomplete? How does the database of analyses get updated over time for existing installed versions?

full ./configure help log:

`configure' configures Rivet 3.1.11 to adapt to many kinds of systems.

Usage: /tmp/debug/build-rivet/source/./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/Rivet]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

Program names:
  --program-prefix=PREFIX            prepend PREFIX to installed program names
  --program-suffix=SUFFIX            append SUFFIX to installed program names
  --program-transform-name=PROGRAM   run sed PROGRAM on installed program names

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-silent-rules   less verbose build output (undo: "make V=1")
  --disable-silent-rules  verbose build output (undo: "make V=0")
  --enable-dependency-tracking
                          do not reject slow dependency extractors
  --disable-dependency-tracking
                          speeds up one-time build
  --enable-static[=PKGS]  build static libraries [default=no]
  --enable-shared[=PKGS]  build shared libraries [default=yes]
  --enable-fast-install[=PKGS]
                          optimize for fast installation [default=yes]
  --disable-libtool-lock  avoid locking (might break parallel builds)
  --disable-analyses      don't try to build or install standard analyses
  --enable-pdfmanual      backward compatility dummy flag, no effect and
                          deprecated
  --disable-doxygen       don't try to make Doxygen documentation
  --disable-pyext         don't build Python module (default=build)
  --enable-debug          build with debugging symbols [default=no]
  --enable-extra-warnings build with extra compiler warnings (recommended for
                          developers) [default=no]

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-pic[=PKGS]       try to use only PIC/non-PIC objects [default=use
                          both]
  --with-aix-soname=aix|svr4|both
                          shared library versioning (aka "SONAME") variant to
                          provide on AIX, [default=aix].
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-sysroot[=DIR]    Search for dependent libraries within DIR (or the
                          compiler's sysroot if not specified).
  --with-yoda             path to YODA [$prefix and various standard
                          locations]
  --with-yoda             path to YODA [$prefix and various standard
                          locations]
  --with-yoda-libpath     path to directory containing YODA library
                          [YODAPATH/lib or YODAPATH/lib/YODA]
  --with-yoda-libname     name to be used when linking the YODA library [YODA]
  --with-yoda             path to YODA [$prefix and various standard
                          locations]
  --with-yoda-incpath     path to directory containing YODA headers
                          [YODAPATH/include]
  --with-hepmc3           path to HepMC3 [$prefix and various standard
                          locations]
  --with-hepmc3-libpath   path to directory containing HepMC3 library
                          [HEPMC3PATH/lib or HEPMC3PATH/lib/HepMC3]
  --with-hepmc3-libname   name to be used when linking the HepMC3 library
                          [HepMC3]
  --with-hepmc3           path to HepMC3 [$prefix and various standard
                          locations]
  --with-hepmc3-incpath   path to directory containing HepMC3 headers
                          [HEPMC3PATH/include]
  --with-hepmc            path to HepMC2 [$prefix and various standard
                          locations]
  --with-hepmc-libpath    path to directory containing HepMC library
                          [HEPMCPATH/lib or HEPMCPATH/lib/HepMC]
  --with-hepmc-libname    name to be used when linking the HepMC library
                          [HepMC]
  --with-hepmc            path to HepMC [$prefix and various standard
                          locations]
  --with-hepmc-incpath    path to directory containing HepMC headers
                          [HEPMCPATH/include]
  --with-fastjet-libpath  path to directory containing fastjet library
                          [FASTJETPATH/lib or FASTJETPATH/lib/fastjet]
  --with-fastjet-libname  name to be used when linking the fastjet library
                          [fastjet]
  --with-fastjet          path to fastjet [$prefix and various standard
                          locations]
  --with-fastjet-incpath  path to directory containing fastjet headers
                          [FASTJETPATH/include]
  --with-fjcontrib        path to FastJet Contrib [$FASTJETPATH]
  --with-python-sys-prefix
                          use Python's sys.prefix and sys.exec_prefix values
  --with-python_prefix    override the default PYTHON_PREFIX
  --with-python_exec_prefix
                          override the default PYTHON_EXEC_PREFIX
  --with-zlib=DIR         root directory path of zlib installation [defaults to
                          /usr/local or /usr if not found in /usr/local]
  --without-zlib          to disable zlib usage completely

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags
  LT_SYS_LIBRARY_PATH
              User-defined run-time library search path.
  CXXCPP      C++ preprocessor
  YODAPATH    path to YODA [$prefix and various standard locations]
  YODALIBPATH path to the directory containing the YODA library [YODAPATH/lib
              or YODAPATH/lib/YODA]
  YODALIBNAME name to be used when linking the YODA library [YODA]
  YODAINCPATH path to the directory containing the YODA header files
              [YODAPATH/include]
  HEPMC3PATH  path to HepMC3 [$prefix and various standard locations]
  HEPMC3LIBPATH
              path to the directory containing the HepMC3 library
              [HEPMC3PATH/lib or HEPMC3PATH/lib/HepMC3]
  HEPMC3LIBNAME
              name to be used when linking the HepMC3 library [HepMC3]
  HEPMC3INCPATH
              path to the directory containing the HepMC3 header files
              [HEPMC3PATH/include]
  HEPMCPATH   path to HepMC [$prefix and various standard locations]
  HEPMCLIBPATH
              path to the directory containing the HepMC library
              [HEPMCPATH/lib or HEPMCPATH/lib/HepMC]
  HEPMCLIBNAME
              name to be used when linking the HepMC library [HepMC]
  HEPMCINCPATH
              path to the directory containing the HepMC header files
              [HEPMCPATH/include]
  FASTJETPATH path to fastjet [$prefix and various standard locations]
  FASTJETLIBPATH
              path to the directory containing the fastjet library
              [FASTJETPATH/lib or FASTJETPATH/lib/fastjet]
  FASTJETLIBNAME
              name to be used when linking the fastjet library [fastjet]
  FASTJETINCPATH
              path to the directory containing the fastjet header files
              [FASTJETPATH/include]
  PYTHON      the Python interpreter
  PYTHON_VERSION
              The installed Python version to use, for example '2.3'. This
              string will be appended to the Python interpreter canonical
              name.

Note that for Rivet v3.1.11 (starting with older Rivet builds to provide greater support) that just trying to build with all analyses

./configure --prefix=$PREFIX --enable-shared=yes --with-yoda=$PREFIX --with-hepmc3=$PREFIX --with-fastjet=$PREFIX --with-fjcontrib=$PREFIX --with-zlib=$PREFIX PYTHON=$PYTHON && \
make --jobs=10

results in a bash: Argument list too long build error

/tmp/debug/build-rivet/.pixi/envs/default/bin/x86_64-conda-linux-gnu-c++ -o "RivetTristanAnalyses.so" -shared -fPIC  -I../include -I/tmp/debug/build-rivet/.pixi/envs/default/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp/debug/build-rivet/.pixi/envs/default/include -DENABLE_HEPMC_3=true -I/tmp/debug/build-rivet/.pixi/envs/default/include -I/tmp/debug/build-rivet/.pixi/envs/default/include -I/tmp/debug/build-rivet/.pixi/envs/default/include -I/tmp/debug/build-rivet/.pixi/envs/default/include -I/tmp/debug/build-rivet/.pixi/envs/default/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /tmp/debug/build-rivet/.pixi/envs/default/include -DENABLE_HEPMC_3=true -I/tmp/debug/build-rivet/.pixi/envs/default/include -I/tmp/debug/build-rivet/.pixi/envs/default/include  -pedantic -Wall -Wno-long-long -Wno-format -Werror=uninitialized -Werror=delete-non-virtual-dtor -fopenmp -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp/debug/build-rivet/.pixi/envs/default/include  -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /tmp/debug/build-rivet/.pixi/envs/default/include -Wl,--no-as-needed  -L../src/.libs -L/tmp/debug/build-rivet/.pixi/envs/default/lib -L/tmp/debug/build-rivet/.pixi/envs/default/lib -L/tmp/debug/build-rivet/.pixi/envs/default/lib -lfastjettools -lfastjet -lm -lfastjetplugins -lsiscone_spherical -lsiscone -L/tmp/debug/build-rivet/.pixi/envs/default/lib -lfastjetcontribfragile -lfastjettools -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp/debug/build-rivet/.pixi/envs/default/lib -Wl,-rpath-link,/tmp/debug/build-rivet/.pixi/envs/default/lib -L/tmp/debug/build-rivet/.pixi/envs/default/lib  -lRivet -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/tmp/debug/build-rivet/.pixi/envs/default/lib -Wl,-rpath-link,/tmp/debug/build-rivet/.pixi/envs/default/lib -L/tmp/debug/build-rivet/.pixi/envs/default/lib -L/tmp/debug/build-rivet/.pixi/envs/default/lib -L/tmp/debug/build-rivet/.pixi/envs/default/lib   pluginTristan/AMY_1990_I283337.cc pluginTristan/AMY_1990_I294525.cc pluginTristan/AMY_1990_I295160.cc pluginTristan/AMY_1990_I298238.cc pluginTristan/AMY_1995_I406129.cc pluginTristan/TOPAZ_1989_I279575.cc pluginTristan/TOPAZ_1990_I283003.cc pluginTristan/TOPAZ_1993_I353845.cc pluginTristan/TOPAZ_1993_I361661.cc pluginTristan/TOPAZ_1995_I381777.cc pluginTristan/TOPAZ_1995_I381900.cc pluginTristan/TOPAZ_1997_I454183.cc pluginTristan/VENUS_1987_I251274.cc pluginTristan/VENUS_1990_I283774.cc pluginTristan/VENUS_1990_I296392.cc pluginTristan/VENUS_1995_I392360.cc pluginTristan/VENUS_1997_I440852.cc pluginTristan/VENUS_1998_I453613.cc pluginTristan/VENUS_1999_I500179.cc  -I../include
make[2]: Entering directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
mkdir -p ./data
make[1]: bash: Argument list too long
make[1]: *** [Makefile:575: all-local] Error 127
make[1]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make: *** [Makefile:569: all-recursive] Error 1

[matthewfeickert]

To get rivet v3.1.11 working with yoda will also need to backport a yoda v1.9.x release more recent than v1.9.6 (current newest v1.9.x release on conda-forge) (like https://gitlab.com/hepcedar/yoda/-/tags/yoda-1.9.11)

[agbuckley]

Is a Rivet install without all these analyses considered a full/valid install? Or is it considered incomplete? How does the database of analyses get updated over time for existing installed versions?

Hi @matthewfeickert , without analyses it would be useful for developers (including physics users just wanting to develop their own analysis routines) but not for the (I think) majority who want to run against existing MC-validation or data-comparison analyses. Since a main headline for Rivet is the ~2000 analyses "out of the box", it would effectively be incomplete. You could split into a rivet and rivet-analyses package pair, I suppose, but it's probably easier just to build and bundle the whole thing.

There's no deployment system to update existing installations with new analyses: we've discussed it often, but in practice we still change interfaces and maintain analyses and it's just better to upgrade the whole thing. Unofficially, anyone can get the C++ codes of new analyses before the Rivet release they'll appear in, and try to compile against their older installation -- most of the time it will work, but sometimes an analysis needs a new feature. That is basically what Contur does to get access to latest and greatest analyses without waiting for a Rivet release: works fine, and the analyses are loaded as dlopen plugins so no problem to just stick the .so's in a dir and tell Rivet where to find the extras, but we don't have any auto-updating magic for it.

./configure --prefix=$PREFIX --enable-shared=yes --with-yoda=$PREFIX --with-hepmc3=$PREFIX --with-fastjet=$PREFIX --with-fjcontrib=$PREFIX --with-zlib=$PREFIX PYTHON=$PYTHON && \
make --jobs=10

results in a bash: Argument list too long build error

...

make[2]: Entering directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make[2]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
mkdir -p ./data
make[1]: bash: Argument list too long
make[1]: *** [Makefile:575: all-local] Error 127
make[1]: Leaving directory '/tmp/debug/build-rivet/source/analyses'
make: *** [Makefile:569: all-recursive] Error 1

Interesting, we've not seen that in our testing. But note that it wasn't the compilation line: if that was an issue, the many ATLAS analyses would have triggered it, not the small number of TRISTAN ones. I think they were just the last to build, then you see all the threads leaving the directory and getting ready to copy the files to the install prefix. There are a lot of files there, but we'd not seen this bash overflow. We'll put a fix in: probably will just look like splitting into multiple copies, for each file type.

[agbuckley]

Not exactly clear how to enable ONNX support in https://gitlab.com/hepcedar/rivet/-/tags/rivet-4.1.0.

You can configure with --enable-onnxrt, which will check for the runtime, enable build of the ONNX example analysis, and add the ONNX flags to analysis builds. It is still a soft dependency, so users building their own ONNX-requiring analyses can just put -lonnxruntime on their rivet-build command line (cf. https://gitlab.com/hepcedar/rivet/-/blob/release-4-1-x/.gitlab-ci.yml?ref_type=heads#L558) and that should work. So it's a bit of a free choice whether that is an automatic flag for every analysis build or to leave it up to the user to install and add the flag. Later, when we have more ONNX analyses in the main library I guess it will become mandatory.

[matthewfeickert]

You could split into a rivet and rivet-analyses package pair, I suppose, but it's probably easier just to build and bundle the whole thing.

@agbuckley Good to know. Packaging the whole thing isn't an issue. The goal is just to package what people would normally expect as the minimum canonical install, so this information is very helpful. 👍

Interesting, we've not seen that in our testing. But note that it wasn't the compilation line: if that was an issue, the many ATLAS analyses would have triggered it, not the small number of TRISTAN ones. I think they were just the last to build, then you see all the threads leaving the directory and getting ready to copy the files to the install prefix. There are a lot of files there, but we'd not seen this bash overflow. We'll put a fix in: probably will just look like splitting into multiple copies, for each file type.

I'll debug more than the cursory pass I took at this. I will open up an actual issue on Rivet's GitLab repo if I get more information. This can probably get ignored as I'm only able to reproduce it in a local shell on my machine and not during the rattler-build process, which for the purposes of packaging on conda-forge is all that matters.

You can configure with --enable-onnxrt, which will check for the runtime, enable build of the ONNX example analysis, and add the ONNX flags to analysis builds.

Okay, great. I think ONNX support was only added in Rivet v4.0.3(?) so that won't affect the building of initial older releases like in conda-forge/staged-recipes#29440, but that's useful to know once we hit the v4.0.x series.

[matthewfeickert]

@agbuckley @20DM We only have Rivet for linux-64 so far

Name	Downloads	Version	Platforms

(can work on getting osx-64, linux-aarch64, linux-ppc64le, and osx-arm64 support in the future) but can you take a look at what is currently getting built for https://github.com/conda-forge/rivet-feedstock and let us know in Issues there if there's anything that seems immediately off/wrong?

Easiest way to do this if you're not already conda people is with pixi:

# curl -fsSL https://pixi.sh/install.sh | bash  # Install pixi

restart shell or source your shell rc file and then

# mkdir -p /tmp/example
# cd /tmp/example
# pixi init
# pixi add rivet
# pixi shell

I also intentionally started with v3.1.11 to get a v3 build that is well understood and cross-platform before moving onto modern v4 builds.