Adjusting error printouts when ≥x86_64-v2 not available

[jreic]

Hi @smuzaffar,

I noticed that some automated WLCG tests were failing at my site (T3_US_Rutgers) in a relatively silent way (see https://monit-grafana.cern.ch/d/siYq3DxZz/wlcg-sitemon-test-details?orgId=20&var-metric=org.cms.WN-22frontier-/cms-ce-token&var-dst_hostname=ruhex-osgce.rutgers.edu&var-timestamp=1741372246000, and a snippet copied at the bottom).

This was running on a test node VM where only x86_64 instructions were enabled--which I was able to adjust (since the host did have x86_64-v2 available). Since the issue wasn't obvious at first, I was wondering if it would be possible to catch (probably in https://github.com/cms-sw/cmssw-config/blob/scramv3/SCRAM/hooks/project/00-multi-targets) when only the older instruction set is available, and print a more verbose error that would point to the underlying issue more quickly.

Thanks!
Joey

[1]

setting up environment:
-----------------------
   env.variable CVMFS_MOUNT_DIR is set to /cvmfs
   env.variable CVMFS is not set
   sourcing cmsset_default.sh ...
   env.variable SCRAM_ARCH is set to el8_amd64_gcc12
      /cvmfs/cms.cern.ch/common/scram
   env.variable SITECONFIG_PATH is set to "/cvmfs/cms.cern.ch/SITECONF/local"
   env.variable CMS_PATH is set to "/cvmfs/cms.cern.ch"
   CMSSW_15_0_1 selected
   making and changing to swnft subdirectory
   creating CMSSW_15_0_1 project...
WARNING: In non-interactive mode release checks e.g. deprecated releases, production architectures are disabled.
WARNING: Unable to find project's default micro-architecture or system supported psABI micro-architectures.
Development area is created without multi-microarchitectures support. Default micro-architecture will be used for build and runtime.
   setting up CMSSW_15_0_1 environment...

...

checking DB FroNtier access in cmsRun (@51 sec):
------------------------------------------------
/srv/wn_frontier.sh: line 385:   337 Illegal instruction     (core dumped) cmsRun ecal_pedestals.py > output.log 2>&1
   [E] cmsRun failed, rc=132

[2]

setting up environment:
-----------------------
   env.variable CVMFS_MOUNT_DIR is set to /cvmfs
   env.variable CVMFS is not set
   sourcing cmsset_default.sh ...
   env.variable SCRAM_ARCH is set to el8_amd64_gcc12
      /cvmfs/cms.cern.ch/common/scram
   env.variable SITECONFIG_PATH is set to "/cvmfs/cms.cern.ch/SITECONF/local"
   env.variable CMS_PATH is set to "/cvmfs/cms.cern.ch"
   CMSSW_15_0_1 selected
   making and changing to swnft subdirectory
   creating CMSSW_15_0_1 project...
WARNING: In non-interactive mode release checks e.g. deprecated releases, production architectures are disabled.
WARNING: You host supported psABI(x86-64-v2 ) do not match SCRAM default psABI x86-64-v3.
         Additional build/runtime micro-architecture x86-64-v2 is enabled.
   setting up CMSSW_15_0_1 environment...
IMPORTANT: Setting CMSSW environment to use 'x86-64-v2' target.

...

checking DB FroNtier access in cmsRun (@23 sec):
------------------------------------------------
   cmsRun complete (@104 sec)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @jreic.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

@jreic , really sorry, I have not noticed this issue.
Ideally job should not have failed and scram should have properly used libs valid for x86-64-v2 . Can you try to reproduce the test and share the acrtual log file e.g. in which library the crash happened?

[jreic]

Hi @smuzaffar, no problem, it was a minor issue anyway. I downgraded one of my worker node VMs to see if I could reproduce it, but it seems that the current CMS SAM tests run as expected even when x86_64-v2 isn't available. An example is at https://etf-cms-prod.cern.ch/etf/check_mk/view.py?host=ruhex-osgce.rutgers.edu&service=org.cms.WN-22frontier-%2Fcms-ce-token&site=etf&view_name=service.

From the printouts I had pasted in the original message, I'm guessing it's a change between 14_0_21 (which the SAM tests currently use) and 15_0_1.

I can clone the cmssam repo (https://gitlab.cern.ch/etf/cmssam) and do a local run successfully with its current defaults:

bash-4.4$ ./wn_frontier.sh 
   env.variable SCRAM_ARCH is set to el8_amd64_gcc12
      /cvmfs/cms.cern.ch/common/scram
OK: DB FroNtier check ok

or get the same type of failure when manually switching to CMSSW_15_0_1:

bash-4.4$ ./wn_frontier.sh 
   env.variable SCRAM_ARCH is set to el8_amd64_gcc12
      /cvmfs/cms.cern.ch/common/scram
WARNING: Unable to find project's default micro-architecture or system supported psABI micro-architectures.
Development area is created without multi-microarchitectures support. Default micro-architecture will be used for build and runtime.
./wn_frontier.sh: line 384: 31188 Illegal instruction     (core dumped) cmsRun ecal_pedestals.py > output.log 2>&1
   [E] cmsRun failed, rc=132
   [E] direct/non-proxy connection to CERN DB FroNtier server(s)
   [W] connection via one (or more) squid proxy services failed
ERROR: cmsRun failed

(output.log itself is empty in this failed case, or I would share it)

Maybe this is enough of an edge case to not bother: presumably there aren't many machines left these days that genuinely don't have support for ≥x86_64-v2 (and any misconfigured VMs like mine might be rare). I think I had made the issue since the lack of x86_64-v3 support had a very clear warning in [2] from my original message (even though it ran successfully), whereas [1] failed in a silent manner.