Pyhf giving inconsistent results depending on machine, with identical setups

[ldcorpe]

Summary

Hi PyHF devs,

I'm working with the theorists in cc to make just of the monojet search PyHF workspace we recently released on HEPData (https://www.hepdata.net/record/ins1847779).

However, we are having difficulty extracting sensible results, and even getting compatible results using the same setup.

Hence we have a series of questions... but the first thing is to start from a reproducible base, which as I said I've been unable to achieve.

To demonstrate this, we have a reproducer script attached, which gives different results on my laptop (MaBook Pro, M2 chip running Sequoia) compared to lxplus.
Python 3.12 is 3.12.11 on my laptop and 3.12.09 on lxplus, but I really hope this would not make a difference.

Returning totally opposite CLs values, as well as different nllo and p-values.

[Laptop]
((myenv) ) louiecorpe_cern2@clratlmac02 mark % ./run_hacked.py
(<pyhf.tensor.jax_backend.jax_backend object at 0x1021c7380>, <pyhf.optimize.scipy_optimizer object at 0x100963c40>)
CLs stuff:
CLs observed: 0.5000
CLs expected: [0.0, 0.0, 0.0, 0.0, 0.0]
Discovery stuff:
chi2 value: 45807.19992672275, -2nllo1: 46212.52641658511; -2nll0: 405.32648986235756
pval observed: 2.1060e-01
CLb: 1.0000e+00
Expected pvals: [6.771459918967592e-223, 2.8790775099245594e-209, 4.507997367881358e-196, 2.599584415440002e-183, 5.521386053743319e-171]

[Lxplus]
(myenv) [lcorpe@lxplus912 markMonojetDebug]$ ./run_hacked.py
(<pyhf.tensor.jax_backend.jax_backend object at 0x7f182c0956c0>, <pyhf.optimize.scipy_optimizer object at 0x7f17d6feb100>)
CLs stuff:
CLs observed: 1.0000
CLs expected: [1.0, 1.0, 1.0, 1.0, 1.0]
Discovery stuff:
chi2 value: -0.6469362112932799, -2nllo1: 404.67955366722083; -2nll0: 405.3264898785141
pval observed: 2.1060e-01
CLb: 4.9838e-01
Expected pvals: [0.002552986461221048, 0.03590893287350567, 0.2117769154952523, 0.5791537539982395, 0.8848777067346829]

Do you have any ideas why we might be getting such wild different results from seemingly identicaltical setups?

Many thanks for your help!

Louie

OS / Environment

My laptop (MaBook Pro, M2 chip running Sequoia) compared to lxplus.
Python 3.12 is 3.12.11 on my laptop and 3.12.09 on lxplus.

>>> import jax
>>> jax.__version__
'0.5.3'
>>> import pyhf
>>> pyhf.__version__
'0.7.6'
>>> import scipy
>>> scipy.__version__
'1.14.0'
>>> import numpy
>>> numpy.__version__
'1.26.4'

Steps to Reproduce

In both cases I ran:

python3.12 - m venv myenv
source  /myenv/bin/activate
pip install -r requirements
unzip Archive.zip
./run_hacked.py

File Upload (optional)

No response

Expected Results

I expected the results on both machines to be the same or similar within float precision.

Actual Results

I expected the results on both machines to be the same or similar within float precision, instead they appear to give contradictory answers.

pyhf Version

((mark_env) ) louiecorpe_cern2@clratlmac02 mark % pyhf --version
pyhf, version 0.7.6

Code of Conduct

• I agree to follow the Code of Conduct

[ldcorpe]

files did not attach properly, sorry!

requirements.txt
Archive.zip