Status of `histfactory_dist` distributions

[tprocter46]

I was trying to use this project (thanks a lot for it, by the way!) on some of the HS3 jsons provided by ATLAS in https://www.hepdata.net/record/157024.

Unfortunately, when running e.g.

model = pyhs3.Workspace.load("ANA-HIGP-2024-32_HHH_SplusBfit_Likelihoods/NonResonantFits_NN/Likelihood_TRSM_X375_S250.json")

they all crash with

ValidationError: 2 validation errors for Workspace
distributions.`ValidationInfo(config={'title': 'Distributions'}, context=None, data={'metadata': Metadata(hs3_version='0.2', packages=[PackageInfo(name='ROOT', version='6.35.01')], authors=None, publications=None, description=None)}, field_name='root')`.0
  Unknown distribution type 'histfactory_dist' does not match any of the expected distributions: 'gaussian_dist', 'mixture_dist', 'product_dist', 'crystalball_doublesided_dist', 'generic_dist', 'poisson_dist' [type=union_tag_invalid, input_value={'axes': [{'edges': [0.1,...pe': 'histfactory_dist'}, input_type=dict]
distributions.`ValidationInfo(config={'title': 'Distributions'}, context=None, data={'metadata': Metadata(hs3_version='0.2', packages=[PackageInfo(name='ROOT', version='6.35.01')], authors=None, publications=None, description=None)}, field_name='root')`.2
  Unknown distribution type 'histfactory_dist' does not match any of the expected distributions: 'gaussian_dist', 'mixture_dist', 'product_dist', 'crystalball_doublesided_dist', 'generic_dist', 'poisson_dist' [type=union_tag_invalid, input_value={'axes': [{'edges': [0.1,...pe': 'histfactory_dist'}, input_type=dict]

As far as I can tell from the histfactory.md in the HS3 github, histfactory_dist is a real part of the HS3 standard. Will it be supported in pyhs3 sometime soon? And/or is there some workaround that we can use in the meantime?
(Or am I simply doing something wrong?)

People using a full-on ROOT based approach have got reasonable looking numbers out of these models, but it would be cool to get them running in python, too.

Thanks again,
Tomasz

[kratsg]

Thanks - this is expected. Right now, if you're working on an HS3 that's just only HistFactory -- you will need to convert that into a HiFa JSON that can be consumed by pyhf. There is a conversion script by @cburgard (I keep misplacing it) that can do the trick.

The current scope is to cover things that are orthogonal to pyhf right now, and tackle the simpler distributions and develop the interface that makes the most sense from an intuitive point of view. This is related to #50 (same analysis there). This still needs to be worked out with IRIS-HEP and the pyhf dev team (disclaimer: includes me on both counts) to make sure this fits into the ecosystem, rather than just trampling over existing work or breaking current user flows.

This project does welcome contributions as it had been very much a hobby-project -- which means I've been a little bit more cautious about trying to implement a full-on distribution and then later completely rewrite it in order to make sure it integrates with existing packages.

[cburgard]

Hey!

This is the conversion script I wrote some time ago. It's not actively maintained, so it might need some massaging to work with the latest version of hs3. Also, it only does the direction "hs3 -> pyhf" - for the other direction, you should probably use ROOT and do pyhf->XML->ROOT->hs3, which should work out of the box.

https://gitlab.cern.ch/cburgard/RooFitUtils/-/blob/master/scripts/json-roofit2pyhf.py?ref_type=heads

Feel free to play with it. If you change anything, feel free to upload your fixed version wherever you find it belongs.

[kratsg]

As a follow up -- there's an initial implementation here that's minimally validated: #110. @tprocter46 - would be great to have some feedback on it. It is in v0.4.0.

[tprocter46]

Hi @kratsg , thanks very much for trying this out.

At least for the HHH examples, I still can't load the workspace. There are a few different errors reported (listed below). Its possible that maybe the json's released on hepdata don't perfectly adhere to the schema? But if that's the case, then we should probably look into whatever tool the analysis team used to right them (and they do seem to run ~fine with roofit)

Errors for debugging (first 20):
  1. distributions[0] -> histfactory_dist -> samples[0] -> data -> errors: Field required
  2. distributions[0] -> histfactory_dist -> samples[0] -> modifiers[1] -> shapesys -> parameters('shapefactor'): Field required
  3. distributions[0] -> histfactory_dist -> samples[0] -> modifiers[1] -> shapesys -> constraint: Input should be 'Poisson'
  4. distributions[0] -> histfactory_dist -> samples[1] -> modifiers[158] -> staterror -> parameters('staterror'): Field required
  5. distributions[0] -> histfactory_dist -> samples[1] -> modifiers[158] -> staterror -> constraint: Input should be 'Gauss'
  6. distributions[0] -> histfactory_dist -> samples[1] -> modifiers[158] -> staterror -> data('staterror'): Field required
  7. distributions[2] -> histfactory_dist -> samples[0] -> data -> errors: Field required
  8. distributions[2] -> histfactory_dist -> samples[0] -> modifiers[5] -> shapesys -> parameters('shapefactor'): Field required
  9. distributions[2] -> histfactory_dist -> samples[0] -> modifiers[5] -> shapesys -> constraint: Input should be 'Poisson'
  10. distributions[2] -> histfactory_dist -> samples[1] -> data -> errors: Field required
  11. distributions[2] -> histfactory_dist -> samples[2] -> data -> errors: Field required
  12. distributions[2] -> histfactory_dist -> samples[3] -> modifiers[158] -> staterror -> parameters('staterror'): Field required
  13. distributions[2] -> histfactory_dist -> samples[3] -> modifiers[158] -> staterror -> constraint: Input should be 'Gauss'
  14. distributions[2] -> histfactory_dist -> samples[3] -> modifiers[158] -> staterror -> data('staterror'): Field required

The likelihood I was using for testing was Likelihood_Heavy_X1500_S1000_width0p2.json -- I tried a few of the other ones provided by that analysis, and they all gave similar errors.

[kratsg]

So the problem is that ROOT likely doesn't follow the spec. Some of them are incorrect and can be fixed -- you're mostly referring to the hhh->6b one right? #50 . I will need to understand what it means to have sample data without errors -- I guess it means that the sample has no statistical error associated with it? Not sure that can be the case. The others like the constraint terms -- I can definitely fix. Some others where it says the field is required is where it's missing parameter names -- I really am uncomfortable with having some parameters auto-generated by some internal pattern to ROOT.

[tprocter46]

I think the samples without errors come from the data-driven background estimate (so its not a "real" MC sample) -- perhaps the errors on this data-driven estimate are included as modifiers in some way?

The analysis team have been very helpful to us reinterpreters, so I'm sure if you wanted some more info from them about the likelihoods they'd be happy to help (I can pass questions along too, if helpful).

On the missing parameter names: how likely is it we could get ROOT to try and tidy this sort of thing up? If this keeps happening it rather defies the point of HS3, no?

[cburgard]

Hi!
Speaking as one of the main people pushing the HS3 effort in ROOT: I'm happy to make sure that ROOT and the standard are in sync. If we have identified an area where this is not the case, we can discuss which action makes more sense: adjust the ROOT implementation to adhere to the standard, to adjust the standard to allow for what ROOT is currently doing.

In the case of the missing errors of the samples, the mismatch might be my own fault: HistFactory (as implemented in ROOT) allows for BBlight to selectively apply to only some samples, while not being applied to others. In such cases, the ROOT workspaces that are used to write the HS3 JSONs simply do not contain the information of the sample errors, so I made the conscious decision to omit the field rather than to fill an array full of 0s. I would also advocate to allow this in the standard as it helps to keep the files smaller, adding a small note to the standard stating that if the errors field is missing, this should be understood as equivalent to having an array of 0s here of the same length as the array of entries.
If, however, you feel strongly about this, I'm also happy to patch the ROOT implementation and keep the standard as-is. Let me know if you can follow my reasoning and what resolution you would prefer in this case.

As for the missing parameter names, the very latest versions of ROOT should now include them always, this was an oversight in earlier implementations that should be fixed now. If you import the HS3 file in ROOT and export it using the latest master, the parameter names should hopefully all be there - if they are not, please let me know where they are missing!

Please feel free to continue tagging me on issues like this one!