-
|
Dear pyhf developers, I have been starting to use pyhf on some HistFitter workspaces (~100) which I am converting to JSON. They each have 35 regions / bins and ~150 systematics with up and down variations. Converting the xml files to json takes roughly 30 hours. Is there any way we could speed this up? Looking at the source code it seems as if each file containing a histogram were opened one-by-one. Do you think there'd be any way to aggregate the files and histogram names first and then to read them out in one go? While this might require a slight refactoring of the readxml, it might increase the speed substantially. Looking forward to you oppinion and potentially further speed-up suggestions, |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 17 replies
-
|
Hi! is the 30 hours a single workspace of all 100 workspaces? We du use a cache |
Beta Was this translation helpful? Give feedback.
-
|
For debugging purposes it may also be interesting to know how this scales with the number of regions. You should be able to remove regions from the top-level xml, and checking runtime with a single region might be a bit easier to handle. What is the memory usage like? I would be surprised if that would be the problem here, but could you be running out of memory with too many open files, and that somehow slows things down significantly? |
Beta Was this translation helpful? Give feedback.
-
|
do you have some numbers as tto how many modifiers/s and sampes/s you gett (this should be shown by the cmd line tool). Have you tried simplifying the workspace (one chanel, one sample) for dbeugging? |
Beta Was this translation helpful? Give feedback.
-
|
I have been constantly monitoring the memory consumption and that did not seem to be an issue, so far. I have nevertheless reduced the running to one region for the moment. @lukasheinrich there are 9 samples and 272 modifiers and from the progress bar (which I only just turned on, i.e. it was not slowing things down in the 30h case), it's taking ~5.3 s to process one |
Beta Was this translation helpful? Give feedback.
-
|
Are you running this locally or on some shared filesystem where maybe file access could be a bottleneck? |
Beta Was this translation helpful? Give feedback.
-
|
This is running on DESY's |
Beta Was this translation helpful? Give feedback.
-
|
Hi again, To compare the performance, I wrote a little script that compares the performance of python quickAccessTest.py
...
It took 0 seconds to parse with pyRoot
It took 8 seconds to parse with upRootI have also attached the profiling of these two. I am not sure whether the comparison of this to Any further help would be highly appreciated! Cheers, |
Beta Was this translation helpful? Give feedback.
-
|
This smells like a filesystem issue unfortunately. I've seen this in the past with other slow file mounts. How large are the input XML and ROOT files? Can you give me access to them so I can try running them on a different filesystem to confirm? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @kratsg, |
Beta Was this translation helpful? Give feedback.
-
|
Ok, this is interesting. ~80% of your modifiers will load in about 1second from the ROOT file, but a handful (primarily |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @kratsg! :) |
Beta Was this translation helpful? Give feedback.
-
|
Ok, the problem is partially in In |
Beta Was this translation helpful? Give feedback.
-
|
Quickly fixing, I have a nice speedup: |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thank you so much @kratsg! Would you mind sharing your fix? Then I can run on all of my files in no time! |
Beta Was this translation helpful? Give feedback.
-
|
Here's the changes made on master of pyhf, but I should have a PR soon. diff --git a/src/pyhf/readxml.py b/src/pyhf/readxml.py
index c078ce3e..2e6dec51 100644
--- a/src/pyhf/readxml.py
+++ b/src/pyhf/readxml.py
@@ -59,19 +59,21 @@ def import_root_histogram(rootdir, filename, path, name, filecache=None):
fullpath = str(Path(rootdir).joinpath(filename))
if fullpath not in filecache:
f = uproot.open(fullpath)
- filecache[fullpath] = f
+ keys = set(f.keys(cycle=False))
+ filecache[fullpath] = (f, keys)
else:
- f = filecache[fullpath]
- try:
+ f, keys = filecache[fullpath]
+
+ fullname = "/".join([path, name])
+
+ if name in keys:
hist = f[name]
- except (KeyError, uproot.deserialization.DeserializationError):
- fullname = "/".join([path, name])
- try:
- hist = f[fullname]
- except KeyError:
- raise KeyError(
- f'Both {name} and {fullname} were tried and not found in {fullpath}'
- )
+ elif fullname in keys:
+ hist = f[fullname]
+ else:
+ raise KeyError(
+ f'Both {name} and {fullname} were tried and not found in {fullpath}'
+ )
return hist.to_numpy()[0].tolist(), extract_error(hist) |
Beta Was this translation helpful? Give feedback.
-
|
Wonderful, thank you! :) |
Beta Was this translation helpful? Give feedback.
-
|
For posterity, this was fixed in PR #1691. |
Beta Was this translation helpful? Give feedback.
Ok, the problem is partially in
uprootand inpyhf. Inuprootcode (thanks @jpivarski !) there is adamerau_levenshteinfunction being called when we hit a missing key that takes a long time because the number of keys in this file is very large (https://github.com/scikit-hep/uproot4/blob/85f219a36e76dffc18da4756227a7beb760657a0/src/uproot/_util.py#L810-L858).In
pyhf, when we hit the name of a histogram that is not retrievable without trying the full path first - then it causes a (slow)DeserializationErrorwhich is caught by an expected exception inpyhf. We need to change the way we check if a key exists in the file. This is a bug.