Reading encoded detectorID

[tdixon97]

It was proposed by @oschulz to add a detector ID encoding to the spec, and this was now merged (since noone objected) link-to-spec

This is used in the current Juleana data production, which currently cannot be read into python (big issue).

Apparently generating an implementation of this decoding should be easy.

@gipert / @ggmarshall / @iguinn

[tdixon97]

It is not clear to me if we want to de-code the int into a string when the file is read? Because then we have for example numpy and ak.Arrays of strings which has horrible performance.
Couldn't we just read it in as an int and then make a function to decode? Choice between eg:

dets = lh5.read_as(...) # with decoding so ak.Array([["V0"],["V1"]]])
det_sel = (dets == "V0")

# or 
dets = lh5.read_as( ...) # without decoding so still ak.Array([[0],[1]])
det_sel = (dets == decode_det_name("V0"))

the second would in most cases have far better performance I guess, and for most analysis is equivalent.

[ggmarshall]

I used bytestrings to have things work at the evt tier, not sure how that compares performance wise, how is it used on the juleana end?

[tdixon97]

I have seen the bytestrings in the current pygama evt tier. I think the performance is not particularly good

[iguinn]

Where is the performance bottleneck? Is it an issue with IO, some specific operations, or just generally it is slow? I haven't really done any serious testing/comparison here...

[tdixon97]

I also did nothing very serious, but eg slicing was a bit slow, stuff like

evts.geds.energy[evts.geds.channel_name==...]

Also I noticed that "view_as("ak")" becomes extremely slow

[ggmarshall]

I guess if we implement this we can use the encoding in the evt tier also, maybe 2nd option would be a better move then

[ggmarshall]

we could have a dpsi call tomorrow to discuss if want this done fast

[gipert]

i think we should clearly show and quantify these performance problems before deciding what to do, it should be quite simple to work out a simple test

[tdixon97]

Isn't the mist pressing thing to be able to read the data with this? Even if its just to ints

[oschulz]

In the Julia implementation, we don't decode the Int32 at all, we just wrap it in a custom type the can convert to String. Maybe you can do something like this in Python too, or make a custom string type or so that decodes to a standard string on the fly when required? Most of the time, the only thing you want to do with the detector name is comparing it and using it as as dict key and so on, you only need the "string version" for printing or when comparing to detector name stored as strings.

[gipert]

Isn't the mist pressing thing to be able to read the data with this? Even if its just to ints

this should be rather simple and an issue of https://github.com/legend-exp/legend-lh5io

[iguinn]

I added a class for ArrayOfDetectorIDs, which currently simply wraps Array. For now this will let us read in juleana data with pygama, and treat it as an array of uint32s. In the future, we can implement functionality for string conversions and that sort of thing (overloading view_as may be particularly useful for getting a useful automatic behavior)

[gipert]

Yes I agree @iguinn, maybe we want to convert them to strings when we view the data as Awkward Arrays. But we should quickly check the performance first.