Appending arrays during uproot.iterate and writing them to pandas

[alpetukhov]

Hello everyone,

I'm going through the root files with the uproot.iterate() function, so I've created an array to store the results

hits_store = ak.Array([])

then on every step I'm doing some transormation and appending the result to that array

hits_store = ak.concatenate([hits_store, result])

---

I want to get a DataFrame that looks something like this

               hits  tracks
0          [[1, 2]]      -1
1  [[1, 2], [3, 4]]      -2
2          [[5, 6]]      -3

So if I have an array that looks like this:

arr_hand = ak.Array([
    [[1, 2]], 
    [[1, 2], [3, 4]], 
    [[5, 6]]
])

I need to make it an ApacheArrow and then pass it to the DataFrame constructor (as I've read in Conversion through Apache Arrow)

arrow_hand = ak.to_arrow(arr_hand)
data_hand = {'hits': arrow_hand, 'tracks' : [-1, -2, -3]}
df_hand = pd.DataFrame(data_hand)

But the problem arises them I'm trying to run the ak.to_arrow() on array that was created with ak.concatenate()

arr_1 = ak.Array([
    [[1, 2]],
    [[1, 2], [3, 4]]
])
arr_2 = ak.Array([
       [[5, 6]]
])
arr_conc = ak.concatenate([arr_1, arr_2])
ak.to_arrow(arr_conc)

The arr_conc looks the same as arr_hand but the last line fails with the following error:

ArrowInvalid                              Traceback (most recent call last)
<ipython-input-74-dc961b3eeec4> in <module>
----> 1 ak.to_arrow(arr_conc)

/cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc8-opt/lib/python3.8/site-packages/awkward/operations/convert.py in to_arrow(array, list_to32, string_to32, bytestring_to32)
   2224             )
   2225 
-> 2226     return recurse(layout, None, False)
   2227 
   2228 

/cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc8-opt/lib/python3.8/site-packages/awkward/operations/convert.py in recurse(layout, mask, is_option)
   1933             )
   1934             if mask is None:
-> 1935                 arrow_arr = pyarrow.Array.from_buffers(
   1936                     pyarrow.large_list(content_type),
   1937                     len(offsets) - 1,

/cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc8-opt/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.Array.from_buffers()

/cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc8-opt/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.Array.validate()

/cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc8-opt/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: First or last list offset out of bounds

---

Is this the correct way to store the results during the iteration? If so, why does the ak.to_arrow() fails?

Thanks in advance,
Aleksandr

P.S. Sorry if I'm flooding the GitHub discussions. Tell me if there's more appropriate place for such questions.

[jpivarski]

As a diagnostic step, call

ak.validity_error(arr_conc)

to see if the array is malformed. That's what Arrow is saying, but it would be good to corroborate it with our own error message.

[alpetukhov]

ak.validity_error(arr_conc)

yields nothing and

print(ak.validity_error(arr_conc))

yields None

[jpivarski]

I just tried reproducing your example and I get no errors. What are your versions?

>>> import awkward as ak
>>> import pyarrow as pa
>>> ak.__version__
'1.3.0'
>>> pa.__version__
'4.0.1'
>>> arr_1 = ak.Array([
...     [[1, 2]],
...     [[1, 2], [3, 4]]
... ])
>>> arr_2 = ak.Array([
...        [[5, 6]]
... ])
>>> arr_conc = ak.concatenate([arr_1, arr_2])
>>> ak.to_arrow(arr_conc)
<pyarrow.lib.LargeListArray object at 0x7fc776f09760>
[
  [
    [
      1,
      2
    ]
  ],
  [
    [
      1,
      2
    ],
    [
      3,
      4
    ]
  ],
  [
    [
      5,
      6
    ]
  ]
]
>>> import pandas as pd
>>> pd.DataFrame({"hits": ak.to_arrow(arr_conc)})
               hits
0          [[1, 2]]
1  [[1, 2], [3, 4]]
2          [[5, 6]]