Coffea with virtual arrays and executors discussion

[ikrommyd]

Hi all,

We have made three coffea pre-releases that you can see under the releases page.
You can install them via pip install --pre coffea.

These pre-releases are the first pre-releases including the coffea 0.7-like executors in coffea 2025 that are to be used with the newly developed virtual arrays.
One major thing to note is that the default nanoevents behavior switched from reading with dask-awkward by default, to reading with virtual arrays.
The delayed kwarg of nanoevents' from_root method has been removed and now there's only a mode kwarg with the default value being "virtual".
The 3 available options for this kwarg are explained below:

• "virtual": the new default, loads data using virtual arrays
• "dask": was the previous default and is identical to the olddelayed=True kwarg, loads data using dask-awkward
• "eager": identical to the old delayed=False, loads the whole file/chunk on the spot eagerly

If you want to load things with dask-awkward, all you need to do is specify mode="dask" in from_root and you get exactly the previous default behavior.
No other parts of the code that change behavior have been touched so that's the only thing you need to change to keep your existing dask-awkward code running
Also, a coffea analysis facility image has just been built using the pre-releases so if you use those or run into them through coffea casa for example, please keep in mind the from_root breaking change to adapt your code to.

The executors API is identical to coffea 0.7 at the moment but we expect changes there and some kind of overlap of the executors with the dask specific utils like the preprocessing and apply_to_fileset. Feel free to try the pre-releases out and start migrating your coffea 0.7 code to coffea 2025 using virtual arrays. We are looking forward to your feedback!

Cheers!
Iason for the coffea developers team

[ikrommyd]

There is an open issue #1331 regarding fsspec-xrootd that, while it has nothing to do with virtual arrays and executors, is currently blocking running the executors with files over xrootd.