Skip to content

Conversation

@ikrommyd
Copy link
Collaborator

This allows us to disable the caching of the materialized array on the virtual array object.
The virtual arrays will still be aware of their generators and be able to materialize again (they just need to call the generator again).
The option is passed through from_buffers which is currently the only publicly exposed way to create virtual arrays.
We also have the freedom with this PR to disable the caching only for specific types of buffers. You may want to keep it for offsets only for example but disable it for all the contents. This PR lets you do that.
The reason for this PR is to enable us the ability to handle caching outside of awkward and be able to handle memory and clean up loaded arrays if we want to. This becomes important in low memory workers where you may have needed to materialize some branches at the beginning of the analyses to do a couple of cuts but you don't need them anymore and want to "dematerialize" them to free up memory.

@pfackeldey this is in line with what we discussed regarding what options to allow and what not.

@ikrommyd ikrommyd requested a review from pfackeldey November 20, 2025 09:46
@codecov
Copy link

codecov bot commented Nov 20, 2025

Codecov Report

❌ Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.71%. Comparing base (b749e49) to head (197a4b5).
⚠️ Report is 474 commits behind head on main.

Files with missing lines Patch % Lines
src/awkward/operations/ak_from_buffers.py 83.33% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/_nplikes/array_module.py 95.25% <ø> (+8.87%) ⬆️
src/awkward/_nplikes/virtual.py 90.68% <100.00%> (ø)
src/awkward/_pickle.py 78.94% <ø> (-8.36%) ⬇️
src/awkward/forms/form.py 81.33% <ø> (-0.94%) ⬇️
src/awkward/highlevel.py 78.30% <ø> (+1.63%) ⬆️
src/awkward/operations/ak_from_avro_file.py 80.95% <ø> (ø)
src/awkward/operations/ak_from_iter.py 92.00% <ø> (ø)
src/awkward/operations/ak_from_safetensors.py 88.88% <ø> (ø)
src/awkward/operations/ak_from_buffers.py 94.17% <83.33%> (+0.05%) ⬆️

... and 193 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3741

Copy link
Collaborator

@pfackeldey pfackeldey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd please find my review in the ak.from_buffers changes :)

@ikrommyd ikrommyd requested a review from pfackeldey November 26, 2025 22:11
Copy link
Collaborator

@pfackeldey pfackeldey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ikrommyd, this looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants