-
Notifications
You must be signed in to change notification settings - Fork 99
feat: ability to optionally disable the caching of virtual arrays on themselves #3741
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files
🚀 New features to boost your workflow:
|
|
The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3741 |
pfackeldey
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ikrommyd please find my review in the ak.from_buffers changes :)
pfackeldey
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @ikrommyd, this looks good to me.
This allows us to disable the caching of the materialized array on the virtual array object.
The virtual arrays will still be aware of their generators and be able to materialize again (they just need to call the generator again).
The option is passed through
from_bufferswhich is currently the only publicly exposed way to create virtual arrays.We also have the freedom with this PR to disable the caching only for specific types of buffers. You may want to keep it for offsets only for example but disable it for all the contents. This PR lets you do that.
The reason for this PR is to enable us the ability to handle caching outside of awkward and be able to handle memory and clean up loaded arrays if we want to. This becomes important in low memory workers where you may have needed to materialize some branches at the beginning of the analyses to do a couple of cuts but you don't need them anymore and want to "dematerialize" them to free up memory.
@pfackeldey this is in line with what we discussed regarding what options to allow and what not.