Skip to content

Conversation

lshaw8317
Copy link
Collaborator

@lshaw8317 lshaw8317 commented Oct 4, 2025

Although there was no double evaluation in old versions of blosc2 for reductions as there was an is_inside_new_expr() stack check which aborted the evaluation and returned blosc2.zeros during the eval call, we have added a shape parser to avoid having to arrive at reduce_slices for reductions. In addition, this allows us to handle other non-numexpr functions without adding the same is_inside_newexpr() call.
This PR is thus useful since 1) it adds the possibility to compute with arbitrary blosc2 functions (such as matmul or even logaddexp) in a lazy imperative way via the changes to ne_evaluate and 2) adds a reasonably extensive shape parser function, which is a bit more explicit than the returning blosc2.zeros hack.

For the following benchmark:

import time
import blosc2

# --- Experiment Setup ---
N = 100000
X = blosc2.ones((N,N))
t0 = time.time()
blosc2.evaluate('sum(X)')
dt = time.time()
print(f"Time for blosc2.evaluate('sum(X)'):{round(dt, 3)} s")

one can confirm that the new method also evaluates the reduction only once on the full operands, and takes the same time as in blosc2<=3.9.1.

Copy link
Member

@FrancescAlted FrancescAlted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only have a couple of comments, but mainly cosmetic.

Also, make sure that the new way to compute, e.g. reductions, does perform similarly to the previous one.

@lshaw8317 lshaw8317 changed the title Fixed double evaluations for reductions Add general lazy evaluation for all blosc2 functions Oct 7, 2025
@lshaw8317 lshaw8317 merged commit 8077b08 into main Oct 8, 2025
15 checks passed
@lshaw8317 lshaw8317 deleted the extend_array_handling branch October 8, 2025 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants