Add support for new pandas UDF engine #418
Conversation
src/blosc2/proxy.py
Outdated
| """ | ||
| data = cls._ensure_numpy_data(data) | ||
| func = decorator(func) | ||
| if data.ndim in (1, 2): |
There was a problem hiding this comment.
Why this check? I am pretty sure that Blosc2 can handle arrays up to 8 dims (can be made larger by recompiling the underlying C-Blosc2 library). If that is not the case, this is a bug.
There was a problem hiding this comment.
Good point. pandas is the one that shouldn't be sending data with more than 2D. I had this check first as I had separate if branches for 1D and 2D, and I wanted to raise for unknown cases, but this is indeed not really needed. I removed it now.
|
BTW, do you have some preliminary benchmarks to check speed-ups? Thanks for your time! |
Not at this point. I'm not sure about implementing proper benchmarking, since from the pandas side we'll just allow anyone to run the |
|
Yes, Blosc2 is using numexpr behind the scenes, so it will work well with any mathematical function (sin, log, exp), conditionals, where (in the sense of numpy.where); see https://www.blosc.org/python-blosc2/reference/array_operations.html for the full list. In addition, it also supports most of the ufuncs implemented by NumPy, although the performance may not be as stellar as the ones supported by numexpr (listed above). |
|
I forgot to mention that |
|
Great, thanks for confirming. I'm wondering now if it'd be clearer for users if we just raise a NotImplementedError for map. I thought at first that it could make sense to run the vectorized function, but maybe that's confusing for users, if they try the same code with different engines. Maybe better to make engines always behave like the default engine, and raise if the exact behaviour is not supported. What do you think? |
|
I think |
|
I updated the PR so the blosc2 engine passes to the udf the data in the way is expected. And I raise |
|
Thank you Marc! |
|
BTW, I see that pandas-dev/pandas#61467 has been merged into main. Would that mean that we can start benchmarking with pandas main already? If so, can you suggest an existing benchmark that would suitable for this? If there is not an existing one, I can do some quick bench with some largish dataframe in combination with some function. Thanks again! |
|
I'm not sure about a particular benchmark. What I have in mind for the pandas docs is to find some use cases that are a good usage for the different JIT compilers, and benchmark those against not using a JIT compiler. I may be wrong, but I don't think it makes too much sense to have a benchmark suite to compare Blosc2 with Numba or with Bodo.ai, since they solve very different use cases for what I know. I'm not sure if it's easy, but personally I'd like to use real-world cases for that. Like, instead of using |
|
@datapythonista I am giving a try at this implementation for the UDF engine, and I am detecting a strange regression for forthcoming pandas 3.0. With the next script: import numpy as np
import pandas
import blosc2
df = pandas.DataFrame([[4., 9.]] * 3, columns=["A", "B"])
res = df.apply(np.sqrt, engine=blosc2.jit)
print(res)the output for pandas 2.3.2 is: but for pandas-3.0.0.dev0+2416.g10a53051e7: so, current main in pandas is returning a numpy array instead of a dataframe. Is this expected? |
|
Also, I am trying to combine two columns of the dataframe in an expression, but I cannot realize the way of doing it. For example, this is not working: import numpy as np
import pandas
import blosc2
df = pandas.DataFrame([[4., 9.]] * 3, columns=["A", "B"])
# res = df.apply(np.sqrt, engine=blosc2.jit)
res = df.apply(lambda x: x.A + x.B, engine=blosc2.jit)
print(res)which errors (using 3.0.0 main branch here) with: Can you suggest the way (if any) to send different columns to the UDF engine? |
|
I don't think returning a numpy array from apply is expected. But I'm not
working on pandas anymore, since there was a lot of pushback in the project
on anything I was trying to work on including this API. I guess it's a bug,
but better to open an issue in case that's a new expected behavior I don't
understand.
For the exception, the API is quite badly designed (my work on trying to
make it clearer was also often blocked and I discountinued), but I think
you need axis=1 on apply to get data row by row.
…On Sat, Sep 20, 2025, 18:25 Francesc Alted ***@***.***> wrote:
*FrancescAlted* left a comment (Blosc/python-blosc2#418)
<#418 (comment)>
Also, I am trying to combine two columns of the dataframe in an
expression, but I cannot realize the way of doing it. For example, this is
not working:
import numpy as npimport pandasimport blosc2
df = pandas.DataFrame([[4., 9.]] * 3, columns=["A", "B"])
# res = df.apply(np.sqrt, engine=blosc2.jit)res = df.apply(lambda x: x.A + x.B, engine=blosc2.jit)print(res)
which errors (using 3.0.0 main branch here) with:
/Users/faltet/miniforge3/envs/blosc2/bin/python /Users/faltet/blosc/python-blosc2/prova-pandas2.py
Traceback (most recent call last):
File "/Users/faltet/blosc/python-blosc2/prova-pandas2.py", line 8, in <module>
res = df.apply(lambda x: x.A + x.B, engine=blosc2.jit)
File "/Users/faltet/software/pandas/pandas/core/frame.py", line 10784, in apply
result = engine.__pandas_udf__.apply(
data=data,
...<4 lines>...
axis=axis,
)
File "/Users/faltet/blosc/python-blosc2/src/blosc2/proxy.py", line 751, in apply
result = [func(data[:, row_idx], *args, **kwargs) for row_idx in range(data.shape[1])]
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/faltet/blosc/python-blosc2/src/blosc2/proxy.py", line 682, in wrapper
retval = func(*new_args, **func_kwargs)
File "/Users/faltet/blosc/python-blosc2/prova-pandas2.py", line 8, in <lambda>
res = df.apply(lambda x: x.A + x.B, engine=blosc2.jit)
^^^
AttributeError: 'SimpleProxy' object has no attribute 'A'
Process finished with exit code 1
Can you suggest the way (if any) to send different columns to the UDF
engine?
—
Reply to this email directly, view it on GitHub
<#418 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACMXUAB56ZHUUO2OFT55CFT3TU2S5AVCNFSM6AAAAAB52OGVO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJUHEYTCMBYG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Sad to hear that you discontinued your work on pandas. Ok, I'll open a a ticket so that don't fell into the cracks. For the record, specifying axis=1 raises an error in numba (?): |
2 participants
Closes #383
With these changes, it'll be possible to do things like this with pandas and blosc2:
In this PR I'm assuming that
blosc2.jitis intended to be used with vectorized numpy operations. If that's not the case, and you want to support scalar functions and jit Python loops (see example below), let me know and we can implement this for map:Also,
applyis designed in pandas to call the udf for each column or row in a dataframe. I'm passing the whole array since it's a numpy array and I think operations should be vectorized, but I can make it work by columns or rows.