[BSE-4737] Implement interface for using bodo as engine in Pandas UDFs. #410

scott-routledge2 · 2025-04-16T18:49:03Z

Changes included in this PR

Subclass BaseExecutionEngine and implement BodoExecutionEngine, add as an attribute to bodo.jit / return_wrapped_fn.
See pandas-dev/pandas#61032 for context.

As a followup we can also implement kwargs (but not key word only arguments, basically do something similar to numba where they fold keyword arguments into args using the function signature.)

Also need to do Series.map support, however that cannot be tested right now because it hasn't been implemented in Pandas.

Testing strategy

In order to test, I needed to build Pandas from source. Here are the steps I took to test:

Remove pandas from environment i.e. pixi remove pandas. Also need to update pyproject.toml. Note pyproject.toml and pixi.lock/toml will be restored before merging.
Uninstall bodo, etc: pixi run clean
Clone pandas and install from source: cd pandas; pip install .
Build bodo: pixi run build-bodo

Ran PR CI for changes to jit decorator.

Run tests locally in spawn mode.

User facing changes

Once Pandas 3 is released, users can use Bodo to accelerate their UDFs in Pandas.
e.g.

df.apply(func, axis=1, engine=bodo.jit, args=args)

Checklist

Pipelines passed before requesting review. To run CI you must include [run CI] in your commit message.
I am familiar with the Contributing Guide
I have installed + ran pre-commit hooks.

for more information, see https://pre-commit.ci

scott-routledge2 · 2025-04-16T19:52:50Z

bodo/pandas_compat.py

+
+    class BodoExecutionEngine(BaseExecutionEngine):
+        @staticmethod
+        def map(


This is not implemented yet in Pandas so there is no way to test. Leaving as a followup

I'll open a PR adding engine to Series.map shortly. I have the implementation already, but map and apply tests are structured very differently, and I need to see how to implement the common fixtures without making the mess bigger or refactoring all the tests.

…ai/Bodo into scott/pandas_bodo_executor

for more information, see https://pre-commit.ci

…ai/Bodo into scott/pandas_bodo_executor

codecov · 2025-04-16T22:31:47Z

Codecov Report

Attention: Patch coverage is 18.96552% with 47 lines in your changes missing coverage. Please review.

Project coverage is 66.52%. Comparing base (76705aa) to head (e9cc70d).
Report is 11 commits behind head on main.

❌ Your project check has failed because the head coverage (66.52%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #410      +/-   ##
==========================================
- Coverage   66.59%   66.52%   -0.07%     
==========================================
  Files         176      176              
  Lines       63177    63288     +111     
  Branches     8835     8856      +21     
==========================================
+ Hits        42072    42103      +31     
- Misses      18483    18561      +78     
- Partials     2622     2624       +2

scott-routledge2 · 2025-04-17T16:01:06Z

pixi.toml

@@ -70,7 +70,6 @@ pip = "*"
 # Core Python Deps
 numba = "==0.61.0"
 numpy = ">=1.24,<2.2"
-pandas = ">=2.2,<2.3"


For testing locally, will restore before merging.

for more information, see https://pre-commit.ci

…ai/Bodo into scott/pandas_bodo_executor

for more information, see https://pre-commit.ci

DrTodd13

LGTM. Thanks.

scott-routledge2 · 2025-04-17T18:16:31Z

bodo/pandas_compat.py

+                args_str += ","
+            else:
+                # Add dummy value for args for spawn mode compatibility.
+                # TODO: fix in spawn mode.


I have a PR to fix this here: #414

ehsantn

Thanks @scott-routledge2 ! Looks good to me.

ehsantn · 2025-04-17T18:10:33Z

pyproject.toml

-    "pandas>=2.2,<2.3",
+    "pandas>=2.2,<3.1",


ehsantn · 2025-04-17T18:17:42Z

bodo/pandas_compat.py

+                def f(func_text, glbls, loc_vars, __name__):
+                    bodo_exec(func_text, glbls, loc_vars, __name__)
+
+            spawner.submit_func_to_workers(f, [], apply_func_text, glbls, {}, __name__)


Wrong indentation looks like. Should be inside if spawn_mode.

Good catch, I just realized this codepath is taken even when engine=bodo.jit(spawn=False, distributed=False). Would there be a way to also case on the decorator?

What do you mean?

It will exec the func text on the workers even when spawn=False. Discussed offline it would be better to just avoid func_text altogether.

ehsantn · 2025-04-17T18:20:09Z

bodo/tests/test_pandas_udf_compat.py

+
+def test_udf_cache():
+    """Tests that we can call the same UDF multiple times with cache flag on
+    without any errors. TODO: check cache."""


Have you verified caching manually at least?

…s. (#410) Subclass BaseExecutionEngine and implement BodoExecutionEngine, add as an attribute to bodo.jit / return_wrapped_fn. As a followup we can also implement kwargs (but not key word only arguments, basically do something similar to numba where they fold keyword arguments into args using the function signature.) Also need to do Series.map support, however that cannot be tested right now because it hasn't been implemented in Pandas. (cherry picked from commit 30b2bc1)

scott-routledge2 and others added 2 commits April 16, 2025 14:48

add BodoExecutionEngine class and implement basic apply method

2662e72

[pre-commit.ci] auto fixes from pre-commit.com hooks

cafbc21

for more information, see https://pre-commit.ci

scott-routledge2 commented Apr 16, 2025

View reviewed changes

scott-routledge2 and others added 9 commits April 16, 2025 16:20

add more tests

bda9f6b

get some tests to pass

4391e8c

Merge branch 'scott/pandas_bodo_executor' of https://github.com/bodo-…

bc916d7

…ai/Bodo into scott/pandas_bodo_executor

tests passing

9407fe9

[pre-commit.ci] auto fixes from pre-commit.com hooks

3da1514

for more information, see https://pre-commit.ci

restore env

e01bda1

[run ci]

48d1694

Merge branch 'scott/pandas_bodo_executor' of https://github.com/bodo-…

e13ff24

…ai/Bodo into scott/pandas_bodo_executor

[run ci]

a6cf81e

support kwargs

d76983b

scott-routledge2 commented Apr 17, 2025

View reviewed changes

pre-commit-ci bot and others added 6 commits April 17, 2025 16:01

[pre-commit.ci] auto fixes from pre-commit.com hooks

a177ea6

for more information, see https://pre-commit.ci

simplify func_text

2325d0a

Merge branch 'scott/pandas_bodo_executor' of https://github.com/bodo-…

e6d3bf7

…ai/Bodo into scott/pandas_bodo_executor

docstring

b6eed0c

[pre-commit.ci] auto fixes from pre-commit.com hooks

fea5452

for more information, see https://pre-commit.ci

restore pixi files [run ci]

e9cc70d

scott-routledge2 marked this pull request as ready for review April 17, 2025 17:15

scott-routledge2 changed the title ~~[WIP] Implement interface for using bodo as engine in Pandas UDFs.~~ [BSE-4737] Implement interface for using bodo as engine in Pandas UDFs. Apr 17, 2025

scott-routledge2 requested review from ehsantn and DrTodd13 April 17, 2025 17:41

scott-routledge2 mentioned this pull request Apr 17, 2025

Support passing empty tuple as argument in spawn mode #414

Merged

3 tasks

DrTodd13 approved these changes Apr 17, 2025

View reviewed changes

scott-routledge2 commented Apr 17, 2025

View reviewed changes

ehsantn approved these changes Apr 17, 2025

View reviewed changes

pr comments

ef33644

typo

e97d895

scott-routledge2 merged commit 30b2bc1 into main Apr 17, 2025
10 checks passed

scott-routledge2 deleted the scott/pandas_bodo_executor branch April 17, 2025 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BSE-4737] Implement interface for using bodo as engine in Pandas UDFs. #410

[BSE-4737] Implement interface for using bodo as engine in Pandas UDFs. #410

Uh oh!

scott-routledge2 commented Apr 16, 2025 •

edited

Loading

Uh oh!

scott-routledge2 Apr 16, 2025

Uh oh!

datapythonista Apr 20, 2025

Uh oh!

codecov bot commented Apr 16, 2025 •

edited

Loading

Uh oh!

scott-routledge2 Apr 17, 2025

Uh oh!

DrTodd13 left a comment •

edited

Loading

Uh oh!

scott-routledge2 Apr 17, 2025

Uh oh!

ehsantn left a comment

Uh oh!

ehsantn Apr 17, 2025

Uh oh!

ehsantn Apr 17, 2025

Uh oh!

scott-routledge2 Apr 17, 2025

Uh oh!

ehsantn Apr 17, 2025

Uh oh!

scott-routledge2 Apr 17, 2025

Uh oh!

ehsantn Apr 17, 2025

Uh oh!

scott-routledge2 Apr 17, 2025

Uh oh!

Uh oh!

Uh oh!

[BSE-4737] Implement interface for using bodo as engine in Pandas UDFs. #410

[BSE-4737] Implement interface for using bodo as engine in Pandas UDFs. #410

Uh oh!

Conversation

scott-routledge2 commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes included in this PR

Testing strategy

User facing changes

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DrTodd13 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehsantn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

scott-routledge2 commented Apr 16, 2025 •

edited

Loading

codecov bot commented Apr 16, 2025 •

edited

Loading

DrTodd13 left a comment •

edited

Loading