docs: user guide + runnable examples for distributing expressions#1547
Open
timsaucer wants to merge 4 commits into
Open
docs: user guide + runnable examples for distributing expressions#1547timsaucer wants to merge 4 commits into
timsaucer wants to merge 4 commits into
Conversation
This was referenced May 15, 2026
Wraps up the Expr-pickle work with the user-facing material: * docs/source/user-guide/io/distributing_work.rst — new user guide page covering the multiprocessing, Ray, and datafusion-distributed patterns. Includes the Security section that is the canonical home for the cloudpickle / pickle.loads threat model. * docs/source/user-guide/io/index.rst — toctree entry. * examples/multiprocessing_pickle_expr.py — runnable example: a Pool.map of a closure-capturing UDF across processes, with worker context registration in the initializer. * examples/ray_pickle_expr.py — Ray actor analogue. * examples/datafusion-ffi-example/python/tests/_test_pickle_strict_ffi.py — exercises the strict-mode refusal end to end against an FFI capsule scalar UDF (kept under the FFI example crate because the test needs that crate's compiled artifacts). * examples/README.md — index entries for the new files. Also tightens three docstrings that previously duplicated the security warning so they point at the canonical Security section instead: * PythonLogicalCodec::with_python_udf_inlining (rustdoc): one-line summary plus a relative pointer to distributing_work.rst and the upstream Python pickle module security warning. * SessionContext.with_python_udf_inlining: one-sentence summary plus :doc: link to the user guide. * datafusion.ipc module docstring: cross-reference to the user guide for the full pattern. The crate-level codec.rs module rustdoc also updates "pure-Python scalar UDFs" to "scalar / aggregate / window UDFs" now that all three are covered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
timsaucer
force-pushed
the
pr4-docs-examples
branch
from
20b3338 to
f7550ec
Compare
…ne UDFs Reviewer feedback on the Expr-pickle PRs (apache#1544) asked that the cloudpickle portability caveats be discoverable on the user-facing page, not only in docstrings. The distributing_work.rst page is the designated canonical home for the distribution story, so add them here: * New 'Portability requirements for inline Python UDFs' subsection covering the matching-Python-minor-version requirement and the by-value vs by-reference import-capture rule (imported modules must be importable on the worker). * Qualify the 'fully portable' Python-UDF bullet to point at the new requirements. * Cross-reference the new subsection from the closure-capture note. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
timsaucer
changed the title
Two codec.rs docstrings were reworded in PR4 in ways that dropped information: * try_encode_python_scalar_udf: restore the `DFPYUDF` family prefix + version byte description of the payload framing (PR4 had collapsed it to `DFPYUDF1` prefix, dropping the version-byte mention). * cloudpickle cached-handle comment: restore "The encode/decode helpers above" wording.
The 'Worker layout' docstring described tasks as `(expr, label)` but the code builds and unpacks them as `(label, expr)`. Correct the doc to match.
11 tasks
ntjohnson1
approved these changes
May 28, 2026
Contributor
ntjohnson1
left a comment
ntjohnson1
left a comment
There was a problem hiding this comment.
The user guide is a bit dense, however until I use this day to day a bit more no thoughts on where this could be simplified (if at all).
I think the examples are really clear though, so someone less familiar and wanting to just get moving should be fine with just the examples. Maybe add a note to the examples at the beginning (in addition to see also at the end) just in case so it's clear.
Overall excited for all of this to land in the next release!
| evaluate, | ||
| [(expr, [1, 2, 3]), (expr, [10, 20, 30])], | ||
| ) | ||
| print(results) # [[2, 4, 6], [20, 40, 60]] |
Contributor
There was a problem hiding this comment.
NIT: I think technically if you use the >>> doctest format it can run and verify this result even in rst
| ) | ||
| print(results) # [[2, 4, 6], [20, 40, 60]] | ||
|
|
||
| When saved to a ``.py`` file and executed with the ``spawn`` or |
Contributor
There was a problem hiding this comment.
NIT I don't know if you use a standard note call out elsewhere in the docs. This is also mostly instructions related to multiprocessing in python rather than datafusion specific so maybe an external link to those docs for those unfamiliar.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1520
Rationale for this change
PRs 1-3 close the round-trip for Python UDFs and add the toggle that controls it. None of that is discoverable without user-facing documentation. This PR ships the user guide page that explains the multiprocessing / Ray / datafusion-distributed patterns, the runnable examples, and the centralized Security section that nails down what the toggle does and does not protect against.
What changes are included in this PR?
User guide.
datafusion.ipc.set_worker_ctx).datafusion.ipc.set_sender_ctxandSessionContext.with_python_udf_inlining).pickle.loadsthreat model.Runnable examples.
examples/multiprocessing_pickle_expr.py—Pool.mapof a closure-capturing UDF across processes, with the worker initializer wiring the worker context. The closure carries non-trivial state to demonstrate that captured state survives the round-trip.examples/ray_pickle_expr.py— Ray actor analogue.Are there any user-facing changes?
Docs and examples only. No code behavior changes, no new public APIs.