Skip to content

Commit ad9b779

Browse files
authored
Implement physical and logical codecs in FFI (#19079)
## Which issue does this PR close? Addresses part of #18671 but does not close it. ## Rationale for this change We have a major restriction in our current FFI crates. They use protobuf encoding and decoding to pass around logical expressions. This encoding and decoding will fail when we encounter a user defined function because we only currently use the default extension codecs. This PR introduces the FFI variants for both logical and physical extension codecs. A follow on PR will enable these in our code base, on order to keep these PRs to a reasonable length for review. In addition to the benefit to the FFI crate for passing expressions, this will also enable sending plans across FFI boundaries, which may be useful for `datafusion-distributed`. ## What changes are included in this PR? - Implement `FFI_LogicalExtensionCodec` - Implement `FFI_PhysicalExtensionCodec` This PR does _not_ use these structures in the current code. That is coming as part of a later PR in an effort to keep the size of the PRs small for effective code review. ## Are these changes tested? Unit tests are added. Coverage report: <img width="523" height="222" alt="Screenshot 2025-12-03 at 4 37 23 PM" src="https://github.com/user-attachments/assets/4e139052-c74c-4e32-a057-66bd4bd68e21" /> ## Are there any user-facing changes? No.
1 parent 83736ef commit ad9b779

File tree

8 files changed

+1444
-7
lines changed

8 files changed

+1444
-7
lines changed

Cargo.lock

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

datafusion/ffi/Cargo.toml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,15 @@ arrow-schema = { workspace = true }
4747
async-ffi = { version = "0.5.0", features = ["abi_stable"] }
4848
async-trait = { workspace = true }
4949
datafusion = { workspace = true, default-features = false }
50+
datafusion-catalog = { workspace = true }
5051
datafusion-common = { workspace = true }
52+
datafusion-datasource = { workspace = true }
5153
datafusion-execution = { workspace = true }
5254
datafusion-expr = { workspace = true }
5355
datafusion-functions-aggregate-common = { workspace = true }
5456
datafusion-physical-expr = { workspace = true }
5557
datafusion-physical-expr-common = { workspace = true }
58+
datafusion-physical-plan = { workspace = true }
5659
datafusion-proto = { workspace = true }
5760
datafusion-proto-common = { workspace = true }
5861
futures = { workspace = true }
@@ -63,6 +66,10 @@ tokio = { workspace = true }
6366

6467
[dev-dependencies]
6568
datafusion = { workspace = true, default-features = false, features = ["sql"] }
69+
datafusion-functions = { workspace = true }
70+
datafusion-functions-aggregate = { workspace = true }
71+
datafusion-functions-aggregate-common = { workspace = true }
72+
datafusion-functions-window = { workspace = true }
6673
doc-comment = { workspace = true }
6774

6875
[features]

datafusion/ffi/src/execution_plan.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -301,7 +301,7 @@ impl ExecutionPlan for ForeignExecutionPlan {
301301
}
302302

303303
#[cfg(test)]
304-
mod tests {
304+
pub(crate) mod tests {
305305
use super::*;
306306
use arrow::datatypes::{DataType, Field, Schema};
307307
use datafusion::{

datafusion/ffi/src/lib.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ pub mod expr;
3434
pub mod insert_op;
3535
pub mod physical_expr;
3636
pub mod plan_properties;
37+
pub mod proto;
3738
pub mod record_batch_stream;
3839
pub mod schema_provider;
3940
pub mod session_config;

0 commit comments

Comments
 (0)