2023-10-05 kernel meeting notes

[zachschuermann]

summary

POC from @nicklan to derisk C FFI (that we can entirely encapsulate C FFI unsafe behind rust trait). Discussion regarding constraints we must impose on C FFI implementors (e.g. keep pointers/memory live for pieces we require). UC discussion started - case: we want to read UC-managed tables from e.g. polars. Kernel does the delta side, but need to integrate with UC somehow.

action items from last time:

• list of connectors to target (add Clickhouse and pytorch) - List of connectors to target #48
• [LATER] look into stacked PRs (graphite, can we use internal tool, etc.)

attendees

@rtyler @roeap @nicklan @MrPowers @zachschuermann @tdas

notes stream

from @nicklan

• there was concern over C FFI: can we lock all the unsafe code in FFI layer or does it have to bubble up to e.g. Rust API
• did a quick POC (c-dyn prototype) to prove we can lock all the unsafe in the C FFI
• TLDR rust people can write trait that requires methods/functions, C FFI and unsafe is entirely encapsulated in safe rust trait:
  • rust we write rust impl of the trait
  • C FFI we write our own impl of the traits that use lower level types that we pass back and forth across the C boundary
    • took model of passing function pointers back and forth
    • impl of the trait makes a rust function that calls into the function pointers (unsafe)
    • but this is down in FFI layer and all hidden behind a nice happy kind sweet fun safe rust trait
• some of the types in kernel have rust-level semantics (e.g. Send)
  • we have to very carefully document these for the C-side
  • Send describes/implies ownership?
  • ownership passing back and forth FFI boundary
  • @rtyler : still thinks Send trait bound is useful (but have to communicate this to C side somehow)
• kernel has 2-sided interface (engine -> kernel and kernel -> engine like for parquet read)
  • are concerns above regarding inbound or outbound calls?
  • both
• if you give kernel something from C side, there are 2 requirements: (1) don't free it, (2) don't access/move/modify it. That is, you transfer ownership to rust.
  • concern: things that cross FFI are pointers or raw types (repr C structs)
  • todo fill in
• big challenge is being consistent on the API layer
• c-dyn proto proves that we can effectively pass function pointers across FFI and use them in a sensible way
• zach thought: types! no rust types in C (have to make our own iterators, etc.)?
• ownership in function pointers? there's a void * something. in rust world, function pointers are 'static -> they are just code and never get freed, etc. Rust assume no crazy stuff going on.
  • if the functions are coming from python they may be dynamic?
  • but for all practical purposes they are static?
  • should 'live' for the life of the kernel
  • we own the function pointer for as long as we need it. this likely needs to be explicitly documented/clear for the engine API
• LDUNLOAD could blow stuff up lol
  • someone like redshit could be doing crazy code gen etc
  • these people will likely be used to these sorts of constraints
  • don't pull the rug out from under the kernel
• summary: we have to explicitly call out lifetimes esp for FFI
  • while a snapshot is live we require pointers from the engine to live

from @ryan-johnson-databricks

The other thing with kernel and pointer liveness -- ideally, most kernel method calls take engine state as args, so kernel doesn't need to hang onto anything between method calls. even iterators, which need to hang onto some internal state, should minimize that internal state to the extent possible. And then the engine only has to guarantee that the pointer remains live until the kernel method call returns.

UC thread

• UC in delta-rs right now: just do a simple prod.zach.table -> s3://bucket/table translation
• no credentials given out to actually read the table
• have to have credentials to the underlying storage set up orthogonally
• ideally, you just bring databricks/UC credentials and it hands out fine-grained (at least table-level) access
• ask: kernel as the leader -> come up with a UC story here
• case study: you want to allow polars users to do scan_delta(prod.zach.table) for UC-managed tables
• complicated by vendor attitudes about catalogs
• need Delta table to know to commit via something other than file system (as table feature)
  • takes care of commit path (can commit with catalog/commit service in the path)
• catalog that governs discovery, permissions, etc. -> UC
• vision: open catalog. problem: not communicated. current status: needs design/eng work.
• UC being pushed on customers means there is all the overhead and none of the benefit for e.g. Scribd since there isn't open source integration they can use.

aside: pointer to type T you can turn into reference. pointer is unsafe, reference is managed. unsafe cast from pointer to reference. (from_raw). to turn reference into_raw (into pointer). unwrap box (Box::into_inner? unstable?)

• tricky ownership
• if we own the pointer, then are responsible for freeing, it won't free when it goes out of scope if it is a reference
• we might require things to point us a ref with a destructor
• if you take a non-ref in function, you own it. if you return it then you give back ownership, if not it is destroyed at the end of function scope.
• for ref, it is 'borrowed' by the function and handed back to calling scope after function return
• ryan thought: maximize the number of references
• iterators are tricky? internal state? -> try to minimize internal state.
• if kernel takes memory from the engine and we have to free it then we have to be given a destructor
  • but we don't want this :)
  • we can have things passed by ref
  • engine governs memory

from @zachschuermann

• scan.execute() returns a vector? Why not iterator?

[rtyler]

One of the follow ups from this is that @nicklan created a repo called cdyn which demonstrated the approach for creating the C FFI bridge with our traits and function pointers for user-defined behavior. I've since taken that prototype and have been starting to pull FileSystemClient and dyn Iterator into the C FFI bridge to further explore the API 🥳

[zachschuermann]

awesome thanks @rtyler!!!

[ryan-johnson-databricks]

@rtyler, one question about that: Doesn't dyn do reference counting, which means Rust will eventually try to "free" the object when its refcount goes to zero? Or did I mix that up with something else we discussed in the meeting?

[wjones127]

Doesn't dyn do reference counting

No, the only thing that does reference counting in Rust is Arc and Rc (the equivalents of C++ shared_ptr).