2025-07-10

[OussamaSaoudi]

Summary

0.12.0 released, and 0.12.1 (probably) coming very soon! (fixing azure issues)

current PRs

• IN/NOT IN PR to merge @roeap (waiting on opaque expressions. then will go in to default engine)
• !!!! opaque expressions @scovich
  • arrow opaque expressions need another level of indirection
  • need an 'arrow expression'
• #[derive(IntoEngineData)] (zach)
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)

upcoming

• zach: metrics PR!
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory

Attendees

@nicklan @scovich @mightyshazam @joonyoo181 @hntd187

notes stream

• @mightyshazam Regarding error handling, kernel asks for void functions that may produce errors. THe ask is to have these functions return an error so that it can bubble up and be handled gracefully. There's also an issue with panics. Currently panics produced in rust are not caught by the foreign code.
• @scovich To be clear, if a panic is thrown in rust, control is passed into host language (ex: c++), and control is returned to rust, then the panic is caught. Rust probably can't catch foreign exceptions though. FFI code must catch their exceptions and pass it gracefully to rust.
• @mightyshazam If rust panics, it comes to dotnet as an unmanaged exception if we're lucky. In the worst case it's a fault.
• @scovich TODO: make our ffi extern "c-unwind". Otherwise a panic is undefined behaviour. We may also need a "rust-unwind". The other thing to check is if dot-net throws an exception, will it be caught be rust's exception catching. It would be bad if we did that, because rust shouldn't swallow the exception
• @mightyshazam currently the function signatures are treated as infallible, but that's not really the case.
• @scovich there's also edge cases with double faults, even if we catch/unwind the panic.
• @mightyshazam certain faults may abort the entire VM. Others can be handled.
• @scovich If an unwind from foreign code is initiated, does rust catch that panic/exception unwind.
• @scovich c-abi says that if you have a fallible rust function calling a foreign callback that throws an exception, we need to decide how to handle. Either: 1) the callback must catch its own exception and return an error. 2) try to figure out how to handle panic unwinds.
• @mightyshazam I prefer explicit errors.
• @oussama: are all exceptions in foreign code the same? C++ and Java likely have different exception types.
• @nicklan To be clear, this is the case where Rust detects an error and we request dotnet to allocate the error. THe initial approach was to throw an exception directly from the error allocator. But rust still "has control", so it interferes with the exception throwing.
• @scovich We need an opaque expression that's a Box so that there's a way to wrap up the foreign error such that rust can proppagate it back to the user.
• @OussamaSaoudi This also indicates an area for focus from user docs
• @nicklan We need to consider all FFI methods to potentially be fallible since something further down can fail (ex(oussama): memory allocation)
• @mightyshazam void methods communicate that the method is infallible. The assumption is that it'd be okay to throw an exception. On the other hand, if the method returns a result, it communicates to the FFI implementer that they should handle the error.

TODO:

• Look into exception unwind from foreign code

[mightyshazam]

@nicklan @OussamaSaoudi @scovich @hntd187 This is a sample of code throwing an exception in a place it probably shouldn't.
This method is a top-level method. It calls the ffi method scan_metdata_next and passes the delegate VisitScanData, which calls another ffi method and passes another delegate. ProcessScanData is our fallible method.
The call chain is C#->Rust->C#->Rust->C#->Rust. The lowest level of C# has the ability to handle an error, but there is no way to pass it up the chain. Currently, I am throwing an exception, and that ultimately shuts down the process. It would be better to kills the scan while keeping the program running.