feat: add `tiledb_query_add_predicate` API to parse a SQL expression string into a `QueryCondition` #5566

rroelke · 2025-06-29T14:17:44Z

Resolves CORE-25.

#5546 enabled QueryCondition to evaluate datafusion expression trees in its apply_sparse function. This was used only in test scenarios, with a configuration parameter which would have QueryCondition translate its syntax tree into a datafusion Expr and then evaluate it.

This pull request builds upon this capability and connects it to C and C++ level APIs. We add tiledb_query_add_predicate(tiledb_ctx_t*, tiledb_query_t*, const char* predicate) which uses datafusion to parse predicate into an Expr which can be evaluated by QueryCondition.

This enables users to add a much more expressive set of predicates to their queries, which can perform arithmetic, function calls, conditional (CASE) expressions, and so on.

Design

Datafusion has a SessionContext which contains the resources and catalog of a Datafusion session. This object is used to parse and evaluate expressions and is where one registers tables, UDFs, etc.

We will lazily attach a SessionContext to a Query when tiledb_query_add_predicate is first invoked. Doing so lazily means that existing applications should observe no changes.

The tiledb_query_add_predicate initializes the SessionContext if needed and uses it to parse the expression into LogicalExpr. We accumulate a list of LogicalExpr and then combine them into a single conjunctive expression tree (possibly together with a QueryCondition syntax tree) and embed that in a new QueryCondition.

Enumerations

To reduce the scope of this pull request, we hoped to defer supporting attributes with enumerations in predicates to a future pull request (see CORE-287). However, #5546 demonstrated some examples of rewriting a QueryCondition into a DataFusion expression tree when attributes with enumerations were used in the predicates. This occurred after QueryCondition::rewrite_for_schema, which resolves the literal into a key for comparison with the key stored in the tile. Continuing to pass these tests requires some amount of enumeration support to be implemented here.

The "obvious" thing to do would be to represent enumerations in the schema using the Arrow Dictionary data type. However, this notion has a few shortcomings:

the data type and cell val num of an enumeration are co-located with the enumeration variants in storage, not with the enumeration name in the schema. This means that resolving an enumeration data type requires loading it.
the Rust DictionaryArray requires that each of the keys is a valid index into the dictionary values, whereas TileDB enumerations specifically do not have this requirement.

To work around these things, we sometimes need the Arrow schema that we generate to use the enumeration key type, and sometimes we will want it to use the enumeration value type.

This pull request adds a switch to tiledb::oxidize::arrow::schema::create and the functions coming afterwards which implement this toggle, for the generated schema to be the "storage" schema or the "view" schema. We use the "storage" schema to support the existing tests. CORE-287 will use the "view" schema to support text predicates on attributes with enumerations.

There is some more code related to enumerations added here which was in draft along the way to implementing a correct "view" schema. I left it in but we can remove it if preferred.

Testing

We add examples for the C API and C++ API which demonstrate using this new API for the same conditions used in the query_condition_sparse API examples, as well as some more complicated predicates which cannot be expressed as a query condition.
We add some sanity tests, examples, and API tests in unit-query-add-predicate.cc.
We add a mode to the rapidcheck tests in unit-sparse-global-order-reader.cc to optional render the generated QueryCondition syntax tree as a SQL string, which we then add as a predicate instead of applying it to the query normally.

Unsupported

attributes with enumerations cannot be used in predicates (CORE-287)
dimension labels are of unknown status (issue TODO)
we add errors for non-sparse global order reader for now (CORE-272, CORE-273)

Next Steps

This does not analyze predicates for attributes/dimensions to improve R-tree traversal (CORE-28) or add the desired intersection function (CORE-26).

TYPE: FEATURE
DESC: add tiledb_query_add_predicate API to parse a SQL expression string into a QueryCondition

…han a copy in order to add Rust binding (#5567) Differences in copy and move semantics between Rust and C++ mean that they are generally not very good at passing values across the boundary - most of the time it is necessary to pass references instead. #5566 intends to call `Attribute::get_enumeration_name` from Rust. Prior to these changes, that function returns `std::optional<std::string>`, which cannot be passed across the boundary (neither `std::optional` nor `std::string` can be). We can either pass a pointer to a string, or pass a string contained within a smart pointer. To avoid the additional memory allocations required for both copying the `std::string` and placing it in a `unique_ptr`, we choose the former. However, `std::optional` does not naturally support references, so we cannot do `std::optional<std::string&>`. Instead we change the return type to `std::optional<std::reference_wrapper<std::string>>`. In addition to enabling passing the result of this function to Rust without allocating additional memory, this also avoids making additional copies of the string. This is not likely to matter from a performance perspective, but is still nice! --- TYPE: NO_HISTORY DESC: Rust binding for `Attribute::get_enumeration_name`

teo-tsirpanis

Preliminary review

tiledb/sm/query/query.cc

tiledb/sm/query/query.h

teo-tsirpanis · 2025-07-01T19:57:18Z

tiledb/sm/query/query_condition.cc

      }
    } else {
-      throw std::logic_error(
+      throw QueryConditionException(


Unrelated to this PR, the trend of our exception (and previously status) types is to indicate where the error occured, rather than what the error was, putting us against the paradigm of exception types in C++ and other languages. This is not a good thing, and leads to C APIs almost always returning the monolithic TILEDB_ERR status code when they fail, which leads to poor error handling and reporting practices downstream, including the very fragile parsing of the error message, which I have seen doing both in testing and production code.

Of course we cannot fix this problem at once and in this PR, but what we can do is prevent the "where"-typed exceptions from proliferating in new code, or at the very least in existing code. Which is a long-winded way for me to say "I think we should revert this line (and the one below)".

I definitely agree with the root notion that the error reporting needs improvement. I prefer leaving this in however. What proper error handling looks like is a question for the future; and what this change does today is at least make it easier to identify all the instances of thrown exceptions in this file by searching on a common string.

tiledb/sm/c_api/tiledb_experimental.h

teo-tsirpanis · 2025-07-01T20:12:49Z

tiledb/sm/query/query.h

 #include "tiledb/sm/subarray/subarray.h"

+#ifdef HAVE_RUST
+#include "tiledb/oxidize/rust.h"


How big is rust.h? If it adds large amounts of code to the compilation, we should avoid including it in other headers, for example by converting LogicalExpr and Session to self-contained types, and storing tdb_unique_ptrs of them in the class. There is also the concern that it will affect developers' inner loop, if the header's content is too sensitive to changes in the Rust code's implementation details, it will force recompilations of the C++ code, which will be limited by moving the include to the implementation file.

The contents of this header are pinned to the version of our cxx crate dependency. I wouldn't be shocked if it never changed.

This header is generated by the cxxbridge dependency and provides C++ definition which correspond to Rust standard library types, e.g. rust::Box and rust::Vec and etc. It isn't generated by compiling any of our Rust crates.

teo-tsirpanis · 2025-07-01T20:16:43Z

tiledb/oxidize/cxx-interface/src/sm/array_schema/mod.rs

        fn attribute_by_name(&self, name: &CxxString) -> *const Attribute;

+        #[cxx_name = "get_enumeration"]
+        fn const_enumeration_cxx(&self, name: &CxxString) -> SharedPtr<ConstEnumeration>;


Can we have ConstSharedPtr instead of const editions of the individual classes?

Unfortunately not. The #[cxx::bridge] macro is pretty strict about what types are allowed. SharedPtr is a special type which it knows how to expand to code on both the Rust and C++ side; custom opaque types are never allowed to be returned or passed by value.

test/support/src/query_helpers.cc

tiledb/sm/query/query.cc

teo-tsirpanis · 2025-09-19T17:43:15Z

examples/c_api/query_add_predicate.c

I doubt that an example in C is useful to have. As I have said in the past, the C API is hard to use directly, and users should prefer using a higher-level language.

I think the C examples are useful for developers, if not customers (who I agree are better off using higher-level APIs). I have referred to them often.

teo-tsirpanis · 2025-09-19T17:43:54Z

tiledb/sm/c_api/tiledb.cc

+  // Sanity check
+  if (sanity_check(ctx, query) == TILEDB_ERR) {
+    return TILEDB_ERR;
+  } else if (predicate == nullptr) {


Suggested change

} else if (predicate == nullptr) {

}

if (predicate == nullptr) {

teo-tsirpanis · 2025-09-19T17:47:50Z

tiledb/sm/cpp_api/query.h

+  /**
+   * Adds a predicate. The predicate will be analyzed and evaluated
+   * in the subarray step, query condition step, or both.
+   *
+   * The predicate is parsed as a SQL expression and must evaluate
+   * to a boolean.
+   *
+   * @param predicate a SQL representation of the predicate
+   * @return Reference to this Query
+   */
+  Query& add_predicate(const std::string& predicate) {
+    auto& ctx = ctx_.get();
+    ctx.handle_error(tiledb_query_add_predicate(
+        ctx.ptr().get(), query_.get(), predicate.c_str()));
+    return *this;
+  }
+


Suggested change

/**

* Adds a predicate. The predicate will be analyzed and evaluated

* in the subarray step, query condition step, or both.

*

* The predicate is parsed as a SQL expression and must evaluate

* to a boolean.

*

* @param predicate a SQL representation of the predicate

* @return Reference to this Query

*/

Query& add_predicate(const std::string& predicate) {

auto& ctx = ctx_.get();

ctx.handle_error(tiledb_query_add_predicate(

ctx.ptr().get(), query_.get(), predicate.c_str()));

return *this;

}

Remove experimental API from stable header?

teo-tsirpanis · 2025-09-19T17:48:33Z

tiledb/sm/query/query.cc

  return Status::Ok();
 }

+Status Query::add_predicate([[maybe_unused]] const char* predicate) {


New code should not use Status.

teo-tsirpanis · 2025-09-19T18:04:21Z

tiledb/sm/query/query.cc

+    if (!expr->is_predicate(array_schema())) {
+      return Status_QueryError("Expression does not return a boolean value");
+    }
+    if (expr->has_aggregate_functions()) {
+      return Status_QueryError(
+          "Aggregate functions in predicate is not supported");
+    }


I would slightly prefer to have Rust do the expression's validation. Might make the FFI simpler.

teo-tsirpanis · 2025-09-19T18:06:38Z

tiledb/sm/query/query_condition.h

+#ifdef HAVE_RUST
+  /** Constructor from a datafusion expression tree */
+  QueryCondition(
+      const ArraySchema& array_schema,
+      rust::Box<tiledb::oxidize::datafusion::logical_expr::LogicalExpr>&& expr);
+#endif
+


This looks like it should better belong in a separate DataFusionQueryCondition class, and either have a common ancestor with QueryCondition, or be stored as an std::variant<QueryCondition, DataFusionQueryCondition>.

From the looks of the Datafusion nested class, it looks like we are halfway there.

teo-tsirpanis · 2025-09-19T18:13:47Z

tiledb/oxidize/CMakeLists.txt

+
+cxxbridge(
+  NAME
+    session


session is ambiguous; I would call it df-session.

teo-tsirpanis · 2025-09-19T18:45:45Z

tiledb/sm/query/query_condition.h

+  bool rewrite_to_datafusion(
+      const ArraySchema& array_schema,
+      tiledb::oxidize::arrow::schema::WhichSchema which);


Changing the implementation of a query condition in-place, looks like a code smell to me. 🤔

teo-tsirpanis · 2025-09-19T18:59:51Z

tiledb/sm/c_api/tiledb_experimental.h

+/**
+ * Adds a predicate to be applied to a read query. The added predicate
+ * will be analyzed and evaluated in the subarray step, query condition
+ * step, or both.
+ *
+ * The predicate is parsed as an Apache DataFusion SQL expression and must
+ * evaluate to a boolean.
+ *
+ * **Example:**
+ *
+ * @code{.c}
+ * const char* pred = "(row BETWEEN 1 AND 10) OR (column BETWEEN 1 AND 10)";
+ * tiledb_query_add_predicate(ctx, query, pred);
+ * @endcode
+ *
+ * @param ctx The TileDB context.
+ * @param query The TileDB query.
+ * @param predicate A text representation of the desired predicate.
+ */


We should document here when query predicates are not supported (such as dense queries).

davisp

This is just a first pass review, but skimming things there's a few things that strike me as things that could be improved.

I think the enumeration stuff could be improved if we split the validation logic up. The reason for enumerations to be lazy loaded is so that they're not sent back and forth when making REST API calls. Loading them on the server during query execution is a thing that already happens for normal query conditions, so I don't see why we can't do the same here. The downside here is that we're moving validation errors from tiledb_query_add_predicate to tiledb_query_submit, but I think that's preferable to not supporting enumerations. I'll also add after I wrote the rest of my comment, we're actually not able to shoot our Rust state across the wire which means our evaluation stuff has to live server side where we can easily load the enumerations.

My bigger issue though is that the FFI interface seems to be getting unnecessarily complex. There were a lot of comments about safety of transmuting that I was giving some side eye in terms of "That's seems plausibly true, but I can't prove it" which doesn't give me the warm fuzzies. However, it sure seems like we could make a lot of that go away by using a simpler interface that only exposes the required APIs. Something like such maybe?

extern "Rust" {
    type QueryEvaluator;

    fn new_query_evaluator(schema: ArraySchemaRef) -> Box<QueryEvaluator>;
    fn add_predicates(&self, expr: Vec<CxxString>) -> Result<()>;    
    fn evaulate(&self, data: RecordBatch) -> Result<RecordBatch>; // Or the bitmap or w/e

    fn validate_expr(schema: ArraySchemaRef, expr: &str) -> Result<()>; // Used by add_predicate to do any initial verification
}

On the less important/hand wavy side, it seems like mixing query conditions and predicates is just adding complexity when we don't really need to. I'd just enforce one or the other rather than attempting to convert from query conditions to expressions. It might still be useful in tests for quick validation, but it feels like the sort of thing that'll be easier to not support long term (plus, what's the use case of mixing?).

examples/cpp_api/query_add_predicate.cc

davisp · 2025-10-16T16:37:20Z

test/src/unit-query-add-predicate.cc

+          "received NativeType::UInt64, DataType: UInt64.\nThis was likely "
+          "caused by a bug in DataFusion's code and we would welcome that you "
+          "file an bug report in our issue tracker";


This seems like a not great error message to be spitting out from our API.

test/src/unit-query-add-predicate.cc

rroelke · 2025-10-22T15:24:18Z

The reason for enumerations to be lazy loaded is so that they're not sent back and forth when making REST API calls.

This is a good requirement (i.e. that the enumeration is not loaded on the client side) to add and test for, good (implicit) suggestion.

Loading them on the server during query execution is a thing that already happens for normal query conditions, so I don't see why we can't do the same here.

Here's what I believed a few months ago (in the PR description):

the data type and cell val num of an enumeration are co-located with the enumeration variants in storage, not with the enumeration name in the schema. This means that resolving an enumeration data type requires loading it.

For query conditions this didn't seem to matter because the expression isn't really validated until it is evaluated. But for datafusion we need the schema to even parse the expression into a LogicalExpr. This is why the storage/view schema distinction had to be drawn - if you have a predicate e = 'foo' then we have to know what type e is to generate the right expression tree.

I was wondering if DataType::Null would be useful here... and then I opened the code to jog my memory... and it looks like that's what I use for enumerations - whether loaded or not loaded. I don't think there's anything here that loads enumerations at all - it's just that this piece couldn't completely ignore them and had to do the storage vs. view schemata.

With respect to enumerations is there a concern for this pull request, or are these details you would like to see for CORE-287?

rroelke · 2025-11-17T18:09:46Z

My bigger issue though is that the FFI interface seems to be getting unnecessarily complex... it sure seems like we could make a lot of that go away by using a simpler interface that only exposes the required APIs. Something like such maybe?

I have this in progress and it does simplify quite a lot. The code here ended up this way as I was a little fixated on extending the "evaluator" notion started in #5546 but I agree it is a lot cleaner to step away from that instead.

rroelke marked this pull request as draft June 29, 2025 14:17

rroelke mentioned this pull request Jun 29, 2025

chore: Attribute::get_enumeration_name returns a reference rather than a copy in order to add Rust binding #5567

Merged

rroelke added 4 commits June 30, 2025 10:31

tiledb_query_add_predicate

94eeeff

Remove assert post rebase

ad3f43c

Fill in WHERE a IS NOT NULL example

c10ef48

Fill in other global order unit test examples

6017582

rroelke force-pushed the rr/core-25-add-predicate branch from 3e877a3 to 6017582 Compare June 30, 2025 14:32

rroelke added 12 commits June 30, 2025 11:39

unit-query-add-predicate.cc tests for other readers

303e831

Move datafusion session to query instead of ContextResources

7aaedd5

Tweak example comment

3668cda

Enumeration::create const std::vector

4e2eebf

Tweak example

bd16e7b

cpp example

40cb894

clippy

2139297

Add test on evolved schema

f10d157

Change test names

2b72ba4

Fix non-rust build

1d962ec

Fix osx build errors

b0d36d3

Fix C API example print_elem buffer

f246bd7

teo-tsirpanis reviewed Jul 1, 2025

View reviewed changes

rroelke added 2 commits July 1, 2025 19:21

Comment new/updated test support functions

744728d

Remove unnecessary ExternType impl

bead919

rroelke commented Jul 1, 2025

View reviewed changes

tiledb/sm/query/query.cc Outdated Show resolved Hide resolved

rroelke commented Jul 1, 2025

View reviewed changes

tiledb/sm/query/query.cc Outdated Show resolved Hide resolved

rroelke added 5 commits July 1, 2025 19:33

Self-review code comments

02fb0ca

Attempt to fix query_add_predicate error

ec829e0

Undo clang-format-17 string splits

f2ddb4b

Change C++ API to use std::string

176a021

Remove logger_->status

d1c7680

rroelke added 4 commits July 3, 2025 11:16

Fix wrong write size in unit_query_condition.cc

860d3d5

Fix UTF-8, unit_query_condition passes

9d8e4ff

Stopgap for enumerations in WhichSchema::View

d6889bd

clippy

d5caf2c

rroelke marked this pull request as ready for review July 3, 2025 17:44

rroelke requested review from davisp, teo-tsirpanis and ypatia July 3, 2025 17:44

rroelke and others added 7 commits July 21, 2025 10:10

Merge remote-tracking branch 'origin/main' into rr/core-25-add-predicate

314b169

RestClientFactory can construct in place

9046ef1

Handle TILEDB_RUST=OFF in unit-query-add-predicate.cc

33c8a72

HeapMemoryLinter ignores oxidize dir

a3e6617

Fix empty dimension tuple

e18ccd7

make format

f1d7eb5

Merge remote-tracking branch 'origin/main' into rr/core-25-add-predicate

27a1be3

teo-tsirpanis reviewed Sep 19, 2025

View reviewed changes

davisp reviewed Oct 16, 2025

View reviewed changes

rroelke added 8 commits October 22, 2025 11:40

Remove non-experimental version of add_predicate

572ba80

Merge remote-tracking branch 'origin/main' into rr/core-25-add-predicate

0edb237

Merge remote-tracking branch 'origin/main' into rr/core-25-add-predicate

77c0f85

Fix error message grammar

1823e8a

Split no predicate and WHERE TRUE in examples

c242c16

cargo update and remove cxxbridge version pin

d8eb419

Single QueryPredicates FFI boundary compiles, not tested

66ff1fc

Single QueryPredicates FFI boundary passes existing tests

5d2521e

rroelke added 3 commits November 17, 2025 16:09

unit_query_condition and API test both pass

48195cf

self review

6f7d080

clippy

199ea2f

	} else if (predicate == nullptr) {
	}
	if (predicate == nullptr) {

feat: add tiledb_query_add_predicate API to parse a SQL expression string into a QueryCondition #5566

Are you sure you want to change the base?

feat: add tiledb_query_add_predicate API to parse a SQL expression string into a QueryCondition #5566

Conversation

rroelke commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design

Enumerations

Testing

Unsupported

Next Steps

Uh oh!

teo-tsirpanis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

teo-tsirpanis Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davisp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rroelke commented Oct 22, 2025

Uh oh!

rroelke commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: add `tiledb_query_add_predicate` API to parse a SQL expression string into a `QueryCondition` #5566

feat: add `tiledb_query_add_predicate` API to parse a SQL expression string into a `QueryCondition` #5566

rroelke commented Jun 29, 2025 •

edited

Loading

teo-tsirpanis Jul 1, 2025 •

edited

Loading