alternatives to CEL

[toumorokoshi]

In #247, there is feedback from @thegagne and @ar3s3ru on the fact that CEL may not be a feasible option for filtering syntax. Starting a thread to discuss where CEL is the right option for filtering syntax.

[toumorokoshi]

Summarizing a few points from the thread issue above, I see a few issues that we should consider:

1. the url-escaped CEL queries are ugly

e.g.

# property == whatever
GET /properties?filter=property%20%3D%3D%20%22whatever%22

2: CEL is a powerful language, and it's difficult to support the full syntax

We have worked on things like cel2db, but it's tailor made to convert cel expressions to SQL queries, and it's valid that not all databases use SQL dialects, and may not be willing to support the full set of expressions cel provides. Not to mention getting into really hairy territory like custom functions.

Feel free to add more concerns / issues as a reply.

[toumorokoshi]

Now we're on to possible alternatives.

@thegagne suggests having two variants: simple, and complex syntax: #247 (comment).

As well as including documentation on a subset of CEL which could be supported.

[toumorokoshi]

Thread on "cel expressions are ugly in a URL".

Honestly, I don't think any mildly functional syntax for a filter in a query parameter will look great. The only unreserved characters in a URL are A-Za-z0-9\-\._~. With only dashes, dots, underscores, and tildes to choose from, there will be likely be a need to invoke some escaping in any option (for example, we want equality, at minimum).

[toumorokoshi]

thread on "basic filtering"

For these examples:

GET /books?category=fiction&year=2024
GET /pets?type=dog&limit=10

I definitely agree that they are very easy to read, and probably user friendly too. This particular example is for a list API, so the usage of query parameters is quite limited, as well.

However, that said, I do see an issue with the fact that there is going to be a need to reserve some values (e.g. limit), which may conflict with a field in the resource. If this conflict arises there's basically no way to express a filter as desired.

[thegagne]

Thanks for starting this up. I do think CEL is a good option overall, but there are some hurdles to get over for adoption. I appreciate considering the alternatives, as well as what it will take to make any option as intuitive, user friendly, and flexible as possible.

In regards to avoiding certain field names in a resource, this isn't altogether a bad thing to have a recommendation about somewhere, regardless of filtering.

Anecdotally, this week using TypeSpec I found it has a reserved keyword for "namespace". For our use I worked around this and found a better name.

We may run into similar things here. Things like resource, filter, max_page_size, and obviously path. Some of these could be used without a technical problem, but would invite confusion.

[toumorokoshi]

In the AEP sync, @rofrankel @thegagne and I had a discussion, and we kind of floated toward the following:

1. Having a well-defined syntax for more complex queries is a good thing: this helps cover complex filtering use cases (the long-tail that often has difficulty being folded into an existing simpler syntax, but is still necessary).
2. CEL is complex, and implementing the full spec is unreasonable. Therefore, we would expect that users would gradually support CEL as needed, evolving their API, rather than support full features from the get go.
3. To enable (2), some annotation to machine-describe the supported operations should be devised.

in short: the general direction is toward keeping CEL, but move toward better ways to express the operations supported.

[kindermoumoute]

To enable (2), some annotation to machine-describe the supported operations should be devised

Would it make sense for these annotations to live in protovalidate?

[toumorokoshi]

The annotations would have to have protobuf and openapi equivalents - see https://aep.dev/4/#annotating-resource-types for examples.

I personally was thinking something along the lines of @thegagne's original suggestion:

x-aep-filter:
  supported_operations:
    -   "conditionalOr", # follow the naming of https://github.com/google/cel-spec/blob/master/doc/langdef.md#syntax
  fields: # supported fields, optional. If all fields are supported can be left blank.
    - "path"
    -  "description"

Would protovalidate satisfy those use cases? Perhaps it does allow one to validate the filter expression directly, and could use CEL. But that would probably make it harder for someone who wants to extract values from the protovalidate expression during client generator (e.g. enable a UI which has dropdowns for selecting the fields that you can filter by).