Allow VQL LIMIT to be set from a variable

[misje]

I have many occasions where I want to limit the amount rows from a source before passing it on. I need an upper limit on the data before passing it on to the destination (API endpoint, for example) as a safe guard. I can only use LIMIT with numeric literals. I would like this limit to be specified as an artifact parameter, e.g. SELECT * FROM Artifact.My.Source() LIMIT MaxItems.

Does this conflict with the design and idea of VQL? If so, are there other ways to programmatically limit the amount of rows from a query?

[scudette]

This is a reasonable request. The reason why limit can only accept a number instead of an expression is because it's not clear when the expression should be evaluated.

For example one implementation is that the limit expression is evaluated on each row. Another interpretation is that the expression is evaluated at query compile time or setup time.

In SQL the limit clause is actually not standardized but a quick Google search indicates that the databases that use it only allow an integer argument.

A better and more general approach is to define an exit condition on a query. This is then clearer that it will be evaluated on each row and the first time it returns true then the query is aborted.

The natural place for it to go is an argument to the query () plug-in. This allows terminating the query not just on the number of rows but on any condition

[scudette]

Fixed in #4679

[misje]

The documentation (or auto-completion of the new exit parameter) is not there yet, but here is an example of how this can be used to impose a limit set from a variable (from this test):

LET q <= SELECT _value AS Value
  FROM range(start=0, end=20)

LET my_limit <= 5

SELECT *
FROM query(query=q, exit='x=>x.Value >= my_limit', inherit=true)

[misje]

Thanks for the quick implementation! I like your solution.