[DON'T MERGE] Proof of Concept: ES|QL approximate query execution

[jan-elastic]

Proof of concept for approximate query execution

This is for gathering early feedback; not for merging!

This is targeting queries of the form

FROM data
  | commands_swappable_with_sample
  | STATS aggs
  | more_commands

Approximating rewrites it to

FROM data
  | SAMPLE probability
  | commands_swappable_with_sample
  | STATS sample_corrected_aggs
  | more_commands

The sample probability is such that the approximated results are based on ~1000 docs. It's determined via the total result count:

FROM data
  | commands_swappable_with_sample
  | STATS COUNT(*)

You can use this as follows

POST _query
{  
  "query": """
    FROM kibana_sample_data_ecommerce
     | STATS count=COUNT() BY CATEGORIZE(category)
     | SORT count DESC
  """,
  "approximate": true
}

With "approximate": false, the (correct) results are:

     count     |CATEGORIZE(category)
---------------+--------------------
3927           |.*?Clothing.*?      
2080           |.*?Shoes.*?         
1402           |.*?Accessories.*?   

(based on "documents_found": 4675)

With "approximate": true, the (approxmiate) results are like:

     count     |CATEGORIZE(category)
---------------+--------------------
3791           |.*?Clothing.*?      
2001           |.*?Shoes.*?         
1533           |.*?Accessories.*?   

(based on "documents_found": 990)

[ivancea]

Just a shallow check. To me, it makes sense. I would wait for somebody else to have another opinion though, in case this extra query could lead to something bad somewhere

[ivancea]

This extra query is probably my major "concern". It looks ok, but it's still going to execute evals and wheres, which could end up executing a full query anyway (?). It looks a bit "dangerous".

As an idea, I wonder if we could use some kind of Lucene statistics for this. I don't know if we have them though, or if what we have is enough. Even if they were just approximates, they could let us avoid this extra query, maybe. This would be another block of work though

[jan-elastic]

I get your concern. That's exactly why I wanted some early feedback.

The extra query is pretty similar to the extra query of the inline join subplan though.

[jan-elastic]

In the case of

FROM data | STATS COUNT()

I guess we can get the count directly from Lucene.

But for a more complicated

FROM data | WHERE my_function(x) < 1 | STATS COUNT()

that's obv not possible.

We can use sampling again though to get an approximate count, which is good enough for setting the probability.