Skip to content

Conversation

@luigidellaquila
Copy link
Contributor

@luigidellaquila luigidellaquila commented Feb 26, 2025

Introduce UNPIVOT UNTABLE command, that creates multiple rows starting from columns, eg.

ROW id=1, name1="foo", name2="bar", name3="baz" | UNTABLE fieldName FOR fieldValue IN (name*)
id:integer | fieldName:keyword | fieldValue:keyword
1          | name1             | foo
1          | name2             | bar
1          | name3             | baz

The command syntax is:

UNTABLE <fieldNameColumn> FOR <fieldValueColumn> IN (<fieldPatterns>*)

Untabled columns have to be all the same type.


LIMIT pushdown is similar to MV_EXPAND, but it also takes into consideration how many columns are untabled (that is deterministic and easy to get at planning time), eg.

| UNTABLE fieldName FOR fieldValue IN (col1, col2, col3) | LIMIT 30 

is optimized as

| LIMIT 10 | UNTABLE fieldName FOR fieldValue IN (col1, col2, col3) | LIMIT 30 

(the lower limit becomes 10, that is 30 / <number of untabled fields>


TODO

  • Filter pushdown:
    • Filters for columns that are not untabled can be pushed down.
    • Filters on both pivoted field name and value can be pushed down as well, eg.
        | UNTABLE fieldName FOR fieldValue IN (col1, col2, col3) 
        | WHERE fieldName == "col1" AND fieldValue == 10
      
      can be pushed down as
        | WHERE col1 == 10
        | UNTABLE fieldName FOR fieldValue IN (col1, col2, col3) 
        | WHERE fieldName == "col1" AND fieldValue == 10
      
  • More tests
  • Better memory management (ie. paginate the output, same as MV_EXPAND)
  • Docs

@luigidellaquila luigidellaquila changed the title ES|QL: PoC - Add UNPIVOT command ES|QL: PoC - Add UNTABLE command Jul 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants