ESQL: name qualifiers meta issue

[alex-spies]

ESQL does not have qualified column names yet and deals with name conflicts by discarding conflicting columns from previous commands. E.g.

ROW x = 1 | EVAL x = 2*x, x = 3*x

x:integer
6

That's undesirable for commands like LOOKUP JOIN or ENRICH that bring in potentially many new columns that can have names conflicting with existing columns.

We want to allow users to define qualifiers for column names so that conflicts can be avoided in the first place. The syntax is not finalized, but should eventually look something like this:

ROW x = 1, key = "foo" | LOOKUP JOIN lookup_idx AS lookup ON key | KEEP lookup.x

x:keyword
bar

Note that the KEEP command at the end unambiguously refers to the x field from the lookup_idx, not the x from the ROW command unless there happens to be an index field called lookup.x.

• Initial PoC: [DO NOT MERGE] ESQL: PoC for name qualifiers #126531
• Initial unambiguous syntax [qualifier].[name]: ESQL: Qualifiers syntax #132925

Now

-> Immediate next steps are split out into subissues, see below.

Cleanups

• Replace usage of qualifiedName and qualifiedNamePattern by fieldName and fieldNamePattern in commands that are not allowed to use qualified names, c.f. ESQL: Qualifiers syntax #132925 (comment)
• Fix UnresolvedAttribute#clone - this should allow to make changes when cloning, like for all other attributes.

Next

• Make aliases use qualifiers and enable using qualifiers in EVAL and RENAME. Check every instantiation of the Alias class to see if it should actually use a qualified name; update serialization code and throw an exception if we're trying to serialize an alias with non-null qualifier.
• Investigate how qualifiers can work with subqueries and FORK
• Consider generative tests for FieldNameUtils#resolveFieldNames to ensure correctness in queries that we didn't think of.
• Resolution in commands with wildcards: should only be KEEP/DROP; we need to decide how they behave in case of wildcards for the unqualified name and for the qualifier; should KEEP * include all qualified names or should that require KEEP * *?

Later (after tech preview)

• Introduce shorthand syntax:
  • Add resolution rules that will resolve qualified.name to a qualified name if unambiguous; decide how to behave in case of ambiguity.
  • Make using the qualifier optional if unambiguous.
  • For both, make sure FieldNameUtils#resolveFieldNames still works correctly.
  • Have a chat with @idegtiarenko about the consequences for field resolution and how they may affect performance. (Thanks for offering, Ievgen!) Consider adding a nightly benchmark because the performance of the field caps request/response may degrade in case of ambiguities and indices with many fields.
• Enable using parameters together with qualifiers
• ES|QL: adopt name qualifiers for ENRICH command #112015

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

I'm splitting out some of the tasks from the list into subissues.