Skip to content

ESQL: Support for _index metadata field in CsvTests #121261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 2, 2025
Merged

ESQL: Support for _index metadata field in CsvTests #121261

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
961532a
ESQL: Support for _index metadata field in CsvTests
GalLalouche Jan 30, 2025
f5951e9
Extract INDEX constant to MetadataAttribute
GalLalouche Jan 30, 2025
10267f3
Merge branch 'main' into tests/csv_tests_metadata_index
GalLalouche Jan 30, 2025
157381f
Add comment on capability
GalLalouche Feb 2, 2025
41acedd
Merge branch 'main' into tests/csv_tests_metadata_index
GalLalouche Feb 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion x-pack/plugin/esql/qa/testFixtures/src/main/resources/metadata-remote.csv-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ max:integer |_index:keyword
;

metaIndexAliasedInAggs
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: metadata_fields_remote_test
from employees metadata _index | eval _i = _index | stats max = max(emp_no) by _i | SORT _i;

Expand Down
6 changes: 3 additions & 3 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/metadata.csv-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ max:integer |_index:keyword
;

metaIndexSorted
required_capability: metadata_fields
required_capability: index_metadata_field
from employees metadata _index | sort _index, emp_no desc | keep emp_no, _index | limit 2;


Expand All @@ -50,7 +50,7 @@ emp_no:integer |_index:keyword
;

metaIndexWithInPredicate
required_capability: metadata_fields
required_capability: index_metadata_field
from employees metadata _index | where _index in ("employees", "foobar") | sort emp_no desc | keep emp_no, _index | limit 2;


Expand All @@ -60,7 +60,7 @@ emp_no:integer |_index:keyword
;

metaIndexAliasedInAggs
required_capability: metadata_fields
required_capability: index_metadata_field
from employees metadata _index | eval _i = _index | stats max = max(emp_no) by _i;


Expand Down
54 changes: 27 additions & 27 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/union_types.csv-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ mc:l | count:l

multiIndexIpString
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: casting_operator
required_capability: union_types_remove_fields

Expand Down Expand Up @@ -162,7 +162,7 @@ sample_data_str | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexIpStringRename
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: casting_operator
required_capability: union_types_remove_fields

Expand Down Expand Up @@ -191,7 +191,7 @@ sample_data_str | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexIpStringRenameToString
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_str METADATA _index
Expand Down Expand Up @@ -219,7 +219,7 @@ sample_data_str | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexWhereIpString
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_str METADATA _index
Expand All @@ -237,7 +237,7 @@ sample_data_str | 2023-10-23T12:15:03.360Z | 3450233 | Connected

multiIndexWhereIpStringLike
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_str METADATA _index
Expand Down Expand Up @@ -445,7 +445,7 @@ count:long | message:keyword

multiIndexMissingIpToString
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_missing_field

FROM sample_data, sample_data_str, missing_ip_sample_data METADATA _index
Expand Down Expand Up @@ -480,7 +480,7 @@ sample_data_str | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450

multiIndexMissingIpToIp
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_missing_field

FROM sample_data, sample_data_str, missing_ip_sample_data METADATA _index
Expand Down Expand Up @@ -515,7 +515,7 @@ sample_data_str | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexTsLong
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_long METADATA _index
Expand Down Expand Up @@ -543,7 +543,7 @@ sample_data_ts_long | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexTsLongRename
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_long METADATA _index
Expand Down Expand Up @@ -573,7 +573,7 @@ sample_data_ts_long | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233
multiIndexTsNanosRename
required_capability: to_date_nanos
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_nanos METADATA _index
Expand Down Expand Up @@ -602,7 +602,7 @@ sample_data_ts_nanos | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233
multiIndexTsNanosRenameToNanos
required_capability: to_date_nanos
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_nanos METADATA _index
Expand Down Expand Up @@ -631,7 +631,7 @@ sample_data_ts_nanos | 2023-10-23T12:15:03.360123456Z | 172.21.2.162 | 34502
multiIndex sort millis and nanos as nanos
required_capability: to_date_nanos
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_nanos METADATA _index
Expand Down Expand Up @@ -660,7 +660,7 @@ sample_data | 2023-10-23T12:15:03.360000000Z | 172.21.2.162 | 34502
multiIndex sort millis and nanos as millis
required_capability: to_date_nanos
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_nanos METADATA _index
Expand Down Expand Up @@ -691,7 +691,7 @@ multiIndexTsNanosRenameToNanosWithFiltering
required_capability: to_date_nanos
required_capability: date_nanos_binary_comparison
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_nanos METADATA _index
Expand All @@ -716,7 +716,7 @@ sample_data_ts_nanos | 2023-10-23T13:33:34.937123456Z | 172.21.0.5 | 12323

multiIndexTsLongRenameToString
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_long METADATA _index
Expand Down Expand Up @@ -744,7 +744,7 @@ sample_data_ts_long | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexWhereTsLong
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields

FROM sample_data, sample_data_ts_long METADATA _index
Expand Down Expand Up @@ -979,7 +979,7 @@ count:long | message:keyword

multiIndexIpStringTsLong
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand Down Expand Up @@ -1022,7 +1022,7 @@ sample_data_ts_nanos | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexIpStringTsLongDropped
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: to_date_nanos

FROM sample_data* METADATA _index
Expand Down Expand Up @@ -1064,7 +1064,7 @@ sample_data_ts_nanos | 8268153 | Connection error

multiIndexIpStringTsLongRename
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand Down Expand Up @@ -1107,7 +1107,7 @@ sample_data_ts_nanos | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexIpStringTsLongRenameDropped
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: to_date_nanos

FROM sample_data* METADATA _index
Expand Down Expand Up @@ -1149,7 +1149,7 @@ sample_data_ts_nanos | 8268153 | Connection error

multiIndexIpStringTsLongRenameToString
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand Down Expand Up @@ -1192,7 +1192,7 @@ sample_data_ts_nanos | 2023-10-23T12:15:03.360Z | 172.21.2.162 | 3450233

multiIndexWhereIpStringTsLong
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand Down Expand Up @@ -1226,7 +1226,7 @@ count:long | message:keyword

multiIndexWhereIpStringLikeTsLong
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand Down Expand Up @@ -1260,7 +1260,7 @@ count:long | message:keyword

multiIndexMultiColumnTypesRename
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand All @@ -1279,7 +1279,7 @@ null | null | 8268153 | Connectio

multiIndexMultiColumnTypesRenameAndKeep
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand All @@ -1299,7 +1299,7 @@ sample_data_ts_nanos | 2023-10-23T13:52:55.015Z | 2023-10-23T13:52:55.015123456

multiIndexMultiColumnTypesRenameAndDrop
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: union_types_remove_fields
required_capability: to_date_nanos

Expand Down Expand Up @@ -1591,7 +1591,7 @@ FROM sample_data, sample_data_ts_long

shortIntegerWidening
required_capability: union_types
required_capability: metadata_fields
required_capability: index_metadata_field
required_capability: casting_operator
required_capability: union_types_numeric_widening

Expand Down
6 changes: 5 additions & 1 deletion x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,16 @@ public enum Cap {
* Cast string literals to a desired data type for IN predicate and more types for BinaryComparison.
*/
STRING_LITERAL_AUTO_CASTING_EXTENDED,

/**
* Support for metadata fields.
*/
METADATA_FIELDS,

/**
* Support specifically for *just* the _index METADATA field.
*/
INDEX_METADATA_FIELD,

/**
* Support for timespan units abbreviations
*/
Expand Down
10 changes: 8 additions & 2 deletions ...ql/src/test/java/org/elasticsearch/xpack/esql/planner/TestPhysicalOperationProviders.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@
import org.elasticsearch.xpack.esql.plan.physical.FieldExtractExec;
import org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.LocalExecutionPlannerContext;
import org.elasticsearch.xpack.esql.planner.LocalExecutionPlanner.PhysicalOperation;
import org.elasticsearch.xpack.esql.session.IndexResolver;
import org.elasticsearch.xpack.ml.MachineLearning;

import java.io.IOException;
Expand Down Expand Up @@ -292,6 +293,10 @@ private Block getBlockForMultiType(DocBlock indexDoc, MultiTypeEsField multiType
private Block extractBlockForSingleDoc(DocBlock docBlock, String columnName, TestBlockCopier blockCopier) {
var indexId = docBlock.asVector().shards().getInt(0);
var indexPage = indexPages.get(indexId);
if (IndexResolver.INDEX_METADATA_FIELD.contains(columnName)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way IndexResolver.INDEX_METADATA_FIELD is defined is a bit trappy, since if that set is ever going to be extended, this might match too much. I guess that's defined as a set for convenience, but the name is then misleading. Don't know if it'd be worth it extracting the _index name into a constant (used in MetadataAttribute too) and matching exactly?
Anyways, can also be left as is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I've defined the constant in MetadataAttribute.

return docBlock.blockFactory()
.newConstantBytesRefBlockWith(new BytesRef(indexPage.index), blockCopier.docIndices.getPositionCount());
}
int columnIndex = indexPage.columnIndex(columnName)
.orElseThrow(() -> new EsqlIllegalArgumentException("Cannot find column named [{}] in {}", columnName, indexPage.columnNames));
var originalData = indexPage.page.getBlock(columnIndex);
Expand Down Expand Up @@ -410,8 +415,9 @@ private Block extractBlockForColumn(
) {
foreachIndexDoc(docBlock, indexDoc -> {
TestBlockCopier blockCopier = blockCopier(dataType, extractPreference, indexDoc.asVector().docs());
Block blockForIndex = extractBlock.apply(indexDoc, blockCopier);
blockBuilder.copyFrom(blockForIndex, 0, blockForIndex.getPositionCount());
try (Block blockForIndex = extractBlock.apply(indexDoc, blockCopier)) {
blockBuilder.copyFrom(blockForIndex, 0, blockForIndex.getPositionCount());
}
});
var result = blockBuilder.build();
assert result.getPositionCount() == docBlock.getPositionCount()
Expand Down