Merge branch 'main' into feature/list_query_api

Author: GalLalouche

.buildkite/hooks/pre-command

@@ -94,6 +94,14 @@ if [[ "${USE_PROD_DOCKER_CREDENTIALS:-}" == "true" ]]; then
   fi
 fi
 
+# Authenticate to the Docker Hub public read-only registry
+if which docker > /dev/null 2>&1; then
+  DOCKERHUB_REGISTRY_USERNAME="$(vault read -field=username secret/ci/elastic-elasticsearch/docker_hub_public_ro_credentials)"
+  DOCKERHUB_REGISTRY_PASSWORD="$(vault read -field=password secret/ci/elastic-elasticsearch/docker_hub_public_ro_credentials)"
+
+  echo "$DOCKERHUB_REGISTRY_PASSWORD" | docker login --username "$DOCKERHUB_REGISTRY_USERNAME" --password-stdin docker.io
+fi
+
 if [[ "$BUILDKITE_AGENT_META_DATA_PROVIDER" != *"k8s"* ]]; then
   # Run in the background, while the job continues
   nohup .buildkite/scripts/setup-monitoring.sh </dev/null >/dev/null 2>&1 &

REST_API_COMPATIBILITY.md

@@ -154,9 +154,11 @@ if (request.getRestApiVersion() == RestApiVersion.V_7 && request.hasParam("limit
 
 The above code checks the request's compatible version and if the request has the parameter in question. In this case the deprecation warning is not automatic and requires the developer to manually log the warning. `request.param` is also required since it consumes the value as to avoid the error of unconsumed parameters.
 
-### Testing
+### Testing Backwards Compatibility
 
-The primary means of testing compatibility is via the prior major version's YAML REST tests. The build system will download the latest prior version of the YAML rest tests and execute them against the current cluster version. Prior to execution the tests will be transformed by injecting the correct headers to enable compatibility as well as other custom changes to the tests to allow the tests to pass. These customizations are configured via the build.gradle and happen just prior to test execution. Since the compatibility tests are manipulated version of the tests stored in Github (via the past major version), it is important to find the local (on disk) version for troubleshooting compatibility tests.
+The primary means of testing compatibility is via the prior major version's YAML REST tests. The build system will download the latest prior version of the YAML rest tests and execute them against the current cluster version. For example if you are testing main versioned as 9.0.0 the build system will download the yaml tests in the 8.x branch and execute those against the current cluster version for 9.0.0.
+
+Prior to execution the tests will be transformed by injecting the correct headers to enable compatibility as well as other custom changes to the tests to allow the tests to pass. These customizations are configured via the build.gradle and happen just prior to test execution. Since the compatibility tests are manipulated version of the tests stored in Github (via the past major version), it is important to find the local (on disk) version for troubleshooting compatibility tests.
 
 The tests are wired into the `check` task, so that is the easiest way to test locally prior to committing.  More specifically the task is called `yamlRestCompatTest`. These behave nearly identical to it's non-compat `yamlRestTest` task. The only variance is that the tests are sourced from the prior version branch and the tests go through a transformation phase before execution. The transformation task is `yamlRestCompatTestTransform`.
 
@@ -170,6 +172,36 @@ Since these are a variation of backward compatibility testing, the entire suite
 
 In some cases the prior version of the YAML REST tests are not sufficient to fully test changes. This can happen when the prior version has insufficient test coverage. In those cases, you can simply add more testing to the prior version or you can add custom REST tests that will run along side of the other compatibility tests. These custom tests can be found in the `yamlRestCompatTest` sourceset. Custom REST tests for compatibility will not be modified prior to execution, so the correct headers need to be manually added.
 
+#### Breaking Changes
+
+It is possible to be in a state where you have intentionally made a breaking change and the compatibility tests will fail irrespective of checks for `skip` or `requires` cluster or test features in the current version such as 9.0.0. In this state, assuming the breaking changes are reasonable and agreed upon by the breaking change committee, the correct behavior is to skip the test in the `build.gradle` in 9.0.0. For example, if you make a breaking change that causes the `range/20_synthetic_source/Date range` to break then this test can be disabled temporarily in this file `rest-api-spec/build.gradle` like within this snippet:
+
+```groovy
+tasks.named("yamlRestCompatTestTransform").configure({task ->
+    task.skipTest("range/20_synthetic_source/Date range", "date range breaking change causes tests to produce incorrect values for compatibility")
+    task.skipTest("indices.sort/10_basic/Index Sort", "warning does not exist for compatibility")
+    task.skipTest("search/330_fetch_fields/Test search rewrite", "warning does not exist for compatibility")
+    task.skipTestsByFilePattern("indices.create/synthetic_source*.yml", "@UpdateForV9 -> tests do not pass after bumping API version to 9 [ES-9597]")
+})
+```
+
+When skipping a test temporarily in 9.0.0, we have to implement the proper `skip` and `requires` conditions to previous branches, such as 8.latest. After these conditions are implemented in 8.latest, you can re-enable the test in 9.0.0 by removing the `skipTest` condition.
+
+The team implementing the changes can decide how to clean up or modify tests based on how breaking changes were backported. e.g.:
+
+In 8.latest:
+
+* Add `skip` / `requires` conditions to existing tests that check the old behavior. This prevents those tests from failing during backward compatibility or upgrade testing from 8.latest to 9.0.0
+
+In 9.0.0:
+
+* Add `requires` conditions for new tests that validate the updated API or output format
+* Add `skip` conditions for older tests that would break in 9.0.0
+
+#### Test Features
+
+Both cluster and test features exist. Cluster features are meant for new capability and test features can specifically be used to gate and manage `skip` and `requires` yaml test operations.  For more information, see [Versioning.md](docs/internal/Versioning.md#cluster-features). When backporting and using these features they can not overlap in name and must be consistent when backported so that clusters built with these features are compatible.
+
 ### Developer's workflow
 
 There should not be much, if any, deviation in a developers normal workflow to introduce and back-port changes. Changes should be applied in main, then back ported as needed.

benchmarks/src/main/java/org/elasticsearch/benchmark/compute/operator/EvalBenchmark.java

@@ -24,6 +24,7 @@
 import org.elasticsearch.compute.data.DoubleVector;
 import org.elasticsearch.compute.data.LongBlock;
 import org.elasticsearch.compute.data.LongVector;
+import org.elasticsearch.compute.data.OrdinalBytesRefVector;
 import org.elasticsearch.compute.data.Page;
 import org.elasticsearch.compute.operator.DriverContext;
 import org.elasticsearch.compute.operator.EvalOperator;
@@ -127,7 +128,9 @@ static void selfTest() {
             "mv_min_ascending",
             "rlike",
             "to_lower",
-            "to_upper" }
+            "to_lower_ords",
+            "to_upper",
+            "to_upper_ords" }
     )
     public String operation;
 
@@ -235,12 +238,12 @@ private static EvalOperator.ExpressionEvaluator evaluator(String operation) {
                 RLike rlike = new RLike(Source.EMPTY, keywordField, new RLikePattern(".ar"));
                 yield EvalMapper.toEvaluator(FOLD_CONTEXT, rlike, layout(keywordField)).get(driverContext);
             }
-            case "to_lower" -> {
+            case "to_lower", "to_lower_ords" -> {
                 FieldAttribute keywordField = keywordField();
                 ToLower toLower = new ToLower(Source.EMPTY, keywordField, configuration());
                 yield EvalMapper.toEvaluator(FOLD_CONTEXT, toLower, layout(keywordField)).get(driverContext);
             }
-            case "to_upper" -> {
+            case "to_upper", "to_upper_ords" -> {
                 FieldAttribute keywordField = keywordField();
                 ToUpper toUpper = new ToUpper(Source.EMPTY, keywordField, configuration());
                 yield EvalMapper.toEvaluator(FOLD_CONTEXT, toUpper, layout(keywordField)).get(driverContext);
@@ -414,13 +417,15 @@ private static void checkExpected(String operation, Page actual) {
                     }
                 }
             }
-            case "to_lower" -> checkBytes(operation, actual, new BytesRef[] { new BytesRef("foo"), new BytesRef("bar") });
-            case "to_upper" -> checkBytes(operation, actual, new BytesRef[] { new BytesRef("FOO"), new BytesRef("BAR") });
+            case "to_lower" -> checkBytes(operation, actual, false, new BytesRef[] { new BytesRef("foo"), new BytesRef("bar") });
+            case "to_lower_ords" -> checkBytes(operation, actual, true, new BytesRef[] { new BytesRef("foo"), new BytesRef("bar") });
+            case "to_upper" -> checkBytes(operation, actual, false, new BytesRef[] { new BytesRef("FOO"), new BytesRef("BAR") });
+            case "to_upper_ords" -> checkBytes(operation, actual, true, new BytesRef[] { new BytesRef("FOO"), new BytesRef("BAR") });
             default -> throw new UnsupportedOperationException(operation);
         }
     }
 
-    private static void checkBytes(String operation, Page actual, BytesRef[] expectedVals) {
+    private static void checkBytes(String operation, Page actual, boolean expectOrds, BytesRef[] expectedVals) {
         BytesRef scratch = new BytesRef();
         BytesRefVector v = actual.<BytesRefBlock>getBlock(1).asVector();
         for (int i = 0; i < BLOCK_LENGTH; i++) {
@@ -430,6 +435,15 @@ private static void checkBytes(String operation, Page actual, BytesRef[] expecte
                 throw new AssertionError("[" + operation + "] expected [" + expected + "] but was [" + b + "]");
             }
         }
+        if (expectOrds) {
+            if (v.asOrdinals() == null) {
+                throw new IllegalArgumentException("expected ords but got " + v);
+            }
+        } else {
+            if (v.asOrdinals() != null) {
+                throw new IllegalArgumentException("expected non-ords but got " + v);
+            }
+        }
     }
 
     private static Page page(String operation) {
@@ -510,6 +524,16 @@ private static Page page(String operation) {
                 }
                 yield new Page(builder.build().asBlock());
             }
+            case "to_lower_ords", "to_upper_ords" -> {
+                var bytes = blockFactory.newBytesRefVectorBuilder(BLOCK_LENGTH);
+                bytes.appendBytesRef(new BytesRef("foo"));
+                bytes.appendBytesRef(new BytesRef("bar"));
+                var ordinals = blockFactory.newIntVectorFixedBuilder(BLOCK_LENGTH);
+                for (int i = 0; i < BLOCK_LENGTH; i++) {
+                    ordinals.appendInt(i % 2);
+                }
+                yield new Page(new OrdinalBytesRefVector(ordinals.build(), bytes.build()).asBlock());
+            }
             default -> throw new UnsupportedOperationException();
         };
     }

build-conventions/settings.gradle

@@ -8,7 +8,7 @@
  */
 
 plugins {
-  id "com.gradle.develocity" version "3.18.1"
+  id "com.gradle.develocity" version "3.19.2"
 }
 
 rootProject.name = 'build-conventions'

build-tools-internal/settings.gradle

@@ -9,7 +9,7 @@ pluginManagement {
 }
 
 plugins {
-  id "com.gradle.develocity" version "3.18.1"
+  id "com.gradle.develocity" version "3.19.2"
 }
 
 dependencyResolutionManagement {

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/ConcatFilesTask.java

@@ -8,6 +8,8 @@
  */
 package org.elasticsearch.gradle.internal;
 
+import com.google.common.collect.Iterables;
+
 import org.gradle.api.DefaultTask;
 import org.gradle.api.file.FileCollection;
 import org.gradle.api.tasks.Input;
@@ -85,7 +87,7 @@ public void setAdditionalLines(List<String> additionalLines) {
     public void concatFiles() throws IOException {
         if (getHeaderLine() != null) {
             getTarget().getParentFile().mkdirs();
-            Files.write(getTarget().toPath(), (getHeaderLine() + '\n').getBytes(StandardCharsets.UTF_8));
+            Files.writeString(getTarget().toPath(), getHeaderLine() + '\n');
         }
 
         // To remove duplicate lines
@@ -95,11 +97,12 @@ public void concatFiles() throws IOException {
                 uniqueLines.addAll(Files.readAllLines(f.toPath(), StandardCharsets.UTF_8));
             }
         }
-        Files.write(getTarget().toPath(), uniqueLines, StandardCharsets.UTF_8, StandardOpenOption.APPEND);
-
-        for (String additionalLine : additionalLines) {
-            Files.write(getTarget().toPath(), (additionalLine + '\n').getBytes(StandardCharsets.UTF_8), StandardOpenOption.APPEND);
-        }
+        Files.write(
+            getTarget().toPath(),
+            Iterables.concat(uniqueLines, additionalLines),
+            StandardCharsets.UTF_8,
+            StandardOpenOption.APPEND
+        );
     }
 
 }

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/DependenciesInfoTask.java

@@ -155,7 +155,7 @@ public void generateDependenciesInfo() throws IOException {
             output.append(dep.getGroup() + ":" + dep.getName() + "," + dep.getVersion() + "," + url + "," + licenseType + "\n");
         }
 
-        Files.write(outputFile.toPath(), output.toString().getBytes("UTF-8"), StandardOpenOption.CREATE);
+        Files.writeString(outputFile.toPath(), output.toString(), StandardOpenOption.CREATE);
     }
 
     @Input

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/InternalDistributionArchiveCheckPlugin.java

@@ -19,6 +19,8 @@
 import org.gradle.api.Task;
 import org.gradle.api.file.ArchiveOperations;
 import org.gradle.api.plugins.BasePlugin;
+import org.gradle.api.provider.ListProperty;
+import org.gradle.api.provider.Provider;
 import org.gradle.api.tasks.Copy;
 import org.gradle.api.tasks.TaskProvider;
 
@@ -103,22 +105,26 @@ private static TaskProvider<Task> registerCheckMlCppNoticeTask(
     ) {
         TaskProvider<Task> checkMlCppNoticeTask = project.getTasks().register("checkMlCppNotice", task -> {
             task.dependsOn(checkExtraction);
+            final Provider<Path> noticePath = checkExtraction.map(
+                c -> c.getDestinationDir()
+                    .toPath()
+                    .resolve("elasticsearch-" + VersionProperties.getElasticsearch() + "/modules/x-pack-ml/NOTICE.txt")
+            );
+            ListProperty<String> expectedMlLicenses = extension.expectedMlLicenses;
             task.doLast(new Action<Task>() {
                 @Override
                 public void execute(Task task) {
                     // this is just a small sample from the C++ notices,
                     // the idea being that if we've added these lines we've probably added all the required lines
-                    final List<String> expectedLines = extension.expectedMlLicenses.get();
-                    final Path noticePath = checkExtraction.get()
-                        .getDestinationDir()
-                        .toPath()
-                        .resolve("elasticsearch-" + VersionProperties.getElasticsearch() + "/modules/x-pack-ml/NOTICE.txt");
+                    final List<String> expectedLines = expectedMlLicenses.get();
                     final List<String> actualLines;
                     try {
-                        actualLines = Files.readAllLines(noticePath);
+                        actualLines = Files.readAllLines(noticePath.get());
                         for (final String expectedLine : expectedLines) {
                             if (actualLines.contains(expectedLine) == false) {
-                                throw new GradleException("expected [" + noticePath + " to contain [" + expectedLine + "] but it did not");
+                                throw new GradleException(
+                                    "expected [" + noticePath.get() + " to contain [" + expectedLine + "] but it did not"
+                                );
                             }
                         }
                     } catch (IOException ioException) {
@@ -133,43 +139,37 @@ public void execute(Task task) {
     private TaskProvider<Task> registerCheckNoticeTask(Project project, TaskProvider<Copy> checkExtraction) {
         return project.getTasks().register("checkNotice", task -> {
             task.dependsOn(checkExtraction);
-            task.doLast(new Action<Task>() {
-                @Override
-                public void execute(Task task) {
-                    final List<String> noticeLines = Arrays.asList("Elasticsearch", "Copyright 2009-2024 Elasticsearch");
-                    final Path noticePath = checkExtraction.get()
-                        .getDestinationDir()
-                        .toPath()
-                        .resolve("elasticsearch-" + VersionProperties.getElasticsearch() + "/NOTICE.txt");
-                    assertLinesInFile(noticePath, noticeLines);
-                }
+            var noticePath = checkExtraction.map(
+                copy -> copy.getDestinationDir().toPath().resolve("elasticsearch-" + VersionProperties.getElasticsearch() + "/NOTICE.txt")
+            );
+            task.doLast(t -> {
+                final List<String> noticeLines = Arrays.asList("Elasticsearch", "Copyright 2009-2024 Elasticsearch");
+                assertLinesInFile(noticePath.get(), noticeLines);
             });
         });
     }
 
     private TaskProvider<Task> registerCheckLicenseTask(Project project, TaskProvider<Copy> checkExtraction) {
         TaskProvider<Task> checkLicense = project.getTasks().register("checkLicense", task -> {
             task.dependsOn(checkExtraction);
-            task.doLast(new Action<Task>() {
-                @Override
-                public void execute(Task task) {
-                    String licenseFilename = null;
-                    if (project.getName().contains("oss-") || project.getName().equals("integ-test-zip")) {
-                        licenseFilename = "AGPL-3.0+SSPL-1.0+ELASTIC-LICENSE-2.0.txt";
-                    } else {
-                        licenseFilename = "ELASTIC-LICENSE-2.0.txt";
-                    }
-                    final List<String> licenseLines;
-                    try {
-                        licenseLines = Files.readAllLines(project.getRootDir().toPath().resolve("licenses/" + licenseFilename));
-                        final Path licensePath = checkExtraction.get()
-                            .getDestinationDir()
-                            .toPath()
-                            .resolve("elasticsearch-" + VersionProperties.getElasticsearch() + "/LICENSE.txt");
-                        assertLinesInFile(licensePath, licenseLines);
-                    } catch (IOException ioException) {
-                        ioException.printStackTrace();
-                    }
+            String projectName = project.getName();
+            Provider<Path> licensePathProvider = checkExtraction.map(
+                copy -> copy.getDestinationDir().toPath().resolve("elasticsearch-" + VersionProperties.getElasticsearch() + "/LICENSE.txt")
+            );
+            File rootDir = project.getLayout().getSettingsDirectory().getAsFile();
+            task.doLast(t -> {
+                String licenseFilename = null;
+                if (projectName.contains("oss-") || projectName.equals("integ-test-zip")) {
+                    licenseFilename = "AGPL-3.0+SSPL-1.0+ELASTIC-LICENSE-2.0.txt";
+                } else {
+                    licenseFilename = "ELASTIC-LICENSE-2.0.txt";
+                }
+                final List<String> licenseLines;
+                try {
+                    licenseLines = Files.readAllLines(rootDir.toPath().resolve("licenses/" + licenseFilename));
+                    assertLinesInFile(licensePathProvider.get(), licenseLines);
+                } catch (IOException ioException) {
+                    ioException.printStackTrace();
                 }
             });
         });