Merge branch 'main' into esql/settings_stack_telemetry

Author: luigidellaquila

.github/CODEOWNERS

@@ -107,20 +107,20 @@ docs/*.yml @elastic/core-docs
 docs/community-clients @elastic/core-docs
 docs/release-notes @elastic/core-docs
 docs/extend @elastic/admin-docs
-docs/java-rest @elastic/dev-docs
+docs/java-rest @elastic/developer-docs
 docs/reference @elastic/core-docs
-docs/reference/aggregations @elastic/dev-docs
-docs/reference/community-contributed @elastic/dev-docs
+docs/reference/aggregations @elastic/developer-docs
+docs/reference/community-contributed @elastic/developer-docs
 docs/reference/elasticsearch/command-line-tools @elastic/admin-docs
-docs/reference/elasticsearch/index-lifecycle-actions @elastic/dev-docs
-docs/reference/elasticsearch/mapping-reference @elastic/dev-docs
-docs/reference/elasticsearch/rest-apis @elastic/dev-docs
+docs/reference/elasticsearch/index-lifecycle-actions @elastic/developer-docs
+docs/reference/elasticsearch/mapping-reference @elastic/developer-docs
+docs/reference/elasticsearch/rest-apis @elastic/developer-docs
 docs/reference/elasticsearch/elasticsearch-audit-events.md @elastic/admin-docs
 docs/reference/elasticsearch/jvm-settings.md @elastic/admin-docs
 docs/reference/elasticsearch/roles.md @elastic/admin-docs
 docs/reference/enrich-processor @elastic/admin-docs
-docs/reference/query-languages @elastic/dev-docs
+docs/reference/query-languages @elastic/developer-docs
 docs/reference/scripting-languages @elastic/admin-docs
-docs/reference/search-connectors @elastic/dev-docs
+docs/reference/search-connectors @elastic/developer-docs
 docs/reference/setup/install/docker @elastic/admin-docs
-docs/reference/text-analysis @elastic/dev-docs
+docs/reference/text-analysis @elastic/developer-docs

.gitignore

@@ -2,6 +2,9 @@
 # claude
 .claude
 
+# cursor
+.cursorrules
+
 # JetBrainsAI/Junie
 .junie/
 
@@ -87,3 +90,6 @@ server/src/main/resources/transport/definitions/manifest.txt
 
 # JEnv
 .java-version
+
+# Vector data files
+qa/vector/.data

AGENTS.md

@@ -0,0 +1,98 @@
+# Elasticsearch
+
+## Toolchain Snapshot
+- **Java**: JDK 25 via `JAVA_HOME`; use the bundled Gradle wrapper (`./gradlew`).
+- **Build tooling**: Gradle composite build with `build-conventions`, `build-tools`, and `build-tools-internal`; Docker is required for some packaging/tests.
+- **OS packages**: Packaging and QA jobs expect ephemeral hosts; do not run packaging suites on your workstation.
+- **Security**: Default dev clusters enable security; use `elastic-admin:elastic-password` or disable with `-Dtests.es.xpack.security.enabled=false`.
+- **Cursor/Copilot rules**: None provided in repo; follow this guide plus CONTRIBUTING.md.
+
+## Build & Run Commands
+- Refer to CONTRIBUTING.md & TESTING.asciidoc for comprehensive build/test instructions.
+
+## Verification & Lint Tasks
+- `./gradlew spotlessJavaCheck` / `spotlessApply` (or `:server:spotlessJavaCheck`): enforce formatter profile in `build-conventions/formatterConfig.xml`.
+
+## Project Structure
+The repository is organized into several key directories:
+*   `server`: The core Elasticsearch server.
+*   `modules`: Features shipped with Elasticsearch by default.
+*   `plugins`: Officially supported plugins.
+*   `libs`: Internal libraries used by other parts of the project.
+*   `qa`: Integration and multi-version tests.
+*   `docs`: Project documentation.
+*   `distribution`: Logic for building distribution packages.
+*   `x-pack`: Additional code modules and plugins under Elastic License.
+*   `build-conventions`, `build-tools`, `build-tools-internal`: Gradle build logic.
+
+## Testing Cheatsheet
+- Standard suite: `./gradlew test` (respects cached results; add `-Dtests.timestamp=$(date +%s)` to bypass caches when reusing seeds).
+- Single project: `./gradlew :server:test` (or other subproject path).
+- Single class: `./gradlew :server:test --tests org.elasticsearch.package.ClassName`.
+- Single package: `./gradlew :server:test --tests 'org.elasticsearch.package.*'`.
+- Single method / repeated runs: `./gradlew :server:test --tests org.elasticsearch.package.ClassName.methodName -Dtests.iters=N`.
+- Deterministic seed: append `-Dtests.seed=DEADBEEF` (each method uses derived seeds).
+- JVM tuning knobs: `-Dtests.jvms=8`, `-Dtests.heap.size=4G`, `-Dtests.jvm.argline="-verbose:gc"`, `-Dtests.output=always`, etc.
+- Debugging: append `--debug-jvm` to the Gradle test task and attach a debugger on port 5005.
+- CI reproductions: copy the `REPRODUCE WITH` line from CI logs; it includes project path, seed, and JVM flags.
+- Yaml REST tests: `./gradlew ":rest-api-spec:yamlRestTest" --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=<relative_test_file_path>}"`
+- Use the Elasticsearch testing framework where possible for unit and yaml tests and be consistent in style with other elasticsearch tests.
+- Use real classes over mocks or stubs for unit tests, unless the real class is complex then either a simplified subclass should be created within the test or, as a last resort, a mock or stub can be used. Unit tests must be as close to real-world scenarios as possible.
+- Ensure mocks or stubs are well-documented and clearly indicate why they were necessary. 
+
+### Test Types
+- Unit Tests: Preferred. Extend `ESTestCase`.
+- Single Node: Extend `ESSingleNodeTestCase` (lighter than full integ test).
+- Integration: Extend `ESIntegTestCase`.
+- REST API: Extend `ESRestTestCase` or `ESClientYamlSuiteTestCase`. **YAML based REST tests are preferred** for integration/API testing.
+
+## Dependency Hygiene
+- Never add a dependency without checking for existing alternatives in the repo.
+
+## Formatting & Imports
+- Absolutely no wildcard imports; keep existing import order and avoid reordering untouched lines.
+
+## Types, Generics, and Suppressions
+- Prefer type-safe constructs; avoid raw types and unchecked casts.
+- If suppressing warnings, scope `@SuppressWarnings` narrowly (ideally a single statement or method).
+- Document non-obvious casts or type assumptions via Javadoc/comments for reviewers.
+
+## Naming Conventions
+- REST handlers typically use the `Rest*Action` pattern; transport-layer handlers mirror them with `Transport*Action` classes.
+- REST classes expose routes via `RestHandler#routes`; when adding endpoints ensure naming matches existing REST/Transport patterns to aid discoverability.
+- Transport `ActionType` strings encode scope (`indices:data/read/...`, `cluster:admin/...`, etc.); align new names with these conventions to integrate with privilege resolution.
+
+## Logging & Error Handling
+- Elasticsearch should prefer its own logger `org.elasticsearch.logging.LogManager` & `org.elasticsearch.logging.Logger`; declare `private static final Logger logger = LogManager.getLogger(Class.class)`.
+- Always use parameterized logging (`logger.debug("operation [{}]", value)`); never build strings via concatenation.
+- Wrap expensive log-message construction in `() -> Strings.format(...)` suppliers when logging at `TRACE`/`DEBUG` to avoid unnecessary work.
+- Log levels:
+  - `TRACE`: highly verbose developer diagnostics; usually read alongside code.
+  - `DEBUG`: detailed production troubleshooting; ensure volume is bounded.
+  - `INFO`: default-enabled operational milestones; prefer factual language.
+  - `WARN`: actionable problems users must investigate; include context and, if needed, exception stack traces.
+  - `ERROR`: reserve for unrecoverable states (e.g., storage health failures); prefer `WARN` otherwise.
+- Only log client-caused exceptions when the cluster admin can act on them; otherwise rely on API responses.
+- Tests can assert logging via `MockLog` for complex flows.
+
+## Javadoc & Comments
+- New packages/classes/public or abstract methods require Javadoc explaining the "why" rather than the implementation details.
+- Avoid documenting trivial getters/setters; focus on behavior, preconditions, or surprises.
+- For tests, Javadoc can describe scenario setup/expectations to aid future contributors.
+- Do not remove existing comments from code unless the code is also being removed or the comment has become incorrect.
+
+## License Headers
+- Default header (outside `x-pack`): Elastic License 2.0, SSPL v1, or AGPL v3—they are already codified at the top of Java files; copy from existing sources.
+- Files under `x-pack` require the Elastic License 2.0-only header; IDEs configured per CONTRIBUTING.md can insert correct text automatically.
+
+## Best Practices for Automation Agents
+- Never edit unrelated files; keep diffs tightly scoped to the task at hand.
+- Prefer Gradle tasks over ad-hoc scripts.
+- When scripting CLI sequences, leverage `gradlew` task.
+- Unrecognized changes: assume other agent; keep going; focus your changes. If it causes issues, stop + ask user.
+- Do not add "Co-Authored-By" lines to commit messages. commit messages should adhere to the 50/72 rule: use a maximum of 50 columns for the commit summary
+
+## Backwards compatibility
+- For changes to a `Writeable` implementation (`writeTo` and constructor from `StreamInput`), add a new `public static final <UNIQUE_DESCRIPTIVE_NAME> = TransportVersion.fromName("<unique_descriptive_name>")` and use it in the new code paths. Confirm the backport branches and then generate a new version file with `./gradlew generateTransportVersion`.
+
+Stay aligned with `CONTRIBUTING.md`, `BUILDING.md`, and `TESTING.asciidoc`; this AGENTS guide summarizes—but does not replace—those authoritative docs.

BUILDING.md

@@ -216,7 +216,7 @@ The benefit of a dedicated project for these tests are:
 
 Sometimes we want to share test fixtures to set up the code under test across multiple projects. There are basically two ways doing so.
 
-Ideally we would use the build-in [java-test-fixtures](https://docs.gradle.org/current/userguide/java_testing.html#sec:java_test_fixtures) Gradle plugin.
+Ideally we would use the built-in [java-test-fixtures](https://docs.gradle.org/current/userguide/java_testing.html#sec:java_test_fixtures) Gradle plugin.
 This plugin relies on having a separate sourceSet for the test fixtures code.
 
 In the Elasticsearch codebase we have test fixtures and actual tests within the same sourceSet. Therefore we introduced the `elasticsearch.internal-test-artifact` plugin to provides another build artifact of your project based on the `test` sourceSet.
@@ -232,7 +232,7 @@ dependencies {
 ```
 
 This test artifact mechanism makes use of the concept of [component capabilities](https://docs.gradle.org/current/userguide/component_capabilities.html)
-similar to how the Gradle build-in `java-test-fixtures` plugin works.
+similar to how the Gradle built-in `java-test-fixtures` plugin works.
 
 `testArtifact` is a shortcut declared in the Elasticsearch build. Alternatively you can declare the dependency via
 

REST_API_COMPATIBILITY.md

@@ -39,7 +39,7 @@ The headers mirrors the 4 supported media types for both the `Accept` and `Conte
 "application/vnd.elasticsearch+cbor;compatible-with=7"
 ```
 
-The version request is found in compatible-with=7. When sent to a version 7 server will do nothing special. When sent to a version 8 server will apply compatiblity if needed. When sent to a version 9 server will fail since compatibility is only supported across 1 major version.
+The version request is found in compatible-with=7. When sent to a version 7 server will do nothing special. When sent to a version 8 server will apply compatibility if needed. When sent to a version 9 server will fail since compatibility is only supported across 1 major version.
 
 A consumer/client may not mix "compatible-with" versions between headers. Attempts to mix and match compatible versions between `Accept` and `Content-Type` headers will result in an error.  A consumer/client may mix the media type (i.e. send `yaml` and get back `json`).
 

benchmarks/src/main/java/org/elasticsearch/benchmark/_nightly/esql/TopNBenchmark.java

@@ -23,6 +23,7 @@
 import org.elasticsearch.compute.data.ElementType;
 import org.elasticsearch.compute.data.Page;
 import org.elasticsearch.compute.operator.Operator;
+import org.elasticsearch.compute.operator.topn.SharedMinCompetitive;
 import org.elasticsearch.compute.operator.topn.TopNEncoder;
 import org.elasticsearch.compute.operator.topn.TopNOperator;
 import org.elasticsearch.indices.breaker.CircuitBreakerMetrics;
@@ -56,6 +57,10 @@
 @State(Scope.Thread)
 @Fork(1)
 public class TopNBenchmark {
+    static {
+        LogConfigurator.configureESLogging();
+    }
+
     private static final BlockFactory blockFactory = BlockFactory.getInstance(
         new NoopCircuitBreaker("noop"),
         BigArrays.NON_RECYCLING_INSTANCE
@@ -133,6 +138,19 @@ private static Operator operator(String data, int topCount, boolean sortedInput)
             List.of(),
             ClusterSettings.createBuiltInClusterSettings()
         );
+        SharedMinCompetitive.Supplier minCompetitive = new SharedMinCompetitive.Supplier(
+            blockFactory.breaker(),
+            IntStream.range(0, encoders.size())
+                .mapToObj(
+                    i -> new SharedMinCompetitive.KeyConfig(
+                        elementTypes.get(i),
+                        encoders.get(i),
+                        sortOrders.get(i).asc(),
+                        sortOrders.get(i).nullsFirst()
+                    )
+                )
+                .toList()
+        );
         return new TopNOperator(
             blockFactory,
             breakerService.getBreaker(CircuitBreaker.REQUEST),
@@ -141,7 +159,8 @@ private static Operator operator(String data, int topCount, boolean sortedInput)
             encoders,
             sortOrders,
             8 * 1024,
-            sortedInput ? TopNOperator.InputOrdering.SORTED : TopNOperator.InputOrdering.NOT_SORTED
+            sortedInput ? TopNOperator.InputOrdering.SORTED : TopNOperator.InputOrdering.NOT_SORTED,
+            minCompetitive // This is optional, but doesn't add much overhead either way
         );
     }
 

benchmarks/src/main/java/org/elasticsearch/benchmark/_nightly/esql/ValuesSourceReaderBenchmark.java

@@ -43,7 +43,7 @@
 import org.elasticsearch.compute.data.Page;
 import org.elasticsearch.compute.lucene.AlwaysReferencedIndexedByShardId;
 import org.elasticsearch.compute.lucene.IndexedByShardIdFromSingleton;
-import org.elasticsearch.compute.lucene.LuceneSourceOperator;
+import org.elasticsearch.compute.lucene.query.LuceneSourceOperator;
 import org.elasticsearch.compute.lucene.read.ValuesSourceReaderOperator;
 import org.elasticsearch.compute.lucene.read.ValuesSourceReaderOperatorStatus;
 import org.elasticsearch.compute.operator.DriverContext;
@@ -237,7 +237,7 @@ private static BlockLoader blockLoader(String name) {
                     break;
             }
             ft.freeze();
-            return new KeywordFieldMapper.KeywordFieldType(w.name, ft, syntheticSource).blockLoader(blContext());
+            return new KeywordFieldMapper.KeywordFieldType(w.name, ft, syntheticSource).blockLoader(new BenchContext());
         }
         throw new IllegalArgumentException("can't read [" + name + "]");
     }
@@ -279,7 +279,7 @@ private static BlockLoader numericBlockLoader(WhereAndBaseName w, NumberFieldMap
             null,
             null,
             false
-        ).blockLoader(blContext());
+        ).blockLoader(new BenchContext());
     }
 
     /**
@@ -586,52 +586,60 @@ public void teardownIndex() throws IOException {
         IOUtils.close(reader, directory);
     }
 
-    private static MappedFieldType.BlockLoaderContext blContext() {
-        return new MappedFieldType.BlockLoaderContext() {
-            @Override
-            public String indexName() {
-                return "benchmark";
-            }
+    private static class BenchContext implements MappedFieldType.BlockLoaderContext {
+        @Override
+        public String indexName() {
+            return "benchmark";
+        }
 
-            @Override
-            public IndexSettings indexSettings() {
-                throw new UnsupportedOperationException();
-            }
+        @Override
+        public IndexSettings indexSettings() {
+            throw new UnsupportedOperationException();
+        }
 
-            @Override
-            public MappedFieldType.FieldExtractPreference fieldExtractPreference() {
-                return MappedFieldType.FieldExtractPreference.NONE;
-            }
+        @Override
+        public MappedFieldType.FieldExtractPreference fieldExtractPreference() {
+            return MappedFieldType.FieldExtractPreference.NONE;
+        }
 
-            @Override
-            public SearchLookup lookup() {
-                throw new UnsupportedOperationException();
-            }
+        @Override
+        public SearchLookup lookup() {
+            throw new UnsupportedOperationException();
+        }
 
-            @Override
-            public Set<String> sourcePaths(String name) {
-                return Set.of(name);
-            }
+        @Override
+        public Set<String> sourcePaths(String name) {
+            return Set.of(name);
+        }
 
-            @Override
-            public String parentField(String field) {
-                throw new UnsupportedOperationException();
-            }
+        @Override
+        public String parentField(String field) {
+            throw new UnsupportedOperationException();
+        }
 
-            @Override
-            public FieldNamesFieldMapper.FieldNamesFieldType fieldNames() {
-                return FieldNamesFieldMapper.FieldNamesFieldType.get(true);
-            }
+        @Override
+        public FieldNamesFieldMapper.FieldNamesFieldType fieldNames() {
+            return FieldNamesFieldMapper.FieldNamesFieldType.get(true);
+        }
 
-            @Override
-            public MappingLookup mappingLookup() {
-                return null;
-            }
+        @Override
+        public MappingLookup mappingLookup() {
+            return null;
+        }
 
-            @Override
-            public BlockLoaderFunctionConfig blockLoaderFunctionConfig() {
-                return null;
-            }
-        };
+        @Override
+        public BlockLoaderFunctionConfig blockLoaderFunctionConfig() {
+            return null;
+        }
+
+        @Override
+        public ByteSizeValue ordinalsByteSize() {
+            return DEFAULT_ORDINALS_BYTE_SIZE;
+        }
+
+        @Override
+        public ByteSizeValue scriptByteSize() {
+            return DEFAULT_SCRIPT_BYTE_SIZE;
+        }
     }
 }