Merge branch 'main' into filesettings

Author: jdconrad

build-tools/src/main/java/org/elasticsearch/gradle/testclusters/RunTask.java

@@ -24,6 +24,7 @@
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
@@ -45,6 +46,8 @@ public abstract class RunTask extends DefaultTestClustersTask {
 
     private Boolean apmServerEnabled = false;
 
+    private List<String> plugins = List.of();
+
     private Boolean preserveData = false;
 
     private Path dataDir = null;
@@ -101,6 +104,22 @@ public void setApmServerEnabled(Boolean apmServerEnabled) {
         this.apmServerEnabled = apmServerEnabled;
     }
 
+    @Option(option = "with-plugins", description = "Run distribution with plugins installed")
+    public void setPlugins(String plugins) {
+        this.plugins = Arrays.asList(plugins.split(","));
+        for (var cluster : getClusters()) {
+            for (String plugin : this.plugins) {
+                cluster.plugin(":plugins:" + plugin);
+            }
+            dependsOn(cluster.getPluginAndModuleConfigurations());
+        }
+    }
+
+    @Input
+    public List<String> getPlugins() {
+        return plugins;
+    }
+
     @Option(option = "data-dir", description = "Override the base data directory used by the testcluster")
     public void setDataDir(String dataDirStr) {
         dataDir = Paths.get(dataDirStr).toAbsolutePath();

docs/changelog/123712.yaml

@@ -0,0 +1,5 @@
+pr: 123712
+summary: Process ILM cluster state updates on another thread
+area: ILM+SLM
+type: enhancement
+issues: []

docs/changelog/124690.yaml

@@ -0,0 +1,5 @@
+pr: 124690
+summary: Avoid creating known_fields for every check in Alias
+area: Indices APIs
+type: enhancement
+issues: []

docs/reference/query-languages/esql/esql-commands.md

@@ -677,17 +677,11 @@ This functionality is in technical preview and may be changed or removed in a fu
 
 **Syntax**
 
-```
+```esql
 FROM <source_index>
 | LOOKUP JOIN <lookup_index> ON <field_name>
 ```
 
-```esql
-FROM firewall_logs
-| LOOKUP JOIN threat_list ON source.IP
-| WHERE threat_level IS NOT NULL
-```
-
 **Parameters**
 
 `<lookup_index>`
@@ -717,6 +711,14 @@ FROM firewall_logs
 | LOOKUP JOIN threat_list ON source.IP
 ```
 
+To filter only for those rows that have a matching `threat_list` entry, use `WHERE ... IS NOT NULL` with a field from the lookup index:
+
+```esql
+FROM firewall_logs
+| LOOKUP JOIN threat_list ON source.IP
+| WHERE threat_level IS NOT NULL
+```
+
 **Host metadata correlation**: This query pulls in environment or ownership details for each host to correlate with your metrics data.
 
 ```esql

docs/reference/query-languages/esql/esql-enrich-data.md

@@ -19,10 +19,10 @@ For example, you can use `ENRICH` to:
 
 * Enrichment data doesn't change frequently
 * You can accept index-time overhead
-* You are working with structured enrichment patterns
 * You can accept having multiple matches combined into multi-values
 * You can accept being limited to predefined match fields
-* `ENRICH` has a simplified security model. There are no restirctions to specific enrich policies or document and field level security.
+* You do not need fine-grained security: There are no restrictions to specific enrich policies or document and field level security.
+* You want to match using ranges or spatial relations
 
 ### How the `ENRICH` command works [esql-how-enrich-works]
 

docs/reference/query-languages/esql/esql-lookup-join.md

@@ -20,11 +20,11 @@ For example, you can use `LOOKUP JOIN` to:
 
 * Your enrichment data changes frequently
 * You want to avoid index-time processing
-* You're working with regular indices
-* You need to preserve distinct matches
+* You want SQL-like behavior, so that multiple matches result in multiple rows
 * You need to match on any field in a lookup index
 * You use document or field level security
-* You want to restrict users to a specific lookup indices that they can you
+* You want to restrict users to use only specific lookup indices
+* You do not need to match using ranges or spatial relations
 
 ## How the `LOOKUP JOIN` command works [esql-how-lookup-join-works]
 
@@ -108,7 +108,7 @@ FROM employees
 To use `LOOKUP JOIN`, the following requirements must be met:
 
 * **Compatible data types**: The join key and join field in the lookup index must have compatible data types. This means:
-  * The data types must either be identical or be internally represented as the same type in Elasticsearch's type system
+  * The data types must either be identical or be internally represented as the same type in {esql}
   * Numeric types follow these compatibility rules:
     * `short` and `byte` are compatible with `integer` (all represented as `int`)
     * `float`, `half_float`, and `scaled_float` are compatible with `double` (all represented as `double`)
@@ -120,9 +120,9 @@ For a complete list of supported data types and their internal representations,
 
 The following are the current limitations with `LOOKUP JOIN`
 
-* `LOOKUP JOIN` will be successful if the join field in the lookup index is a `KEYWORD` type. If the main index's join field is `TEXT` type, it must have an exact `.keyword` subfield that can be matched with the lookup index's `KEYWORD` field.
 * Indices in [lookup](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting) mode are always single-sharded.
 * Cross cluster search is unsupported. Both source and lookup indices must be local.
+* Currently, only matching on equality is supported.
 * `LOOKUP JOIN` can only use a single match field and a single index. Wildcards, aliases, datemath, and datastreams are not supported.
-* The name of the match field in `LOOKUP JOIN lu_idx ON match_field` must match an existing field in the query. This may require renames or evals to achieve.
+* The name of the match field in `LOOKUP JOIN lu_idx ON match_field` must match an existing field in the query. This may require `RENAME`s or `EVAL`s to achieve.
 * The query will circuit break if there are too many matching documents in the lookup index, or if the documents are too large. More precisely, `LOOKUP JOIN` works in batches of, normally, about 10,000 rows; a large amount of heap space is needed if the matching documents from the lookup index for a batch are multiple megabytes or larger. This is roughly the same as for `ENRICH`.

libs/entitlement/src/test/java/org/elasticsearch/entitlement/runtime/policy/PolicyManagerTests.java

@@ -508,7 +508,7 @@ public void testFilesEntitlementsWithExclusive() {
         assertThat(
             iae.getMessage(),
             allOf(
-                containsString("Path [/base/test] is already exclusive"),
+                containsString("Path [" + testPath1 + "] is already exclusive"),
                 containsString("[plugin1][test.module1]"),
                 containsString("[plugin2][test.module2]"),
                 containsString("cannot add exclusive access")