Merge branch 'main' of github.com:elastic/elasticsearch into knn_patience

Author: tteofili

.buildkite/scripts/dra-workflow.trigger.sh

@@ -2,10 +2,34 @@
 
 set -euo pipefail
 
-echo "steps:"
-
 source .buildkite/scripts/branches.sh
 
+# We use that filtering to keep different schedule for different branches
+if [ -n "${INCLUDED_BRANCHES:-}" ]; then
+  # If set, only trigger the pipeline for the specified branches
+  IFS=',' read -r -a BRANCHES <<< "${INCLUDED_BRANCHES}"
+elif [ -n "${EXCLUDED_BRANCHES:-}" ]; then
+  # If set, listed branches will be excluded from the list of branches in branches.json
+  IFS=',' read -r -a EXCLUDED_BRANCHES <<< "${EXCLUDED_BRANCHES}"
+  FILTERED_BRANCHES=()
+    for BRANCH in "${BRANCHES[@]}"; do
+      EXCLUDE=false
+      for EXCLUDED_BRANCH in "${EXCLUDED_BRANCHES[@]}"; do
+        if [ "$BRANCH" == "$EXCLUDED_BRANCH" ]; then
+          EXCLUDE=true
+          break
+        fi
+      done
+      if [ "$EXCLUDE" = false ]; then
+        FILTERED_BRANCHES+=("$BRANCH")
+      fi
+    done
+    BRANCHES=("${FILTERED_BRANCHES[@]}")
+fi
+
+
+echo "steps:"
+
 for BRANCH in "${BRANCHES[@]}"; do
   INTAKE_PIPELINE_SLUG="elasticsearch-intake"
   BUILD_JSON=$(curl -sH "Authorization: Bearer ${BUILDKITE_API_TOKEN}" "https://api.buildkite.com/v2/organizations/elastic/pipelines/${INTAKE_PIPELINE_SLUG}/builds?branch=${BRANCH}&state=passed&per_page=1" | jq '.[0] | {commit: .commit, url: .web_url}')

BUILDING.md

@@ -82,7 +82,7 @@ In case of updating a dependency, ensure to remove the unused entry of the outda
 
 You can also automate the generation of this entry by running your build using the `--write-verification-metadata` commandline option:
 ```
->./gradlew --write-verification-metadata sha256 precommit
+./gradlew --write-verification-metadata sha256 precommit
 ```
 
 The `--write-verification-metadata` Gradle option is generally able to resolve reachable configurations,
@@ -92,10 +92,10 @@ uses the changed dependencies. In most cases, `precommit` or `check` are good ca
 We prefer sha256 checksums as md5 and sha1 are not considered safe anymore these days. The generated entry
 will have the `origin` attribute been set to `Generated by Gradle`.
 
->A manual confirmation of the Gradle generated checksums is currently not mandatory.
->If you want to add a level of verification you can manually confirm the checksum (e.g. by looking it up on the website of the library)
->Please replace the content of the `origin` attribute by `official site` in that case.
->
+> [!Tip]  
+> A manual confirmation of the Gradle generated checksums is currently not mandatory.
+> If you want to add a level of verification you can manually confirm the checksum (e.g. by looking it up on the website of the library)
+> Please replace the content of the `origin` attribute by `official site` in that case.
 
 #### Custom plugin and task implementations
 
@@ -229,13 +229,9 @@ In addition to snapshot builds JitPack supports building Pull Requests. Simply u
 3. Run the Gradle build as needed. Keep in mind the initial resolution might take a bit longer as this needs to be built
 by JitPack in the background before we can resolve the adhoc built dependency.
 
----
-
-**NOTE**
-
-You should only use that approach locally or on a developer branch for production dependencies as we do
+> [!Note]  
+> You should only use that approach locally or on a developer branch for production dependencies as we do
 not want to ship unreleased libraries into our releases.
----
 
 #### How to use a custom third party artifact?
 
@@ -265,12 +261,9 @@ allprojects {
   ```
 4. Run the Gradle build as needed with `--write-verification-metadata` to ensure the Gradle dependency verification does not fail on your custom dependency.
 
----
-**NOTE**
-
-As Gradle prefers to use modules whose descriptor has been created from real meta-data rather than being generated,
+> [!Note]  
+> As Gradle prefers to use modules whose descriptor has been created from real meta-data rather than being generated,
 flat directory repositories cannot be used to override artifacts with real meta-data from other repositories declared in the build.
-For example, if Gradle finds only `jmxri-1.2.1.jar` in a flat directory repository, but `jmxri-1.2.1.pom` in another repository
+> For example, if Gradle finds only `jmxri-1.2.1.jar` in a flat directory repository, but `jmxri-1.2.1.pom` in another repository
 that supports meta-data, it will use the second repository to provide the module.
-Therefore, it is recommended to declare a version that is not resolvable from public repositories we use (e.g. Maven Central)
----
+> Therefore, it is recommended to declare a version that is not resolvable from public repositories we use (e.g. Maven Central)

CONTRIBUTING.md

@@ -401,7 +401,8 @@ It is important that the only code covered by the Elastic licence is contained
 within the top-level `x-pack` directory. The build will fail its pre-commit
 checks if contributed code does not have the appropriate license headers.
 
-> **NOTE:** If you have imported the project into IntelliJ IDEA the project will
+> [!NOTE]
+> If you have imported the project into IntelliJ IDEA the project will
 > be automatically configured to add the correct license header to new source
 > files based on the source location.
 

build-tools-internal/src/main/resources/checkstyle.xml

@@ -68,6 +68,11 @@
 
     <!-- Unused imports are forbidden -->
     <module name="UnusedImports" />
+    <module name="SuppressionCommentFilter">
+      <property name="offCommentFormat" value="begin generated imports"/>
+      <property name="onCommentFormat" value="end generated imports"/>
+      <property name="checkFormat" value="UnusedImports"/>
+    </module>
 
     <!-- Non-inner classes must be in files that match their names. -->
     <module name="OuterTypeFilename" />

distribution/docker/src/docker/Dockerfile.ess-fips

@@ -96,15 +96,6 @@ COPY fips/resources/fips_java.policy /usr/share/elasticsearch/config/fips_java.p
 
 WORKDIR /usr/share/elasticsearch/config
 
-## Add fips specific JVM options
-RUN cat <<EOF > /usr/share/elasticsearch/config/jvm.options.d/fips.options
--Djavax.net.ssl.keyStoreType=BCFKS
--Dorg.bouncycastle.fips.approved_only=true
--Djava.security.properties=config/fips_java.security
--Djava.security.policy=config/fips_java.policy
-EOF
-
-
 ################################################################################
 # Build stage 2 (the actual Elasticsearch image):
 #
@@ -136,6 +127,10 @@ ENV ELASTIC_CONTAINER=true
 WORKDIR /usr/share/elasticsearch
 
 COPY --from=builder --chown=0:0 /usr/share/elasticsearch /usr/share/elasticsearch
+COPY --from=builder --chown=0:0 /fips/libs/*.jar /usr/share/elasticsearch/lib/
+COPY --from=builder --chown=0:0 /opt /opt
+
+ENV ES_PLUGIN_ARCHIVE_DIR=/opt/plugins/archive
 ENV PATH=/usr/share/elasticsearch/bin:\$PATH
 ENV SHELL=/bin/bash
 COPY ${bin_dir}/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
@@ -159,6 +154,28 @@ RUN chmod g=u /etc/passwd && \\
 
 RUN ln -sf /etc/ssl/certs/java/cacerts /usr/share/elasticsearch/jdk/lib/security/cacerts
 
+# Convert cacerts (PKCS12) to BCFKS format using POSIX-compatible shell syntax
+RUN printf "\\n" | jdk/bin/keytool -importkeystore \
+  -srckeystore /usr/share/elasticsearch/jdk/lib/security/cacerts \
+  -srcstoretype PKCS12 \
+  -destkeystore config/cacerts.bcfks \
+  -deststorepass passwordcacert \
+  -deststoretype BCFKS \
+  -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \
+  -providerpath lib/bc-fips-1.0.2.5.jar \
+  -destprovidername BCFIPS
+
+
+## Add fips specific JVM options
+RUN cat <<EOF > /usr/share/elasticsearch/config/jvm.options.d/fips.options
+-Djavax.net.ssl.keyStoreType=BCFKS
+-Dorg.bouncycastle.fips.approved_only=true
+-Djava.security.properties=config/fips_java.security
+-Djava.security.policy=config/fips_java.policy
+-Djavax.net.ssl.trustStore=config/cacerts.bcfks
+-Djavax.net.ssl.trustStorePassword=passwordcacert
+EOF
+
 EXPOSE 9200 9300
 
 LABEL org.label-schema.build-date="${build_date}" \\
@@ -196,11 +213,6 @@ CMD ["/app/elasticsearch.sh"]
 
 USER 1000:0
 
-COPY --from=builder --chown=0:0 /opt /opt
-ENV ES_PLUGIN_ARCHIVE_DIR=/opt/plugins/archive
-WORKDIR /usr/share/elasticsearch
-COPY --from=builder --chown=0:0 /fips/libs/*.jar /usr/share/elasticsearch/lib/
-
 ################################################################################
 # End of multi-stage Dockerfile
 ################################################################################

docs/changelog/120488.yaml

@@ -0,0 +1,5 @@
+pr: 120488
+summary: Publish queue latency metrics from tracked thread pools
+area: "Infra/Metrics"
+type: enhancement
+issues: []

docs/changelog/127383.yaml

@@ -0,0 +1,5 @@
+pr: 127383
+summary: Don't push down filters on the right hand side of an inlinejoin
+area: ES|QL
+type: bug
+issues: []

docs/changelog/127658.yaml

@@ -0,0 +1,5 @@
+pr: 127658
+summary: Append all data to Chat Completion buffer
+area: Machine Learning
+type: bug
+issues: []

docs/release-notes/breaking-changes.md

@@ -12,10 +12,12 @@ If you are migrating from a version prior to version 9.0, you must first upgrade
 
 % ## Next version [elasticsearch-nextversion-breaking-changes]
 
-## 9.1.0 [elasticsearch-910-breaking-changes]
+## 9.0.1 [elasticsearch-9.0.1-breaking-changes]
+```{applies_to}
+stack: coming 9.0.1
+```
 
-ES|QL
-:   * Allow partial results by default in ES|QL [#125060](https://github.com/elastic/elasticsearch/pull/125060)
+No breaking changes in this version.
 
 ## 9.0.0 [elasticsearch-900-breaking-changes]
 

docs/release-notes/deprecations.md

@@ -16,6 +16,13 @@ To give you insight into what deprecated features you’re using, {{es}}:
 
 % ## Next version [elasticsearch-nextversion-deprecations]
 
+## 9.0.1 [elasticsearch-9.0.1-deprecations]
+```{applies_to}
+stack: coming 9.0.1
+```
+
+No deprecations in this version.
+
 ## 9.0.0 [elasticsearch-900-deprecations]
 
 ES|QL: