Merge branch 'main' into entitlements/improve-fileaccesstree-logging

Author: ldematte

docs/release-notes/known-issues.md

@@ -7,3 +7,23 @@ mapped_pages:
 # Elasticsearch known issues [elasticsearch-known-issues]
 Known issues are significant defects or limitations that may impact your implementation. These issues are actively being worked on and will be addressed in a future release. Review the Elasticsearch known issues to help you make informed decisions, such as upgrading to a new version.
 
+## 9.0.0 [elasticsearch-9.0.0-known-issues]
+* Elasticsearch on Windows might fail to start, or might forbid some file-related operations, when referencing paths with a case different from the one stored by the filesystem. Windows treats paths as case-insensitive, but the filesystem stores them with case. Entitlements, the new security system used by Elasticsearch, treat all paths as case-sensitive, and can therefore prevent access to a path that should be accessible.
+
+  For example: If Elasticsearch is installed in  `C:\ELK\elasticsearch`, and you try to launch it as `c:\elk\elasticsearch\bin\elasticsearch.bat`, you will get a `NotEntitledException` while booting. This is because Elasticsearch blocks access to `c:\elk\elasticsearch`, because does not match `C:\ELK\elasticsearch`. \
+This issue will be fixed in a future patch release (see [PR #126990](https://github.com/elastic/elasticsearch/pull/126990)).
+
+  As a workaround, make sure that all paths you specify have the same casing as the paths stored in the filesystem. Files and directory names should be entered as they appear in Windows Explorer or in a command prompt. This applies to paths specified in the command line, config files, environment variables and secure settings.
+
+* Active Directory authentication is blocked by default. Entitlements, the new security system used by Elasticsearch, has a policy for the `x-pack-core` module that is too restrictive, and does not allow the LDAP library used for AD authentication to perform outbound network connections. This issue will be fixed in a future patch release (see [PR #126992](https://github.com/elastic/elasticsearch/pull/126992)).
+
+  As a workaround, you can temporarily patch the policy using a JVM option:
+
+  1. Create a file called `${ES_CONF_PATH}/jvm_options/workaround-127061.options`.
+  2. Add the following line to the new file:
+
+     ```
+     -Des.entitlements.policy.x-pack-core=dmVyc2lvbnM6CiAgLSA4LjE4LjAKICAtIDkuMC4wCnBvbGljeToKICB1bmJvdW5kaWQubGRhcHNkazoKICAgIC0gc2V0X2h0dHBzX2Nvbm5lY3Rpb25fcHJvcGVydGllcwogICAgLSBvdXRib3VuZF9uZXR3b3Jr
+     ```
+
+  For information about editing your JVM settings, refer to [JVM settings](https://www.elastic.co/docs/reference/elasticsearch/jvm-settings).

libs/entitlement/bridge/src/main/java/org/elasticsearch/entitlement/bridge/package-info.java

@@ -0,0 +1,26 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+/**
+ * Contains classes that need to be used directly from instrumented methods.
+ * It's a minimal shim that is patched into the {@code java.base} module so that it is callable from the class library methods instrumented
+ * by the agent. The shim retains a {@link org.elasticsearch.entitlement.bridge.EntitlementChecker} instance (inside its
+ * {@link org.elasticsearch.entitlement.bridge.EntitlementCheckerHandle} holder) and forwards the entitlement checks to the main library,
+ * that exists in the system classloader.
+ * {@link org.elasticsearch.entitlement.bridge.EntitlementChecker} holds all the entitlements check definitions, one for each instrumented
+ * method.
+ * <p>
+ * In order to work across multiple Java versions, this project uses multi-release jars via the {@code mrjar} plugin, which makes it is
+ * possible to specify classes for specific Java versions in specific {@code src} folders (e.g. {@code main23} for classes available to
+ * Java 23+).
+ * All the versioned Java classes are merged into the bridge jar. Therefore, we must prefix the class name
+ * with the version, e.g. {@code Java23EntitlementCheckerHandle} and {@code Java23EntitlementChecker}.
+ * </p>
+ */
+package org.elasticsearch.entitlement.bridge;

libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java

@@ -104,7 +104,42 @@ public static EntitlementChecker checker() {
         return manager;
     }
 
-    // Note: referenced by agent reflectively
+    /**
+     * Initializes the Entitlement system:
+     * <ol>
+     * <li>
+     * Finds the version-specific subclass of {@link EntitlementChecker} to use
+     * </li>
+     * <li>
+     * Builds the set of methods to instrument using {@link InstrumentationService#lookupMethods}
+     * </li>
+     * <li>
+     * Augment this set “dynamically” using {@link InstrumentationService#lookupImplementationMethod}
+     * </li>
+     * <li>
+     * Creates an {@link Instrumenter} via {@link InstrumentationService#newInstrumenter}, and adds a new {@link Transformer} (derived from
+     * {@link java.lang.instrument.ClassFileTransformer}) that uses it. Transformers are invoked when a class is about to load, after its
+     * bytes have been deserialized to memory but before the class is initialized.
+     * </li>
+     * <li>
+     * Re-transforms all already loaded classes: we force the {@link Instrumenter} to run on classes that might have been already loaded
+     * before entitlement initialization by calling the {@link java.lang.instrument.Instrumentation#retransformClasses} method on all
+     * classes that were already loaded.
+     * </li>
+     * </ol>
+     * <p>
+     * The third step is needed as the JDK exposes some API through interfaces that have different (internal) implementations
+     * depending on the JVM host platform. As we cannot instrument an interfaces, we find its concrete implementation.
+     * A prime example is {@link FileSystemProvider}, which has different implementations (e.g. {@code UnixFileSystemProvider} or
+     * {@code WindowsFileSystemProvider}). At runtime, we find the implementation class which is currently used by the JVM, and add
+     * its methods to the set of methods to instrument. See e.g. {@link EntitlementInitialization#fileSystemProviderChecks}.
+     * </p>
+     * <p>
+     * <strong>NOTE:</strong> this method is referenced by the agent reflectively
+     * </p>
+     *
+     * @param inst the JVM instrumentation class instance
+     */
     public static void initialize(Instrumentation inst) throws Exception {
         manager = initChecker();
 

libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/InstrumentationService.java

@@ -22,6 +22,32 @@ record InstrumentationInfo(MethodKey targetMethod, CheckMethod checkMethod) {}
 
     Instrumenter newInstrumenter(Class<?> clazz, Map<MethodKey, CheckMethod> methods);
 
+    /**
+     * This method uses the method names of the provided class to identify the JDK method to instrument; it examines all methods prefixed
+     * by {@code check$}, and parses the rest of the name to extract the JDK method:
+     * <ul>
+     * <li>
+     * Instance methods have the fully qualified class name (with . replaced by _), followed by $, followed by the method name. Example:
+     * {@link org.elasticsearch.entitlement.bridge.EntitlementChecker#check$java_lang_Runtime$halt}
+     * </li>
+     * <li>
+     * Static methods have the fully qualified class name (with . replaced by _), followed by $$, followed by the method name. Example:
+     * {@link org.elasticsearch.entitlement.bridge.EntitlementChecker#check$java_lang_System$$exit}
+     * </li>
+     * <li>
+     * Constructors have the fully qualified class name (with . replaced by _), followed by $ and nothing else. Example:
+     * {@link org.elasticsearch.entitlement.bridge.EntitlementChecker#check$java_lang_ClassLoader$}
+     * </li>
+     * </ul>
+     * <p>
+     * <strong>NOTE:</strong> look up of methods using this convention is the primary way we use to identify which methods to instrument,
+     * but other methods can be added to the map of methods to instrument. See
+     * {@link org.elasticsearch.entitlement.initialization.EntitlementInitialization#initialize} for details.
+     * </p>
+     *
+     * @param clazz the class to inspect to find methods to instrument
+     * @throws ClassNotFoundException if the class is not defined or cannot be inspected
+     */
     Map<MethodKey, CheckMethod> lookupMethods(Class<?> clazz) throws ClassNotFoundException;
 
     InstrumentationInfo lookupImplementationMethod(

libs/entitlement/src/main/java/org/elasticsearch/entitlement/instrumentation/Instrumenter.java

@@ -10,5 +10,30 @@
 package org.elasticsearch.entitlement.instrumentation;
 
 public interface Instrumenter {
+
+    /**
+     * Instruments the appropriate methods of a class by adding a prologue that checks for entitlements.
+     * The prologue:
+     * <ol>
+     * <li>
+     * gets the {@link org.elasticsearch.entitlement.bridge.EntitlementChecker} instance from the
+     * {@link org.elasticsearch.entitlement.bridge.EntitlementCheckerHandle} holder;
+     * </li>
+     * <li>
+     * identifies the caller class and pushes it onto the stack;
+     * </li>
+     * <li>
+     * forwards the instrumented function parameters;
+     * </li>
+     * <li>
+     * calls the {@link org.elasticsearch.entitlement.bridge.EntitlementChecker} method corresponding to the method it is injected into
+     * (e.g. {@code check$java_net_DatagramSocket$receive} for {@link java.net.DatagramSocket#receive}).
+     * </li>
+     * </ol>
+     * @param className the name of the class to instrument
+     * @param classfileBuffer its bytecode
+     * @param verify whether we should verify the bytecode before and after instrumentation
+     * @return the instrumented class bytes
+     */
     byte[] instrumentClass(String className, byte[] classfileBuffer, boolean verify);
 }