Skip to content

Commit 93dd903

Browse files
committed
Merge remote-tracking branch 'upstream/main' into unmute_file_service_test
2 parents fb0fa0f + 03173af commit 93dd903

File tree

88 files changed

+4043
-1279
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

88 files changed

+4043
-1279
lines changed

.buildkite/pipelines/periodic-micro-benchmarks.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
steps:
22
- label: periodic-micro-benchmarks
33
command: |
4-
.ci/scripts/run-gradle.sh -p benchmarks/ run --args 'org.elasticsearch.benchmark._nightly -rf json -rff build/result.json'
4+
.ci/scripts/run-gradle.sh :benchmarks:run --args 'org.elasticsearch.benchmark._nightly -rf json -rff build/result.json'
55
timeout_in_minutes: 300
66
agents:
77
provider: gcp

.ci/scripts/run-gradle.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,4 +41,4 @@ else
4141
fi
4242

4343
set -e
44-
$GRADLEW -S --max-workers=$MAX_WORKERS $@
44+
$GRADLEW -S --max-workers=$MAX_WORKERS "$@"

docs/changelog/128405.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 128405
2+
summary: Modify the mechanism to pause indexing
3+
area: Distributed
4+
type: bug
5+
issues: []

docs/changelog/128650.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
pr: 128650
2+
summary: Update shardGenerations for all indices on snapshot finalization
3+
area: Snapshot/Restore
4+
type: enhancement
5+
issues:
6+
- 108907

docs/reference/enrich-processor/dissect-processor.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ and result in a document with the following fields:
4646

4747
A dissect pattern is defined by the parts of the string that will be discarded. In the previous example, the first part to be discarded is a single space. Dissect finds this space, then assigns the value of `clientip` everything up until that space. Next, dissect matches the `[` and then `]` and then assigns `@timestamp` to everything in-between `[` and `]`. Paying special attention to the parts of the string to discard will help build successful dissect patterns.
4848

49-
Successful matches require all keys in a pattern to have a value. If any of the `%{{keyname}}` defined in the pattern do not have a value, then an exception is thrown and may be handled by the [`on_failure`](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#handling-pipeline-failures) directive. An empty key `%{}` or a [named skip key](#dissect-modifier-named-skip-key) can be used to match values, but exclude the value from the final document. All matched values are represented as string data types. The [convert processor](/reference/enrich-processor/convert-processor.md) may be used to convert to expected data type.
49+
Successful matches require all keys in a pattern to have a value. If any of the `%{keyname}` defined in the pattern do not have a value, then an exception is thrown and may be handled by the [`on_failure`](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#handling-pipeline-failures) directive. An empty key `%{}` or a [named skip key](#dissect-modifier-named-skip-key) can be used to match values, but exclude the value from the final document. All matched values are represented as string data types. The [convert processor](/reference/enrich-processor/convert-processor.md) may be used to convert to expected data type.
5050

5151
Dissect also supports [key modifiers](#dissect-key-modifiers) that can change dissect’s default behavior. For example you can instruct dissect to ignore certain fields, append fields, skip over padding, etc. See [below](#dissect-key-modifiers) for more information.
5252

@@ -75,7 +75,7 @@ $$$dissect-options$$$
7575

7676
## Dissect key modifiers [dissect-key-modifiers]
7777

78-
Key modifiers can change the default behavior for dissection. Key modifiers may be found on the left or right of the `%{{keyname}}` always inside the `%{` and `}`. For example `%{+keyname ->}` has the append and right padding modifiers.
78+
Key modifiers can change the default behavior for dissection. Key modifiers may be found on the left or right of the `%{keyname}` always inside the `%{` and `}`. For example `%{+keyname ->}` has the append and right padding modifiers.
7979

8080
$$$dissect-key-modifiers-table$$$
8181

@@ -89,9 +89,9 @@ $$$dissect-key-modifiers-table$$$
8989

9090
### Right padding modifier (`->`) [dissect-modifier-skip-right-padding]
9191

92-
The algorithm that performs the dissection is very strict in that it requires all characters in the pattern to match the source string. For example, the pattern `%{{fookey}} %{{barkey}}` (1 space), will match the string "foo bar" (1 space), but will not match the string "foo bar" (2 spaces) since the pattern has only 1 space and the source string has 2 spaces.
92+
The algorithm that performs the dissection is very strict in that it requires all characters in the pattern to match the source string. For example, the pattern `%{fookey} %{barkey}` (1 space), will match the string "foo bar" (1 space), but will not match the string "foo bar" (2 spaces) since the pattern has only 1 space and the source string has 2 spaces.
9393

94-
The right padding modifier helps with this case. Adding the right padding modifier to the pattern `%{fookey->} %{{barkey}}`, It will now will match "foo bar" (1 space) and "foo bar" (2 spaces) and even "foo bar" (10 spaces).
94+
The right padding modifier helps with this case. Adding the right padding modifier to the pattern `%{fookey->} %{barkey}`, It will now will match "foo bar" (1 space) and "foo bar" (2 spaces) and even "foo bar" (10 spaces).
9595

9696
Use the right padding modifier to allow for repetition of the characters after a `%{keyname->}`.
9797

@@ -101,7 +101,7 @@ Right padding modifier example
101101

102102
| | |
103103
| --- | --- |
104-
| **Pattern** | `%{ts->} %{{level}}` |
104+
| **Pattern** | `%{ts->} %{level}` |
105105
| **Input** | 1998-08-10T17:15:42,466 WARN |
106106
| **Result** | * ts = 1998-08-10T17:15:42,466<br>* level = WARN<br> |
107107

@@ -111,7 +111,7 @@ Right padding modifier with empty key example
111111

112112
| | |
113113
| --- | --- |
114-
| **Pattern** | `[%{{ts}}]%{->}[%{{level}}]` |
114+
| **Pattern** | `[%{ts}]%{->}[%{level}]` |
115115
| **Input** | [1998-08-10T17:15:42,466] [WARN] |
116116
| **Result** | * ts = 1998-08-10T17:15:42,466<br>* level = WARN<br> |
117117

@@ -153,7 +153,7 @@ Named skip key modifier example
153153

154154
| | |
155155
| --- | --- |
156-
| **Pattern** | `%{{clientip}} %{?ident} %{?auth} [%{@timestamp}]` |
156+
| **Pattern** | `%{clientip} %{?ident} %{?auth} [%{@timestamp}]` |
157157
| **Input** | 1.2.3.4 - - [30/Apr/1998:22:00:52 +0000] |
158158
| **Result** | * clientip = 1.2.3.4<br>* @timestamp = 30/Apr/1998:22:00:52 +0000<br> |
159159

@@ -167,7 +167,7 @@ Reference key modifier example
167167

168168
| | |
169169
| --- | --- |
170-
| **Pattern** | `[%{{ts}}] [%{{level}}] %{*p1}:%{&p1} %{*p2}:%{&p2}` |
170+
| **Pattern** | `[%{ts}] [%{level}] %{*p1}:%{&p1} %{*p2}:%{&p2}` |
171171
| **Input** | [2018-08-10T17:15:42,466] [ERR] ip:1.2.3.4 error:REFUSED |
172172
| **Result** | * ts = 2018-08-10T17:15:42,466<br>* level = ERR<br>* ip = 1.2.3.4<br>* error = REFUSED<br> |
173173

libs/entitlement/src/main/java/module-info.java

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,13 @@
2020
requires static org.elasticsearch.entitlement.bridge; // At runtime, this will be in java.base
2121

2222
exports org.elasticsearch.entitlement.runtime.api;
23-
exports org.elasticsearch.entitlement.runtime.policy;
24-
exports org.elasticsearch.entitlement.runtime.policy.entitlements to org.elasticsearch.server;
2523
exports org.elasticsearch.entitlement.instrumentation;
2624
exports org.elasticsearch.entitlement.bootstrap to org.elasticsearch.server;
2725
exports org.elasticsearch.entitlement.initialization to java.base;
2826

27+
// TODO: Most of the things in the policy package should be internal implementation details that are not exported.
28+
exports org.elasticsearch.entitlement.runtime.policy;
29+
exports org.elasticsearch.entitlement.runtime.policy.entitlements to org.elasticsearch.server;
30+
2931
uses org.elasticsearch.entitlement.instrumentation.InstrumentationService;
3032
}

libs/entitlement/src/main/java/org/elasticsearch/entitlement/bootstrap/EntitlementBootstrap.java

Lines changed: 5 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,9 @@
1414
import com.sun.tools.attach.AttachNotSupportedException;
1515
import com.sun.tools.attach.VirtualMachine;
1616

17-
import org.elasticsearch.core.Nullable;
1817
import org.elasticsearch.core.PathUtils;
1918
import org.elasticsearch.core.SuppressForbidden;
2019
import org.elasticsearch.entitlement.initialization.EntitlementInitialization;
21-
import org.elasticsearch.entitlement.runtime.policy.PathLookup;
2220
import org.elasticsearch.entitlement.runtime.policy.PathLookupImpl;
2321
import org.elasticsearch.entitlement.runtime.policy.Policy;
2422
import org.elasticsearch.entitlement.runtime.policy.PolicyManager;
@@ -33,35 +31,11 @@
3331
import java.util.function.Function;
3432
import java.util.stream.Stream;
3533

36-
import static java.util.Objects.requireNonNull;
37-
3834
public class EntitlementBootstrap {
3935

40-
public record BootstrapArgs(
41-
@Nullable Policy serverPolicyPatch,
42-
Map<String, Policy> pluginPolicies,
43-
Function<Class<?>, PolicyManager.PolicyScope> scopeResolver,
44-
PathLookup pathLookup,
45-
Map<String, Path> sourcePaths,
46-
Set<Package> suppressFailureLogPackages
47-
) {
48-
public BootstrapArgs {
49-
requireNonNull(pluginPolicies);
50-
requireNonNull(scopeResolver);
51-
requireNonNull(pathLookup);
52-
requireNonNull(sourcePaths);
53-
requireNonNull(suppressFailureLogPackages);
54-
}
55-
}
56-
57-
private static BootstrapArgs bootstrapArgs;
58-
59-
public static BootstrapArgs bootstrapArgs() {
60-
return bootstrapArgs;
61-
}
62-
6336
/**
64-
* Activates entitlement checking. Once this method returns, calls to methods protected by Entitlements from classes without a valid
37+
* Main entry point that activates entitlement checking. Once this method returns,
38+
* calls to methods protected by entitlements from classes without a valid
6539
* policy will throw {@link org.elasticsearch.entitlement.runtime.api.NotEntitledException}.
6640
*
6741
* @param serverPolicyPatch a policy with additional entitlements to patch the embedded server layer policy
@@ -98,10 +72,10 @@ public static void bootstrap(
9872
Set<Package> suppressFailureLogPackages
9973
) {
10074
logger.debug("Loading entitlement agent");
101-
if (EntitlementBootstrap.bootstrapArgs != null) {
102-
throw new IllegalStateException("plugin data is already set");
75+
if (EntitlementInitialization.initializeArgs != null) {
76+
throw new IllegalStateException("initialization data is already set");
10377
}
104-
EntitlementBootstrap.bootstrapArgs = new BootstrapArgs(
78+
EntitlementInitialization.initializeArgs = new EntitlementInitialization.InitializeArgs(
10579
serverPolicyPatch,
10680
pluginPolicies,
10781
scopeResolver,

libs/entitlement/src/main/java/org/elasticsearch/entitlement/initialization/EntitlementInitialization.java

Lines changed: 51 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,9 @@
1010
package org.elasticsearch.entitlement.initialization;
1111

1212
import org.elasticsearch.core.Booleans;
13-
import org.elasticsearch.entitlement.bootstrap.EntitlementBootstrap;
13+
import org.elasticsearch.core.Nullable;
1414
import org.elasticsearch.entitlement.bridge.EntitlementChecker;
15-
import org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker;
15+
import org.elasticsearch.entitlement.runtime.policy.ElasticsearchEntitlementChecker;
1616
import org.elasticsearch.entitlement.runtime.policy.PathLookup;
1717
import org.elasticsearch.entitlement.runtime.policy.Policy;
1818
import org.elasticsearch.entitlement.runtime.policy.PolicyChecker;
@@ -22,8 +22,12 @@
2222
import java.lang.instrument.Instrumentation;
2323
import java.lang.reflect.Constructor;
2424
import java.lang.reflect.InvocationTargetException;
25+
import java.nio.file.Path;
2526
import java.util.Map;
2627
import java.util.Set;
28+
import java.util.function.Function;
29+
30+
import static java.util.Objects.requireNonNull;
2731

2832
/**
2933
* Called by the agent during {@code agentmain} to configure the entitlement system,
@@ -36,6 +40,7 @@ public class EntitlementInitialization {
3640

3741
private static final Module ENTITLEMENTS_MODULE = PolicyManager.class.getModule();
3842

43+
public static InitializeArgs initializeArgs;
3944
private static ElasticsearchEntitlementChecker checker;
4045

4146
// Note: referenced by bridge reflectively
@@ -63,32 +68,60 @@ public static EntitlementChecker checker() {
6368
* @param inst the JVM instrumentation class instance
6469
*/
6570
public static void initialize(Instrumentation inst) throws Exception {
66-
checker = initChecker(inst, createPolicyManager());
71+
// the checker _MUST_ be set before _any_ instrumentation is done
72+
checker = initChecker(createPolicyManager());
73+
initInstrumentation(inst);
74+
}
75+
76+
/**
77+
* Arguments to {@link #initialize}. Since that's called in a static context from the agent,
78+
* we have no way to pass arguments directly, so we stuff them in here.
79+
*
80+
* @param serverPolicyPatch
81+
* @param pluginPolicies
82+
* @param scopeResolver
83+
* @param pathLookup
84+
* @param sourcePaths
85+
* @param suppressFailureLogPackages
86+
*/
87+
public record InitializeArgs(
88+
@Nullable Policy serverPolicyPatch,
89+
Map<String, Policy> pluginPolicies,
90+
Function<Class<?>, PolicyManager.PolicyScope> scopeResolver,
91+
PathLookup pathLookup,
92+
Map<String, Path> sourcePaths,
93+
Set<Package> suppressFailureLogPackages
94+
) {
95+
public InitializeArgs {
96+
requireNonNull(pluginPolicies);
97+
requireNonNull(scopeResolver);
98+
requireNonNull(pathLookup);
99+
requireNonNull(sourcePaths);
100+
requireNonNull(suppressFailureLogPackages);
101+
}
67102
}
68103

69104
private static PolicyCheckerImpl createPolicyChecker(PolicyManager policyManager) {
70-
EntitlementBootstrap.BootstrapArgs bootstrapArgs = EntitlementBootstrap.bootstrapArgs();
71105
return new PolicyCheckerImpl(
72-
bootstrapArgs.suppressFailureLogPackages(),
106+
initializeArgs.suppressFailureLogPackages(),
73107
ENTITLEMENTS_MODULE,
74108
policyManager,
75-
bootstrapArgs.pathLookup()
109+
initializeArgs.pathLookup()
76110
);
77111
}
78112

79113
private static PolicyManager createPolicyManager() {
80-
EntitlementBootstrap.BootstrapArgs bootstrapArgs = EntitlementBootstrap.bootstrapArgs();
81-
Map<String, Policy> pluginPolicies = bootstrapArgs.pluginPolicies();
82-
PathLookup pathLookup = bootstrapArgs.pathLookup();
114+
Map<String, Policy> pluginPolicies = initializeArgs.pluginPolicies();
115+
PathLookup pathLookup = initializeArgs.pathLookup();
83116

84117
FilesEntitlementsValidation.validate(pluginPolicies, pathLookup);
85118

86119
return new PolicyManager(
87-
HardcodedEntitlements.serverPolicy(pathLookup.pidFile(), bootstrapArgs.serverPolicyPatch()),
120+
HardcodedEntitlements.serverPolicy(pathLookup.pidFile(), initializeArgs.serverPolicyPatch()),
88121
HardcodedEntitlements.agentEntitlements(),
89122
pluginPolicies,
90-
EntitlementBootstrap.bootstrapArgs().scopeResolver(),
91-
EntitlementBootstrap.bootstrapArgs().sourcePaths(),
123+
initializeArgs.scopeResolver(),
124+
initializeArgs.sourcePaths(),
92125
pathLookup
93126
);
94127
}
@@ -115,7 +148,7 @@ private static void ensureClassesSensitiveToVerificationAreInitialized() {
115148
}
116149
}
117150

118-
static ElasticsearchEntitlementChecker initChecker(Instrumentation inst, PolicyManager policyManager) throws Exception {
151+
static ElasticsearchEntitlementChecker initChecker(PolicyManager policyManager) {
119152
final PolicyChecker policyChecker = createPolicyChecker(policyManager);
120153
final Class<?> clazz = EntitlementCheckerUtils.getVersionSpecificCheckerClass(
121154
ElasticsearchEntitlementChecker.class,
@@ -136,17 +169,20 @@ static ElasticsearchEntitlementChecker initChecker(Instrumentation inst, PolicyM
136169
throw new AssertionError(e);
137170
}
138171

172+
return checker;
173+
}
174+
175+
static void initInstrumentation(Instrumentation instrumentation) throws Exception {
139176
var verifyBytecode = Booleans.parseBoolean(System.getProperty("es.entitlements.verify_bytecode", "false"));
140177
if (verifyBytecode) {
141178
ensureClassesSensitiveToVerificationAreInitialized();
142179
}
143180

144181
DynamicInstrumentation.initialize(
145-
inst,
182+
instrumentation,
146183
EntitlementCheckerUtils.getVersionSpecificCheckerClass(EntitlementChecker.class, Runtime.version().feature()),
147184
verifyBytecode
148185
);
149186

150-
return checker;
151187
}
152188
}

libs/entitlement/src/main/java/org/elasticsearch/entitlement/package-info.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@
189189
* <h2>Checks</h2>
190190
* <p>
191191
* The injected prologue calls a {@code check$} method on {@link org.elasticsearch.entitlement.bridge.EntitlementChecker}; its
192-
* implementation (normally on {@link org.elasticsearch.entitlement.runtime.api.ElasticsearchEntitlementChecker}, unless it is a
192+
* implementation (normally on {@link org.elasticsearch.entitlement.runtime.policy.ElasticsearchEntitlementChecker}, unless it is a
193193
* version-specific method) calls the appropriate methods on {@link org.elasticsearch.entitlement.runtime.policy.PolicyManager},
194194
* forwarding the caller class and a specific set of arguments. These methods all start with check, roughly matching an entitlement type
195195
* (e.g. {@link org.elasticsearch.entitlement.runtime.policy.PolicyChecker#checkInboundNetworkAccess},
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the "Elastic License
4+
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
5+
* Public License v 1"; you may not use this file except in compliance with, at
6+
* your election, the "Elastic License 2.0", the "GNU Affero General Public
7+
* License v3.0 only", or the "Server Side Public License, v 1".
8+
*/
9+
10+
/**
11+
* The public API for the Entitlements system.
12+
* All other packages are implementation details that should use selective exports.
13+
*/
14+
package org.elasticsearch.entitlement.runtime.api;

0 commit comments

Comments
 (0)