Skip to content

Commit 1145b14

Browse files
authored
Merge branch 'elastic:main' into EvaluatorImplementer-conditional-warnings
2 parents 6aea041 + 651314e commit 1145b14

File tree

30 files changed

+1140
-179
lines changed

30 files changed

+1140
-179
lines changed

docs/changelog/135337.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 135337
2+
summary: Do not pass `ProjectMetadata` to lazy index permissions builder
3+
area: Security
4+
type: enhancement
5+
issues: []

docs/reference/enrich-processor/geoip-processor.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -206,14 +206,18 @@ PUT my_ip_locations
206206

207207
If you can’t [automatically update](#geoip-automatic-updates) your IP geolocation databases from the Elastic endpoint, you have a few other options:
208208

209-
* [Use a proxy endpoint](#use-proxy-geoip-endpoint)
209+
* [Use a reverse proxy endpoint](#use-proxy-geoip-endpoint)
210210
* [Use a custom endpoint](#use-custom-geoip-endpoint)
211211
* [Manually update your IP geolocation databases](#manually-update-geoip-databases)
212212

213213
$$$use-proxy-geoip-endpoint$$$
214-
**Use a proxy endpoint**
214+
**Use a reverse proxy endpoint**
215215

216-
If you can’t connect directly to the Elastic GeoIP endpoint, consider setting up a secure proxy. You can then specify the proxy endpoint URL in the [`ingest.geoip.downloader.endpoint`](#ingest-geoip-downloader-endpoint) setting of each node’s `elasticsearch.yml` file.
216+
If you can’t connect directly to the Elastic GeoIP endpoint, consider setting up a secure reverse proxy. You can then specify the reverse proxy endpoint URL in the [`ingest.geoip.downloader.endpoint`](#ingest-geoip-downloader-endpoint) setting of each node’s `elasticsearch.yml` file.
217+
218+
:::{note}
219+
True HTTP proxy support for GeoIP database downloads is not currently available in {{es}}.
220+
:::
217221

218222
In a strict setup the following domains may need to be added to the allowed domains list:
219223

docs/reference/query-languages/esql/_snippets/functions/description/count_distinct.md

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/kibana/definition/functions/count_distinct.json

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/reference/query-languages/esql/kibana/docs/functions/count_distinct.md

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

muted-tests.yml

Lines changed: 3 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -549,12 +549,6 @@ tests:
549549
- class: org.elasticsearch.xpack.test.rest.XPackRestIT
550550
method: test {p0=transform/transforms_start_stop/Test stop transform with force and wait_for_checkpoint true}
551551
issue: https://github.com/elastic/elasticsearch/issues/135135
552-
- class: org.elasticsearch.lucene.RollingUpgradeSearchableSnapshotIndexCompatibilityIT
553-
method: testSearchableSnapshotUpgrade {p0=[9.2.0, 9.2.0, 9.2.0]}
554-
issue: https://github.com/elastic/elasticsearch/issues/135150
555-
- class: org.elasticsearch.lucene.RollingUpgradeSearchableSnapshotIndexCompatibilityIT
556-
method: testMountSearchableSnapshot {p0=[9.2.0, 9.2.0, 9.2.0]}
557-
issue: https://github.com/elastic/elasticsearch/issues/135151
558552
- class: org.elasticsearch.discovery.ClusterDisruptionIT
559553
method: testAckedIndexing
560554
issue: https://github.com/elastic/elasticsearch/issues/117024
@@ -591,27 +585,12 @@ tests:
591585
- class: org.elasticsearch.xpack.esql.expression.function.scalar.score.DecayTests
592586
method: "testEvaluateBlockWithoutNulls {TestCase=<date_nanos>, <date_nanos>, <time_duration>, <_source> #12}"
593587
issue: https://github.com/elastic/elasticsearch/issues/135394
594-
- class: org.elasticsearch.upgrades.DataStreamsUpgradeIT
595-
method: testDataStreamValidationDoesNotBreakUpgrade
596-
issue: https://github.com/elastic/elasticsearch/issues/135406
597-
- class: org.elasticsearch.upgrades.IndexingIT
598-
method: testIndexing
599-
issue: https://github.com/elastic/elasticsearch/issues/135407
600-
- class: org.elasticsearch.upgrades.QueryableBuiltInRolesUpgradeIT
601-
method: testBuiltInRolesSyncedOnClusterUpgrade
602-
issue: https://github.com/elastic/elasticsearch/issues/135194
603588
- class: org.elasticsearch.gradle.TestClustersPluginFuncTest
604589
method: override jdk usage via ES_JAVA_HOME for known jdk os incompatibilities
605590
issue: https://github.com/elastic/elasticsearch/issues/135413
606591
- class: org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT
607592
method: test {csv-spec:spatial_shapes.ConvertCartesianShapeFromStringParseError}
608593
issue: https://github.com/elastic/elasticsearch/issues/135455
609-
- class: org.elasticsearch.upgrades.SearchableSnapshotsRollingUpgradeIT
610-
method: testBlobStoreCacheWithPartialCopyInMixedVersions
611-
issue: https://github.com/elastic/elasticsearch/issues/135473
612-
- class: org.elasticsearch.upgrades.SearchableSnapshotsRollingUpgradeIT
613-
method: testBlobStoreCacheWithFullCopyInMixedVersions
614-
issue: https://github.com/elastic/elasticsearch/issues/135474
615594
- class: org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT
616595
method: test
617596
issue: https://github.com/elastic/elasticsearch/issues/134407
@@ -636,6 +615,9 @@ tests:
636615
- class: org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT
637616
method: testAggTooManyMvLongs
638617
issue: https://github.com/elastic/elasticsearch/issues/135585
618+
- class: org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT
619+
method: test
620+
issue: https://github.com/elastic/elasticsearch/issues/134407
639621

640622
# Examples:
641623
#

server/src/main/java/module-info.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -491,4 +491,5 @@
491491
exports org.elasticsearch.inference.telemetry;
492492
exports org.elasticsearch.index.codec.vectors.diskbbq to org.elasticsearch.test.knn;
493493
exports org.elasticsearch.index.codec.vectors.cluster to org.elasticsearch.test.knn;
494+
exports org.elasticsearch.search.crossproject;
494495
}

server/src/main/java/org/elasticsearch/ElasticsearchException.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
import org.elasticsearch.search.aggregations.AggregationExecutionException;
4646
import org.elasticsearch.search.aggregations.MultiBucketConsumerService;
4747
import org.elasticsearch.search.aggregations.UnsupportedAggregationOnDownsampledIndex;
48+
import org.elasticsearch.search.crossproject.NoMatchingProjectException;
4849
import org.elasticsearch.search.query.SearchTimeoutException;
4950
import org.elasticsearch.transport.TcpTransport;
5051
import org.elasticsearch.xcontent.ParseField;
@@ -79,6 +80,7 @@
7980
import static org.elasticsearch.cluster.metadata.IndexMetadata.INDEX_UUID_NA_VALUE;
8081
import static org.elasticsearch.common.xcontent.XContentParserUtils.ensureExpectedToken;
8182
import static org.elasticsearch.common.xcontent.XContentParserUtils.ensureFieldName;
83+
import static org.elasticsearch.search.crossproject.CrossProjectIndexExpressionsRewriter.NO_MATCHING_PROJECT_EXCEPTION_VERSION;
8284

8385
/**
8486
* A base class for all elasticsearch exceptions.
@@ -2022,6 +2024,12 @@ private enum ElasticsearchExceptionHandle {
20222024
184,
20232025
TransportVersions.REMOTE_EXCEPTION,
20242026
TransportVersions.REMOTE_EXCEPTION_8_19
2027+
),
2028+
NO_MATCHING_PROJECT_EXCEPTION(
2029+
NoMatchingProjectException.class,
2030+
NoMatchingProjectException::new,
2031+
185,
2032+
NO_MATCHING_PROJECT_EXCEPTION_VERSION
20252033
);
20262034

20272035
final Class<? extends ElasticsearchException> exceptionClass;

server/src/main/java/org/elasticsearch/cluster/metadata/IndexAbstractionResolver.java

Lines changed: 92 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -48,92 +48,114 @@ public ResolvedIndexExpressions resolveIndexAbstractions(
4848
boolean includeDataStreams
4949
) {
5050
final ResolvedIndexExpressions.Builder resolvedExpressionsBuilder = ResolvedIndexExpressions.builder();
51-
5251
boolean wildcardSeen = false;
5352
for (String index : indices) {
54-
String indexAbstraction;
55-
boolean minus = false;
56-
if (index.charAt(0) == '-' && wildcardSeen) {
57-
indexAbstraction = index.substring(1);
58-
minus = true;
59-
} else {
60-
indexAbstraction = index;
61-
}
53+
wildcardSeen = resolveIndexAbstraction(
54+
resolvedExpressionsBuilder,
55+
index,
56+
indicesOptions,
57+
projectMetadata,
58+
allAuthorizedAndAvailableBySelector,
59+
isAuthorized,
60+
includeDataStreams,
61+
wildcardSeen
62+
);
63+
}
64+
return resolvedExpressionsBuilder.build();
65+
}
6266

63-
// Always check to see if there's a selector on the index expression
64-
final Tuple<String, String> expressionAndSelector = IndexNameExpressionResolver.splitSelectorExpression(indexAbstraction);
65-
final String selectorString = expressionAndSelector.v2();
66-
if (indicesOptions.allowSelectors() == false && selectorString != null) {
67-
throw new UnsupportedSelectorException(indexAbstraction);
68-
}
69-
indexAbstraction = expressionAndSelector.v1();
70-
IndexComponentSelector selector = IndexComponentSelector.getByKeyOrThrow(selectorString);
67+
private boolean resolveIndexAbstraction(
68+
ResolvedIndexExpressions.Builder resolvedExpressionsBuilder,
69+
String index,
70+
IndicesOptions indicesOptions,
71+
ProjectMetadata projectMetadata,
72+
Function<IndexComponentSelector, Set<String>> allAuthorizedAndAvailableBySelector,
73+
BiPredicate<String, IndexComponentSelector> isAuthorized,
74+
boolean includeDataStreams,
75+
boolean wildcardSeen
76+
) {
77+
String indexAbstraction;
78+
boolean minus = false;
79+
if (index.charAt(0) == '-' && wildcardSeen) {
80+
indexAbstraction = index.substring(1);
81+
minus = true;
82+
} else {
83+
indexAbstraction = index;
84+
}
85+
86+
// Always check to see if there's a selector on the index expression
87+
final Tuple<String, String> expressionAndSelector = IndexNameExpressionResolver.splitSelectorExpression(indexAbstraction);
88+
final String selectorString = expressionAndSelector.v2();
89+
if (indicesOptions.allowSelectors() == false && selectorString != null) {
90+
throw new UnsupportedSelectorException(indexAbstraction);
91+
}
92+
indexAbstraction = expressionAndSelector.v1();
93+
IndexComponentSelector selector = IndexComponentSelector.getByKeyOrThrow(selectorString);
7194

72-
// we always need to check for date math expressions
73-
indexAbstraction = IndexNameExpressionResolver.resolveDateMathExpression(indexAbstraction);
95+
// we always need to check for date math expressions
96+
indexAbstraction = IndexNameExpressionResolver.resolveDateMathExpression(indexAbstraction);
7497

75-
if (indicesOptions.expandWildcardExpressions() && Regex.isSimpleMatchPattern(indexAbstraction)) {
76-
wildcardSeen = true;
77-
final HashSet<String> resolvedIndices = new HashSet<>();
78-
for (String authorizedIndex : allAuthorizedAndAvailableBySelector.apply(selector)) {
79-
if (Regex.simpleMatch(indexAbstraction, authorizedIndex)
98+
if (indicesOptions.expandWildcardExpressions() && Regex.isSimpleMatchPattern(indexAbstraction)) {
99+
wildcardSeen = true;
100+
final HashSet<String> resolvedIndices = new HashSet<>();
101+
for (String authorizedIndex : allAuthorizedAndAvailableBySelector.apply(selector)) {
102+
if (Regex.simpleMatch(indexAbstraction, authorizedIndex)
103+
&& isIndexVisible(
104+
indexAbstraction,
105+
selectorString,
106+
authorizedIndex,
107+
indicesOptions,
108+
projectMetadata,
109+
indexNameExpressionResolver,
110+
includeDataStreams
111+
)) {
112+
resolveSelectorsAndCollect(authorizedIndex, selectorString, indicesOptions, resolvedIndices, projectMetadata);
113+
}
114+
}
115+
if (resolvedIndices.isEmpty()) {
116+
// es core honours allow_no_indices for each wildcard expression, we do the same here by throwing index not found.
117+
if (indicesOptions.allowNoIndices() == false) {
118+
throw new IndexNotFoundException(indexAbstraction);
119+
}
120+
resolvedExpressionsBuilder.addLocalExpressions(index, new HashSet<>(), SUCCESS);
121+
} else {
122+
if (minus) {
123+
resolvedExpressionsBuilder.excludeFromLocalExpressions(resolvedIndices);
124+
} else {
125+
resolvedExpressionsBuilder.addLocalExpressions(index, resolvedIndices, SUCCESS);
126+
}
127+
}
128+
} else {
129+
final HashSet<String> resolvedIndices = new HashSet<>();
130+
resolveSelectorsAndCollect(indexAbstraction, selectorString, indicesOptions, resolvedIndices, projectMetadata);
131+
if (minus) {
132+
resolvedExpressionsBuilder.excludeFromLocalExpressions(resolvedIndices);
133+
} else {
134+
final boolean authorized = isAuthorized.test(indexAbstraction, selector);
135+
if (authorized) {
136+
final boolean visible = indexExists(projectMetadata, indexAbstraction)
80137
&& isIndexVisible(
81138
indexAbstraction,
82139
selectorString,
83-
authorizedIndex,
140+
indexAbstraction,
84141
indicesOptions,
85142
projectMetadata,
86143
indexNameExpressionResolver,
87144
includeDataStreams
88-
)) {
89-
resolveSelectorsAndCollect(authorizedIndex, selectorString, indicesOptions, resolvedIndices, projectMetadata);
90-
}
91-
}
92-
if (resolvedIndices.isEmpty()) {
93-
// es core honours allow_no_indices for each wildcard expression, we do the same here by throwing index not found.
94-
if (indicesOptions.allowNoIndices() == false) {
95-
throw new IndexNotFoundException(indexAbstraction);
96-
}
97-
resolvedExpressionsBuilder.addLocalExpressions(index, new HashSet<>(), SUCCESS);
98-
} else {
99-
if (minus) {
100-
resolvedExpressionsBuilder.excludeFromLocalExpressions(resolvedIndices);
101-
} else {
102-
resolvedExpressionsBuilder.addLocalExpressions(index, resolvedIndices, SUCCESS);
103-
}
104-
}
105-
} else {
106-
final HashSet<String> resolvedIndices = new HashSet<>();
107-
resolveSelectorsAndCollect(indexAbstraction, selectorString, indicesOptions, resolvedIndices, projectMetadata);
108-
if (minus) {
109-
resolvedExpressionsBuilder.excludeFromLocalExpressions(resolvedIndices);
145+
);
146+
final LocalIndexResolutionResult result = visible ? SUCCESS : CONCRETE_RESOURCE_NOT_VISIBLE;
147+
resolvedExpressionsBuilder.addLocalExpressions(index, resolvedIndices, result);
148+
} else if (indicesOptions.ignoreUnavailable()) {
149+
// ignoreUnavailable implies that the request should not fail if an index is not authorized
150+
// so we map this expression to an empty list,
151+
resolvedExpressionsBuilder.addLocalExpressions(index, new HashSet<>(), CONCRETE_RESOURCE_UNAUTHORIZED);
110152
} else {
111-
final boolean authorized = isAuthorized.test(indexAbstraction, selector);
112-
if (authorized) {
113-
final boolean visible = indexExists(projectMetadata, indexAbstraction)
114-
&& isIndexVisible(
115-
indexAbstraction,
116-
selectorString,
117-
indexAbstraction,
118-
indicesOptions,
119-
projectMetadata,
120-
indexNameExpressionResolver,
121-
includeDataStreams
122-
);
123-
final LocalIndexResolutionResult result = visible ? SUCCESS : CONCRETE_RESOURCE_NOT_VISIBLE;
124-
resolvedExpressionsBuilder.addLocalExpressions(index, resolvedIndices, result);
125-
} else if (indicesOptions.ignoreUnavailable()) {
126-
// ignoreUnavailable implies that the request should not fail if an index is not authorized
127-
// so we map this expression to an empty list,
128-
resolvedExpressionsBuilder.addLocalExpressions(index, new HashSet<>(), CONCRETE_RESOURCE_UNAUTHORIZED);
129-
} else {
130-
// store the calculated expansion as unauthorized, it will be rejected later
131-
resolvedExpressionsBuilder.addLocalExpressions(index, resolvedIndices, CONCRETE_RESOURCE_UNAUTHORIZED);
132-
}
153+
// store the calculated expansion as unauthorized, it will be rejected later
154+
resolvedExpressionsBuilder.addLocalExpressions(index, resolvedIndices, CONCRETE_RESOURCE_UNAUTHORIZED);
133155
}
134156
}
135157
}
136-
return resolvedExpressionsBuilder.build();
158+
return wildcardSeen;
137159
}
138160

139161
private static void resolveSelectorsAndCollect(

0 commit comments

Comments
 (0)