Skip to content

Commit eb91164

Browse files
iveraseiverase
authored
Merge branch 'main' into quantizeVectorWithIntervals
2 parents 058c145 + 8ffbf4a commit eb91164

File tree

5 files changed

+183
-37
lines changed

5 files changed

+183
-37
lines changed

‎docs/changelog/129725.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 129725
2+
summary: Throw a 400 when sorting for all types of range fields
3+
area: Search
4+
type: bug
5+
issues: []

‎docs/reference/search-connectors/index.md‎

Lines changed: 49 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ mapped_pages:
88
- https://www.elastic.co/guide/en/enterprise-search/current/connectors.html
99
---
1010

11-
# Search connectors
11+
# Content connectors
1212

1313
$$$es-connectors-native$$$
1414

1515

1616
:::{note}
17-
This page is about Search connectors that synchronize third-party data into {{es}}. If you’re looking for Kibana connectors to integrate with services like generative AI model providers, refer to [Kibana Connectors](docs-content://deploy-manage/manage-connectors.md).
17+
This page is about Content connectors that synchronize third-party data into {{es}}. If you’re looking for Kibana connectors to integrate with services like generative AI model providers, refer to [Kibana Connectors](docs-content://deploy-manage/manage-connectors.md).
1818
:::
1919

2020
A _connector_ is an Elastic integration that syncs data from an original data source to {{es}}. Use connectors to create searchable, read-only replicas of your data in {{es}}.
@@ -30,46 +30,60 @@ These connectors are written in Python and the source code is available in the [
3030
As of Elastic 9.0, managed connectors on Elastic Cloud Hosted are no longer available. All connectors must be [self-managed](/reference/search-connectors/self-managed-connectors.md).
3131
::::
3232

33-
34-
Connectors are available for the following third-party data sources:
35-
36-
- [Azure Blob Storage](/reference/search-connectors/es-connectors-azure-blob.md)
37-
- [Box](/reference/search-connectors/es-connectors-box.md)
38-
- [Confluence](/reference/search-connectors/es-connectors-confluence.md)
39-
- [Dropbox](/reference/search-connectors/es-connectors-dropbox.md)
40-
- [GitHub](/reference/search-connectors/es-connectors-github.md)
41-
- [Gmail](/reference/search-connectors/es-connectors-gmail.md)
42-
- [Google Cloud Storage](/reference/search-connectors/es-connectors-google-cloud.md)
43-
- [Google Drive](/reference/search-connectors/es-connectors-google-drive.md)
44-
- [GraphQL](/reference/search-connectors/es-connectors-graphql.md)
45-
- [Jira](/reference/search-connectors/es-connectors-jira.md)
46-
- [MicrosoftSQL](/reference/search-connectors/es-connectors-ms-sql.md)
47-
- [MongoDB](/reference/search-connectors/es-connectors-mongodb.md)
48-
- [MySQL](/reference/search-connectors/es-connectors-mysql.md)
49-
- [Network drive](/reference/search-connectors/es-connectors-network-drive.md)
50-
- [Notion](/reference/search-connectors/es-connectors-notion.md)
51-
- [OneDrive](/reference/search-connectors/es-connectors-onedrive.md)
52-
- [OpenText Documentum](/reference/search-connectors/es-connectors-opentext.md)
53-
- [Oracle](/reference/search-connectors/es-connectors-oracle.md)
54-
- [Outlook](/reference/search-connectors/es-connectors-outlook.md)
55-
- [PostgreSQL](/reference/search-connectors/es-connectors-postgresql.md)
56-
- [Redis](/reference/search-connectors/es-connectors-redis.md)
57-
- [S3](/reference/search-connectors/es-connectors-s3.md)
58-
- [Salesforce](/reference/search-connectors/es-connectors-salesforce.md)
59-
- [ServiceNow](/reference/search-connectors/es-connectors-servicenow.md)
60-
- [SharePoint Online](/reference/search-connectors/es-connectors-sharepoint-online.md)
61-
- [SharePoint Server](/reference/search-connectors/es-connectors-sharepoint.md)
62-
- [Slack](/reference/search-connectors/es-connectors-slack.md)
63-
- [Teams](/reference/search-connectors/es-connectors-teams.md)
64-
- [Zoom](/reference/search-connectors/es-connectors-zoom.md)
33+
This table provides an overview of our available connectors, their current support status, and the features they support.
34+
35+
The columns provide specific information about each connector:
36+
37+
- **Status**: Indicates whether the connector is in General Availability (GA), Technical Preview, Beta, or is an Example connector.
38+
- **Advanced sync rules**: Specifies the versions in which advanced sync rules are supported, if applicable.
39+
- **Local binary extraction service**: Specifies the versions in which the local binary extraction service is supported, if applicable.
40+
- **Incremental syncs**: Specifies the version in which incremental syncs are supported, if applicable.
41+
- **Document level security**: Specifies the version in which document level security is supported, if applicable.
42+
43+
44+
45+
| Connector | Status | [Advanced sync rules](./es-sync-rules.md#es-sync-rules-advanced) | [Local binary extraction service](./es-connectors-content-extraction.md#es-connectors-content-extraction-local) | [Incremental syncs](./content-syncs.md#es-connectors-sync-types-incremental) | [Document level security](./document-level-security.md) | Source code |
46+
| ------- | --------------- | -- | -- | -- | -- | -- |
47+
| [Azure Blob](/reference/search-connectors/es-connectors-azure-blob.md) | **GA** | - | 8.11+ | 8.13+ | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/azure_blob_storage.py) |
48+
| [Box](/reference/search-connectors/es-connectors-box.md) | **Preview** | - | - | 8.13+ | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/box.py) |
49+
| [Confluence Cloud](/reference/search-connectors/es-connectors-confluence.md) | **GA** | 8.9+ | 8.11+ | 8.13+ | 8.10 | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/confluence.py) |
50+
| [Confluence Data Center](/reference/search-connectors/es-connectors-confluence.md) | **Preview** | 8.13+ | 8.13+ | 8.13+ | 8.14+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/confluence.py) |
51+
| [Confluence Server](/reference/search-connectors/es-connectors-confluence.md)| **GA** | 8.9+ | 8.11+ | 8.13+ | 8.14+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/confluence.py) |
52+
| [Dropbox](/reference/search-connectors/es-connectors-dropbox.md)| **GA** | - | 8.11+ | 8.13+ | 8.12+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/dropbox.py) |
53+
| [GitHub](/reference/search-connectors/es-connectors-github.md)| **GA** | 8.10+ | 8.11+ | 8.13+ | 8.12+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/github.py) |
54+
| [Gmail](/reference/search-connectors/es-connectors-gmail.md)| **GA** | - | - | 8.13+ | 8.10+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/gmail.py) |
55+
| [Google Cloud Storage](/reference/search-connectors/es-connectors-google-cloud.md)| **GA** | - | 8.11+ | 8.13+ | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/google_cloud_storage.py) |
56+
| [Google Drive](/reference/search-connectors/es-connectors-google-drive.md)| **GA** | - | 8.11+ | 8.13+ | 8.10+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/google_drive.py) |
57+
| [GraphQL](/reference/search-connectors/es-connectors-graphql.md)| **Preview** | - | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/graphql.py) |
58+
| [Jira Cloud](/reference/search-connectors/es-connectors-jira.md)| **GA** | 8.9+ | 8.11+ | 8.13+ | 8.10+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/jira.py) |
59+
| [Jira Data Center](/reference/search-connectors/es-connectors-jira.md)| **Preview** | 8.13+ | 8.13+ | 8.13+ | 8.13+*| [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/jira.py) |
60+
| [Jira Server](/reference/search-connectors/es-connectors-jira.md)| **GA** | 8.9+ | 8.11+ | 8.13+ | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/jira.py) |
61+
| [Microsoft SQL Server](/reference/search-connectors/es-connectors-ms-sql.md)| **GA** | 8.11+ | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/mssql.py) |
62+
| [MongoDB](/reference/search-connectors/es-connectors-mongodb.md)| **GA** | 8.8 native/ 8.12 self-managed | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/mongo.py) |
63+
| [MySQL](/reference/search-connectors/es-connectors-mysql.md)| **GA** | 8.8+ | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/mysql.py) |
64+
| [Network drive](/reference/search-connectors/es-connectors-network-drive.md)| **GA** | 8.10+ | 8.14+ | 8.13+ | 8.11+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/network_drive.py) |
65+
| [Notion](/reference/search-connectors/es-connectors-notion.md)| **GA** | 8.14+ | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/notion.py) |
66+
| [OneDrive](/reference/search-connectors/es-connectors-onedrive.md)| **GA** | 8.11+ | 8.11+ | 8.13+ | 8.11+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/onedrive.py) |
67+
| [Opentext Documentum](/reference/search-connectors/es-connectors-opentext.md)| **Example** | n/a | n/a | n/a | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/opentext_documentum.py) |
68+
| [Oracle](/reference/search-connectors/es-connectors-oracle.md)| **GA** | - | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/oracle.py) |
69+
| [Outlook](/reference/search-connectors/es-connectors-outlook.md)| **GA** | - | 8.11+ | 8.13+ | 8.14+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/outlook.py) |
70+
| [PostgreSQL](/reference/search-connectors/es-connectors-postgresql.md)| **GA** | 8.11+ | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/postgresql.py) |
71+
| [Redis](/reference/search-connectors/es-connectors-redis.md)| **Preview** | - | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/redis.py) |
72+
| [Amazon S3](/reference/search-connectors/es-connectors-s3.md)| **GA** | 8.12+ | 8.11+ | - | - |[View code](https://github.com/elastic/connectors/tree/main/connectors/sources/s3.py) |
73+
| [Salesforce](/reference/search-connectors/es-connectors-salesforce.md)| **GA** | 8.12+ | 8.11+ | 8.13+ | 8.13+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/salesforce.py) |
74+
| [ServiceNow](/reference/search-connectors/es-connectors-servicenow.md)| **GA** | 8.10+ | 8.11+ | 8.13+ | 8.13+ | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/servicenow.py) |
75+
| [Sharepoint Online](/reference/search-connectors/es-connectors-sharepoint-online.md)| **GA** | 8.9+ | 8.9+ | 8.9+ | 8.9+ |[View code](https://github.com/elastic/connectors/tree/main/connectors/sources/sharepoint_online.py) |
76+
| [Sharepoint Server](/reference/search-connectors/es-connectors-sharepoint.md)| **Beta** | - | 8.11+ | 8.13+ | 8.15+ |[View code](https://github.com/elastic/connectors/tree/main/connectors/sources/sharepoint_server.py) |
77+
| [Slack](/reference/search-connectors/es-connectors-slack.md)| **Preview** | - | - | - | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/slack.py) |
78+
| [Teams](/reference/search-connectors/es-connectors-teams.md)| **Preview** | - | - | 8.13+ | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/teams.py) |
79+
| [Zoom](/reference/search-connectors/es-connectors-zoom.md)| **Preview** | - | 8.11+ | 8.13+ | - | [View code](https://github.com/elastic/connectors/tree/main/connectors/sources/zoom.py) |
6580

6681
:::{tip}
6782
Because prerequisites and configuration details vary by data source, you’ll need to refer to the individual connector references for specific details.
6883
:::
6984

7085
## Overview
7186

72-
7387
Because connectors are self-managed on your own infrastructure, they run outside of your Elastic deployment.
7488

7589
You can run them from source or in a Docker container.

‎docs/reference/search-connectors/toc.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
project: 'Search connectors reference'
1+
project: 'Content connectors reference'
22
toc:
33
- file: index.md
44
- file: connector-reference.md

‎server/src/internalClusterTest/java/org/elasticsearch/search/sort/RangeFieldSortIT.java‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the "Elastic License
4+
* 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
5+
* Public License v 1"; you may not use this file except in compliance with, at
6+
* your election, the "Elastic License 2.0", the "GNU Affero General Public
7+
* License v3.0 only", or the "Server Side Public License, v 1".
8+
*/
9+
10+
package org.elasticsearch.search.sort;
11+
12+
import org.elasticsearch.common.settings.Settings;
13+
import org.elasticsearch.index.mapper.RangeType;
14+
import org.elasticsearch.rest.RestStatus;
15+
import org.elasticsearch.test.ESSingleNodeTestCase;
16+
import org.elasticsearch.xcontent.XContentBuilder;
17+
import org.elasticsearch.xcontent.XContentFactory;
18+
19+
import static org.elasticsearch.index.query.QueryBuilders.matchAllQuery;
20+
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertFailures;
21+
import static org.hamcrest.Matchers.containsString;
22+
23+
public class RangeFieldSortIT extends ESSingleNodeTestCase {
24+
25+
private static final String FIELD_NAME = "range";
26+
27+
public void testSortingOnIntegerRangeFieldThrows400() throws Exception {
28+
String indexName = "int_range_index";
29+
createIndex(indexName, FIELD_NAME, RangeType.INTEGER.typeName());
30+
assertFailures(
31+
client().prepareSearch(indexName).setQuery(matchAllQuery()).addSort(SortBuilders.fieldSort(FIELD_NAME).order(SortOrder.ASC)),
32+
RestStatus.BAD_REQUEST,
33+
containsString("Sorting by range field [" + FIELD_NAME + "] is not supported")
34+
);
35+
}
36+
37+
public void testSortingOnLongRangeFieldThrows400() throws Exception {
38+
String indexName = "long_range_index";
39+
createIndex(indexName, FIELD_NAME, RangeType.LONG.typeName());
40+
assertFailures(
41+
client().prepareSearch(indexName).setQuery(matchAllQuery()).addSort(SortBuilders.fieldSort(FIELD_NAME).order(SortOrder.ASC)),
42+
RestStatus.BAD_REQUEST,
43+
containsString("Sorting by range field [" + FIELD_NAME + "] is not supported")
44+
);
45+
}
46+
47+
public void testSortingOnFloatRangeFieldThrows400() throws Exception {
48+
String indexName = "float_range_index";
49+
createIndex(indexName, FIELD_NAME, RangeType.FLOAT.typeName());
50+
assertFailures(
51+
client().prepareSearch(indexName).setQuery(matchAllQuery()).addSort(SortBuilders.fieldSort(FIELD_NAME).order(SortOrder.ASC)),
52+
RestStatus.BAD_REQUEST,
53+
containsString("Sorting by range field [" + FIELD_NAME + "] is not supported")
54+
);
55+
}
56+
57+
public void testSortingOnDoubleRangeFieldThrows400() throws Exception {
58+
String indexName = "double_range_index";
59+
createIndex(indexName, FIELD_NAME, RangeType.DOUBLE.typeName());
60+
assertFailures(
61+
client().prepareSearch(indexName).setQuery(matchAllQuery()).addSort(SortBuilders.fieldSort(FIELD_NAME).order(SortOrder.ASC)),
62+
RestStatus.BAD_REQUEST,
63+
containsString("Sorting by range field [" + FIELD_NAME + "] is not supported")
64+
);
65+
}
66+
67+
public void testSortingOnIpRangeFieldThrows400() throws Exception {
68+
String indexName = "ip_range_index";
69+
createIndex(indexName, FIELD_NAME, RangeType.IP.typeName());
70+
assertFailures(
71+
client().prepareSearch(indexName).setQuery(matchAllQuery()).addSort(SortBuilders.fieldSort(FIELD_NAME).order(SortOrder.ASC)),
72+
RestStatus.BAD_REQUEST,
73+
containsString("Sorting by range field [" + FIELD_NAME + "] is not supported")
74+
);
75+
}
76+
77+
public void testSortingOnDateRangeFieldThrows400() throws Exception {
78+
String indexName = "date_range_index";
79+
createIndex(indexName, FIELD_NAME, RangeType.DATE.typeName());
80+
assertFailures(
81+
client().prepareSearch(indexName).setQuery(matchAllQuery()).addSort(SortBuilders.fieldSort(FIELD_NAME).order(SortOrder.ASC)),
82+
RestStatus.BAD_REQUEST,
83+
containsString("Sorting by range field [" + FIELD_NAME + "] is not supported")
84+
);
85+
}
86+
87+
private void createIndex(String indexName, String rangeFieldName, String rangeFieldType) throws Exception {
88+
int numShards = randomIntBetween(1, 3);
89+
client().admin()
90+
.indices()
91+
.prepareCreate(indexName)
92+
.setSettings(Settings.builder().put("index.number_of_shards", numShards))
93+
.setMapping(createMapping(rangeFieldName, rangeFieldType))
94+
.get();
95+
}
96+
97+
private XContentBuilder createMapping(String fieldName, String fieldType) throws Exception {
98+
return XContentFactory.jsonBuilder()
99+
.startObject()
100+
.startObject("properties")
101+
.startObject(fieldName)
102+
.field("type", fieldType)
103+
.endObject()
104+
.endObject()
105+
.endObject();
106+
}
107+
}

‎server/src/main/java/org/elasticsearch/index/mapper/RangeFieldMapper.java‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
package org.elasticsearch.index.mapper;
1111

1212
import org.apache.lucene.search.Query;
13+
import org.apache.lucene.search.SortField;
1314
import org.apache.lucene.util.BytesRef;
1415
import org.elasticsearch.ElasticsearchException;
1516
import org.elasticsearch.common.Explicit;
@@ -20,13 +21,17 @@
2021
import org.elasticsearch.common.time.DateFormatter;
2122
import org.elasticsearch.common.time.DateMathParser;
2223
import org.elasticsearch.common.util.LocaleUtils;
24+
import org.elasticsearch.core.Nullable;
2325
import org.elasticsearch.core.Tuple;
2426
import org.elasticsearch.features.NodeFeature;
2527
import org.elasticsearch.index.fielddata.FieldDataContext;
2628
import org.elasticsearch.index.fielddata.IndexFieldData;
29+
import org.elasticsearch.index.fielddata.IndexFieldDataCache;
2730
import org.elasticsearch.index.fielddata.plain.BinaryIndexFieldData;
2831
import org.elasticsearch.index.query.SearchExecutionContext;
32+
import org.elasticsearch.indices.breaker.CircuitBreakerService;
2933
import org.elasticsearch.search.DocValueFormat;
34+
import org.elasticsearch.search.MultiValueMode;
3035
import org.elasticsearch.search.aggregations.support.CoreValuesSourceType;
3136
import org.elasticsearch.xcontent.XContentBuilder;
3237
import org.elasticsearch.xcontent.XContentParser;
@@ -227,7 +232,22 @@ public RangeType rangeType() {
227232
@Override
228233
public IndexFieldData.Builder fielddataBuilder(FieldDataContext fieldDataContext) {
229234
failIfNoDocValues();
230-
return new BinaryIndexFieldData.Builder(name(), CoreValuesSourceType.RANGE);
235+
return new BinaryIndexFieldData.Builder(name(), CoreValuesSourceType.RANGE) {
236+
@Override
237+
public BinaryIndexFieldData build(IndexFieldDataCache cache, CircuitBreakerService breakerService) {
238+
return new BinaryIndexFieldData(name(), CoreValuesSourceType.RANGE) {
239+
@Override
240+
public SortField sortField(
241+
@Nullable Object missingValue,
242+
MultiValueMode sortMode,
243+
XFieldComparatorSource.Nested nested,
244+
boolean reverse
245+
) {
246+
throw new IllegalArgumentException("Sorting by range field [" + name() + "] is not supported");
247+
}
248+
};
249+
}
250+
};
231251
}
232252

233253
@Override

0 commit comments

Comments
 (0)