-
Notifications
You must be signed in to change notification settings - Fork 25.6k
[DOCS]: Move ES connectors Known issues page in 9.0+
#126600
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
5a690fc
add known issues page to es connectors
charlotte-hoblik 825435a
update known issues
charlotte-hoblik 126cb5b
Update docs/reference/search-connectors/es-connectors-known-issues.md
charlotte-hoblik a99610d
Update docs/reference/search-connectors/es-connectors-known-issues.md
charlotte-hoblik 6ea9c19
Merge branch 'main' into charlotte-search-connector-issues-281
charlotte-hoblik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
132 changes: 132 additions & 0 deletions
132
docs/reference/search-connectors/es-connectors-known-issues.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| --- | ||
| navigation_title: "Known issues" | ||
| --- | ||
|
|
||
| # Connector known issues [es-connectors-known-issues] | ||
|
|
||
| :::{important} | ||
| Enterprise Search is not available in {{stack}} 9.0+. | ||
| ::: | ||
|
|
||
| ## Connector service [es-connectors-known-issues-connector-service] | ||
|
|
||
| The connector service has the following known issues: | ||
|
|
||
| * **OOM errors when syncing large database tables** | ||
|
|
||
| Syncs after the initial sync can cause out-of-memory (OOM) errors when syncing large database tables. This occurs because database connectors load and store IDs in memory. For tables with millions of records, this can lead to memory exhaustion if the connector service has insufficient RAM. | ||
|
|
||
| To mitigate this issue, you can: | ||
|
|
||
| * **Increase RAM allocation**: | ||
|
|
||
| * **Self-managed**: Increase RAM allocation for the machine/container running the connector service. | ||
|
|
||
| ::::{dropdown} RAM sizing guidelines | ||
| The following table shows the estimated RAM usage for loading IDs into memory. | ||
|
|
||
| | | | | ||
| | --- | --- | | ||
| | **Number of IDs** | **Memory Usage in MB (2X buffer)** | | ||
| | 1,000,000 | ≈ 45.78 MB | | ||
| | 10,000,000 | ≈ 457.76 MB | | ||
| | 50,000,000 | ≈ 2288.82 MB (≈ 2.29 GB) | | ||
| | 100,000,000 | ≈ 4577.64 MB (≈ 4.58 GB) | | ||
|
|
||
| :::: | ||
|
|
||
| * **Optimize** [**sync rules**](/reference/search-connectors/es-sync-rules.md): | ||
|
|
||
| * Review and optimize sync rules to filter and reduce data retrieved from the source before syncing. | ||
|
|
||
| * **Use a self-managed connector** instead of a managed connector: | ||
|
|
||
| * Because self-managed connectors run on your infrastructure, they are not subject to the same RAM limitations of the Enterprise Search node. | ||
|
|
||
| * **Upgrades from deployments running on versions earlier than 8.9.0 can cause sync job failures** | ||
|
|
||
| Due to a bug, the `job_type` field mapping will be missing after upgrading from deployments running on versions earlier than 8.9.0. Sync jobs won’t be displayed in the Kibana UI (job history) and the connector service won’t be able to start new sync jobs. **This will only occur if you have previously scheduled sync jobs.** | ||
|
|
||
| To resolve this issue, you can manually add the missing field with the following command and trigger a sync job: | ||
|
|
||
| ```console | ||
| PUT .elastic-connectors-sync-jobs-v1/_mapping | ||
| { | ||
| "properties": { | ||
| "job_type": { | ||
| "type": "keyword" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| * **The connector service will fail to sync when the connector tries to fetch more more than 2,147,483,647 (*2^31-1*) documents from a data source** | ||
|
|
||
| A workaround is to manually partition the data to be synced using multiple search indices. | ||
|
|
||
| * **Custom scheduling might break when upgrading from version 8.6 or earlier.** | ||
|
|
||
| If you encounter the error `'custom_schedule_triggered': undefined method 'each' for nil:NilClass (NoMethodError)`, it means the custom scheduling feature migration failed. You can use the following manual workaround: | ||
|
|
||
| ```console | ||
| POST /.elastic-connectors/_update/connector-id | ||
| { | ||
| "doc": { | ||
| "custom_scheduling": {} | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| This error can appear on Connectors or Crawlers that aren’t the cause of the issue. If the error continues, try running the above command for every document in the `.elastic-connectors` index. | ||
|
|
||
| * **Connectors upgrading from 8.7 or earlier can be missing configuration fields** | ||
|
|
||
| A connector that was created prior to 8.8 can sometimes be missing configuration fields. This is a known issue for the MySQL connector but could also affect other connectors. | ||
|
|
||
| If the self-managed connector raises the error `Connector for <connector_id> has missing configuration fields: <field_a>, <field_b>...`, you can resolve the error by manually adding the missing configuration fields via the Dev Tools. Only the following two field properties are required, as the rest will be autopopulated by the self-managed connector: | ||
|
|
||
| * `type`: one of `str`, `int`, `bool`, or `list` | ||
| * `value`: any value, as long as it is of the correct `type` (`list` type values should be saved as comma-separated strings) | ||
|
|
||
| ```console | ||
| POST /.elastic-connectors/_update/connector_id | ||
| { | ||
| "doc" : { | ||
| "configuration": { | ||
| "field_a": { | ||
| "type": "str", | ||
| "value": "" | ||
| }, | ||
| "field_b": { | ||
| "type": "bool", | ||
| "value": false | ||
| }, | ||
| "field_c": { | ||
| "type": "int", | ||
| "value": 1 | ||
| }, | ||
| "field_d": { | ||
| "type": "list", | ||
| "value": "a,b" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| * **Python connectors that upgraded from 8.7.1 will report document volumes in gigabytes (GB) instead of megabytes (MB)** | ||
|
|
||
| As a result, true document volume will be under-reported by a factor of 1024. | ||
|
|
||
| * **The following Elastic managed connectors will not run correctly on Elastic Cloud in 8.9.0.** They are still available as self-managed connectors. | ||
|
|
||
| * Azure Blob Storage | ||
| * Confluence Cloud & Server | ||
| * Jira Cloud & Server | ||
| * Network drives | ||
|
|
||
|
|
||
charlotte-hoblik marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Individual connector known issues [es-connectors-known-issues-specific] | ||
|
|
||
| Individual connectors may have additional known issues. Refer to [each connector’s reference documentation](/reference/search-connectors/index.md) for connector-specific known issues. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.