adding reference architectures section #114653
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Documentation preview: |
elasticsearchmachine
added
needs:triage
georgewallace
added
>docs
|
Pinging @elastic/es-docs (Team:Docs) |
elasticsearchmachine
removed
the
needs:triage
…dex mode (elastic#114573) Here we check for the existence of a `host.name` field in index sort settings when the index mode is `logsdb` and decide to inject the field in the mapping depending on whether it exists or not. By default `host.name` is required for sorting in LogsDB. This reduces the chances for errors at mapping or template composition time as a result of injecting the `host.name` field only if strictly required. A user who wants to override index sort settings without including a `host.name` field would be able to do so without finding an additional `host.name` field in the mappings (injected automatically). If users override the sort settings and a `host.name` field is not included we don't need to inject such field since sorting does not require it anymore. As a result of this change we have the following: * the user does not provide any index sorting configuration: we are responsible for injecting the default sort fields and their mapping (for `logsdb`) * the user explicitly provides non-empty index sorting configuration: the user is also responsible for providing correct mappings and we do not modify index sorting or mappings Note also that all sort settings `index.sort.*` are `final` which means doing this check once, when mappings are merged at template composition time, is enough.
We will deprecate the `_source.mode` mapping level configuration in favor of the index-level `index.mapping.source.mode` setting. As a result, we go through the documentation and update it to reflect the introduction of the setting.
* (Doc+) link video for resolving shards too large 👋 howdy, team (cc: @anniegale9538 )! Playing forward elastic#111254, [this video](https://www.youtube.com/watch?v=sHyNYnwbYro) demonstrates an example resolving shards too large via reindex under [this section](https://www.elastic.co/guide/en/elasticsearch/reference/master/size-your-shards.html#shard-size-recommendation) as it's a top support ask. --------- Co-authored-by: Liam Thompson <[email protected]>
* (Doc+) Cross-link max shards 👋 It appears we have two docs of similar content about max open shards. This one contains the error users search (so is what we linked the error to in elastic#110993) but the other I believe is a placeholder doc for the health api code. Should maybe consolidate some day but in the mean time at least cross-link. --------- Co-authored-by: Liam Thompson <[email protected]>
@masseyke noticed this in his review of elastic#114847. I fixed it in the backport to `8.x` via elastic#114872, but this PR is needed to get the same fix into `main`.
These tests should be fixed and can be unmuted. The associated github issues have already been closed.
This links to our 6 newest [Support Troubleshooting](https://www.youtube.com/playlist?list=PL_mJOmq4zsHbQlfEMEh_30_LuV_hZp-3d) videos which are about resolving general ILM Health & the top five ILM rollover errors to the existing [Troubleshooting ILM errors](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-error-handling.html). It side quests to link the watermark error to [its troubleshooting doc](https://www.elastic.co/guide/en/elasticsearch/reference/master/fix-watermark-errors.html).
…ding requests (elastic#114870) * Make ESQL EnrichPolicyResolver try to do proper connection before sending requests * Make encureConnected be !skipUnavailable
With recent changes in Lucene 9.12 around not forking execution when not necessary (see apache/lucene#13472), we have removed the search worker thread pool in elastic#111099. The worker thread pool had unlimited queue, and we feared that we couuld have much more queueing on the search thread pool if we execute segment level searches on the same thread pool as the shard level searches, because every shard search would take up to a thread per slice when executing the query phase. We have then introduced an additional conditional to stop parallelizing when there is a queue. That is perhaps a bit extreme, as it's a decision made when creating the searcher, while a queue may no longer be there once the search is executing. This has caused some benchmarks regressions, given that having a queue may be a transient scenario, especially with short-lived segment searches being queued up. We may end up disabling inter-segment concurrency more aggressively than we would want, penalizing requests that do benefit from concurrency. At the same time, we do want to have some kind of protection against rejections of shard searches that would be caused by excessive slicing. When the queue is above a certain size, we can turn off the slicing and effectively disable inter-segment concurrency. With this commit we set that threshold to be the number of threads in the search pool.
Currently the incremental and non-incremental bulk variations will return different error codes when the json body provided is invalid. This commit ensures both version return status code 400. Additionally, this renames the incremental rest tests to bulk tests and ensures that all tests work with both bulk api versions. We set these tests to randomize which version of the api we test each run.
…y under snapshot (elastic#114784) * make named parameter for identifier and pattern snapshot
…#114271) * skip validating remote cluster index names in parser
This remove all recovery source specific SFM singletons. Whether recovery source is enabled can be checked via `DocumentParserContext`. This reduces the number of SFM instances by half.
…115459) The blob store may be triggered to create a local directory while in a reduced privilege context. This commit guards the creation of directories with doPrivileged.
…=reference/esql/esql-across-clusters/line_197} elastic#115575
…ic#114580) It was deprecated in elastic#104209 (8.13) and shouldn't be set or returned in 9.0 The Desired Nodes API is an internal API, and users shouldn't depend on its backward compatibility.
This PR adds detailed documentation for `logsdb` mode, covering several key aspects of its default behavior and configuration options. It includes: - default settings for index sorting (`index.sort.field`, `index.sort.order`, etc.). - usage of synthetic `_source` by default. - information about specialized codecs and how users can override them. - default behavior for `ignore_malformed` and `ignore_above` settings, including precedence rules. - explanation of how fields without `doc_values` are handled and what we do if they are missing.
…2569) KibanaThreadPoolIT checks the Kibana system user can write (using the system read/write threadpools) even when the normal read/write threadpools are blocked. This commit re-enables a key part of the test which was disabled. closes elastic#107625
* Propagate root subobjects setting to downsample indexes * exclude tests from rest compat * remove subobjects propagation
* changelog entry
The test issue was fixed by elastic#110807 closes elastic#110801
…#115138) * fix: correctly update search status for a nonexistent local index * Check for cluster existence before updation * Remove unnecessary `println` * Address review comment: add an explanatory code comment * Further clarify code comment
`MetadataStateFormat.FORMAT.loadLatestState` can actually return null when the state directory hasn't been initialized yet, so we have to keep the null check when loading retention leases during the initialization of the engine. See elastic#39359
…ic#114698) Since metadata storage was moved to Lucene in elastic#50907 (7.16.0), we shouldn't encounter any on-disk global metadata files, so we can remove support for loading them.
…ic#114665) * Fixing remote ENRICH by pushing the Enrich inside FragmentExec * Improve handling of more complex cases such as several enriches
…present (elastic#115586) Sometimes the test framework adds a global legacy template. When this happens, a test that is using another legacy template to create an index emits a warning since the index matches two legacy templates. This PR allows that warning.
…ative long values (elastic#115594)
This change introduces a new index mode, lookup, for indices intended for lookup operations in ES|QL. Lookup indices must have a single shard and be replicated to all data nodes by default. Aside from these requirements, they function as standard indices. Documentation will be added later when the lookup operator in ES|QL is implemented.
…uiteIT test {yaml=cluster.stats/30_ccs_stats/cross-cluster search stats search} elastic#115600
…gic (elastic#115487) The approach taken by `ExpressionList` becomes very expensive for large numbers of indices/datastreams. It implies that large lists of concrete names (as they are passed down from the transport layer via e.g. security) are copied at least twice during iteration. Removing the intermediary list and inlining the logic brings down the latency of searches targetting many shards/indices at once and allows for subsequent optimizations. The removed tests appear redundant as they tested an implementation detail of the IndexNameExpressionResolver which itself is well covered by its own tests.
Fixes a timeout in the Inference API where if connecting to an existing deployment and that deployment does not exist the listener was not called.
…ices.create/10_basic/Create lookup index} elastic#115605
We can randomly inject a global template that defaults to 2 shards instead of 1. This causes the lookup index YAML tests to fail. To avoid this, the change requires specifying the default_shards setting for these tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
76 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.