From b8fc5beb004257c3dcfde3166daf6981d0e83b40 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Wed, 26 Feb 2025 12:56:04 +0100 Subject: [PATCH 1/3] Tweak search toc, add apis and tools page, move stuff --- deploy-manage/tools/snapshot-and-restore.md | 2 +- explore-analyze/query-filter/tools.md | 14 ++-- .../data-store}/near-real-time-search.md | 1 - manage-data/toc.yml | 1 + .../serverless/elasticsearch-dev-tools.md | 7 -- .../snapshot-restore.md | 2 +- raw-migrated-files/toc.yml | 1 - solutions/search/apis-and-tools.md | 67 +++++++++++++++++-- solutions/search/search-applications.md | 15 +++-- .../search-application-api.md | 4 +- .../search-application-client.md | 4 +- .../search-application-security.md | 4 +- solutions/toc.yml | 9 ++- 13 files changed, 91 insertions(+), 40 deletions(-) rename {solutions/search/search-approaches => manage-data/data-store}/near-real-time-search.md (99%) delete mode 100644 raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md diff --git a/deploy-manage/tools/snapshot-and-restore.md b/deploy-manage/tools/snapshot-and-restore.md index 7cf4a63354..962c755151 100644 --- a/deploy-manage/tools/snapshot-and-restore.md +++ b/deploy-manage/tools/snapshot-and-restore.md @@ -106,7 +106,7 @@ In Elasticsearch 8.0 and later versions, feature states are the only way to back ## How snapshots work -Snapshots are **automatically deduplicated** to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](/solutions/search/search-approaches/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot. +Snapshots are **automatically deduplicated** to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](/manage-data/data-store/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot. Each snapshot is **logically independent**. When you delete a snapshot, Elasticsearch only deletes the segments used exclusively by that snapshot. Elasticsearch doesn’t delete segments used by other snapshots in the repository. diff --git a/explore-analyze/query-filter/tools.md b/explore-analyze/query-filter/tools.md index 925c6af3b8..56232579be 100644 --- a/explore-analyze/query-filter/tools.md +++ b/explore-analyze/query-filter/tools.md @@ -8,16 +8,16 @@ mapped_pages: # Query tools [devtools-kibana] -Elasticsearch offers tools that you can use to query your data, manage those queries, and optimize them to be as efficient as possible. +Access these specialized tools in Kibana and the Serverless UI to develop, debug, and refine your search queries while monitoring their performance and efficiency. -| | | -| --- | --- | -| [Saved queries](tools/saved-queries.md) | Save your searches and queries to reuse them later. | +| Tool | Function | +|------|----------| +| [Saved queries](tools/saved-queries.md) | Save your searches and queries to reuse them later. | | [Console](tools/console.md) | Interact with the REST APIs of {{es}} and {{kib}}, including sending requests and viewing API documentation. | | [{{searchprofiler}}](tools/search-profiler.md) | Inspect and analyze your search queries. | -| [Grok Debugger   ](tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. | -| [Painless Lab](../scripting/painless-lab.md) | [beta] Test and debug Painless scripts in real-time. | -| [Playground](tools/playground.md) | Combine your Elasticsearch data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface. | +| [Grok Debugger](tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. | +| [Painless Lab](../scripting/painless-lab.md) | [beta] Test and debug Painless scripts in real-time. | +| [Playground](tools/playground.md) | Combine your Elasticsearch data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface. | diff --git a/solutions/search/search-approaches/near-real-time-search.md b/manage-data/data-store/near-real-time-search.md similarity index 99% rename from solutions/search/search-approaches/near-real-time-search.md rename to manage-data/data-store/near-real-time-search.md index c2387ee26a..72c0bb9bc8 100644 --- a/solutions/search/search-approaches/near-real-time-search.md +++ b/manage-data/data-store/near-real-time-search.md @@ -3,7 +3,6 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/near-real-time.html applies_to: stack: - serverless: --- # Near real-time search [near-real-time] diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 1c5e611a73..89b2b5eeb1 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -4,6 +4,7 @@ toc: - file: data-store.md children: - file: data-store/index-basics.md + - file: data-store/near-real-time-search.md - file: data-store/data-streams.md children: - file: data-store/data-streams/set-up-data-stream.md diff --git a/raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md b/raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md deleted file mode 100644 index f7fa379f57..0000000000 --- a/raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md +++ /dev/null @@ -1,7 +0,0 @@ -# Developer tools [elasticsearch-dev-tools] - -A number of developer tools are available in your project’s UI under the **Dev Tools** section. - -* [Console](https://www.elastic.co/guide/en/serverless/current/devtools-run-api-requests-in-the-console.html): Make API calls to your {{es}} instance using the Query DSL and view the responses. -* [Search Profiler](https://www.elastic.co/guide/en/serverless/current/devtools-profile-queries-and-aggregations.html): Inspect and analyze your search queries to identify performance bottlenecks. -* [Grok Debugger](https://www.elastic.co/guide/en/serverless/current/devtools-debug-grok-expressions.html): Build and debug grok patterns before you use them in your data processing pipelines. diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshot-restore.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshot-restore.md index 29e97dd1bb..72d1a6dedf 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshot-restore.md +++ b/raw-migrated-files/elasticsearch/elasticsearch-reference/snapshot-restore.md @@ -62,7 +62,7 @@ In {{es}} 8.0 and later versions, feature states are the only way to back up and ## How snapshots work [how-snapshots-work] -Snapshots are automatically deduplicated to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](../../../solutions/search/search-approaches/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot. +Snapshots are automatically deduplicated to save storage space and reduce network transfer costs. To back up an index, a snapshot makes a copy of the index’s [segments](../../../manage-data/data-store/near-real-time-search.md) and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository’s last snapshot. Each snapshot is also logically independent. When you delete a snapshot, {{es}} only deletes the segments used exclusively by that snapshot. {{es}} doesn’t delete segments used by other snapshots in the repository. diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index b98d82321f..eeb061d804 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -160,7 +160,6 @@ toc: - file: docs-content/serverless/connect-to-byo-llm.md - file: docs-content/serverless/cspm-required-permissions.md - file: docs-content/serverless/detections-logsdb-index-mode-impact.md - - file: docs-content/serverless/elasticsearch-dev-tools.md - file: docs-content/serverless/elasticsearch-differences.md - file: docs-content/serverless/elasticsearch-explore-your-data.md - file: docs-content/serverless/elasticsearch-http-apis.md diff --git a/solutions/search/apis-and-tools.md b/solutions/search/apis-and-tools.md index 09adb1bc5a..9ae02198ea 100644 --- a/solutions/search/apis-and-tools.md +++ b/solutions/search/apis-and-tools.md @@ -9,10 +9,67 @@ applies_to: # APIs and tools -% What needs to be done: Write from scratch +This page is handy list of the most important APIs and tools you need to build, test, and manage your search app built with {{es}}. -% Use migrated content from existing pages that map to this page: +## API endpoints -% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-dev-tools.md -% - [ ] https://www.elastic.co/guide/en/enterprise-search/current/search-ui.html -% Notes: mostly redirect purposes, this page wasn't migrated, but you can pull from the live URL if needed. \ No newline at end of file +### Query & search APIs + +| Endpoint | Function | +|----------|----------| +| [`_search`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-search) | Searches and aggregations written in [Query DSL](/explore-analyze/query-filter/languages/querydsl.md) and [retrievers](retrievers-overview.md) syntax | +| [`_query`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-esql)| Endpoint for [{{esql}}](/explore-analyze/query-filter/languages/esql.md) queries | +| [`_explain`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-explain) | Provides detailed explanation of how a specific document matches a query with scoring breakdown | +| [`_count`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-count) | Returns count of documents matching a query without retrieving results | +| [`_validate/query`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-validate-query) | Validates query syntax without executing the search | +| [`_analyze`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-analyze.html) | Performs analysis for [full-text search](./full-text.md) on a text string and returns the resulting tokens. | + +### Ingestion & mapping APIs + +| Endpoint | Function | +|----------|----------| +| [`_mapping`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-get-field-mapping) | Retrieves or updates field mappings with options for specific field inspection | +| [`_reindex`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-reindex) | Copies documents from one index to another, useful for mapping changes | +| [`_update_by_query`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-update-by-query) | Updates documents matching a query without reindexing | +| [`_bulk`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-bulk-1) | Performs multiple index/update/delete operations in a single request | +| [`_refresh`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-refresh-3) | Forces a refresh to make recent operations searchable | +| [`_ingest/pipeline`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ingest) | Creates and manages document processing pipelines before indexing | + +### Search optimization APIs + +| Endpoint | Function | +|----------|----------| +| [`_rank_eval`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-rank-eval)| Evaluates search quality against known relevant documents | +| [`_settings`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-indices-get-settings-1) | Configures settings including slow logs, refresh intervals, and replicas (only index-level settings available in serverless) | +| [`_scripts`](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-script) | Creates or updates stored scripts for reuse in queries and aggregations | + + +## UI tools [elasticsearch-dev-tools] + +### Dev tools + +Access these specialized tools in Kibana and the Serverless UI to develop, debug, and refine your search queries while monitoring their performance and efficiency. + +These tools are documented in the **Explore & Analyze** section: + +| Tool | Function | +|------|----------| +| [Saved queries](/explore-analyze/query-filter/tools/saved-queries.md) | Save your searches and queries to reuse them later. | +| [Console](/explore-analyze/query-filter/tools/console.md) | Interact with the REST APIs of {{es}} and {{kib}}, including sending requests and viewing API documentation. | +| [{{searchprofiler}}](/explore-analyze/query-filter/tools/search-profiler.md) | Inspect and analyze your search queries. | +| [Grok Debugger](/explore-analyze/query-filter/tools/grok-debugger.md) | Build and debug grok patterns before you use them in your data processing pipelines. | +| [Painless Lab](/explore-analyze/scripting/painless-lab.md) | Test and debug Painless scripts in real-time. | + +### Playground + +Use [Playground](rag/playground.md) to combine your {{es}} data with the power of large language models (LLMs) for retrieval augmented generation (RAG), using a chat interface. Playground is also very useful for testing and debugging your {{es}} queries, using the [retrievers](retrievers-overview.md) syntax with the `_search` endpoint. + +### Search UI + +[Elastic Search UI](./site-or-app/search-ui) is a library of JavaScript and React tools for building search experiences, optimized for use with {{es}}. + +:::{tip} +Check out the Elasticsearch Labs [blog](https://www.elastic.co/search-labs) to learn how to use Elastic to build advanced search experiences including generative AI, embedding models, reranking capabilities and more. + +The accompanying [GitHub repository](https://www.github.com/elastic/elasticsearch-labs) contains hands-on Python notebooks and sample apps to help you get started with these advanced search features. +::: \ No newline at end of file diff --git a/solutions/search/search-applications.md b/solutions/search/search-applications.md index f533a4b1f3..fa0ad18505 100644 --- a/solutions/search/search-applications.md +++ b/solutions/search/search-applications.md @@ -3,12 +3,10 @@ navigation_title: "Search Applications" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-overview.html applies_to: - stack: - serverless: + stack: beta + serverless: beta --- - - # Search applications [search-application-overview] @@ -32,14 +30,15 @@ The Search Applications feature was introduced in Elastic version **8.8.0**. ::::{note} Search Applications is a beta feature. Beta features are subject to change and are not covered by the support SLA of general release (GA) features. Elastic plans to promote this feature to GA in a future release. - :::: -This feature is available to all **Elastic Cloud** deployments. +This feature is available to all **{{ech}}** deployments. This feature is also available to **self-managed** deployments when Elastic subscription requirements are satisfied. View the requirements for this feature under the **Elastic Search** section of the [Elastic Stack subscriptions](https://www.elastic.co/subscriptions) page. +For Serverless users, this is an API-only feature. You can create and manage search applications using the [Search Application APIs](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/group/endpoint-search_application). + Your deployment must include the {{es}} and {{kib}} services. Managing search applications requires the `manage_search_application` cluster privilege, and also requires the `manage` [index privilege](../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md#privileges-list-indices) on all indices associated with the search application. @@ -57,6 +56,10 @@ Search Applications use [search templates](search-templates.md) to simplify the ### Option 1: Get started in the UI [search-application-overview-get-started-ui] +```{applies_to} +serverless: unavailable +``` + You can create build, and manage your search applications directly in the {{kib}} UI under **Search**. Make sure you have at least one {{es}} index to work with on your deployment. The indices underlying your search application are searched together, similar to how an [alias](../../manage-data/data-store/aliases.md) searches over multiple indices. To create a new search application in {{kib}}: diff --git a/solutions/search/search-applications/search-application-api.md b/solutions/search/search-applications/search-application-api.md index 7124fb0e73..0b484a1f9b 100644 --- a/solutions/search/search-applications/search-application-api.md +++ b/solutions/search/search-applications/search-application-api.md @@ -3,8 +3,8 @@ navigation_title: "Search API and templates" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-api.html applies_to: - stack: - serverless: + stack: beta + serverless: beta --- diff --git a/solutions/search/search-applications/search-application-client.md b/solutions/search/search-applications/search-application-client.md index 252e9db287..72bd71bc37 100644 --- a/solutions/search/search-applications/search-application-client.md +++ b/solutions/search/search-applications/search-application-client.md @@ -3,8 +3,8 @@ navigation_title: "Search Application client guide" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-client.html applies_to: - stack: - serverless: + stack: beta + serverless: beta --- diff --git a/solutions/search/search-applications/search-application-security.md b/solutions/search/search-applications/search-application-security.md index 7bd7ec3e64..f557e56a88 100644 --- a/solutions/search/search-applications/search-application-security.md +++ b/solutions/search/search-applications/search-application-security.md @@ -3,8 +3,8 @@ navigation_title: "Security" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-application-security.html applies_to: - stack: - serverless: + stack: beta + serverless: beta --- diff --git a/solutions/toc.yml b/solutions/toc.yml index f64fdde774..e76f5be4c3 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -644,6 +644,9 @@ toc: children: - file: search/ranking/learning-to-rank-model-training.md - file: search/ranking/learning-to-rank-search-usage.md + - file: search/cross-cluster-search.md + children: + - file: search/using-resolve-cluster-endpoint-before-cross-cluster-search.md - file: search/querying-for-search.md children: - file: search/the-search-api.md @@ -661,8 +664,4 @@ toc: - file: search/search-applications/search-application-api.md - file: search/search-applications/search-application-security.md - file: search/search-applications/search-application-client.md - - file: search/apis-and-tools.md - - file: search/search-approaches/near-real-time-search.md - - file: search/cross-cluster-search.md - children: - - file: search/using-resolve-cluster-endpoint-before-cross-cluster-search.md \ No newline at end of file + - file: search/apis-and-tools.md \ No newline at end of file From af47f2d807980d1b8ab8038bd1e0325dd9a8fd3c Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Wed, 26 Feb 2025 13:12:14 +0100 Subject: [PATCH 2/3] Add redirect, link fix --- redirects.yml | 1 + solutions/search/apis-and-tools.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/redirects.yml b/redirects.yml index f53ec18cb6..db8d5f5b4c 100644 --- a/redirects.yml +++ b/redirects.yml @@ -1,4 +1,5 @@ redirects: + 'solutions/search/search-approaches/near-real-time-search.md': '!manage-data/data-store/near-real-time-search.md' 'get-started/installing-elastic-stack.md': '!deploy-manage/deploy/self-managed.md' 'deploy-manage/deploy/elastic-cloud/ec-configure-deployment-settings.md': '!deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md' 'deploy-manage/users-roles/cluster-or-deployment-auth/user-authentication.md': diff --git a/solutions/search/apis-and-tools.md b/solutions/search/apis-and-tools.md index 9ae02198ea..68f1486309 100644 --- a/solutions/search/apis-and-tools.md +++ b/solutions/search/apis-and-tools.md @@ -66,7 +66,7 @@ Use [Playground](rag/playground.md) to combine your {{es}} data with the power o ### Search UI -[Elastic Search UI](./site-or-app/search-ui) is a library of JavaScript and React tools for building search experiences, optimized for use with {{es}}. +[Elastic Search UI](./site-or-app/search-ui.md) is a library of JavaScript and React tools for building search experiences, optimized for use with {{es}}. :::{tip} Check out the Elasticsearch Labs [blog](https://www.elastic.co/search-labs) to learn how to use Elastic to build advanced search experiences including generative AI, embedding models, reranking capabilities and more. From 6dd50dedf14e32e3133a3b0d5b8eeee64d3e5531 Mon Sep 17 00:00:00 2001 From: Liam Thompson Date: Wed, 26 Feb 2025 13:24:22 +0100 Subject: [PATCH 3/3] image paths --- manage-data/data-store/near-real-time-search.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/manage-data/data-store/near-real-time-search.md b/manage-data/data-store/near-real-time-search.md index 72c0bb9bc8..57b3fc8bd9 100644 --- a/manage-data/data-store/near-real-time-search.md +++ b/manage-data/data-store/near-real-time-search.md @@ -13,7 +13,7 @@ Lucene, the Java libraries on which {{es}} is based, introduced the concept of p Sitting between {{es}} and the disk is the filesystem cache. Documents in the in-memory indexing buffer ([Figure 1](#img-pre-refresh)) are written to a new segment ([Figure 2](#img-post-refresh)). The new segment is written to the filesystem cache first (which is cheap) and only later is it flushed to disk (which is expensive). However, after a file is in the cache, it can be opened and read just like any other file. -:::{image} ../../../images/elasticsearch-reference-lucene-in-memory-buffer.png +:::{image} /images/elasticsearch-reference-lucene-in-memory-buffer.png :alt: A Lucene index with new documents in the in-memory buffer :title: A Lucene index with new documents in the in-memory buffer :name: img-pre-refresh @@ -21,7 +21,7 @@ Sitting between {{es}} and the disk is the filesystem cache. Documents in the in Lucene allows new segments to be written and opened, making the documents they contain visible to search ​without performing a full commit. This is a much lighter process than a commit to disk, and can be done frequently without degrading performance. -:::{image} ../../../images/elasticsearch-reference-lucene-written-not-committed.png +:::{image} /images/elasticsearch-reference-lucene-written-not-committed.png :alt: The buffer contents are written to a segment, which is searchable, but is not yet committed :title: The buffer contents are written to a segment, which is searchable, but is not yet committed :name: img-post-refresh