diff --git a/explore-analyze/machine-learning/nlp/inference-processing.md b/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md similarity index 51% rename from explore-analyze/machine-learning/nlp/inference-processing.md rename to explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md index 6452b7cee7..f9a3a820a3 100644 --- a/explore-analyze/machine-learning/nlp/inference-processing.md +++ b/explore-analyze/machine-learning/machine-learning-in-kibana/inference-processing.md @@ -3,50 +3,42 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-pipeline-search-inference.html --- -# Inference processing [ingest-pipeline-search-inference] +# {{infer-cap}} processing [ingest-pipeline-search-inference] -When you create an index through the **Content** UI, a set of default ingest pipelines are also created, including a ML inference pipeline. The [ML inference pipeline](/solutions/search/search-pipelines.md#ingest-pipeline-search-details-specific-ml-reference) uses inference processors to analyze fields and enrich documents with the output. Inference processors use ML trained models, so you need to use a built-in model or [deploy a trained model in your cluster^](ml-nlp-deploy-models.md) to use this feature. +When you create an index through the **Content** UI, a set of default ingest pipelines are also created, including a {{ml}} {{infer}} pipeline. The [ML {{infer}} pipeline](/solutions/search/search-pipelines.md#ingest-pipeline-search-details-specific-ml-reference) uses {{infer}} processors to analyze fields and enrich documents with the output. Inference processors use ML trained models, so you need to use a built-in model or [deploy a trained model in your cluster](../nlp/ml-nlp-deploy-models.md) to use this feature. -This guide focuses on the ML inference pipeline, its use, and how to manage it. +This guide focuses on the ML {{infer}} pipeline, its use, and how to manage it. ::::{important} This feature is not available at all Elastic subscription levels. Refer to the Elastic subscriptions pages for [Elastic Cloud](https://www.elastic.co/subscriptions/cloud) and [self-managed](https://www.elastic.co/subscriptions) deployments. :::: - - ## NLP use cases [ingest-pipeline-search-inference-nlp-use-cases] -[Natural Language Processing (NLP)^](ml-nlp-overview.md) allows developers to create rich search experiences that go beyond the standards of lexical search. A few examples of ways to improve search experiences through the use of NLP models: - - -## ELSER text expansion [ingest-pipeline-search-inference-elser] +[Natural Language Processing (NLP)](../nlp/ml-nlp-overview.md) enables developers to create rich search experiences that go beyond the standards of lexical search. A few examples of ways to improve search experiences through the use of NLP models: -Using Elastic’s [ELSER machine learning model^](ml-nlp-elser.md) you can easily incorporate text expansion for your queries. This works by using ELSER to provide semantic enrichments to your documents upon ingestion, combined with the power of [Elastic Search Application templates](../../../solutions/search/applications.md) to provide automated text expansion at query time. +### ELSER text expansion [ingest-pipeline-search-inference-elser] +Using Elastic’s [ELSER machine learning model](../nlp/ml-nlp-elser.md) you can easily incorporate text expansion for your queries. This works by using ELSER to provide semantic enrichments to your documents upon ingestion, combined with the power of [Elastic Search Application templates](../../../solutions/search/applications.md) to provide automated text expansion at query time. -## Named entity recognition (NER) [ingest-pipeline-search-inference-ner] +### Named entity recognition (NER) [ingest-pipeline-search-inference-ner] -Most commonly used to detect entities such as People, Places, and Organization information from text, [NER^](ml-nlp-extract-info.md#ml-nlp-ner) can be used to extract key information from text and group results based on that information. A sports news media site could use NER to automatically extract names of professional athletes, stadiums, and sports teams in their articles and link to season stats or schedules. +Most commonly used to detect entities such as People, Places, and Organization information from text, [NER](../nlp/ml-nlp-extract-info.md#ml-nlp-ner) can be used to extract key information from text and group results based on that information. A sports news media site could use NER to automatically extract names of professional athletes, stadiums, and sports teams in their articles and link to season stats or schedules. +### Text classification [ingest-pipeline-search-inference-text-classification] -## Text classification [ingest-pipeline-search-inference-text-classification] +[Text classification](../nlp/ml-nlp-classify-text.md#ml-nlp-text-classification) is commonly used for sentiment analysis and can be used for similar tasks, such as labeling content as containing hate speech in public forums, or triaging and labeling support tickets so they reach the correct level of escalation automatically. -[Text classification^](ml-nlp-classify-text.md#ml-nlp-text-classification) is commonly used for sentiment analysis and can be used for similar tasks, such as labeling content as containing hate speech in public forums, or triaging and labeling support tickets so they reach the correct level of escalation automatically. +### Text embedding [ingest-pipeline-search-inference-text-embedding] - -## Text embedding [ingest-pipeline-search-inference-text-embedding] - -Analyzing a text field using a [Text embedding^](ml-nlp-search-compare.md#ml-nlp-text-embedding) model will generate a [dense vector](https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html) representation of the text. This array of numeric values encodes the semantic *meaning* of the text. Using the same model with a user’s search query will produce a vector that can then be used to search, ranking results based on vector similarity - semantic similarity - as opposed to traditional word or text similarity. +Analyzing a text field using a [Text embedding](../nlp/ml-nlp-search-compare.md#ml-nlp-text-embedding) model will generate a [dense vector](https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html) representation of the text. This array of numeric values encodes the semantic *meaning* of the text. Using the same model with a user’s search query will produce a vector that can then be used to search, ranking results based on vector similarity - semantic similarity - as opposed to traditional word or text similarity. A common use case is a user searching FAQs, or a support agent searching a knowledge base, where semantically similar content may be indexed with little similarity in phrasing. +## NLP in the Content UI [ingest-pipeline-search-inference-nlp-in-enterprise-search] -## NLP in Content UI [ingest-pipeline-search-inference-nlp-in-enterprise-search] - - -### Overview of ML inference pipeline [ingest-pipeline-search-inference-overview] +### Overview of the ML {{infer}} pipeline [ingest-pipeline-search-inference-overview] The diagram below shows how documents are processed during ingestion. @@ -55,32 +47,29 @@ The diagram below shows how documents are processed during ingestion. ::: * Documents are processed by the `my-index-0001` pipeline, which happens automatically when indexing through a an Elastic connector or crawler. -* The `_run_ml_inference` field is set to `true` to ensure the ML inference pipeline (`my-index-0001@ml-inference`) is executed. This field is removed during the ingestion process. -* The inference processor analyzes the `message` field on the document using the `my-positivity-model-id` trained model. The inference output is stored in the `ml.inference.positivity_prediction` field. +* The `_run_ml_inference` field is set to `true` to ensure the ML {{infer}} pipeline (`my-index-0001@ml-inference`) is executed. This field is removed during the ingestion process. +* The {{infer}} processor analyzes the `message` field on the document using the `my-positivity-model-id` trained model. The {{infer}} output is stored in the `ml.inference.positivity_prediction` field. * The resulting enriched document is then indexed into the `my-index-0001` index. * The `ml.inference.positivity_prediction` field can now be used at query time to search for documents above or below a certain threshold. - ## Find, deploy, and manage trained models [ingest-pipeline-search-inference-find-deploy-manage-trained-models] -This feature is intended to make it easier to use your ML trained models. First, you need to figure out which model works best for your data. Make sure to use a [compatible third party NLP model^](ml-nlp-model-ref.md). Since these are publicly available, it is not possible to fine-tune models before [deploying them^](ml-nlp-deploy-models.md). +This feature is intended to make it easier to use your ML trained models. First, you need to figure out which model works best for your data. Make sure to use a [compatible third party NLP model](../nlp/ml-nlp-model-ref.md). Since these are publicly available, it is not possible to fine-tune models before [deploying them](../nlp/ml-nlp-deploy-models.md). -Trained models must be available in the current [Kibana Space^](../../../deploy-manage/manage-spaces.md) and running in order to use them. By default, models should be available in all Kibana Spaces that have the **Analytics** > **Machine Learning** feature enabled. To manage your trained models, use the Kibana UI and navigate to **Stack Management → Machine Learning → Trained Models**. Spaces can be controlled in the **spaces** column. To stop or start a model, go to the **Machine Learning** tab in the **Analytics** menu of Kibana and click **Trained Models** in the **Model Management** section. +Trained models must be available in the current [Kibana Space](../../../deploy-manage/manage-spaces.md) and running in order to use them. By default, models should be available in all Kibana Spaces that have the **Analytics** > **Machine Learning** feature enabled. To manage your trained models, use the Kibana UI and navigate to **Stack Management → Machine Learning → Trained Models**. Spaces can be controlled in the **spaces** column. To stop or start a model, go to the **Machine Learning** tab in the **Analytics** menu of Kibana and click **Trained Models** in the **Model Management** section. ::::{note} -The `monitor_ml` [Elasticsearch cluster privilege](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) is required to manage ML models and ML inference pipelines which use those models. +The `monitor_ml` [Elasticsearch cluster privilege](../../../deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) is required to manage ML models and ML {{infer}} pipelines which use those models. :::: +### Add {{infer}} processors to your ML {{infer}} pipeline [ingest-pipeline-search-inference-add-inference-processors] - -### Add inference processors to your ML inference pipeline [ingest-pipeline-search-inference-add-inference-processors] - -To create the index-specific ML inference pipeline, go to **Search → Content → Indices → → Pipelines** in the Kibana UI. +To create the index-specific ML {{infer}} pipeline, go to **Search → Content → Indices → → Pipelines** in the Kibana UI. If you only see the `search-default-ingestion` pipeline, you will need to click **Copy and customize** to create index-specific pipelines. This will create the `{{index_name}}@ml-inference` pipeline. -Once your index-specific ML inference pipeline is ready, you can add inference processors that use your ML trained models. To add an inference processor to the ML inference pipeline, click the **Add Inference Pipeline** button in the **Machine Learning Inference Pipelines** card. +Once your index-specific ML {{infer}} pipeline is ready, you can add {{infer}} processors that use your ML trained models. To add an {{infer}} processor to the ML {{infer}} pipeline, click the **Add Inference Pipeline** button in the **Machine Learning Inference Pipelines** card. :::{image} ../../../images/elasticsearch-reference-document-enrichment-add-inference-pipeline.png :alt: Add Inference Pipeline @@ -98,39 +87,36 @@ Here, you’ll be able to: * The model must be deployed before you can select it. To begin deployment of a model, click the **Deploy** button. -3. Select one or more source fields as input for the inference processor. +3. Select one or more source fields as input for the {{infer}} processor. * If there are no source fields available, your index will need a [field mapping](../../../manage-data/data-store/mapping.md). -4. (Optional) Choose a name for your target field(s). This is where the output of the inference model will be stored. Changing the default name is only possible if you have a single source field selected. +4. (Optional) Choose a name for your target field(s). This is where the output of the {{infer}} model will be stored. Changing the default name is only possible if you have a single source field selected. 5. Add the source-target field mapping to the configuration by clicking the **Add** button. 6. Repeat steps 3-5 for each field mapping you want to add. 7. (Optional) Test the pipeline with a sample document. 8. (Optional) Review the pipeline definition before creating it with the **Create pipeline** button. +### Manage and delete {{infer}} processors from your ML {{infer}} pipeline [ingest-pipeline-search-inference-manage-inference-processors] -### Manage and delete inference processors from your ML inference pipeline [ingest-pipeline-search-inference-manage-inference-processors] +Inference processors added to your index-specific ML {{infer}} pipelines are normal Elasticsearch pipelines. Once created, each processor will have options to **View in Stack Management** and **Delete Pipeline**. Deleting an {{infer}} processor from within the **Content** UI deletes the pipeline and also removes its reference from your index-specific ML {{infer}} pipeline. -Inference processors added to your index-specific ML inference pipelines are normal Elasticsearch pipelines. Once created, each processor will have options to **View in Stack Management** and **Delete Pipeline**. Deleting an inference processor from within the **Content** UI deletes the pipeline and also removes its reference from your index-specific ML inference pipeline. +These pipelines can also be viewed, edited, and deleted in Kibana via **Stack Management → Ingest Pipelines**, just like all other Elasticsearch ingest pipelines. You may also use the [Ingest pipeline APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-apis.html). If you delete any of these pipelines outside of the **Content** UI in Kibana, make sure to edit the ML {{infer}} pipelines that reference them. -These pipelines can also be viewed, edited, and deleted in Kibana via **Stack Management → Ingest Pipelines**, just like all other Elasticsearch ingest pipelines. You may also use the [Ingest pipeline APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-apis.html). If you delete any of these pipelines outside of the **Content** UI in Kibana, make sure to edit the ML inference pipelines that reference them. +## Test your ML {{infer}} pipeline [ingest-pipeline-search-inference-test-inference-pipeline] +You can verify the expected structure of the {{infer}} output before indexing any documents while creating the {{ml}} {{infer}} pipeline under the **Test** tab. Provide a sample document, click **Simulate**, and look for the `ml.inference` object in the results. -## Test your ML inference pipeline [ingest-pipeline-search-inference-test-inference-pipeline] - -You can verify the expected structure of the inference output before indexing any documents while creating the {{ml}} inference pipeline under the **Test** tab. Provide a sample document, click **Simulate**, and look for the `ml.inference` object in the results. - -To ensure the ML inference pipeline will be run when ingesting documents, you must make sure the documents you are ingesting have a field named `_run_ml_inference` that is set to `true` and you must set the pipeline to `{{index_name}}`. For connector and crawler indices, this will happen automatically if you’ve configured the settings appropriately for the pipeline name `{{index_name}}`. To manage these settings: +To ensure the ML {{infer}} pipeline will be run when ingesting documents, you must make sure the documents you are ingesting have a field named `_run_ml_inference` that is set to `true` and you must set the pipeline to `{{index_name}}`. For connector and crawler indices, this will happen automatically if you’ve configured the settings appropriately for the pipeline name `{{index_name}}`. To manage these settings: 1. Go to **Search > Content > Indices > > Pipelines**. 2. Click on the **Settings** link in the **Ingest Pipelines** card for the `{{index_name}}` pipeline. -3. Ensure **ML inference pipelines** is selected. If it is not, select it and save the changes. - +3. Ensure **ML {{infer}} pipelines** is selected. If it is not, select it and save the changes. ## Learn More [ingest-pipeline-search-inference-learn-more] * See [Overview](/solutions/search/search-pipelines.md#ingest-pipeline-search-in-enterprise-search) for information on the various pipelines that are created. -* Learn about [ELSER](ml-nlp-elser.md), Elastic’s proprietary retrieval model for semantic search with sparse vectors. +* Learn about [ELSER](../nlp/ml-nlp-elser.md), Elastic’s proprietary retrieval model for semantic search with sparse vectors. * [NER HuggingFace Models](https://huggingface.co/models?library=pytorch&pipeline_tag=token-classification&sort=downloads) * [Text Classification HuggingFace Models](https://huggingface.co/models?library=pytorch&pipeline_tag=text-classification&sort=downloads) * [Text Embedding HuggingFace Models](https://huggingface.co/models?library=pytorch&pipeline_tag=sentence-similarity&sort=downloads) diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-examples.md b/explore-analyze/machine-learning/nlp/ml-nlp-examples.md index 680374370d..32f0b985bc 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-examples.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-examples.md @@ -7,9 +7,7 @@ mapped_pages: The following pages contain end-to-end examples of how to use the different {{nlp}} tasks in the {{stack}}. +* [An NLP end-to-end tutorial](nlp-end-to-end-tutorial.md) * [Multilingual semantic search - notebook example](https://colab.research.google.com/github/elastic/elasticsearch-labs/blob/main/notebooks/search/04-multilingual.ipynb) * [Named entity recognition](ml-nlp-ner-example.md) * [Text embedding and semantic search](ml-nlp-text-emb-vector-search-example.md) - - - diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md b/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md index fe8e0a28cf..dcebdf9e59 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-limitations.md @@ -7,13 +7,10 @@ mapped_pages: The following limitations and known problems apply to the 9.0.0-beta1 release of the Elastic {{nlp}} trained models feature. - -## Document size limitations when using `semantic_text` fields [ml-nlp-large-documents-limit-10k-10mb] +## Document size limitations when using `semantic_text` fields [ml-nlp-large-documents-limit-10k-10mb] When using semantic text to ingest documents, chunking takes place automatically. The number of chunks is limited by the [`index.mapping.nested_objects.limit`](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) cluster setting, which defaults to 10k. Documents that are too large will cause errors during ingestion. To avoid this issue, please split your documents into roughly 1MB parts before ingestion. - -## ELSER semantic search is limited to 512 tokens per field that inference is applied to [ml-nlp-elser-v1-limit-512] +## ELSER semantic search is limited to 512 tokens per field that inference is applied to [ml-nlp-elser-v1-limit-512] When you use ELSER for semantic search, only the first 512 extracted tokens from each field of the ingested documents that ELSER is applied to are taken into account for the search process. If your data set contains long documents, divide them into smaller segments before ingestion if you need the full text to be searchable. - diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md b/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md index aceda41ffc..8366adb606 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-ner-example.md @@ -4,14 +4,10 @@ mapped_pages: - https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-ner-example.html --- - - # Named entity recognition [ml-nlp-ner-example] - You can use these instructions to deploy a [named entity recognition (NER)](ml-nlp-extract-info.md#ml-nlp-ner) model in {{es}}, test the model, and add it to an {{infer}} ingest pipeline. The model that is used in the example is publicly available on [HuggingFace](https://huggingface.co/). - ## Requirements [ex-ner-requirements] To follow along the process on this page, you must have: @@ -20,7 +16,6 @@ To follow along the process on this page, you must have: * The [appropriate subscription](https://www.elastic.co/subscriptions) level or the free trial period activated. * [Docker](https://docs.docker.com/get-docker/) installed. - ## Deploy a NER model [ex-ner-deploy] You can use the [Eland client](https://www.elastic.co/guide/en/elasticsearch/client/eland/current) to install the {{nlp}} model. Use the prebuilt Docker image to run the Eland install model commands. Pull the latest image with: @@ -51,7 +46,6 @@ Since the `--start` option is used at the end of the Eland import command, {{es} Go to the **{{ml-app}} > Trained Models** page and synchronize your trained models. A warning message is displayed at the top of the page that says *"ML job and trained model synchronization required"*. Follow the link to *"Synchronize your jobs and trained models."* Then click **Synchronize**. You can also wait for the automatic synchronization that occurs in every hour, or use the [sync {{ml}} objects API](https://www.elastic.co/guide/en/kibana/current/ml-sync.html). - ## Test the NER model [ex-ner-test] Deployed models can be evaluated in {{kib}} under **{{ml-app}}** > **Trained Models** by selecting the **Test model** action for the respective model. @@ -112,10 +106,8 @@ The API returns a response similar to the following: :::: - Using the example text "Elastic is headquartered in Mountain View, California.", the model finds three entities: an organization "Elastic", and two locations "Mountain View" and "California". - ## Add the NER model to an {{infer}} ingest pipeline [ex-ner-ingest] You can perform bulk {{infer}} on documents as they are ingested by using an [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html) in your ingest pipeline. The novel *Les Misérables* by Victor Hugo is used as an example for {{infer}} in the following example. [Download](https://github.com/elastic/stack-docs/blob/8.5/docs/en/stack/ml/nlp/data/les-miserables-nd.json) the novel text split by paragraph as a JSON file, then upload it by using the [Data Visualizer](../../../manage-data/ingest.md#upload-data-kibana). Give the new index the name `les-miserables` when uploading the file. @@ -187,7 +179,6 @@ POST _reindex 1. The default batch size for reindexing is 1000. Reducing `size` to a smaller number makes the update of the reindexing process quicker which enables you to follow the progress closely and detect errors early. - Take a random paragraph from the source document as an example: ```js @@ -240,7 +231,6 @@ The request returns the document marked up with one identified person: (...) ``` - ## Visualize results [ex-ner-visual] You can create a tag cloud to visualize your data processed by the {{infer}} pipeline. A tag cloud is a visualization that scales words by the frequency at which they occur. It is a handy tool for viewing the entities found in the data. diff --git a/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md b/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md index a1330764a0..0641d38dc7 100644 --- a/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md +++ b/explore-analyze/machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md @@ -4,18 +4,14 @@ mapped_pages: - https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-text-emb-vector-search-example.html --- - - # Text embedding and semantic search [ml-nlp-text-emb-vector-search-example] - You can use these instructions to deploy a [text embedding](ml-nlp-search-compare.md#ml-nlp-text-embedding) model in {{es}}, test the model, and add it to an {{infer}} ingest pipeline. It enables you to generate vector representations of text and perform vector similarity search on the generated vectors. The model that is used in the example is publicly available on [HuggingFace](https://huggingface.co/). The example uses a public data set from the [MS MARCO Passage Ranking Task](https://microsoft.github.io/msmarco/#ranking). It consists of real questions from the Microsoft Bing search engine and human generated answers for them. The example works with a sample of this data set, uses a model to produce text embeddings, and then runs vector search on it. You can find [this example as a Jupyter notebook](https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/integrations/hugging-face/loading-model-from-hugging-face.ipynb) using the Python client in the `elasticsearch-labs` repo. - ## Requirements [ex-te-vs-requirements] To follow along the process on this page, you must have: @@ -24,7 +20,6 @@ To follow along the process on this page, you must have: * The [appropriate subscription](https://www.elastic.co/subscriptions) level or the free trial period activated. * [Docker](https://docs.docker.com/get-docker/) installed. - ## Deploy a text embedding model [ex-te-vs-deploy] You can use the [Eland client](https://www.elastic.co/guide/en/elasticsearch/client/eland/current) to install the {{nlp}} model. Use the prebuilt Docker image to run the Eland install model commands. Pull the latest image with: @@ -55,7 +50,6 @@ Since the `--start` option is used at the end of the Eland import command, {{es} Go to the **{{ml-app}} > Trained Models** page and synchronize your trained models. A warning message is displayed at the top of the page that says *"ML job and trained model synchronization required"*. Follow the link to *"Synchronize your jobs and trained models."* Then click **Synchronize**. You can also wait for the automatic synchronization that occurs in every hour, or use the [sync {{ml}} objects API](https://www.elastic.co/guide/en/kibana/current/ml-sync.html). - ## Test the text embedding model [ex-text-emb-test] Deployed models can be evaluated in {{kib}} under **{{ml-app}}** > **Trained Models** by selecting the **Test model** action for the respective model. @@ -98,10 +92,8 @@ The API returns a response similar to the following: :::: - The result is the predicted dense vector transformed from the example text. - ## Load data [ex-text-emb-data] In this step, you load the data that you later use in an ingest pipeline to get the embeddings. @@ -115,7 +107,6 @@ Upload the file by using the [Data Visualizer](../../../manage-data/ingest.md#up :class: screenshot ::: - ## Add the text embedding model to an {{infer}} ingest pipeline [ex-text-emb-ingest] Process the initial data with an [{{infer}} processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-processor.html). It adds an embedding for each passage. For this, create a text embedding ingest pipeline and then reindex the initial data with this pipeline. @@ -195,7 +186,6 @@ POST _reindex?wait_for_completion=false 1. The default batch size for reindexing is 1000. Reducing `size` to a smaller number makes the update of the reindexing process quicker which enables you to follow the progress closely and detect errors early. - The API call returns a task ID that can be used to monitor the progress: ```js @@ -211,7 +201,6 @@ You can also open the model stat UI to follow the progress. After the reindexing is finished, the documents in the new index contain the {{infer}} results – the vector embeddings. - ## Semantic search [ex-text-emb-vect-search] After the dataset has been enriched with vector embeddings, you can query the data using [semantic search](../../../solutions/search/vector/knn.md#knn-semantic-search). Pass a `query_vector_builder` to the k-nearest neighbor (kNN) vector search API, and provide the query text and the model you have used to create vector embeddings. This example searches for "How is the weather in Jamaica?": diff --git a/explore-analyze/machine-learning/nlp/nlp-example.md b/explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md similarity index 95% rename from explore-analyze/machine-learning/nlp/nlp-example.md rename to explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md index f9094ecdc2..bf503edeb2 100644 --- a/explore-analyze/machine-learning/nlp/nlp-example.md +++ b/explore-analyze/machine-learning/nlp/nlp-end-to-end-tutorial.md @@ -1,10 +1,10 @@ --- -navigation_title: "NLP tutorial" +navigation_title: "End-to-end tutorial" mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/nlp-example.html --- -# NLP example [nlp-example] +# An NLP end-to-end tutorial [nlp-example] This guide focuses on a concrete task: getting a machine learning trained model loaded into Elasticsearch and set up to enrich your documents. @@ -41,7 +41,7 @@ Follow the steps to **Create** a new deployment. Make sure to add capacity to th Enriching documents using machine learning was introduced in Enterprise Search **8.5.0**, so be sure to use version **8.5.0 or later**. -## Clone Eland [nlp-example-clone-eland] +## Clone Eland [nlp-example-clone-eland] Elastic’s [Eland](https://github.com/elastic/eland) tool makes it easy to upload trained models to your deployment via Docker. @@ -188,9 +188,9 @@ In this guide, we covered how to: ## Learn more [nlp-example-learn-more] -* [Compatible third party models^](ml-nlp-model-ref.md) -* [NLP Overview^](ml-nlp-overview.md) +* [Compatible third party models](ml-nlp-model-ref.md) +* [NLP Overview](ml-nlp-overview.md) * [Docker section of Eland readme](https://github.com/elastic/eland#docker) -* [Deploying a model ML guide^](ml-nlp-deploy-models.md) -* [Eland Authentication methods^](ml-nlp-import-model.md#ml-nlp-authentication) -* [Adding inference pipelines](inference-processing.md#ingest-pipeline-search-inference-add-inference-processors) +* [Deploying a model ML guide](ml-nlp-deploy-models.md) +* [Eland Authentication methods](ml-nlp-import-model.md#ml-nlp-authentication) +* [Adding inference pipelines](../machine-learning-in-kibana/inference-processing.md#ingest-pipeline-search-inference-add-inference-processors) diff --git a/explore-analyze/toc.yml b/explore-analyze/toc.yml index 2ad008c3cc..f0da5de044 100644 --- a/explore-analyze/toc.yml +++ b/explore-analyze/toc.yml @@ -221,17 +221,16 @@ toc: - file: machine-learning/nlp/ml-nlp-model-ref.md - file: machine-learning/nlp/ml-nlp-examples.md children: + - file: machine-learning/nlp/nlp-end-to-end-tutorial.md - file: machine-learning/nlp/ml-nlp-ner-example.md - file: machine-learning/nlp/ml-nlp-text-emb-vector-search-example.md - file: machine-learning/nlp/ml-nlp-limitations.md - - file: machine-learning/nlp/inference-processing.md - children: - - file: machine-learning/nlp/nlp-example.md - file: machine-learning/machine-learning-in-kibana.md children: - file: machine-learning/machine-learning-in-kibana/xpack-ml-anomalies.md - file: machine-learning/machine-learning-in-kibana/xpack-ml-dfanalytics.md - file: machine-learning/machine-learning-in-kibana/xpack-ml-aiops.md + - file: machine-learning/machine-learning-in-kibana/inference-processing.md - file: machine-learning/aiops-labs.md children: - file: machine-learning/aiops-labs/observability-aiops-detect-anomalies.md