diff --git a/agent_knowledge/elasticsearch-install-and-setup/ICD_Elasticsearch_install_and_setup.md b/agent_knowledge/elasticsearch-install-and-setup/ICD_Elasticsearch_install_and_setup.md new file mode 100644 index 0000000..bd610d3 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/ICD_Elasticsearch_install_and_setup.md @@ -0,0 +1,235 @@ +# How to set up Elasticsearch from IBM Cloud and integrate it with Agent Knowledge in watsonx Orchestrate +This is a documentation about how to set up Elasticsearch from IBM Cloud and create Agent Knowledge in watsonx Orchestrate using Elasticsearch index. + +## Table of contents: +* [Step 1: Provision an Elasticsearch instance on IBM Cloud](#step-1-provision-an-elasticsearch-instance-on-ibm-cloud) +* [Step 2: Set up Kibana to connect to Elasticsearch](#step-2-set-up-kibana-to-connect-to-elasticsearch) +* [Step 3: Create an Elasticsearch index (keyword-search)](#step-3-create-an-elasticsearch-index-keyword-search) +* [Step 4: Enable semantic search with ELSER](#step-5-enable-semantic-search-with-elser) + + +## Step 1: Provision an Elasticsearch instance on IBM Cloud +* Create an [IBM Cloud account](https://cloud.ibm.com/registration) if you don't have one. +* Provision a Databases for Elasticsearch instance from the [IBM Cloud catalog](https://cloud.ibm.com/catalog/databases-for-elasticsearch). + **A platinum plan with at least 4GB RAM is required in order to use the advanced ML features, + such as [Elastic Learned Sparse EncodeR (ELSER)](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html)** +* Create a service credentials from the left-side menu and find the `hostname`, `port`, `username` and `password`. + The credentials will be used to connect to Kibana and watsonx Orchestrate at next steps. You can use admin userid and password as well. + Please refer to [this doc](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-user-management&interface=ui#user-management-elasticsearch-ibm-superuser) to learn more about different user roles. + + +## Step 2: Set up Kibana to connect to Elasticsearch +* Install Docker so that you can pull the Kibana container image later. You can follow the detailed [docker install guide](./how_to_install_docker.md) +* Create a kibana config folder, for example + `mkdir -p ~/.kibana/config` +* Download the certificate from the Elasticsearch instance overview page, and move the downloaded file to the kibana config folder +* Under the kibana config folder, create a YAML file called `kibana.yml`. Inside the file, you need the following Kibana configuration settings: + ```YAML + elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/" + elasticsearch.username: "" + elasticsearch.password: "" + elasticsearch.hosts: ["https://"] + server.name: "kibana" + server.host: "0.0.0.0" + ``` + Notes: + - Find the `hostname`, `port`, `username`, `password` from the service credentials created at Step 1 + - `elasticsearch.ssl.certificateAuthorities` is the location where the kibana deployment will look for the certificate in the docker container. + `/usr/share/kibana/config/` is the default Kibana's config directory in the container + +* Verify the Elasticsearch instance endpoint and find its version + * Run + ```bash + curl -u : --cacert https:// + ``` + * Find the version number from the output + +* Download and start the Kibana container + ```bash + docker run -it --name kibana --rm \ + -v :/usr/share/kibana/config \ + -p 5601:5601 docker.elastic.co/kibana/kibana: + ``` + Once Kibana has connected to your Databases for Elasticsearch deployment and is running successfully, you will see the output in your terminal. + ``` + [2024-01-02T16:43:29.378+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601 + [2024-01-02T16:46:13.777+00:00][INFO ][status] Kibana is now available + ``` + +## Step 3: Create an Elasticsearch index (keyword-search) +This step is to create an Elasticsearch index with default settings for quick testing and verification. +With default settings, an Elasticsearch index does keyword search. + +* Open http://0.0.0.0:5601 in browser and log into Kibana using the `username` and `password` from the service credentials of the Elasticsearch instance +* Navigate to the indices page http://localhost:5601/app/enterprise_search/content/search_indices +* Click on `Create a new index`, choose `Use the API`, and follow the steps there to create a new Elasticsearch index with default settings +* Go to the overview page for your newly created index, follow the steps there to verify your Elasticsearch index. + Notes: + * Generate an API key, and you will use the API key for authentication and authorization for this specific Elasticsearch index + * Use your `hostname` and `port` from the service credentials of the Elasticsearch instance to build `ES_URL` + ```bash + export ES_URL=https:// + ``` + * Append `--cacert ` to the cURL for SSL connection or append `--insecure` to the cURL commands to ignore the certificate + * If you are able to run the `Build your first search query` command at the last step, your Elasticsearch index has been set up successfully! + + +## Step 4: Enable semantic search with ELSER +This step is to enable semantic search using ELSER. Here are the tutorials from Elasticsearch doc: +ELSER v1: https://www.elastic.co/guide/en/elasticsearch/reference/8.10/semantic-search-elser.html +ELSER v2: https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search-elser.html + +**IMPORTANT NOTE**: ELSER v2 has become available since Elasticsearch 8.11. It is preferred to use ELSER v2 if it is available. + +The following steps are based on ELSER v2 model: +### Create environment variables for ES credentials + ```bash + export ES_URL=https:// + export ES_USER= + export ES_PASSWORD= + export ES_CACERT= + ``` +You can find the credentials from the service credentials of your Elasticsearch instance. +  +### Enable ELSER model (v2) +ELSER model is not enabled by default, but you can enable it in Kibana. Please follow the [download-deploy-elser instructions](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html#download-deploy-elser) to do it. + +Note: `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. + + +### Load data into Elasticsearch +In Kibana, you can upload a data file to Elasticsearch cluster using the Data Visualizer in the Machine Learning UI http://localhost:5601/app/ml/filedatavisualizer. + +As an example, you can download [wa-docs-100](./assets/wa_docs_100.tsv) TSV data and upload it to Elasticsearch. +This dataset contains documents processed from the watsonx Assistant product documents. There are three columns in this TSV file, +`title`, `section_title` and `text`. The columns are extracted from the original documents. Specifically, +each `text` value is a small chunk of text split from the original document. + +In Kibana, +* Select your downloaded file to upload + +* Click `Override settings` and then check `Has header row` checkbox because the example dataset has header row + +* Import the data to a new Elasticsearch index and name it `wa-docs` + +Once finished, you have created an index for the data you just uploaded. +### Create an index with mappings for ELSER output + ```bash + curl -X PUT "${ES_URL}/search-wa-docs?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "mappings": { + "_source": { + "excludes": [ + "ml.tokens" + ] + }, + "properties": { + "ml.tokens": { + "type": "sparse_vector" + }, + "text": { + "type": "text" + } + } + } + }' + ``` +Notes: +* `search-wa-docs` will be your index name. +* `ml.tokens` is the field that will keep ELSER output when data is ingested. +* `text` is the input filed for the inference processor. In the example dataset, the name of the input field is `text` which will be used by ELSER model to process. +* `sparse_vector` type is for ELSER v2. For ELSER v1, please use `rank_features` type. +* Learn more about [elser-mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search-elser.html#elser-mappings) from the tutorial. + +### Create an ingest pipeline with an inference processor +Create an ingest pipeline with an inference processor to use ELSER to infer against the data that will be ingested in the pipeline. + ```bash + curl -X PUT "${ES_URL}/_ingest/pipeline/elser-v2-test?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "processors": [ + { + "inference": { + "model_id": ".elser_model_2_linux-x86_64", + "target_field": "ml", + "field_map": { + "text": "text_field" + }, + "inference_config": { + "text_expansion": { + "results_field": "tokens" + } + } + } + } + ] + }' + ``` +Notes: +* `elser-v2-test` is the name of the ingest pipeline with an inference processor using ELSER v2 model. +* `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. +* `"text": "text_field"` maps the `text` field from an index to the input field of the ELSER model. `text_field` is the default input field of the ELSER model when it is deployed. You may need to update it if you configure a different input field when deploying your ELSER model. +* Learn more about [inference-ingest-pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search-elser.html#inference-ingest-pipeline) from the tutorial + +### Ingest the data through the inference ingest pipeline +Create the tokens from the text by reindexing the data through the inference pipeline that uses ELSER as the inference model. + ```bash + curl -X POST "${ES_URL}/_reindex?wait_for_completion=false&pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "source": { + "index": "wa-docs" + }, + "dest": { + "index": "search-wa-docs", + "pipeline": "elser-v2-test" + } + }' + ``` +* `wa-docs` is the index you created when uploading the example file to Elasticsearch cluster. It contains the text data. +* `search-wa-docs` is the search index that has ELSER output field. +* `elser-v2-test` is the ingest pipeline with an inference processor using ELSER v2 model. +### Semantic search by using the text_expansion query +To perform semantic search, use the `text_expansion` query, and provide the query text and the ELSER model ID. +The example below uses the query text "How to set up custom extension?", the `ml.tokens` field contains +the generated ELSER output: + ```bash + curl -X GET "${ES_URL}/search-wa-docs/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_2_linux-x86_64", + "model_text":"how to set up custom extension?" + } + } + } + }' + ``` +Notes: +* You can also use `API_KEY` for authorization. You can generate an `API_KEY` for your search index on the index overview page in Kibana. +* Learn more about [text-expansion-query](https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search-elser.html#text-expansion-query) from the tutorial. + +### Enable semantic search for Agent Knowledge on watsonx Orchestrate +To enable semantic search for your Agent Knowledge on watsonx Orchestrate, you just need to specify the following query body in your Elasticsearch Knowledge source configuration: + ```json + { + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_2_linux-x86_64", + "model_text":"$QUERY" + } + } + } + } + ``` + + +Notes: +* `$QUERY` is the query variable that contains the user search query by default. +* `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. + +Learn more about configuring Agent Knowledge from [Elasticsearch integration with Agent Knowledge in watsonx Orchestrate](README.md#elasticsearch-integration-with-agent-knowledge-in-watsonx-orchestrate) diff --git a/agent_knowledge/elasticsearch-install-and-setup/README.md b/agent_knowledge/elasticsearch-install-and-setup/README.md new file mode 100644 index 0000000..aa65572 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/README.md @@ -0,0 +1,26 @@ +# Elasticsearch Installation and Setup Documentation + +This directory contains documentation for installing and setting up Elasticsearch along with related guides and integrations. + +## Elasticsearch Setup +- [Install Docker or Docker alternatives](how_to_install_docker.md): A guide explaining Docker and Docker Compose installation options, essential for running Elasticsearch-related applications. +- [Set up Elasticsearch from IBM Cloud and integrate it with watsonx Orchestrate](ICD_Elasticsearch_install_and_setup.md): Instructions for provisioning Elasticsearch instance on IBM Cloud and setting up Agent Knowledge in watsonx Orchestrate. +- [Set up watsonx Discovery (aka Elasticsearch on-prem) and integrate it with watsonx Orchestrate on-prem](watsonx_discovery_install_and_setup.md): Documentation for setting up watsonx Discovery (aka Elasticsearch on-prem) and integrating it with watsonx Orchestrate on-prem. + +## Elasticsearch integration with Agent Knowledge in watsonx Orchestrate +### Option 1: Add Knowledge to your agents in the Agent Builder UI +See [Connecting to an Elasticsearch content repository](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=agents-connecting-elasticsearch-content-repository) in watsonx Orchestrate documentation for more details. + +### Option 2: Create Knowledge bases via watsonx Orchestrate ADK (Agent Development Kit) +See [Creating external knowledge bases with Elasticsearch](https://developer.watson-orchestrate.ibm.com/knowledge_base/build_kb#elasticsearch) in ADK documentation for more details. + +### Configure the Advanced Elasticsearch Settings +There are two settings under `Advanced Elasticsearch Settings` for using custom query body and custom filters to achieve advanced search use cases. See the guide [How to configure Advanced Elasticsearch Settings](./how_to_configure_advanced_elasticsearch_settings.md) for more details. + +### Federated search +You can follow the guide [here](federated_search.md) to run queries across multiple indexes within your Elasticsearch cluster. + +## Document Ingestion with Elasticsearch +- [Set up the web crawler in Elasticsearch](how_to_use_web_crawler_in_elasticsearch.md): Guide for setting up and using the web crawler in Elasticsearch and connecting it to Agent Knowledge in watsonx Orchestrate. +- [Working with PDF and office documents in Elasticsearch](how_to_index_pdf_and_office_documents_elasticsearch.md): Guide for working with PDF and Office Documents in Elasticsearch, including indexing and connecting to Agent Knowledge in watsonx Orchestrate. +- [Set up text embedding models in Elasticsearch](text_embedding_deploy_and_use.md): Instructions for setting up and using 3rd-party text embeddings for dense vector search in Elasticsearch. diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/add_crawl_rules_for_web_crawler.png b/agent_knowledge/elasticsearch-install-and-setup/assets/add_crawl_rules_for_web_crawler.png new file mode 100644 index 0000000..0f7aef2 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/add_crawl_rules_for_web_crawler.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/advanced_elasticsearch_settings.png b/agent_knowledge/elasticsearch-install-and-setup/assets/advanced_elasticsearch_settings.png new file mode 100644 index 0000000..59a870a Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/advanced_elasticsearch_settings.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/config_query_source_when_use_es_extension.png b/agent_knowledge/elasticsearch-install-and-setup/assets/config_query_source_when_use_es_extension.png new file mode 100644 index 0000000..7adf0b2 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/config_query_source_when_use_es_extension.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/config_result_content.png b/agent_knowledge/elasticsearch-install-and-setup/assets/config_result_content.png new file mode 100644 index 0000000..2716079 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/config_result_content.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/connect_to_elasticsearch_index.png b/agent_knowledge/elasticsearch-install-and-setup/assets/connect_to_elasticsearch_index.png new file mode 100644 index 0000000..8a40924 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/connect_to_elasticsearch_index.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/conversation-search-example-with-web-crawler.png b/agent_knowledge/elasticsearch-install-and-setup/assets/conversation-search-example-with-web-crawler.png new file mode 100644 index 0000000..ad18427 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/conversation-search-example-with-web-crawler.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/conversational_search_example_python_doc_ingestion.png b/agent_knowledge/elasticsearch-install-and-setup/assets/conversational_search_example_python_doc_ingestion.png new file mode 100644 index 0000000..8ffdf14 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/conversational_search_example_python_doc_ingestion.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/elasticsearch-integration.png b/agent_knowledge/elasticsearch-install-and-setup/assets/elasticsearch-integration.png new file mode 100644 index 0000000..0353e5e Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/elasticsearch-integration.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/elasticsearch-pdfofficedocs-watsonx-example.png b/agent_knowledge/elasticsearch-install-and-setup/assets/elasticsearch-pdfofficedocs-watsonx-example.png new file mode 100644 index 0000000..5ee7296 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/elasticsearch-pdfofficedocs-watsonx-example.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/example_volume_mount.png b/agent_knowledge/elasticsearch-install-and-setup/assets/example_volume_mount.png new file mode 100644 index 0000000..b16c961 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/example_volume_mount.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/federated_search_official_search_extension.png b/agent_knowledge/elasticsearch-install-and-setup/assets/federated_search_official_search_extension.png new file mode 100644 index 0000000..3dc0e7c Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/federated_search_official_search_extension.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/fscrawler-docker-config.zip b/agent_knowledge/elasticsearch-install-and-setup/assets/fscrawler-docker-config.zip new file mode 100644 index 0000000..fa79c4d Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/fscrawler-docker-config.zip differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/import_data_to_new_index.png b/agent_knowledge/elasticsearch-install-and-setup/assets/import_data_to_new_index.png new file mode 100644 index 0000000..d10113a Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/import_data_to_new_index.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/jupyter_play_button.png b/agent_knowledge/elasticsearch-install-and-setup/assets/jupyter_play_button.png new file mode 100644 index 0000000..6f517fc Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/jupyter_play_button.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/override_settings_for_uploaded_file.png b/agent_knowledge/elasticsearch-install-and-setup/assets/override_settings_for_uploaded_file.png new file mode 100644 index 0000000..bde7c1f Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/override_settings_for_uploaded_file.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/podman_extensions.png b/agent_knowledge/elasticsearch-install-and-setup/assets/podman_extensions.png new file mode 100644 index 0000000..6f177e0 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/podman_extensions.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/podman_machine_edit.png b/agent_knowledge/elasticsearch-install-and-setup/assets/podman_machine_edit.png new file mode 100644 index 0000000..b201a0f Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/podman_machine_edit.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/podman_machine_running.png b/agent_knowledge/elasticsearch-install-and-setup/assets/podman_machine_running.png new file mode 100644 index 0000000..acf7ee4 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/podman_machine_running.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/query_body_for_elasticsearch.png b/agent_knowledge/elasticsearch-install-and-setup/assets/query_body_for_elasticsearch.png new file mode 100644 index 0000000..71d910f Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/query_body_for_elasticsearch.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/query_body_with_custom_filters.png b/agent_knowledge/elasticsearch-install-and-setup/assets/query_body_with_custom_filters.png new file mode 100644 index 0000000..4fcf64c Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/query_body_with_custom_filters.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/rancher_init.png b/agent_knowledge/elasticsearch-install-and-setup/assets/rancher_init.png new file mode 100644 index 0000000..da28d93 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/rancher_init.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/sample_pdf_docs/lendyr_preferred_card.pdf b/agent_knowledge/elasticsearch-install-and-setup/assets/sample_pdf_docs/lendyr_preferred_card.pdf new file mode 100644 index 0000000..bc024b9 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/sample_pdf_docs/lendyr_preferred_card.pdf differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/sample_pdf_docs/lendyr_topaz_card.pdf b/agent_knowledge/elasticsearch-install-and-setup/assets/sample_pdf_docs/lendyr_topaz_card.pdf new file mode 100644 index 0000000..605a44f Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/sample_pdf_docs/lendyr_topaz_card.pdf differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/search_for_the_answer.png b/agent_knowledge/elasticsearch-install-and-setup/assets/search_for_the_answer.png new file mode 100644 index 0000000..252bf6b Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/search_for_the_answer.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/synchronize_trained_model.png b/agent_knowledge/elasticsearch-install-and-setup/assets/synchronize_trained_model.png new file mode 100644 index 0000000..2467551 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/synchronize_trained_model.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/upload_file_though_data_visualizer.png b/agent_knowledge/elasticsearch-install-and-setup/assets/upload_file_though_data_visualizer.png new file mode 100644 index 0000000..73d2c34 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/upload_file_though_data_visualizer.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/use_nested_query_in_search_integration_settings.png b/agent_knowledge/elasticsearch-install-and-setup/assets/use_nested_query_in_search_integration_settings.png new file mode 100644 index 0000000..e7daf1c Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/use_nested_query_in_search_integration_settings.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/use_session_variable_in_custom_filters_for_search.png b/agent_knowledge/elasticsearch-install-and-setup/assets/use_session_variable_in_custom_filters_for_search.png new file mode 100644 index 0000000..679844b Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/use_session_variable_in_custom_filters_for_search.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/wa_conversational_search_result.png b/agent_knowledge/elasticsearch-install-and-setup/assets/wa_conversational_search_result.png new file mode 100644 index 0000000..e3f3d1d Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/wa_conversational_search_result.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/wa_docs_100.tsv b/agent_knowledge/elasticsearch-install-and-setup/assets/wa_docs_100.tsv new file mode 100644 index 0000000..bb18414 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/assets/wa_docs_100.tsv @@ -0,0 +1,101 @@ +section_title title text +Previewing and sharing your assistant Previewing and sharing your assistant \nPreviewing and sharing your assistant\nInternal review is a necessary step in any virtual assistant workflow. You need an environment without customer interactions so your team can test your assistant. The draft environment closely resembles the final experience that your users encounter\n so you can ensure that you are publishing the optimal end product.\n +Saving and editing your work in the draft environment Previewing and sharing your assistant \nSaving and editing your work in the draft environment\nThe draft environment contains all your in-progress work in the Actions, Preview, and Publish pages. Use the Draft environment tab to manage the draft environment, including\n adding draft environment integrations (channels and extensions) that you can use for internal testing before deployment. These integrations are unique to the draft environment, and changes to draft integrations don't affect the live environment.\n +The Preview page Previewing and sharing your assistant \nThe Preview page\nUse the Preview page to test your assistant. You can experience your assistant from your customers' perspective. The Preview page includes an interactive web chat widget where you can test out your assistant\n as if you were a customer. The content that is contained in the assistant is the content that you built into your actions or set up with the search integration.\nOn the Preview page, you also find the following elements:\n\nCopy link to share: Share an unauthenticated version of your assistant with your colleagues by sending them a link. For more information, see Copying a link to share.\nChange background: Change the background of the page so you can see what your assistant looks like on different web pages. For more information, see Changing background website.\nCustomize web chat: Customize your draft web chat channel to match your brand or website. For more information, see Web chat setup overview.\n\n\n +Copying a link to share Previewing and sharing your assistant \nCopying a link to share\nYou can share an unauthenticated version of your assistant with your team by sending them a link. The link opens a sample web page with an interactive web chat widget where you can test out your assistant as if you were a customer. Your subject-matter\n experts can test your in-progress assistant without needing access to watsonx Assistant itself. The experience is identical to using Preview this environment on the draft environment tab.\nTo share a link:\n\nOn the Preview page, click Copy link to share.\nSend the link to your team.\n\nThe preview link is not accessible if web chat security is enabled. For more information about web chat security, see Securing the web chat.\n +Changing background website Previewing and sharing your assistant \nChanging background website\nYou can visualize how your assistant would look as a web chat widget on your organization's website. You can enter a URL or upload an image.\n +Entering a URL Previewing and sharing your assistant \nEntering a URL\nYou can enter a URL of your organization's website. Your assistant captures an image of your website to use as the Preview page background.\nYour website must be publicly available to all users. Private or intranet sites can’t be accessed. Any login, splash, cookie, or warning screens might be captured in the image.\nTo enter a URL:\n\nOn the Preview page, click Change background.\nClick Enter URL, then click Continue.\nEnter the path of your website URL, for example, https://www.example.com or example.com.\nClick Continue.\n\n +Uploading an image Previewing and sharing your assistant \nUploading an image\nYou can upload an image of your organization's website. Images are stored for 24 hours. The maximum file size is 1 MB. Supported file types are JPEG and PNG.\nTo upload an image:\n\nOn the Preview page, click Change background.\nClick Upload an image, then click Continue.\nDrag a file or click to upload, then click Change background.\n\nImages are stored for 24 hours. A warning message might appear on the Preview page about the time limit expiration. To clear this message:\n\nOn the Preview page, click Change background.\nClick Clear background setting, then click Continue.\nClick Remove background to finish.\n\n +Removing the background Previewing and sharing your assistant \nRemoving the background\nAfter you enter a URL or upload an image to use as a background, you might decide to remove the background and restore the default background.\nTo remove the background image:\n\nOn the Preview page, click Change background.\nClick Remove background, then click Continue.\nClick Remove background to finish.\n\n +Using variables to manage conversation information Using variables to manage conversation information "Using variables to manage conversation information\nWhen customers reply to your assistant, they share information about themselves and what they want. Your assistant remembers this information, and other information about a conversation, as variables. Your assistant can use variables\n to provide a more personalized and customized experience, and to get users quickly to the solutions they need.\nVariables are a powerful tool that you can use to build a better assistant. Variables make possible all of the following benefits:\n\n\nPersonalization. The best virtual assistant experiences are targeted and personalized for each customer. When an assistant greets a customer by saying ""Hello, Frank! Welcome back,"" it tells Frank that it remembers\n his name and that it has talked to him before. By storing this kind of information in variables and then referencing them in your assistant's output, you can personalize the conversation and help your assistant seem more human.\n\n\nAcceleration. Over the course of a conversation, your customers answer questions and make choices. These customer responses are stored as variables, which your assistant can then use to guide a conversation. By choosing the\n right steps and not wasting your customers' time, you can get them as quickly as possible to the right solution.\n\n\nModularity. Some information might be useful for many different purposes (for example, a customer's current account balance or contact information). Rather than retrieving or recalculating this information in multiple locations,\n you can do so once, by using a variable to store the result and then access it wherever you need it.\n\n\nPrivacy. The privacy of the customer information is critical for all enterprises. In your assistant, you can mask the variables to protect the privacy of the shared information. When you mask a variable to make it private,\n the assistant hides the confidential information in the user input or assistant's responses with asterisks in the conversation logs." +Using variables to manage conversation information Using variables to manage conversation information The feature for masking the confidential customer information is available only for actions in assistants. If you're calling actions from a dialog, the privacy setting is available in your action steps but won't mask the customer information.\nA variable is simply a named container for a piece of information; by referencing this container by name, your assistant can store or retrieve the information at run time. For example, a variable that is called account_balance might store\n your customer's current account balance, a value your assistant can update or retrieve as needed.\nThe data that is stored by a variable is characterized by the type of data that it contains, such as text, a numeric value, a date, or even a list of multiple values. The operations that you can perform with a variable vary depending on its data\n type. +Action variables and session variables Using variables to manage conversation information "Action variables and session variables\nIBM® watsonx™ Assistant supports two categories of variables:\n\n\nAction variables: When a step collects information from the customer, the customer response is automatically stored in an action variable. You can think of action variables as short-term memory: they persist only\n during the current action.\nThe name of an action variable is always the name of the step that defines the customer response. (You cannot change the name of an action variable.) For example, suppose you define a step that asks ""When were you born?"" and accepts\n a date value as a response. The customer response is automatically stored as an action variable called When were you born?, which you can then access from any subsequent step in the same action.\nYou can make an action variable private by selecting the Protect data collected at this step checkbox in the customer response settings." +Action variables and session variables Using variables to manage conversation information Session variables: A value that is not necessarily tied to a particular action can be stored as a session variable. Session variables are long-term memory: they persist throughout the user's interaction with the\n assistant, and your assistant can reference them from any action.\nYou can create a session variable to store the value from an action variable, if you want to keep the value available for other actions to use. You can also define a session variable based on another session variable, or by using a value\n defined in an expression. In addition to variables you create, watsonx Assistant provides a set of built-in session variables for global values like the current time and date.\nTo hide the confidential customer information in the conversation logs, you can select the Protect data stored in this variable checkbox when you create or edit a session variable.{: tip)\nSession variables can help you to modularize your assistant because you can write a single action that collects information that is needed in multiple places. For example, you might have a greeting action that collects basic information\n about the customer and stores the responses in session variables, which any action can then access.\nA session variable that you create persists only during a single session. At the end of the session, the variable's value is cleared. How long a session lasts depends upon how your customers access your assistant, and how your assistant\n is configured. +Creating a session variable Using variables to manage conversation information Creating a session variable\nTo add a session variable that can be accessed by any action:\n\n\nFrom the Actions page, click Variables > Created by you. The list shows all session variables that you created for your assistant.\n\n\nClick New variable.\nYou can also create a new session variable from the step editor. For more information, see Storing a value in a session variable.\n\n\nIn the Name field, type a name for the session variable.\nAs you add the name, an ID is generated for you. Any spaces in the name are replaced with underscores (_) in the ID.\n\n\nOptional: Add a type to set the response type of the variable. (For more information about response types, see Choosing a response type.)\nFrom this field, you can also select any of the saved responses that you created. For more information about saved responses, see Saving and reusing customer responses.\nIn addition to the listed types, a variable can also be created as an array. To create an array variable, select Any as the type, and in the next step, define an initial value that uses the expression [] to\n represent an empty array.\n\n\nOptional: Add an initial value to set the starting value for the variable at the beginning of each user session. For example, your customers can use an assistant to make purchases. You might initialize a Payment due variable with a starting value of 0, and then add to that value as the customer orders items.\nTo specify a complex object or an array as the initial value, or to calculate the initial value based on other variables, you can write an expression. For more information about writing expressions, see Writing expressions.\n\n\nOptional: Add a description.\n\n\nOptional: Select the Protect data stored in this variable checkbox in the Privacy section if the session variable contains confidential information.\n\n\nClick Apply. +Built-in variables Using variables to manage conversation information Built-in variables\nIn addition to the variables you create, watsonx Assistant provides a set of built-in variables you can access from any action. At run time, these variables are automatically set with the appropriate values. For example, the Current time session variable always provides the current time in the user's time zone, at the time of the interaction with the customer.\nTo see these variables, click Variables on the Actions page.\n\n\nThe Set by assistant page shows built-in session variables that are automatically provided for each assistant.\n\n\nThe Set by integration page shows variables that are automatically provided by the integration your customer is using to connect to the assistant. (These variables are not set if no integration is connected.)\n\n\nSet by assistant:\n\nVariables set by assistant\n\n\nVariable name\nVariable ID\nDescription\nExamples\n\n\n\n\nDigressed from\ndigressed_from\nLast action before the customer digressed (or null if not digressed)\nPay bill\n\n\nNow\nnow\nThe current date and time in the user's time zone.\n2021-08-11T11:28:02\n\n\nCurrent time\ncurrent_time\nThe current time in the user's time zone.\n11:28:02\n\n\nCurrent date\ncurrent_date\nThe current date in the user's time zone.\n2021-08-11\n\n\nFallback reason\nfallback_reason\nThe reason why a user is routed to the fallback action\nStep validation failed - Agent requested - No action matches\n\n\nNo action matches count\nno_action_matches_count\nRepresents a count of customer's consecutive unrecognized input attempts\n3\n\n\nSession history IBM Cloud\nsession_history\nThe 24 most recent messages from the customer’s conversation. For more information, see Session history\n[{u : 'book a flight', n : true}, {a : 'sure! from where?'}]\n\n\n\nSet by integration:\n\nVariables set by integration\n\n\nVariable name\nVariable ID\nDescription\nExample\n\n\n\n\nTimezone\ntimezone\nThe user's time zone as specified by the integration or API client. The default time zone (if not specified by the integration) is Coordinated Universal Time.\nAmerica/New_York +Built-in variables Using variables to manage conversation information Locale\nlocale\nThe user's locale as set by the integration or API client. The locale can affect understanding and formatting of dates, times, and numbers.\nen-gb\n\n\nChannel Name\nchannel_name\nThe name of the channel that your user is interacting with.\nWeb chat +Storing a value in a session variable Using variables to manage conversation information Storing a value in a session variable\nAny action can store a value in a session variable so it is available to other actions. To store a value in a session variable:\n\n\nFrom within a step, click Set variable values.\n\n\nClick Set new value.\n\n\nIn the Set drop-down list, your choices are:\n\nSet variable values\n\n\nChoice\nDescription\n\n\n\n\nSession variable\nThe session variable that you want to store the value in. The new value replaces any previous value that is stored.\n\n\nIntegration variable\nThe session variable that you want to store the value in. The new value replaces any previous value that is stored.\n\n\nExpression\nWrite an expression directly without first picking a variable. For more information, see Writing expressions.\n\n\n+ New session variable\nYou can create a new session variable, which is added to the list of session variables for the assistant. For more information, see Creating a session variable.\n\n\n\n\n\nIn the To drop-down list, the choices vary depending on the type of variable you're setting. Possible choices include:\n\nSet variable values\n\n\nChoice\nDescription\n\n\n\n\nScalar value by type\nSet a specific value for each variable type. The choice varies depending on the variable type. For example, for a date variable, the choice is Enter a date, and you can use a date picker to set a date. Other choices\n appear for Boolean, confirmation, currency, date, free text, number, percentage, and time.\n\n\nExpression\nWrite an expression to define the value for the session variable. For more information about expressions, see Writing expressions.\n\n\nAction variables\nSelect an action variable to use the value of a customer response in another step. The choices that are listed match the type of variable that you want to set.\n\n\nSession variables\nSelect another session variable to use its value. The choices that are listed match the type of variable that you want to set.\n\n\nAssistant variables\nSelect a built-in system variable to use its value. The choices that are listed match the type of variable that you want to set.\n\n\nIntegration variables\nIf you are setting an integration variable, you can choose other integration variables as the value.\n\n\n\n\n\nTo set more variable values in the same step, click Set new value. +Using variables to manage conversation flow Using variables to manage conversation information \nUsing variables to manage conversation flow\nOne of the ways you can use variables is to choose the correct path through the conversation, based on customer responses and other values available at run time. You can do this by defining step conditions, which determine whether a specific\n step in an action is executed based on runtime conditions.\nBy defining a condition based on an action variable, you can control whether a step is executed based on the customer's response to a previous step. You can also build step conditions based on session variables, which can store information from\n other actions.\nFor more information about step conditions, see Defining step conditions.\n +Using variables to customize the conversation Using variables to manage conversation information Using variables to customize the conversation\nYou can also use variables in what your assistant says, dynamically referencing information that has been collected during the conversation. This is useful for confirming information the customer has provided (for example, You want to transfer $153.14 to your checking account. Is that correct?),\n and for simply personalizing the conversation to make it more human (Hi, John. How can I help you today?).\nTo reference a variable in what your assistant says:\n\n\nIn the Assistant says field, start typing the text for the response.\n\n\nWhen you reach a point where you want to insert a reference to a variable, type a dollar sign ($) or click the Insert a variable icon (). A list\n appears showing the variables you can choose from.\n\n\nClick a variable to add a reference to it in the text.\n\n\nWhen you reference a variable, it appears using a default format in your assistant's response. The format of the variable might differ from the way the value is stored; for example, a date value of 2021-08-11 is formatted as August 11, 2021 by default.\nThe default formats are as follows:\n\nDefault formats for variables\n\n\nType\nFormat\nExamples\n\n\n\n\nOptions\nAs chosen by the user\nYes No\n\n\nNumber\nNumerals only\n1000\n\n\nDate\nMmm DD, YYYY\nJun 30, 2021\n\n\nTime\nH:MM:SS AM\n5:15:00 PM\n\n\nCurrency\nNumber only, no currency symbol\n20\n\n\nPercent\nNumber only, no percentage symbol\n20\n\n\nFree text\nAs entered by the user\nPlease check that the apples aren't bruised\n\n\n\nWhen building an assistant response that includes variables, you concatenate multiple parts (text strings and variables). A single response can consist of no more than 30 concatenated parts (for example, 15 variables along with 15 text strings). +Referencing expressions Using variables to manage conversation information \nReferencing expressions\nIf you need to reference a dynamic value that is calculated using an expression, you must first assign this value to a session variable. (For more information about how to do this, see Storing a value in a session variable.)\n You can then reference the session variable in the Assistant says field.\nNote that the syntax for referencing expressions in assistant output is not supported in actions.\n +Configuring suggestions Configuring suggestions Configuring suggestions\nSuggestions give your customers a way to try something else when the current exchange with the assistant isn't delivering what they expect. A question mark icon is displayed\n in the web chat that customers can click at any time to see other topics that might be of interest or, if configured, to request support. Customers can click a suggested topic to submit it as input or click the X icon to close\n the suggestions list.\nIf customers select a suggestion and the response is not helpful, they can open the suggestions list again to try a different suggestion. The input generated by the first choice is submitted and recorded as part of the conversation. However, any\n contextual information that is generated by the initial suggestion is reset when the subsequent suggestion is submitted.\nThe suggestions are shown automatically in situations where the customer might otherwise become frustrated. For example, if a customer uses different wording to ask the same question multiple times in succession, and the same action is triggered\n each time, then related topic suggestions are shown in addition to the triggered action's response. The suggestions that are offered give the customer a quick way to get the conversation back on track.\nThe suggestions list is populated with actions that are relevant in some way to the matched action. The actions are ones that the AI model considered to be possible alternatives, but that didn't meet the high confidence threshold that is required\n for an action to be listed as a disambiguation option. Any action can be shown as a suggestion, unless its Ask clarifying question setting is set to Off. For more information about the Ask clarifying question setting, see Asking clarifying questions.\nTo configure suggestions, complete the following steps:\n\n\nOpen the Suggestions tab.\nSuggestions are enabled automatically for new web chat integrations. If you don't want to use suggestions, toggle the switch to Off. +Configuring suggestions Configuring suggestions In the Include a connection to support section, specify when you want an option to connect with support to be included in the list of suggestions. You can specify Always, Never, or After one failed attempt.\nAfter one failed attempt: Adds the option to the list only if the customer reached a node with an anything_else condition in the previous conversation turn or reaches the same action for a second time in succession.\n\n\nIn the Option label field, type the text of the message that requests help from support. This message is shown as the label for the support option, which is included in the Suggestions window under the circumstances\n you specified in the previous step. If the customer clicks this option, the same message is sent to the assistant.\nThe message you specify should trigger an action that gives customers a way to connect with support. By default, the message Connect with agent is used. If your web chat is integrated with a contact center platform, this message\n initiates a transfer to a human agent. (For more information about integrating with a contact center, see Adding contact center support.)\nIf your web chat is not integrated with a contact center, specify a message that helps your customers reach whatever form of support you do offer. If you offer a toll-free support line, you might add Get the support line phone number.\n Or if you offer an online support request form, you might add Open a support ticket.\nWhether you use the default support message or add your own, make sure your action is designed to recognize the message and respond to it appropriately. +Web chat overview Web chat overview \nWeb chat overview\nYou can use the web chat integration to deploy your assistant on your website or embed it as a WebView in your mobile app. The web chat integration provides an easy-to-use chatbot interface that can integrate seamlessly with your site, without\n requiring the time and effort that would be required to build your own custom user interface.\nThe web chat can help your customers start the conversation with common questions or tasks; it can display multimedia and interactive elements such as forms, and it can transfer customers to human agents for more help. A developer can customize\n the web chat to add even more capabilities.\n +Why use the web chat? Web chat overview Why use the web chat?\nBuilding a custom user interface requires spending time and effort solving typical UI problems. For example, you need to design the layout and styling, keep up with browser changes, manage scrolling behavior, validate input, and comply with\n accessibility requirements. The time you spend building and maintaining a custom UI is better spent building a high-quality assistant instead.\nThe web chat widget uses cutting-edge functionality from IBM Design and Research to engage your users when they need help, answer their questions quickly and efficiently, and provide fallback options so there is always a path to a solution.\n The web chat is easy for you to deploy and easy for customers to use, it is secure, and it supports a wide range of desktop and mobile browsers.\nThe web chat is also customizable, which means that you can take advantage of the web chat functionality while still maintaining consistency with your website style and branding, adding custom UI elements, and integrating with external systems\n (such as live agent tools or CRM systems). +What you can do with the web chat Web chat overview What you can do with the web chat\nYou can quickly deploy the web chat to your website (or even to a local test page) and see how it works. You can also use the web chat integration settings in the watsonx Assistant user interface to configure the web chat for your website and\n your customers.\nIf you are a developer, you can further customize and extend the web chat by writing code and using the web chat API. You can also use a WebView with a JavaScript bridge to add the web chat to your mobile app.\nThe following documentation topics provide more information about the capabilities of the web chat integration, how to configure and deploy it, and how to customize it.\n\nHow the web chat works: Overview of web chat capabilities and architecture\nEmbedding the web chat on your page: How to embed the web chat widget on your website\nWeb chat setup: How to configure the web chat using the integration settings\nWeb chat development: How to customize and extend the web chat by writing code\nAdding contact center support: How to integrate the web chat with a contact center platform so you can connect your customers to human agents +Advanced analysis and log-related tasks Advanced analysis and log-related tasks \nAdvanced analysis and log-related tasks\n +Classic experience only Advanced analysis and log-related tasks \nClassic experience only\nThis information applies to dialog skill analytics in the classic experience. For information about analytics in watsonx Assistant, see Use analytics to review your entire assistant at a glance.\nLearn about notebooks and APIs that you can use to access and analyze log data.\n +Using Jupyter notebooks for analysis Advanced analysis and log-related tasks \nUsing Jupyter notebooks for analysis\nIBM created Jupyter notebooks that you can use to analyze the behavior or your assistant. A Jupyter notebook is a web-based environment for interactive computing. You can run small pieces of code that process your data, and you can immediately\n view the results of your computation.\nYou can use the notebooks with English-language skills only.\n +Analysis notebooks Advanced analysis and log-related tasks \nAnalysis notebooks\nAnalysis notebooks are available for:\n\nIBM Watson® Studio\nStandard Python tools\n\nWatson Studio provides an environment where you can:\n\nChoose the tools that you need to analyze and visualize data.\nCleanse and shape data.\nIngest streaming data.\nCreate, train, and deploy machine learning models.\n\nFor more information, see the product documentation.\nThe watsonx Assistant Continuous Improvement Best Practices Guide describes how to get the most out of these notebooks.\n +Using the notebooks with Watson Studio Advanced analysis and log-related tasks Using the notebooks with Watson Studio\nThe following notebooks are available:\n\nDialog skill analysis for watsonx Assistant\nMeasure watsonx Assistant Performance.\nAnalyze watsonx Assistant Effectiveness\nDialog Flow Analysis for watsonx Assistant\n\nIf you choose to use the notebooks that are designed for use with Watson Studio, the steps are:\n\n\nCreate a Watson Studio account, create a project, and add a Cloud Object Storage account to it.\n\n\nFrom the Watson Studio community, choose a notebook.\nEarly in the development process, use the Dialog skill analysis for watsonx Assistant notebook to help you get started. The notebook:\n\nExamines the terms that are correlated with each intent in your training data to find anomalies that might identify problems that you can investigate.\nUses a blind test set that you provide to calculate performance on statistical metrics like Accuracy, Precision, Recall, and F1.\nOffers advanced features that you can use to find the causes of common issues, such as why some sentences are often misidentified.\n\nTo learn more about how this notebook can help you improve your dialog, read Dialog skill analysis.\n\n\nAfter you deploy a version of the assistant and collect conversation log data, run the Measure watsonx Assistant Performance notebook.\n\n\nFollow the step-by-step instructions provided with the notebook to analyze a subset of the dialog exchanges from the logs.\nRun the following notebook first:\n\nMeasure: Gathers metrics that focus on coverage (how often the assistant is confident enough to respond to users) and effectiveness (when the assistant does respond, whether the responses are satisfying user needs).\n\nThe insights are visualized in ways that make it easier to understand areas for improvement in your assistant.\n\n\nExport a sample set of the logs from ineffective conversations, and then analyze and annotate them.\nFor example, indicate whether a response is correct. If correct, mark whether it is helpful. If a response is incorrect, then identify the root cause, the wrong intent or entity was detected, for example, or the wrong dialog node was triggered.\n After you identify the root cause, indicate what is the correct choice.\n\n\nFeed the annotated spreadsheet to the Analyze watsonx Assistant Effectiveness notebook.\n\nEffectiveness: Provides a deeper analysis of your logs to help you understand the steps that you can take to improve your assistant. +Using the notebooks with Watson Studio Advanced analysis and log-related tasks Use the Dialog Flow Analysis for watsonx Assistant notebook to review your dialog. The notebook can help you pinpoint the dialog nodes where customers most frequently abandon the conversation.\nFor more information about how this notebook can help you analyze and assess abandonment, see Do you know where and why users drop off the conversation?.\n\n\nThis process helps you to understand the steps you can take to improve your assistant. +Using the notebooks with standard Python tools Advanced analysis and log-related tasks \nUsing the notebooks with standard Python tools\nIf you choose to use standard Python tools to run the notebooks, you can get the notebooks from GitHub.\n\nDialog Skill Analysis for watsonx Assistant\nIBM® watsonx™ Assistant Recommendation notebooks (Measure and Analyze Effectiveness)\nIBM® watsonx™ Assistant Dialog Flow Analysis notebook\n\nThe watsonx Assistant Continuous Improvement Best Practices Guide outlines which notebook to use at each stage of your improvement process.\n +Using the logs API Advanced analysis and log-related tasks \nUsing the logs API\nYou can use the /logs API to list events from the transcripts of conversations that occurred between your users and your assistant. For conversations created with the v2 /message API, use the instance-level endpoint\n to list log events in all workspaces, and then filter by Assistant ID. For more information, see Filter query reference.\nThe API logs messages that are exchanged in conversations that are defined by a dialog skill only.\nThe number of days that logs are stored differs by service plan type. For more information, see Log limits.\nFor a Python script you can run to export logs and convert them to CSV format, download the export_logs_py.py file from the watsonx Assistant GitHub repository).\n +Understanding terminology Advanced analysis and log-related tasks Understanding terminology\nFirst, review the definitions of terms that are associated with Watson Assistant logs.\n\nTerminology\n\n\nTerm\nDefinition\n\n\n\n\nAssistant\nAn application - sometimes referred to as a 'chat bot' - that implements your Watson Assistant content.\n\n\nAssistant ID\nThe unique identifier of an assistant.\n\n\nConversation\nA set of messages that an individual user sends to your assistant, and the messages your assistant sends back.\n\n\nConversation ID\nUnique identifier that is added to individual message calls to link related message exchanges together. App developers that use the V1 version of the Watson Assistant API add this value to the message calls in a conversation by including\n the ID in the metadata of the context object.\n\n\nCustomer ID\nA unique ID that can be used to label customer data such that it can be deleted if the customer requests the removal of their data.\n\n\nDeployment ID\nA unique label that app developers of the Watson Assistant API V1 version pass with each user message to help identify the deployment environment that produced the message.\n\n\nInstance\nYour deployment of Watson Assistant, accessible with unique credentials. A {{site.data.keassistant_classic_shortnshort}} instance might contain multiple assistants.\n\n\nMessage\nA message is a single utterance that a user sends to the assistant.\n\n\nSkill ID\nThe unique identifier of a skill.\n\n\nUser\nA user is anyone who interacts with your assistant.\n\n\nUser ID\nA unique label that is used to track the level of service usage of a specific user.\n\n\n\nThe User ID property is not equivalent to the Customer ID property, though both can be passed with the message. The User ID field is used to track levels of usage for billing purposes. The Customer ID field is used to support the labeling and subsequent deletion of messages that are associated with users. Customer ID is used consistently across all Watson services and is specified in the X-Watson-Metadata header. User ID is\n used exclusively by the Watson Assistant service and is passed in the context object of each /message API call. +Associating message data with a user for deletion Advanced analysis and log-related tasks \nAssociating message data with a user for deletion\nThere may be cases when you want to completely remove a set of your user's data from a Watson Assistant instance. When the delete feature is used, the Overview metrics don't include those deleted messages, and results in fewer Total Conversations.\n +Before you begin Advanced analysis and log-related tasks "Before you begin\nTo delete messages for one or more individuals, you first need to associate a message with a unique Customer ID for each individual. To specify the Customer ID for any message that is sent with the /message API, include the X-Watson-Metadata: customer_id property in your header. You can pass multiple Customer ID entries with semicolon separated field=value pairs, by using customer_id,\n as in the following example:\ncurl -X POST -u ""apikey:3Df... ...Y7Pc9"" \\n --header \\n ""Content-Type: application/json"" \\n ""X-Watson-Metadata: customer_id={first-customer-ID};customer_id={second-customer-ID}"" \\n --data ""{\""input\"":{\""text\"":\""hello\""}}"" \\n ""{url}/v2/assistants/{assistant_id}/sessions/{session_id}/message?version=2019-02-28""\n\nwhere {url} is the appropriate URL for your instance. For more information, see Endpoint URLs.\nThe customer_id string cannot include the semicolon (;) or equal sign (=) characters. You are responsible for ensuring that each Customer ID parameter is unique across your customers.\nFor instructions on how to delete messages that use customer_id values, see the Labeling and deleting data in watsonx Assistant." +Adding conditions to an action Adding conditions to an action Adding conditions to an action\nIBM Cloud\nAn action condition is a Boolean test, based on some runtime value; the action runs only if the test evaluates as true. This test can be applied to any variable. By defining action conditions, you can do things such as control user access to actions\n or create date-specific actions.\nFor more information about variables, see Using variables to manage conversation information.\nA basic action condition is expressed in the following form:\nIf {variable} {operator} {value}\nwhere:\n\n{variable} is the name of a variable or an expression.\n{operator} is the type of test to apply to the variable value (for example, is or is not).\n{value} is the value to compare to the variable.\n\nFor example, an action condition might read:\nIf User category? is employee\nIn this example, User category can be a list of employees at your organization. This condition evaluates as true if the user is an active employee. If false, you can control access so that former employees can't use the action.\nConditions can be grouped to construct complex tests.\nTo add an action condition:\n\n\nIn an action, click Customer starts with.\n\n\nClick Conditions.\n\n\n\nConditions\n\n\n\n\nChoose the variable for the condition. You can select:\n\nAn action variable with the customer response from a previous step in the action\nA session variable that contains a value that is stored by any action\nA built-in variable set by the assistant or by an integration\n\nYou can also define a complex condition by writing an expression that defines some other value. For more information about expressions, see Writing expressions.\n\n\nSelect the operator that represents the test that you want to perform on the variable (for example, is or is not). The available operators for a particular value depend upon its data type. For more information, see\n Operators.\n\n\nSelect the value that you want to evaluate the condition against. The values available depend upon the type of value that you are testing. For example, a variable that contains an options response can be tested against any of the defined options,\n and a date value can be tested against any date.\n\n\n\nConditions +Adding conditions to an action Adding conditions to an action Conditions\n\n\n\n\nTo add more than one condition to an action, click Add condition.\n\n\nTo add another group of conditions, click New condition group.\nYou can use groups to build complex action conditions. Each group is evaluated true or false as a whole, and then these results are evaluated together. For example, you might build an action that runs only if all conditions in group 1 are\n true or any condition in group 2 is true. (Groups function like parentheses in the Boolean conditions of many programming languages.)\nAfter you add a group, you can define one or more conditions in the new group. Between groups, choose and or or to indicate whether the conditions in both conditional groups or only one of them must be met\n for the step to be included in the conversational flow. +Operators Adding conditions to an action \nOperators\nAn operator specifies the kind of test that you are performing on a value in a condition. The specific operators available in a condition depend on the customer response type of the value, as shown in the following table.\n\nOperators\n\n\nResponse type\nOperators\n\n\n\n\n\n\nOptions\n\n\n\n\nis\nis not\nis any of\nis none of\n\n\n\n\n\n\nRegex\n\n\n\n\nis\nis not\n\n\n\n\n\n\nNumber\nCurrency\nPercent\n\n\n\n\nis defined\nis not defined\nis equal to (==)\nis not equal to (≠)\nis less than (<)\nis less than or equal to (<=)\nis greater than (>)\nis greater than or equal to (>=)\n\n\n\n\n\n\nDate\n\n\n\n\nis defined\nis not defined\nis on (also allows specific day of the week)\nis not on\nis before\nis after\nis on or before\nis on or after\n\n\n\n\n\n\nTime\n\n\n\n\nis defined\nis not defined\nis at\nis not at\nis before\nis after\nis at or before\nis at or after\n\n\n\n\n\n\nFree text\n\n\n\n\nis\nis not\ncontains\ndoes not contain\nmatches\ndoes not match\n\n\n\n\n\n +Creating intents Creating intents \nCreating intents\nIntents are purposes or goals that are expressed in a customer's input, such as answering a question or processing a bill payment. By recognizing the intent expressed in a customer's input, the watsonx Assistant service can choose the\n correct dialog flow for responding to it.\n +Intent creation overview Creating intents Intent creation overview\n\n\nPlan the intents for your application.\nConsider what your customers might want to do, and what you want your application to be able to handle on their behalf. For example, you might want your application to help your customers make a purchase. If so, you can add a #buy_something intent. (The # that is added as a prefix to the intent name helps to clearly identify it as an intent.)\n\n\nTeach watsonx Assistant about your intents.\nAfter you decide which business requests that you want your application to handle for your customers, you must teach watsonx Assistant about them. For each business goal (such as #buy_something), you must provide at least 5\n examples of utterances that your customers typically use to indicate their goal. For example, I want to make a purchase.\nIdeally, find real-world user utterance examples that you can extract from existing business processes. Tailor the user examples to your specific business. For example, if you are an insurance company, a user example might look more like\n this, I want to buy a new XYZ insurance plan.\nYour assistant uses the examples that you provide to build a machine learning model that can recognize the same and similar types of utterances and map them to the appropriate intent.\n\n\nStart with a few intents, and test them as you iteratively expand the scope of the application. +Creating intents Creating intents "Creating intents\n\n\nClick Intents.\n\n\nSelect Create intent.\n\n\nIn the Intent name field, type a name for the intent.\n\nThe intent name can contain letters (in Unicode), numbers, underscores, hyphens, and periods.\nThe name cannot consist of .. or any other string of only periods.\nIntent names cannot contain spaces and must not exceed 128 characters. The following are examples of intent names:\n \n#weather_conditions\n#pay_bill\n#escalate_to_agent\n\n\n\nA number sign # prefix is included in the intent name automatically to help identify the term as an intent. You do not need to add it.\nKeep the name as short as possible for readability in the ""Try it out"" pane and conversation logs.\nOptionally add a description of the intent in the Description field.\n\n\nSelect Create intent to save your intent name.\n\n\n\nNew intent\n\n\n\n\nIn the User example field, type the text of a user example for the intent. An example can be any string up to 1,024 characters in length. The following utterances might be examples for the #pay_bill intent:\n\nI need to pay my bill.\nPay my account balance\nmake a payment\n\nTo learn about the impact of including references to entities in your user examples, see How entity references are treated.\nIntent names and example text can be shown in URLs when an application interacts with watsonx Assistant. Do not include sensitive or personal information in these artifacts.\n\n\nClick Add example to save the user example.\n\n\n\nAdd user example\n\n\n\n\nRepeat the same process to add more examples.\nProvide at least five examples for each intent.\n\n\nWhen you are done adding examples, click the close arrow to finish creating the intent.\n\n\nThe system trains itself on the intent and user examples you added.\nImportant:" +Creating intents Creating intents Intent example data should be representative and typical of data that your users provide. Examples can be collected from actual user data, or from people who are experts in your specific field. The representative and accurate nature of the\n data is important.\nBoth training and test data (for evaluation purposes) should reflect the distribution of intents in real usage. Generally, more frequent intents have relatively more examples, and better response coverage.\nYou can include punctuation in the example text if it appears naturally. If you believe that some users express their intents with examples that include punctuation, and some users will not, include both versions. Generally, the more coverage\n for various patterns, the better the response. +How entity references are treated Creating intents \nHow entity references are treated\nWhen you include an entity mention in a user example, the machine learning model uses the information in different ways in these scenarios:\n\nReferencing entity values and synonyms in intent examples\nAnnotated mentions\nDirectly referencing an entity name in an intent example\n\n +Referencing entity values and synonyms in intent examples Creating intents \nReferencing entity values and synonyms in intent examples\nIf you define, or plan to define, entities that are related to this intent, mention the entity values or synonyms in some of the examples. Doing so helps to establish a relationship between the intent and entities. It is a weak relationship,\n but it does inform the model.\n +Annotated mentions Creating intents \nAnnotated mentions\nAs you define entities, you can annotate mentions of the entity directly from your existing intent user examples. A relationship that you identify in this way between the intent and the entity is not used by the intent classification\n model. However, when you add the mention to the entity, it is also added to that entity as new value. And when you add the mention to an existing entity value, it is also added to that entity value as a new synonym. Intent classification\n does use these types of dictionary references in intent user examples to establish a weak reference between an intent and an entity.\n +Directly referencing an entity name in an intent example Creating intents Directly referencing an entity name in an intent example\nThis approach is advanced. If used, it must be used consistently.\nYou can choose to directly reference entities in your intent examples. For instance, say that you have an entity that is called @PhoneModelName, which contains values Galaxy S8, Moto Z2, LG G6, and Google Pixel 2.\n When you create an intent, for example #order_phone, you might then provide training data as follows:\n\nCan I get a @PhoneModelName?\nHelp me order a @PhoneModelName.\nIs the @PhoneModelName in stock?\nAdd a @PhoneModelName to my order.\n\n\n\n\nIntent definition\n\n\nCurrently, you can only directly reference synonym entities that you define (pattern values are ignored). You cannot use system entities.\nIf you choose to reference an entity as an intent example (for example, @PhoneModelName) anywhere in your training data it cancels the value of using a direct reference (for example, Galaxy S8) in an intent example\n anywhere else. All intents will then use the entity-as-an-intent-example approach. You cannot apply this approach for a specific intent only.\nIn practice, this means that if you have previously trained most of your intents based on direct references (Galaxy S8), and you now use entity references (@PhoneModelName) for just one intent, the change impacts your\n previous training. If you do choose to use @Entity references, you must replace all previous direct references with @Entity references.\nDefining one example intent with an @Entity that has 10 values that are defined for it does not equate to specifying that example intent 10 times. The watsonx Assistant service does not give that much weight to\n that one example intent syntax. +Testing your intents Creating intents "Testing your intents\nAfter you finish creating new intents, you can test the system to see if it recognizes your intents as you expect.\n\n\nClick Try it.\n\n\nIn the ""Try it out"" pane, enter a question or other text string and press Enter to see which intent is recognized. If the wrong intent is recognized, you can improve your model by adding this text as an example to the correct intent.\nIf you make recent changes in your skill, you might see a message that indicates that the system is still retraining. If you see this message, wait until training completes before testing:\nThe response indicates which intent was recognized from your input.\n\n\n\nTest intents\n\n\n\n\nIf the system does not recognize the correct intent, you can correct it. To correct the recognized intent, select the displayed intent and then select the correct intent from the list. After your correction is submitted, the system automatically\n retrains itself to incorporate the new data.\n\n\n\nCorrecting an intent\n\n\n\n\nIf the input is unrelated to any of the intents in your application, you can teach your assistant that by selecting the displayed intent, and then clicking Mark as irrelevant.\n\n\n\nMark as irrelevnt\n\n\n\n\nIf your intents are not being correctly recognized, consider making the following kinds of changes:\n\nAdd the unrecognized text as an example to the correct intent.\nMove existing examples from one intent to another.\nConsider whether your intents are too similar, and redefine them." +Absolute scoring Creating intents \nAbsolute scoring\nThe watsonx Assistant service scores each intent’s confidence independently, not in relation to other intents. This approach adds flexibility; multiple intents can be detected in a single user input. It also means that the system might not return\n an intent at all. If the top intent has a low confidence score (less than 0.2), the top intent is included in the intents array that is returned by the API, but any nodes that condition on the intent are not triggered. If you want to detect\n the case when no intents with good confidence scores were detected, use the irrelevant special condition in your dialog node.\nAs intent confidence scores change, your dialogs might need restructuring. For example, if a dialog node uses an intent in its condition, and the intent's confidence score starts to consistently drop below 0.2, the dialog node stops being processed.\n If the confidence score changes, the behavior of the dialog can also change.\n +Intent limits Creating intents \nIntent limits\nThe number of intents and examples you can create depends on your watsonx Assistant plan type:\n\nPlan details\n\n\nPlan\nIntents per skill\nExamples per skill\n\n\n\n\nEnterprise\n2,000\n25,000\n\n\nPremium (legacy)\n2,000\n25,000\n\n\nPlus\n2,000\n25,000\n\n\nTrial\n100\n25,000\n\n\nLite\n100\n25,000\n\n\n\n +Editing intents Creating intents \nEditing intents\nYou can click any intent in the list to open it for editing. You can make the following changes:\n\nRename the intent\nDelete the intent\nAdd, edit, or delete examples\nMove examples to a different intent\n\nTo move or delete an example, click the checkbox that is associated with it, and then click Move or Delete.\n\n\n\nMove or delete example\n\n\n +Searching intents Creating intents \nSearching intents\nUse the Search feature to find user examples, intent names, and descriptions.\n\n\nFrom the Intents page, click the Search icon .\n\n\nSubmit a search term or phrase. You can also select Include partial match.\nThe first time that you search for something, you might get a message that says the content is being indexed. If so, wait a minute, and then resubmit the search term.\nIntents that contain your search term are displayed.\n\n\n\nSearch results\n\n\n\n\n +Downloading intents Creating intents \nDownloading intents\nYou can download a number of intents to a CSV file, so you can then upload and reuse them in another watsonx Assistant application.\n\nGo to the Intents page.\n\n\n\nTo download all intents, meaning the intents that are listed on this and any additional pages, do not select any individual intents. Instead, click the Download all intents icon .\n\n\nTo download the intents that are listed on the current page only, select the checkbox in the header. This action selects all of the intents on the current page. Then, click the Download button.\n\n\nTo download one or more specific intents, select the intents that you want to download, and then click the Download button.\n\n\n\nSpecify the name and location in which to store the CSV file that is generated, and then click Save.\n\n +Uploading intents and examples Creating intents Uploading intents and examples\nIf you have many intents and examples, you might find it easier to upload them from a comma-separated value (CSV) file than to define them one by one. Be sure to remove any personal data from the user examples that you include in the file.\n\n\nCollect the intents and examples into a CSV file, or export them from a spreadsheet to a CSV file. The required format for each line in the file is as follows:\n,\n\nwhere is the text of a user example, and is the name of the intent you want the example to match. For example:\nTell me the current weather conditions.,weather_conditions\nIs it raining?,weather_conditions\nWhat's the temperature?,weather_conditions\nWhere is your nearest location?,find_location\nDo you have a store in Raleigh?,find_location\n\nImportant: Save the CSV file with UTF-8 encoding and no byte order mark (BOM).\n\n\nFrom the Intents page, click the Upload intents icon .\n\n\nDrag a file or browse to select a file from your computer.\nImportant: The maximum CSV file size is 10 MB. If your CSV file is larger, consider splitting it into multiple files and uploading them separately.\n\n\nClick Upload intents.\nThe file is validated and uploaded, and the system trains itself on the new data.\n\n\nYou can view the uploaded intents and the corresponding examples on the Intents tab. You might need to refresh the page to see the new intents and examples. +Deleting intents Creating intents \nDeleting intents\nYou can select a number of intents for deletion.\nBy deleting intents that you are also deleting all associated examples, and these items cannot be retrieved later. All dialog nodes that reference these intents must be updated manually to no longer reference the deleted content.\n\n\nGo to the Intents page\n\n\nTo delete all intents, meaning the intents that are listed on this and any additional pages, do not select any individual intents. Instead, click the Delete all intents icon. \n\n\nTo delete the intents that are listed on the current page only, select the checkbox in the header. This action selects all of the intents that are listed on the current page. Click Delete.\n\n\nTo delete one or more specific intents, select the intents that you want to delete, and then click Delete.\n\n\n\n\n +Accessibility Accessibility \nAccessibility\nIBM strives to provide products with usable access for everyone, regardless of age or ability.\nIBM® watsonx™ Assistant uses standard Windows navigation keys.\n +Accessibility features in the product documentation Accessibility \nAccessibility features in the product documentation\nAccessibility features help people with a physical disability, such as restricted mobility or limited vision, or with other special needs, use information technology products successfully.\nThe accessibility features in this product documentation allow users to do the following:\n\nUse screen-reader software and digital speech synthesizers to hear what is displayed on the screen. Consult the product documentation of the assistive technology for details on using assistive technologies with HTML-based information.\nUse screen magnifiers to magnify what is displayed on the screen.\nOperate specific or equivalent features by using only the keyboard.\n\nThe documentation content is published in the IBM Cloud Docs site. For information about the accessibility of the site, see Accessibility features for IBM Cloud.\n +Anatomy of a message Anatomy of a message "Anatomy of a message\nA single /message API call is equivalent to a single turn in a conversation, which consists of a message that is submitted by a customer and a corresponding response from your assistant.\nEach reply that a customer makes in response to a prompt from the assistant is passed as an independent /message API call.\nThe body of the /message API call request and response includes the following objects:\n\n\ncontext: Contains variables that are meant to be persisted. For the dialog to reference information that is submitted by the user, you must store the information in the context object. For example, the dialog can collect the user's\n name and then refer to the user by name in subsequent nodes. The following example shows how the context object is represented in the dialog JSON editor:\n{\n ""context"" : {\n ""user_name"" : """"\n }\n\nFor more information, see Retaining information across dialog turns.\n\n\ninput: The string of text that was submitted by the user. The text string can contain up to 2,048 characters. The following example shows how the input object is represented in the dialog JSON editor:\n{\n ""input"" : {\n ""text"" : ""Where's your nearest store?""\n }\n\n\n\noutput: The dialog response to return to the user. The following example shows how the output object is represented in the dialog JSON editor:\n{\n""output"": {\n ""generic"": [\n {\n ""values"": [\n {\n ""text"": ""This is my response text.""\n }\n ],\n ""response_type"": ""text"",\n ""selection_policy"": ""sequential""\n }\n ]\n}\n}\n\n\n\nIn the resulting API /message response, the text response is formatted as follows:\n{\n ""text"": ""This is my response text."",\n ""response_type"": ""text""\n}" +Anatomy of a message Anatomy of a message You can define more response types other than a text response. See Responses for more details.\nFor more information about the /message API call, see the API reference.\nFor information about how to refer to these message objects within a conversation, see Expressions for accessing objects. +Viewing API call details Anatomy of a message \nViewing API call details\nAs you test your conversation, you might want to know what the underlying API calls look like that are being returned from the service. You can use the developer tools that are provided by your web browser to inspect them.\nFrom Chrome, for example:\n\n\nOpen the developer tools.\n\n\nOpen the Network tool. The Name section lists multiple API calls.\n\n\nClick the message call that is associated with your test utterance\n\n\nClick the Response column to see the API response body. It lists the intents and entities that were recognized in the user input with their confidence scores. It also lists the values of context variables at the time of\n the call.\n\n\n\n\n\nUse Chrome developer tools to see API response\n\n\n\nTo view the response body in structured format, click the Preview column.\n\n +Personalizing the dialog with context Personalizing the dialog with context \nPersonalizing the dialog with context\nTo personalize the conversation, your assistant can collect information from the customer and then refer to it later in the conversation.\n +Retaining information across dialog turns Personalizing the dialog with context \nRetaining information across dialog turns\nThe dialog is stateless, meaning that it does not retain information from one interaction with the user to the next. When you add a dialog to an assistant and deploy it, the assistant saves the context from one message call, and then resubmits\n it on the next request throughout the current session. The current session lasts while a user interacts with the assistant plus the designated session inactivity time frame. The maximum session inactivity time allowed ranges from 5 minutes\n to 7 days, depending on your plan type. If you do not add the dialog to an assistant, it is your responsibility as the custom application developer to maintain any continuing information that the application needs.\nThe application can pass information to the dialog, and the dialog can update this information and pass it back to the application, or to a subsequent node. The dialog does so by using context variables.\n +Context variables Personalizing the dialog with context \nContext variables\nA context variable is a variable that you define in a node. You can specify a default value for it. Then, other nodes, application logic, or user input can set or change the value of the context variable.\nYou can condition against context variable values by referencing a context variable from a dialog node condition to determine whether to execute a node. You can also reference a context variable from dialog node response conditions to show different\n responses depending on a value that is provided by an external service or by the user.\nLearn more:\n\nPassing context from the application\nPassing context from node to node\nDefining a context variable\nCommon context variable tasks\nDeleting a context variable\nUpdating a context variable\nHow context variables are processed\nOrder of operation\nAdding context variables to a node with slots\n\n +Passing context from the application Personalizing the dialog with context \nPassing context from the application\nPass information from the application to the dialog by setting a context variable and passing the context variable to the dialog.\nFor example, your application can set a $time_of_day context variable, and pass it to the dialog that can use the information to tailor the greeting it shows to the user.\n\n\n\nTime of day context\n\n\nIn this example, the dialog knows that the application sets the variable to one of these values: morning, afternoon, or evening. It can check for each value, and depending on which value is present, return the appropriate\n greeting. If the variable is not passed or has a value that does not match one of the expected values, then a more generic greeting is displayed to the user.\n +Passing context from node to node Personalizing the dialog with context \nPassing context from node to node\nThe dialog can also add context variables to pass information from one node to another or to update the values of context variables. As the dialog asks for and gets information from the user, it can track the information and reference it later\n in the conversation.\nFor example, in one node you might ask users for their name, and in a later node address them by name.\n\n\n\nPass context from one node to another\n\n\nIn this example, the system entity @name is used to extract the user's name from the input if the user provides one. In the JSON editor, the username context variable is defined and set to the @name value. In a subsequent node, the $username\n context variable is included in the response to address the user by name.\n +Defining a context variable Personalizing the dialog with context "Defining a context variable\nDefine a context variable by adding the variable name to the Variable field and adding a default value for it to the Value field in the node's edit view.\n\n\nClick to open the dialog node to which you want to add a context variable.\n\n\nGo to the Assistant responds section and click the menu icon .\n\n\nClick Open context editor.\n\n\nAdd the variable name and value pair to the Variable and Value fields.\n\n\nThe name can contain any upper- and lowercase alphabet characters, numeric characters (0-9), and underscores.\n\n\nThe value can be any supported JSON type, such as a simple string variable, a number, a JSON array, or a JSON object.\n\n\n\n\nThe following table shows some examples of how to define name and value pairs for different types of values:\n\nDefine name and value pairs\n\n\nVariable\nValue\nValue Type\n\n\n\n\ndessert\n""cake""\nString\n\n\nage\n18\nNumber\n\n\ntoppings_array\n[""onions"",""olives""]\nJSON Array\n\n\nfull_name\n{""first"":""John"",""last"":""Doe""}\nJSON Object\n\n\n\nThen, to refer to these context variables, use the syntax $name where name is the name of the context variable that you defined.\nFor example, you might specify the following expression as the dialog response:\nThe customer, $age-year-old , wants a pizza with , and then $dessert.\nThe resulting output is displayed as follows:\nThe customer, 18-year-old John, wants a pizza with onions and olives, and then cake.\nYou can use the JSON editor to define context variables also. You might prefer to use the JSON editor if you want to add a complex expression as the variable value. See Context variables in the JSON editor for more details." +Common context variable tasks Personalizing the dialog with context Common context variable tasks\nTo store the entire string that was provided by the user as input, use input.text:\n\nCapturing user input\n\n\nVariable\nValue\n\n\n\n\nrepeat\n\n\n\n\nFor example, the user input might be, I want to order a device. If the node response is, You said: $repeat, then the response would be displayed as, You said: I want to order a device.\nTo store the value of an entity in a context variable, use this syntax:\n\nCapturing an entity mention\n\n\nVariable\nValue\n\n\n\n\nplace\n@place\n\n\n\nFor example, the user input might be, I want to go to Paris. If your @place entity recognizes Paris, then your assistant saves Paris in the $place context variable.\nTo store the value of a string that you extract from the user's input, you can include a SpEL expression that uses the extract method to apply a regular expression to the user input. The following expression extracts a number\n from the user input, and saves it to the $number context variable.\n\nUsing a String method\n\n\nVariable\nValue\n\n\n\n\nnumber\n\n\n\n\nTo store the value of a pattern entity, append .literal to the entity name. Using this syntax make sure that the exact span of text from user input that matched the specified pattern is stored in the variable.\n\nCapturing a pattern entity value\n\n\nVariable\nValue\n\n\n\n\nemail\n\n\n\n\nFor example, the user input is Contact me at joe@example.com. Your entity that is named @email recognizes the name@domain.com email format. By configuring the context variable to store @email.literal,\n you indicate that you want to store the part of the input that matched the pattern. If you omit the .literal property from the value expression, then the entity value name that you specified for the pattern is returned instead\n of the segment of user input that matched the pattern. +Deleting a context variable Personalizing the dialog with context \nDeleting a context variable\nTo delete a context variable, set the variable to null.\n\nNulling a context variable\n\n\nVariable\nValue\n\n\n\n\norder_form\nnull\n\n\n\n +Updating a context variable value Personalizing the dialog with context \nUpdating a context variable value\nTo update a context variable's value, define a context variable with the same name as the previous context variable, but this time, specify a different value for it.\nWhen more than one node sets the value of the same context variable, the value for the context variable can change over the course of a conversation with a user. The value that is applied depends on which node is being triggered by the user\n in the course of the conversation. The value specified for the context variable in the last node that is processed overwrites any values that were set for the variable by nodes that were processed previously.\nFor information about how to update the value of a context variable when the value is a JSON object or JSON array data type, see Updating a context variable value in JSON\n +How context variables are processed Personalizing the dialog with context \nHow context variables are processed\nWhere you define the context variable matters. The context variable is not created and set to the value that you specify for it until your assistant processes the part of the dialog node where you defined the context variable. In most cases,\n you define the context variable as part of the node response. When you do so, the context variable is created and given the specified value when your assistant returns the node response.\nFor a node with conditional responses, the context variable is created and set when the condition for a specific response is met and that response is processed. For example, if you define a context variable for conditional response #1 and\n your assistant processes response #2, then the variable that you defined for conditional response #1 is not set.\nFor information about adding context variables that you want your assistant to set as a user interacts with a node with slots, see Adding context variables to a node with slots.\n +Order of operation Personalizing the dialog with context Order of operation\nWhen you define multiple variables to be processed together, the order in which you define them does not determine the order in which they are evaluated by your assistant. Your assistant evaluates the variables in random order. Don't set a\n value in the first context variable and expect to use it in the second variable because the first context variable might not be executed before the second one. For example, do not use two context variables to implement logic that checks\n whether the user input contains the word Yes in it.\n\nUsing two context variables to check for a value in user input\n\n\nVariable\nValue\n\n\n\n\nuser_input\n\n\n\ncontains_yes\n\n\n\n\nInstead, use a slightly more complex expression to avoid having to rely on the value of the first variable in your list (user_input) being evaluated before the second variable (contains_yes) is evaluated.\n\nUsing a single context variable\n\n\nVariable\nValue\n\n\n\n\ncontains_yes\n +Adding context variables to a node with slots Personalizing the dialog with context "Adding context variables to a node with slots\nFor more information about slots, see Gathering information with slots.\nTo add a context variable that is processed after a response condition for a slot is met:\n\nOpen the node with slots in the edit view.\nClick the Customize slot icon .\nClick the Options icon , and then select Enable condition.\nClick the Customize handler icon next to the response with which you want to associate the context variable.\nClick the Options icon in the Assistant responds section, and then click Open context editor.\nAdd the variable name and value pair to the Variable and Value fields.\n\nTo add a context variable that is set or updated after a slot condition is met, complete the following steps:\n\nOpen the node with slots in the edit view.\nClick the Customize slot icon .\nClick the Options icon , and then select Enable condition.\nAdd the variable name and value pair in JSON format.\n\n{\n""time_of_day"": ""morning""\n}\n\nYou can't use the context editor to define context variables that are set during this phase of dialog node evaluation. You must use the JSON editor instead. For more information about using the JSON editor, see Context variables in the JSON editor." +Context variables in the JSON editor Personalizing the dialog with context "Context variables in the JSON editor\nYou can also define a context variable in the JSON editor. Use the JSON editor if you are defining a complex context variable and want to be able to see the full SpEL expression as you add or change it.\nThe name and value pair must meet these requirements:\n\n\nThe name can contain any upper- and lowercase alphabet characters, numeric characters (0-9), and underscores.\nYou can include other characters, such as periods and hyphens, in the name. However, if you do, then you must specify the shorthand syntax $(variable-name) every time you reference the variable. See Expressions for accessing objects for more details.\n\n\nThe value can be any supported JSON type, such as a simple string variable, a number, a JSON array, or a JSON object.\n\n\nThe following JSON sample defines values for the $dessert string, $toppings_array array, $age number, and $full_name object context variables:\n{\n ""context"": {\n ""dessert"": ""cake"",\n ""toppings_array"": [\n ""onions"",\n ""olives""\n ],\n ""age"": 18,\n ""full_name"": {\n ""first"": ""Jane"",\n ""last"": ""Doe""\n }\n },\n ""output"":{}\n}\n\nTo define a context variable in JSON format, complete the following steps:\n\n\nClick to open the dialog node to which you want to add the context variable.\nAny existing context variable values that are defined for this node are displayed in a set of corresponding Variable and Value fields. If you do not want them to be displayed in the edit view of the node,\n you must close the context editor. You can close the editor from the same menu that is used to open the JSON editor; the following steps describe how to access the menu." +Context variables in the JSON editor Personalizing the dialog with context "Click the Options icon for the assistant response, and then click Open JSON editor.\nIf the Multiple conditioned responses setting is enabled for the node, then you must first click the Customize response icon for the response with\n which you want to associate the context variable.\n\n\nAdd a ""context"":{} block if one is not present.\n{\n ""context"":{},\n ""output"":{}\n}\n\n\n\nIn the context block, add a ""name"" and ""value"" pair for each context variable that you want to define.\n{\n ""context"":{\n ""name"": ""value""\n},\n ""output"": {}\n}\n\nIn this example, a variable that is named new_variable is added to a context block that already contains a variable.\n{\n ""context"":{\n ""existing_variable"": ""value"",\n ""new_variable"":""value""\n }\n}\n\nTo reference the context variable, use the syntax $name where name is the name of the context variable that you defined. For example, $new_variable.\n\n\nLearn more:\n\nDeleting a context variable in JSON\nUpdating a context variable value in JSON\nSetting one context variable equal to another" +Deleting a context variable in JSON Personalizing the dialog with context "\nDeleting a context variable in JSON\nTo delete a context variable, set the variable to null.\n{\n ""context"": {\n ""order_form"": null\n }\n}\n\nIf you want to remove all trace of the context variable, you can use the JSONObject.remove(string) method to delete it from the context object. However, you must use a variable to perform the removal. Define the new variable in the message\n output so it isn't saved beyond the current call.\n{\n ""output"": {\n ""text"" : {},\n ""deleted_variable"" : """"\n }\n}\n\nAlternatively you can delete the context variable in your application logic.\n" +Updating a context variable value in JSON Personalizing the dialog with context \nUpdating a context variable value in JSON\nIn general, if a node sets the value of a context variable that is already set, then the previous value is overwritten by the new value.\n +Updating a complex JSON object Personalizing the dialog with context "\nUpdating a complex JSON object\nPrevious values are overwritten for all JSON types except a JSON object. If the context variable is a complex type such as JSON object, a JSON merge procedure is used to update the variable. The merge operation adds any newly defined properties\n and overwrites any existing properties of the object.\nIn this example, a name context variable is defined as a complex object.\n{\n ""context"": {\n ""complex_object"": {\n ""user_firstname"" : ""Paul"",\n ""user_lastname"" : ""Pan"",\n ""has_card"" : false\n }\n }\n}\n\nA dialog node updates the context variable JSON object with the following values:\n{\n ""complex_object"": {\n ""user_firstname"": ""Peter"",\n ""has_card"": true\n }\n}\n\nThe result is this context:\n{\n ""complex_object"": {\n ""user_firstname"": ""Peter"",\n ""user_lastname"": ""Pan"",\n ""has_card"": true\n }\n}\n\n" +Updating arrays Personalizing the dialog with context "Updating arrays\nIf your dialog context data contains an array of values, you can update the array by appending values, removing a value, or replacing all the values.\nChoose one of these actions to update the array. In each case, we see the array before the action, the action, and the array after the action is applied.\n\n\nAppend: To add values to the end of an array, use the append method.\nFor this dialog runtime context:\n{\n ""context"": {\n ""toppings_array"": [""onion"", ""olives""]\n }\n}\n\nMake this update:\n{\n ""context"": {\n ""toppings_array"": """"\n }\n}\n\nResult:\n{\n ""context"": {\n ""toppings_array"": [""onion"", ""olives"", ""ketchup"", ""tomatoes""]\n }\n}\n\n\n\nRemove: To remove an element, use the remove method and specify its value or position in the array.\n\n\nRemove by value removes an element from an array by its value.\nFor this dialog runtime context:\n{\n ""context"": {\n ""toppings_array"": [""onion"", ""olives""]\n }\n}\n\nMake this update:\n{\n ""context"": {\n ""toppings_array"": """"\n }\n}\n\nResult:\n{\n ""context"": {\n ""toppings_array"": [""olives""]\n }\n}\n\n\n\nRemove by position: Removing an element from an array by its index position:\nFor this dialog runtime context:\n{\n ""context"": {\n ""toppings_array"": [""onion"", ""olives""]\n }\n}\n\nMake this update:\n{\n ""context"": {\n ""toppings_array"": """"\n }\n}\n\nResult:\n{\n ""context"": {\n ""toppings_array"": [""olives""]\n }\n}" +Updating arrays Personalizing the dialog with context "Overwrite: To overwrite the values in an array, set the array to the new values:\nFor this dialog runtime context:\n{\n ""context"": {\n ""toppings_array"": [""onion"", ""olives""]\n }\n}\n\nMake this update:\n{\n ""context"": {\n ""toppings_array"": [""ketchup"", ""tomatoes""]\n }\n}\n\nResult:\n{\n ""context"": {\n ""toppings_array"": [""ketchup"", ""tomatoes""]\n }\n}" +Setting one context variable equal to another Personalizing the dialog with context "Setting one context variable equal to another\nWhen you set one context variable equal to another context variable, you define a pointer from one to the other. Later, if the value of one of the variables changes, then the value of the other variable is changed also.\nFor example, if you specify a context variable as follows, then when the value of either $var1 or $var2 later changes, the value of the other changes too.\n\nSetting one context variable equal to another\n\n\nVariable\nValue\n\n\n\n\nvar2\nvar1\n\n\n\nDo not set one variable equal to another to capture a point in time value. With arrays, if you want to capture an array value that is stored in a context variable and use it later, create a new variable based on the current value of the variable\n instead.\nFor example, to create a copy of the values of an array at a certain point, add an array that is populated with the values for the existing array. To do so, you can use the following syntax:\n{\n""context"": {\n ""var2"": """"\n }\n }" +FAQs for watsonx Assistant FAQs for watsonx Assistant \nFAQs for watsonx Assistant\nFind answers to frequently-asked questions and quick fixes for common problems.\n +FAQs about watsonx Assistant FAQs for watsonx Assistant \nFAQs about watsonx Assistant\n +What is watsonx Assistant? FAQs for watsonx Assistant \nWhat is watsonx Assistant?\nIBM® watsonx™ Assistant is an improved way to build, publish, and improve virtual assistants. You use actions to build conversations. Actions are a simple way for anyone to create assistants. For more information, see the Getting Started guide or the documentation.\n +Why can't I see the assistants that I made with the classic experience in watsonx Assistant? FAQs for watsonx Assistant \nWhy can't I see the assistants that I made with the classic experience in watsonx Assistant?\nIBM® watsonx™ Assistant is a clean slate in the same IBM Cloud instance as your classic experience. Assistants that you created in one experience don't appear in the other. However, you can switch back and forth between experiences without\n losing any work. For more information, see Switching between watsonx Assistant and the classic experience.\n +What happens when I switch between the classic experience and watsonx Assistant? FAQs for watsonx Assistant \nWhat happens when I switch between the classic experience and watsonx Assistant?\nThe assistants that you create in one experience don't transfer to the other. However, you can switch experiences, return to your work, and create or use assistants. You don't lose anything by switching. Changing experiences doesn't affect\n other users in the same instance. For more information, see Switching between watsonx Assistant and the classic experience.\n +Is the classic experience still available? FAQs for watsonx Assistant \nIs the classic experience still available?\nIBM has no plans to discontinue the classic experience. However, we encourage you to explore the benefits and capabilities in watsonx Assistant. For more information, see the Getting Started guide. You can also continue to use dialog in watsonx Assistant. For more information, see Migrating to watsonx Assistant.\n +Where are the search skill and channel integrations in watsonx Assistant? FAQs for watsonx Assistant \nWhere are the search skill and channel integrations in watsonx Assistant? \nIn the left navigation, click Integrations . On the Integrations page, you can add search, channel, and extension integrations to your assistant. For more information,\n see Adding integrations.\n +Where is the Assistant ID found in the new product experience? FAQs for watsonx Assistant \nWhere is the Assistant ID found in the new product experience?\nThe assistant ID can be found in Assistant settings.\nIn Assistant settings, the assistant ID is in the Assistant IDs and API details section.\n +What do the draft and live tags mean? FAQs for watsonx Assistant \nWhat do the draft and live tags mean?\nA Draft tag indicates that the information is linked to your draft environment, which means that you can preview these updates but they are not visible to your users. A Live tag indicates that the information is linked\n to your live environment, which means that the content is available to your users to interact with.\nFor more information, see Environments.\n +Why can't I log in? FAQs for watsonx Assistant \nWhy can't I log in?\nIf you can't log in to a service instance or see messages about tokens, such as unable to fetch access token or 400 bad request - header or cookie too large, it might mean that you need to clear your browser cache.\n Open a private browser window, and then try again.\n\nIf the private browsing window fixes the issue, then consider always using a private window or clear the cache of your browser. You can typically find an option for clearing the cache or deleting cookies in the browser's privacy and security\n settings.\nIf the private browsing window doesn't fix the issue, then try deleting the API key for the instance and creating a new one.\n\n +Why am I being asked to log in repeatedly? FAQs for watsonx Assistant \nWhy am I being asked to log in repeatedly?\nIf you keep getting messages, such as you are getting redirected to login, it might be due to one of the following things:\n\nThe Lite plan that you were using expired. Lite plans expire if they are not used within a 30-day span. To begin again, log in to IBM Cloud and create a new service instance of watsonx Assistant.\nAn instance is locked when you exceed the plan limits for the month. To log in successfully, wait until the start of the next month when the plan limit totals are reset.\n\n +Why don't I see the Analytics page? FAQs for watsonx Assistant \nWhy don't I see the Analytics page?\nTo view the Analytics page, you must have a service role of Manager and a platform role of at least Viewer. For more information about access roles and how to request an access role change, see Managing access to resources.\n +Why am I unable to view the API details, API key, or service credentials? FAQs for watsonx Assistant \nWhy am I unable to view the API details, API key, or service credentials?\nIf you cannot view the API details or service credentials, it is likely that you do not have Manager access to the service instance in which the resource was created. Only people with Manager access to the instance can use the service credentials.\n +Why can't I edit intents, entities, or dialog nodes? FAQs for watsonx Assistant \nWhy can't I edit intents, entities, or dialog nodes?\nTo edit a dialog, you must have Writer service access to the service instance and a platform role of at least Viewer. For more information about access roles and how to request an access role change, see Managing access to resources.\n +Can I export the user conversations from the Analytics page? FAQs for watsonx Assistant \nCan I export the user conversations from the Analytics page?\nYou cannot directly export conversations from the conversation page. However, you can use the /logs API to list events from the transcripts of conversations that occurred between your users and your assistant. For more information,\n see the V2 API reference. Or, you can use a Python script to export logs. For more information, see export_logs_py.\n +Can I export and import dialog nodes? FAQs for watsonx Assistant \nCan I export and import dialog nodes?\nNo, you cannot export and import dialog nodes from the product user interface.\nIf you want to copy dialog nodes from one dialog into another dialog, follow these steps:\n\nDownload as JSON files both the dialog that you want to copy the dialog nodes from and the dialog that you want to copy the nodes to.\nIn a text editor, open the JSON file for the dialog that you want to copy the dialog nodes from.\nFind the dialog_nodes array, and copy it.\nIn a text editor, open the JSON file for the dialog skill that you want to copy the dialog nodes to, and then paste the dialog_nodes array into it.\nImport the JSON file that you edited in the previous step to create a new dialog skill with the dialog nodes you wanted.\n\n +Is it possible to recover a deleted dialog? FAQs for watsonx Assistant \nIs it possible to recover a deleted dialog?\nRegularly back up data to prevent problems that might arise from inadvertent deletions. If you do not have a backup, there is a short window of time during which a deleted dialog might be recoverable. Immediately following the deletion, open a case with Support to determine if the data can be recovered. Include the following information in your case:\n\nSkill ID\nInstance ID or name\nRegion where the service instance is hosted from which the dialog was deleted\n\n +Can I change my plan to a Lite plan? FAQs for watsonx Assistant \nCan I change my plan to a Lite plan?\nNo, you cannot change from a Trial, Plus, or Standard plan to a Lite plan. And you cannot upgrade from a Trial to a Standard plan.\n +How many Lite plan instances of watsonx Assistant can I create? FAQs for watsonx Assistant \nHow many Lite plan instances of watsonx Assistant can I create?\nYou can have only one Lite plan instance of watsonx Assistant per resource group.\n +How long are log files kept? FAQs for watsonx Assistant \nHow long are log files kept?\nThe length of time for which messages are retained depends on your service plan. For more information, see Log limits.\n +How do I create a webhook? FAQs for watsonx Assistant \nHow do I create a webhook?\nTo define a webhook and add its details, go to the Live environment page and open the Environment settings page. From the Environment settings page, click Webhooks > Pre-message webhook.\n You can add details about your webhook. For more information, see Making a call before processing a message.\n diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/wa_elasticsearch_result.png b/agent_knowledge/elasticsearch-install-and-setup/assets/wa_elasticsearch_result.png new file mode 100644 index 0000000..83c2e5f Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/wa_elasticsearch_result.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawl_with_chunking_document_example.png b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawl_with_chunking_document_example.png new file mode 100644 index 0000000..6893905 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawl_with_chunking_document_example.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_foreach_processor.png b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_foreach_processor.png new file mode 100644 index 0000000..77e760b Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_foreach_processor.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_ingest_pipeline.png b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_ingest_pipeline.png new file mode 100644 index 0000000..7377a89 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_ingest_pipeline.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_ingest_pipeline_custom.png b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_ingest_pipeline_custom.png new file mode 100644 index 0000000..a4ec28f Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_ingest_pipeline_custom.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_overview_with_crawl_requests.png b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_overview_with_crawl_requests.png new file mode 100644 index 0000000..5909683 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_overview_with_crawl_requests.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_script_processor.png b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_script_processor.png new file mode 100644 index 0000000..58317b9 Binary files /dev/null and b/agent_knowledge/elasticsearch-install-and-setup/assets/web_crawler_script_processor.png differ diff --git a/agent_knowledge/elasticsearch-install-and-setup/federated_search.md b/agent_knowledge/elasticsearch-install-and-setup/federated_search.md new file mode 100644 index 0000000..5b66558 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/federated_search.md @@ -0,0 +1,130 @@ +# Federated Search in Elasticsearch + +This guide dives into performing a single search query across multiple indices within an Elasticsearch cluster. It assumes these indices share the same relevant fields for your search results. + +## Methods for Multi-Index Search + +There are several approaches to searching across multiple indices in Elasticsearch: + +### 1. Comma-Separated Indices + +This method involves listing all relevant indices separated by commas in the search URL. Here's an example using elser model for semantic search across `index-1` and `index-2`: + +```bash +curl -X GET "${ES_URL}/index-1,index-2/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_1", + "model_text":"how to set up custom extension?" + } + } + } +}' +``` + +### 2. Index Patterns + +Index patterns allow you to target multiple indices using a wildcard pattern. Here's how to search across `index-1` and `index-2` using the same example: + +```bash +curl -X GET "${ES_URL}/index-*/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_1", + "model_text":"how to set up custom extension?" + } + } + } +}' +``` + +### 3. Aliases +You can create an alias that represents multiple indices, and then search against that alias. This allows you to dynamically change which indices are included in the search without modifying the search query. + +```bash +curl -X POST "${ES_URL}/_aliases" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "actions": [ + { "add": { "index": "index-1", "alias": "alias-1" }}, + { "add": { "index": "index-2", "alias": "alias-1" }} + ] +}' +``` + +You can then search against the `alias-1` alias, which includes both `index-1` and `index-2`: + +```bash +curl -X GET "${ES_URL}/alias-1/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_1", + "model_text":"how to set up custom extension?" + } + } + } +}' +``` + +### 4. Searching All Indices + +To search across all indices in the cluster, you can omit the index name entirely or use `_all` or `*`: + +```bash +# Omit index name +curl -X GET "${ES_URL}/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_1", + "model_text":"how to set up custom extension?" + } + } + } +}' + +# Using _all +curl -X GET "${ES_URL}/_all_/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_1", + "model_text":"how to set up custom extension?" + } + } + } +}' + +# Using * +curl -X GET "${ES_URL}/*/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "query":{ + "text_expansion":{ + "ml.tokens":{ + "model_id":".elser_model_1", + "model_text":"how to set up custom extension?" + } + } + } +}' +``` + +## Using these Methods with Agent Knowledge in watsonx Orchestrate + +For similar functionality with watsonx Orchestrate's Agent Knowledge, follow the documentation for [Connecting to an Elasticsearch content repository](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=agents-connecting-elasticsearch-content-repository). The key difference is specifying your `Elasticsearch index` field using one of the methods described above. For example: + + diff --git a/agent_knowledge/elasticsearch-install-and-setup/how_to_configure_advanced_elasticsearch_settings.md b/agent_knowledge/elasticsearch-install-and-setup/how_to_configure_advanced_elasticsearch_settings.md new file mode 100644 index 0000000..d6700f5 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/how_to_configure_advanced_elasticsearch_settings.md @@ -0,0 +1,322 @@ +# How to configure the advanced Elasticsearch settings +This guide shows how to configure the advanced Elasticsearch settings for Agent Knowledge in watsonx Orchestrate. Specifically, there are two types of settings under `Advanced Elasticsearch settings`: custom filters and query body. + + + +## Table of contents: +* [How to configure the custom filters](#how-to-configure-the-custom-filters) + * [Filter object examples](#filter-object-examples) + * [AND](#and) + * [OR](#or) + * [NOT](#not) + * [(a AND b) OR c](#a-and-b-or-c) +* [How to configure the query body](#how-to-configure-the-query-body) + * [Semantic search with ELSER](#semantic-search-with-elser) + * [KNN dense vector search](#knn-dense-vector-search) + * [Using a nested query to search over nested documents with ELSER](#using-a-nested-query-to-search-over-nested-documents-with-elser) + * [Hybrid search with combined keyword search and dense vector search](#hybrid-search-with-combined-keyword-search-and-dense-vector-search) + +## How to configure the custom filters +You can configure custom filters for your Elasticsearch knowledge source under `Advanced Elasticsearch settings`. These custom filters will be used as global filters and apply to all user queries. If a custom query body is provided, the `$FILTER` variable needs to be included in the query body to use the custom filters. For example, + + + +### Filter object examples +Using the custom filters, you can achieve advanced filtering use cases for search and conversational search. Here are some examples: + +**Note: The below examples assume `title`, `text`, and `id` fields are available in your Elasticsearch index. Among them, `title` and `text` are text type fields, while `id` is a keyword type field.** + +#### AND +```json +[ + { + "match": { + "title": "A_keyword_in_title" + } + }, + { + "match": { + "text": "A_keyword_in_text" + } + }, + { + "match": { + "id": "A_specific_ID" + } + } +] +``` +This filter object will filter the search results using the following conditions: +* `title` contains "A_keyword_in_title", AND +* `text` contains "A_keyword_in_text", AND +* `id` is equal to "A_specific_ID". + + +#### OR +```json +[ + { + "bool": { + "should": [ + { + "match": { + "title": "A_keyword_in_title" + } + }, + { + "match": { + "text": "A_keyword_in_text" + } + }, + { + "match": { + "id": "A_specific_ID" + } + } + ] + } + } +] +``` +This filter object will filter the search results using the following conditions: +* `title` contains "A_keyword_in_title", OR +* `text` contains "A_keyword_in_text", OR +* `id` is equal to "A_specific_ID". + +#### NOT +```json +[ + { + "bool": { + "must_not": [ + { + "match": { + "title": "A_keyword_in_title" + } + }, + { + "match": { + "text": "A_keyword_in_text" + } + }, + { + "match": { + "id": "A_specific_ID" + } + } + ] + } + } +] +``` +This filter object will filter the search results using the following conditions: +* `title` does not contain "A_keyword_in_title", AND +* `text` does not contain "A_keyword_in_text", AND +* `id` is not equal to "A_specific_ID". + +#### (a AND b) OR c +```json +[ + { + "bool": { + "should": [ + { + "match": { + "id": "A_specific_ID" + } + } + { + "bool": { + "filter": [ + { + "match": { + "title": "A_keyword_in_title" + } + }, + { + "match": { + "text": "A_keyword_in_text" + } + } + ] + } + } + ] + } + } +] +``` +This filter object will filter the search results using the following conditions: +* `title` contains "A_keyword_in_title" AND `text` contains "A_keyword_in_text", OR +* `id` is not equal to "A_specific_ID" + + +Learn more about Elasticsearch filters from the [Elasticsearch boolean query documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) + +## How to configure the query body +By default, keyword search is used for your Search integration, but you can configure the query body in the `Advanced Elasticsearch settings` to enable advanced search such as semantic search with ELSER, KNN dense vector search, and using a nested query to search nested documents, and hybrid search. Here are some query body examples: + +### Semantic search with ELSER +```json +{ + "query": { + "bool": { + "should": [ + { + "text_expansion": { + "ml.tokens": { + "model_id": ".elser_model_2_linux-x86_64", + "model_text": "$QUERY" + } + } + } + ], + "filter": "$FILTER" + } + } +} +``` +* `ml.tokens` refers to the field that stores the ELSER tokens. You may need to update it if you use a different field in your index. If the ELSER tokens are not available, you can also use the field that contains the raw text, but the search quality may degrade. +* `.elser_model_2_linux-x86-64` is the model ID for the optimized version of ELSER v2. It is recommended to use if it is available in your Elasticsearch deployment. Otherwise, use `.elser_model_2` for the regular ELSER v2 model, or `.elser_model_1` for ELSER v1. +* `$QUERY` is the variable for accesing the user query. It will make sure that the user query will be passed to the query body. +* `$FILTER` is the variable for accessing the custom filters configured either in the `Advanced Elasticsearch settings` or when calling the search in an action step. It will make sure that the custom filters will be used in the query body. +* Learn more about ELSER from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search-elser.html) +* Learn more about Elasticsearch boolean query and filters from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) + +### KNN dense vector search +```json +{ + "knn": { + "field": "text_embedding.predicted_value", + "query_vector_builder": { + "text_embedding": { + "model_id": "intfloat__multilingual-e5-small", + "model_text": "$QUERY" + } + }, + "k": 10, + "num_candidates": 100, + "filter" : "$FILTER" + } +} +``` +Notes: +* `text_embedding.predicted_value` refers to the field that stores the dense vectors. You may need to update it if you use a different field in your index. +* `text_embedding` under `query_vector_builder` is the natural language processing task to perform. It has to be `text_embedding` for KNN search. +* `intfloat__multilingual-e5-small` is the embedding model ID. You may need to update it if you want to use a different embedding model. +* `$QUERY` is the variable for accesing the user query. It will make sure that the user query will be passed to the query body. +* `$FILTER` is the variable for accessing the custom filters configured either in the `Advanced Elasticsearch settings` or when calling the search in an action step. It will make sure that the custom filters will be used in the query body. +* Learn more about knn search from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html). +* Learn more about how to set up a text embedding model in Elasticsearch from [here](text_embedding_deploy_and_use.md). + +### Using a nested query to search over nested documents with ELSER +If there are inner documents in your Elasticsearch index, you can use a nested query to search over the inner documents. If there is a match, the query returns the root parent document with matched inner documents. So, if applying filters to the search results, you can choose to filter the outer or inner documents. + +#### Filtering the outer documents +```json +{ + "query": { + "bool": { + "must": [ + { + "nested": { + "path": "passages", + "query": { + "text_expansion": { + "passages.sparse.tokens": { + "model_id": ".elser_model_2_linux-x86_64", + "model_text": "$QUERY" + } + } + }, + "inner_hits": {"_source": {"excludes": ["passages.sparse"]}} + } + } + ], + "filter": "$FILTER" + } + }, + "_source": false +} +``` + +#### Filtering the inner documents +```json +{ + "query": { + "nested": { + "path": "passages", + "query": { + "bool": { + "should": [ + { + "text_expansion": { + "passages.sparse.tokens": { + "model_id": ".elser_model_2_linux-x86_64", + "model_text": "$QUERY" + } + } + } + ], + "filter": "$FILTER" + } + }, + "inner_hits": {"_source": {"excludes": ["passages.sparse"]}} + } + }, + "_source": false +} +``` +Notes: +* `passages` is the nested field that stores inner documents within a parent document. You may need to update it if you use a different nested field in your index. +* `passages.sparse.tokens` refers to the field that stores the ELSER tokens or raw text for the inner documents. You may need to update it if you use a different nested field in your index. If the ELSER tokens are not available, you can also use the field that contains the raw text, but the search quality may degrade. +* `"inner_hits": {"_source": {"excludes": ["passages.sparse"]}}` is to exclude the ELSER tokens from the inner documents in the search results. +* `"_source": false` is to exclude all the top-level fields in the search results because only the inner documents in the search results will be used. +* `$QUERY` is the variable for accesing the user query. It will make sure that the user query will be passed to the query body. +* `$FILTER` is the variable for accessing the custom filters configured either in the `Advanced Elasticsearch settings` or when calling the search in an action step. It will make sure that the custom filters will be used in the query body. If applied on the outer documents, only outer fields are available to use in the filters. If applied on the inner documents, only inner fields are available to use in the filters. +* Learn more about nested queries and fields from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html) + +### Hybrid search with combined keyword search and dense vector search +```json +{ + "query": { + "bool": { + "query": { + "query_string": { + "query": "$QUERY", + "fields": ["$BODY_FIELD_NAME", "$TITLE_FIELD_NAME"], + } + }, + "filter" : "$FILTER" + } + }, + "knn": { + "field": "text_embedding.predicted_value", + "query_vector_builder": { + "text_embedding": { + "model_id": "intfloat__multilingual-e5-small", + "model_text": "$QUERY" + } + }, + "k": 10, + "num_candidates": 100, + "filter" : "$FILTER" + }, + "rank": { + "rrf": { + "window_size": 200 + } + }, + "size": 10, + "_source": {"excludes": ["text_embedding.predicted_value"]} +} +``` +Notes: +* `text_embedding.predicted_value` refers to the field that stores the dense vectors. You may need to update it if you use a different field in your index. +* `text_embedding` under `query_vector_builder` is the natural language processing task to perform. It has to be `text_embedding` for KNN search. +* `intfloat__multilingual-e5-small` is the embedding model ID. You may need to update it if you want to use a different embedding model. +* `$QUERY` is the variable for accesing the user query. It makes sure that the user query will be passed to the query body. +* `$BODY_FIELD_NAME` and `$TITLE_FIELD_NAME` are the variables for accessing the Body field and Title field configured in the Search integration, respectively. +* `$FILTER` is the variable for accessing the custom filters configured either in the `Advanced Elasticsearch settings` or when calling the search in an action step. It makes sure that the custom filters will be used in the query body. +* `rank.rrf` is the Reciprocal rank fusion (rrf) method to combine the search results from keyword search and dense vector search. +* `"_source": {"excludes": ["text_embedding.predicted_value"]}` is to exclude the unnecessary dense vector field in the search results. diff --git a/agent_knowledge/elasticsearch-install-and-setup/how_to_index_pdf_and_office_documents_elasticsearch.md b/agent_knowledge/elasticsearch-install-and-setup/how_to_index_pdf_and_office_documents_elasticsearch.md new file mode 100644 index 0000000..0ebd43e --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/how_to_index_pdf_and_office_documents_elasticsearch.md @@ -0,0 +1,237 @@ +# Working with PDF and Office Documents in Elasticsearch + +## Table of contents: +* [Step 1: Create all the necessary environment variables](#step-1-create-all-the-necessary-environment-variables) +* [Step 2: Create an Elasticsearch index with specific nested mapping](#step-2-create-an-index-with-a-nested-mapping-for-storing-chunked-text-and-tokens) +* [Step 3: Create an ELSER ingest pipeline with regex based chunking](#step-3-create-an-elser-ingest-pipeline-with-regex-based-chunking) +* [Step 4: Run fscrawler app using docker](#step-4-running-the-fscrawler-as-a-docker-service) +* [Step 5: Connecting to Agent Knowledge in watsonx Orchestrate](#step-5-connecting-to-agent-knowledge-in-watsonx-orchestrate) + +## Pre-requisites: + +1. The following tutorial assumes there exists a folder of documents having the supported file [types](https://fscrawler.readthedocs.io/en/latest/user/formats.html) that you would like to index into Elasticsearch. + + Optional: If using IBM COS or other Cloud Object Storage for your files, you can follow instructions to use [s3fs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-s3fs) or [rclone](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-rclone) to synchronize your data to a local mounted filesystem + +2. If you do not have `docker` and `docker-compose`, you can install them by following the [docker and compose install guide](./how_to_install_docker.md). These are needed to run the `fscrawler` app. + + +## Introduction + +Before you start, ensure you have set up your Elasticsearch cluster: + +* For Elasticsearch on IBM Cloud, please refer to [ICD-elasticsearch-install-and-setup](./ICD_Elasticsearch_install_and_setup.md) for more details. +* For Elasticsearch (watsonx Discovery) on CloudPak, please refer to [watsonx-discovery-install-and-setup](./watsonx_discovery_install_and_setup.md) for more details. + +We will use the filesystem crawler [fscrawler](https://fscrawler.readthedocs.io/en/latest/index.html) to help sync the documents from a local filesystem on to an index on Elasticsearch. You can read more in detail about the configuration options available for `fscrawler` in their linked documentation. + +### Step 1: Create all the necessary environment variables + +The `ES_INDEX_NAME` and `ES_PIPELINE_NAME` variables can be whatever you would like to name your index and ingestion pipeline for use throughout this process/guide as references in various steps below. + + ```bash + export ES_URL=https:// + export ES_USER= + export ES_PASSWORD= + export ES_CACERT= + export ES_INDEX_NAME= + export ES_PIPELINE_NAME= + ``` + + +### Step 2: Create an index with a nested mapping for storing chunked text and tokens + +```bash +curl -X PUT "${ES_URL}/${ES_INDEX_NAME}?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "mappings": { + "properties": { + "passages": { + "type": "nested", + "properties": { + "text": { + "type": "text" + }, + "sparse": { + "properties": { + "tokens": { + "type": "sparse_vector" + } + } + } + } + } + } + } +}' +``` + +Note: `sparse_vector` type is for ELSER v2. For ELSER v1, please use `rank_features`. ELSER v2 has been available since Elasticsearch 8.11 and is preferred to use if it is available. Learn more about ELSER v2 from [here](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html) + + +### Step 3: Create an ELSER ingest pipeline with regex based chunking + +Below is a sample request to create a pipeline that transforms the "content" field [extracted](https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#generated-fields) by `fscrawler` using the ELSER model. + +You will be able to reference this pipeline in the next few steps as a part of indexing the documents of choice. + +**NOTE**: You can update the `model_limit` param in the request below if you would like to experiment with different chunk sizes for your data. This controls the length of the chunks in characters. + +In this case, it has been set to `1024` since we are working with the ELSER model which has a 512 token limit( produces tokens only on the first 512 tokens of the text ). A chunk length of `1024` is unlikely to produce more than 512 tokens thereby staying within the token limit of the model. + +```bash +curl -X PUT "${ES_URL}/_ingest/pipeline/${ES_PIPELINE_NAME}?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "processors": [ + { + "script": { + "source": " String[] envSplit = /((? + +NOTE: There are some example documents available [here](./assets/sample_pdf_docs), if you would like to test the setup. + + +### Step 4d. Run fscrawler + +You can now run the fscrawler to ingest your documents. + +NOTE: If the updated date of the documents are older than the current date, you would have to follow instructions as mentioned [here](https://fscrawler.readthedocs.io/en/latest/user/tips.html#moving-files-to-a-watched-directory) to ensure the fscrawler is able to pick it up for indexing. + +Next, navigate to the `fscrawler-docker-config` directory and run: + +``` +docker-compose up -d +``` + +This will start the `fscrawler` app and begin to ingest the documents from the directory it was pointed at. + +a. If you have installed Kibana locally following [ICD-elasticsearch-install-and-setup](./ICD_Elasticsearch_install_and_setup.md#step-2-set-up-kibana-to-connect-to-elasticsearch), then you can verify this by bringing up Kibana. Once Kibana is up and running successfully: + + 1. Navigate to http://localhost:5601/app/enterprise_search/elasticsearch + 2. Click on "Indices" and click on your chosen index. + * The "Overview" tab will be updated with the number of documents indexed or being indexed. + * You can verify the index to make sure the crawler ingested all documents successfully + +b. If not, you can use the following to check the document ingestion count as well which will be displayed in the `count` field of the output from the below API request + +```bash +curl -X GET "${ES_URL}/${ES_INDEX_NAME}/_count?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" +``` + +OPTIONAL: Once all documents are indexed, you can stop the `fscrawler` app or if you would like to, you can leave it running to keep the filesystem in sync if new documents are added or old ones removed. To stop the app , run the following: + +``` +docker-compose down +``` + +Your documents are now available in the index, ready for searching and querying. Follow the steps outlined below to use this index for Agent Knowledge in watsonx Orchestrate. + +### Step 5: Connecting to Agent Knowledge in watsonx Orchestrate + +To configure your index for Agent Knowledge in watsonx Orchestrate, you need to follow the documentation for [Connecting to an Elasticsearch content repository](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=agents-connecting-elasticsearch-content-repository). + +Importantly, you need to use the right fields to configure your result content (In this guide, use `title` for Title and `text` for Body). You also need to use the right query body to make the search integration work with your web crawler index. Here is an screenshot of the configuration: + + + +Here is the query body you need in the `Advanced Elasticsearch Settings` to search over the chunked passages: +```json +{ + "query": { + "nested": { + "path": "passages", + "query": { + "text_expansion": { + "passages.sparse.tokens": { + "model_id": ".elser_model_2_linux-x86_64", + "model_text": "$QUERY" + } + } + }, + "inner_hits": {"_source": {"excludes": ["passages.sparse"]}} + } + }, + "_source": false +} +``` +Notes: +* `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. +* `passages` is the nested field that stores nested documents. You may need to update it if you use a different nested field in your index. +* `passages.sparse.tokens` refers to the field that stores the ELSER tokens for the nested documents. +* `"inner_hits": {"_source": {"excludes": ["passages.sparse"]}}` is to exclude the ELSER tokens from the nested documents in the search results. +* `"_source": false` is to exclude top-level fields in the search results. +* Learn more about nested queries and fields from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html) diff --git a/agent_knowledge/elasticsearch-install-and-setup/how_to_install_docker.md b/agent_knowledge/elasticsearch-install-and-setup/how_to_install_docker.md new file mode 100644 index 0000000..a053b48 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/how_to_install_docker.md @@ -0,0 +1,166 @@ +# Guide to installing Docker or Docker alternatives + +## What is Docker? + +Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. Using Docker, you can quickly deploy and scale applications into any environment and know your code will run. + +And Docker Compose is a tool for defining and running multi-container applications. It is the key to unlocking a streamlined and efficient development and deployment experience. + +You will see references to `docker` and `docker-compose` as you work through some of our guides and this document serves to guide anyone who needs a starting point to install that software or its alternatives. + +Useful links: + +1. [Docker Compose Overview](https://docs.docker.com/compose/) +2. [Docker Overview](https://docs.docker.com/get-docker/) + +## Option 1: Docker Desktop + +**IMPORTANT NOTE**: Docker Desktop is free for small businesses (fewer than 250 employees AND less than 10 dollar million in annual revenue), personal use, education, and non-commercial open source projects. Otherwise, it requires a paid subscription + +Docker provides a simple and easy to use desktop platform that comes integrated with the `docker` engine as well as `docker-compose`. It is easy to use and is maintained regularly by Docker. + +If you have an existing subscription or if your enterprise is able to provide you with a subscription, you can follow instrutions to download and install ***Docker Desktop*** for the OS that you are working with, either [Mac](https://docs.docker.com/desktop/install/mac-install/), [Windows](https://docs.docker.com/desktop/install/windows-install/) or [Linux](https://docs.docker.com/desktop/install/linux-install/). + +Keep in mind that you would need to activate a subscription in case you are not eligible for the free version of Docker Desktop. + +## Option 2: Rancher Desktop + +**IMPORTANT NOTE**: Recommended for users who are new to Docker and prefer an easy one-click install of basic functionality + +### For Mac and Windows: + +Go to the Rancher Desktop [homepage](https://rancherdesktop.io/) and download the appropriate installer for your OS, then proceed to install using your respective installer file which should help install the software automatically on your machine + +### For Linux (non RHEL and Fedora related distributions): + +You can install using either the [.deb](https://docs.rancherdesktop.io/getting-started/installation/#installation-via-deb-package) or [.rpm](https://docs.rancherdesktop.io/getting-started/installation/#installation-via-deb-package) package + +### For Linux (RHEL and Fedora related distributions): + +You can install using [AppImage](https://docs.rancherdesktop.io/getting-started/installation/#installation-via-deb-package) + +### Verifying the install + +Upon completion of the install, open and start the `Rancher Desktop` program and you will see a screen with options as shown in the image. + + + +Here, select `dockerd` as the container engine and select `automatic` PATH configuration and click `OK` + +You can now verify your install. To do so: + +1. Close any open shell(or terminal consoles) and restart a new shell. +2. Run the below command to ensure your can work with `docker` successfully. + +```bash +docker run hello-world +``` + + +#### Useful Links: + +* [https://medium.com/@praveenmidde/seamlessly-switch-to-rancher-desktop-as-a-replacement-for-docker-desktop-on-mac-for-1abcf5844c4d](https://medium.com/@praveenmidde/seamlessly-switch-to-rancher-desktop-as-a-replacement-for-docker-desktop-on-mac-for-1abcf5844c4d) +* [https://docs.rancherdesktop.io/getting-started/installation/](https://docs.rancherdesktop.io/getting-started/installation/) + + +## Option 3: Podman + +**IMPORTANT NOTE**: For Windows and MacOS users, running Podman requires a Linux distribution/subsystem or Linux in a virtual machine + +Make sure to check the [prerequisites](https://podman-desktop.io/docs/podman/installing#prerequisites) before proceeding with the install. Then, + +1. Go to the [downloads](https://podman-desktop.io/downloads) page and download the binary suitable for your machine +2. Follow the steps listed in the [procedure](https://podman-desktop.io/docs/podman/installing#procedure) section to perform the necessary checks and install podman along with other necessities. + +Once installed, Podman initializes a machine with standard configuration and your Podman Desktop dashboard should show `Podman is running` like below + + + +However, it is recommended to update the configurations of the machine by : + +* clicking on the edit button +* choosing a reasonable CPU, Memory and Disk Size +* click on "Update" + + + +This will restart the machine with the updated configurations. + +### Install Docker and Compose Extensions + +Under Settings > Extensions, click on the "install" icon for the Docker and Compose extensions. Follow the prompts on the screen. + + + +### Emulating Docker CLI + +In order to run `docker` commands using podman, you would have to follow steps as listed in the Step 1 and 3 of the [documentation](https://podman-desktop.io/docs/migrating-from-docker/emulating-docker-cli-with-podman#procedure) + +This creates a proxy to podman using the `docker` command on the interface and podman behind the scenes + +### Verifying the install + +You can now run the below on your terminal or console to verify the setup is working. + +```bash +docker run hello-world +``` + +#### Useful Links: + +* [https://thenewstack.io/install-and-use-podman-desktop-gui-to-manage-containers/](https://thenewstack.io/install-and-use-podman-desktop-gui-to-manage-containers/) +* [https://developers.redhat.com/articles/2023/09/27/how-install-and-use-podman-desktop-windows#getting_started_with_podman_desktop](https://developers.redhat.com/articles/2023/09/27/how-install-and-use-podman-desktop-windows#getting_started_with_podman_desktop) +* [https://ubuntuhandbook.org/index.php/2023/05/podman-desktop-1-0-install-in-ubuntu/](https://ubuntuhandbook.org/index.php/2023/05/podman-desktop-1-0-install-in-ubuntu/) + +## Option 4: Colima + +**IMPORTANT NOTE**: This approach requires using a CLI(Command Line Interface) and does not have a GUI. This is recommended for users who prefer a CLI-focused approach to Docker Desktop's GUI focus. + +[Colima](https://github.com/abiosoft/colima) is a new open-source project that bundles the container management tool [lima](https://github.com/lima-vm/lima) with a docker (linux) back-end. This is similar to what Docker Desktop actually does, but Colima and Lima are entirely open-source and just focused on running containers. They work on both `amd64` and `arm64` (M1) macs. Colima does not require installation of Docker Desktop or paying a licensing fee to Docker, Inc. + + +### For MacOS and Linux + +1. First, install `homebrew` package manager by running the below command on a shell: + + ```shell + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + ``` + +2. Next, install `colima`, `docker` and `compose` using homebrew + + ```shell + brew install colima + brew install docker docker-compose + ``` +3. Start `colima` with the following command. This may take a while the first time, so wait for it to fully complete + + ```shell + colima start + ``` + NOTE: The default VM created by Colima has 2 CPUs, 2GiB memory and 60GiB storage. You can customize the CPUs and Memory by instead starting `colima` and specifying the cpu and memory. For example, the below will start up colima with 4 cpu and 6GiB memory: + + ```shell + colima start --cpu 4 --memory 6 + ``` + + Learn more about how to customize colima from [here](https://github.com/abiosoft/colima?tab=readme-ov-file#customizing-the-vm) + +4. You can now verify that `docker` and `compose` can be run successfully + + ```shell + docker run hello-world + ``` + + ```shell + docker-compose + ``` + +#### Useful Links: + +* [https://smallsharpsoftwaretools.com/tutorials/use-colima-to-run-docker-containers-on-macos/](https://smallsharpsoftwaretools.com/tutorials/use-colima-to-run-docker-containers-on-macos/) +* [https://www.linkedin.com/pulse/colima-better-way-run-docker-macos-linux-asutosh-pandya/](https://www.linkedin.com/pulse/colima-better-way-run-docker-macos-linux-asutosh-pandya/) + + + + diff --git a/agent_knowledge/elasticsearch-install-and-setup/how_to_use_web_crawler_in_elasticsearch.md b/agent_knowledge/elasticsearch-install-and-setup/how_to_use_web_crawler_in_elasticsearch.md new file mode 100644 index 0000000..9484f62 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/how_to_use_web_crawler_in_elasticsearch.md @@ -0,0 +1,506 @@ +# How to set up and use the web crawler in Elasticsearch +This is a documentation about how to set up and use the web crawler in Elasticsearch and connect it to Agent Knowledge in watsonx Orchestrate. + +## Tabel of contents: +* [Step 1: Set up Enterprise Search to enable the web crawler in Elasticsearch](#step-1-set-up-enterprise-search-to-enable-the-web-crawler-in-elasticsearch) +* [Step 2: Create and configure a web crawler in Elasticsearch](#step-2-create-and-configure-a-web-crawler-in-elasticsearch) +* [Step 3: Build an ELSER ingest pipeline with a chunking processor](#step-3-build-an-elser-ingest-pipeline-with-a-chunking-processor) +* [Step 4: Connect a web crawler index to Agent Knowledge in watsonx Orchestrate](#step-4-connect-a-web-crawler-index-to-agent-knowledge-in-watsonx-orchestrate) + +## Step 1: Set up Enterprise Search to enable the web crawler in Elasticsearch +Before you start, you will need to install and set up your Elasticsearch cluster, +* For Elasticsearch on IBM Cloud, please refer to [ICD-elasticsearch-install-and-setup](../../docs/elasticsearch-install-and-setup/ICD_Elasticsearch_install_and_setup.md) for more details. +* For Elasticsearch (watsonx Discovery) on CloudPak, please refer to [watsonx-discovery-install-and-setup](../../docs/elasticsearch-install-and-setup/watsonx_discovery_install_and_setup.md) for more details. + +### Set up Enterprise Search for Elasticsearch on IBM Cloud +Assuming you have installed Kibana locally following [ICD-elasticsearch-install-and-setup](../../docs/elasticsearch-install-and-setup/ICD_Elasticsearch_install_and_setup.md), +follow these steps to set up Enterprise Search in Elasticsearch: +**NOTE: Enterprise Search requires at least 4GB of memory, so please make sure you have enough memory allocated to your Docker Engine.** +* Create a docker network + ```shell + docker network create elastic + ``` + NOTE: `elastic` will be the name of your docker network. + + +* Restart Kibana within a network and with enterprise search host + ```shell + KIBANA_CONFIG_FOLDER= + KIBANA_VERSION= + ``` + NOTE: `KIBANA_VERSION` needs to be compatible with the Elasticsearch version. It can be the same as the Elasticsearch version. + Learn more about the compatibility with Elasticsearch from [here](https://www.elastic.co/support/matrix#matrix_compatibility) + ```shell + docker run -it --name kibana --network elastic --rm \ + -v ${KIBANA_CONFIG_FOLDER}:/usr/share/kibana/config \ + -p 5601:5601 \ + --env "ENTERPRISESEARCH_HOST=http://enterprise-search:3002" \ + docker.elastic.co/kibana/kibana:${KIBANA_VERSION} + ``` + NOTE: Instead of passing `ENTERPRISESEARCH_HOST` as an environment variable, another option is to add the enterprise search + host to your kibana config YAML file, for example, + ```shell + enterpriseSearch.host: "http://enterprise-search:3002" + ``` + +* Generate an encryption key for Enterprise Search +In a new terminal window, run the following command to generate an encryption key which will be used in the next step: + ```shell + ENCRYPT_KEY=$( openssl rand -hex 32 ) + ``` + +* Create environment variables for elasticsearch credentials: + ```shell + ES_URL= + ES_USER= + ES_PASSWORD= + + ES_VERSION= + ES_CACERT_FOLDER= + ES_CACERT_NAME= + ``` + NOTE: `ES_CACERT_FOLDER` can be your kibana config folder if you have stored the elasticsearch certificate in the folder following [ICD-elasticsearch-install-and-setup](../../docs/elasticsearch-install-and-setup/ICD_Elasticsearch_install_and_setup.md) + +* Download and start Enterprise Search + ```shell + docker run \ + --name "enterprise-search" \ + --network "elastic" \ + --publish "3002:3002" \ + -v "${ES_CACERT_FOLDER}:/usr/share/enterprise-search/es-config:ro" \ + --interactive \ + --tty \ + --rm \ + --env "secret_management.encryption_keys=[${ENCRYPT_KEY}]" \ + --env "allow_es_settings_modification=true" \ + --env "elasticsearch.host=${ES_URL}" \ + --env "elasticsearch.username=${ES_USER}" \ + --env "elasticsearch.password=${ES_PASSWORD}" \ + --env "elasticsearch.ssl.enabled=true" \ + --env "elasticsearch.ssl.certificate_authority=/usr/share/enterprise-search/es-config/${ES_CACERT_NAME}" \ + --env "kibana.external_url=http://kibana:5601" \ + --env "connector.crawler.content_extraction.enabled=true" \ + --env "connector.crawler.content_extraction.mime_types=["application/pdf", "application/msword"]" \ + --env "connector.crawler.http.response_size.limit=10485760" \ + "docker.elastic.co/enterprise-search/enterprise-search:${ES_VERSION}" + ``` + NOTES: + * `connector.crawler.content_extraction.enabled=true` enables binary content extraction. It is required to extract content from downloadable binary files, such as PDF and DOCX files. + * `connector.crawler.content_extraction.mime_types=["application/pdf", "application/msword"]` specifies which MIME types should have their contents extracted. + * `connector.crawler.http.response_size.limit=10485760` set the maximum size of an HTTP response (in bytes) supported by the Elastic web crawler. The default limit is `10485760` bytes or 10MB. You can increase it if needed. + * To learn more, see [Elastic web crawler configuration](https://www.elastic.co/guide/en/enterprise-search/current/configuration.html#configuration-settings-elastic-crawler). +* Verify the installation + * Open http://localhost:5601 in your browser and log into Kibana using your Elasticsearch username and password. + * Navigate to the Search Overview page http://localhost:5601/app/enterprise_search/overview. + * If you can see `Web Crawler` available as an option to ingest content, your Enterprise Search has been set up successfully. + +### Set up Enterprise Search for watsonx Discovery on CloudPak +Assuming you have successfully installed an Elasticsearch instance in a CloudPak cluster following [watsonx-discovery-install-and-setup](../../docs/elasticsearch-install-and-setup/watsonx_discovery_install_and_setup.md), +you can follow these steps to set up Enterprise Search: + +* Log into your CloudPak cluster following the instructions [here](../../docs/elasticsearch-install-and-setup/watsonx_discovery_install_and_setup.md#log-in-to-your-cloudpak-cluster) + + +* Create environment variables for installation + ```shell + export ES_NAMESPACE="elastic" + export ES_CLUSTER="wxd" + export ENTERPRISE_SEARCH_NAME="enterprise-search-wxd" + export ES_VERSION="8.11.1" + ``` + NOTES: + * `ES_NAMESPACE` is your Elasticsearch namespace. + * `ES_CLUSTER` is your Elasticsearch instance name in your CloudPak cluster. + * `ENTERPRISE_SEARCH_NAME` can be whatever you would like to name your Enterprise Search instance. + * `ES_VERSION` is your Elasticsearch version, and it needs to be the same for Enterprise Search. + + +* Create an Enterprise Search instance and associate it with your Elasticsearch cluster + ```shell + cat < + NOTE: The two rules tell the web crawler to only crawl URLs that begin with https://www.nationalparks.org/explore/parks. + Learn more about crawl rules from [here](https://www.elastic.co/guide/en/app-search/current/web-crawler-reference.html#web-crawler-reference-crawl-rule) + + +* Don't start the web crawler (by clicking on `Crawl` in the upper right corner) yet, because that would cause it to + start ingesting with the default index mappings and pipeline. Instead, continue on to the next section to build a + custom ingest pipeline before starting the crawl. + + +## Step 3: Build an ELSER ingest pipeline with a chunking processor +To use ELSER for text expansion queries on chunked texts, you need to build an ingest pipeline with a chunking processor +that uses the ELSER model. + +NOTE: ELSER model is not enabled by default, and you can enable it in Kibana, following the [download-deploy-elser instructions](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html#download-deploy-elser). +Depending on your Elasticsearch version, you can choose to deploy either ELSER v1 or v2 model. `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. + +### Update the index mappings of your web crawler +Define your web crawler index name as an environment variable: +```shell +ES_INDEX_NAME= +``` +Update the index mappings: +```shell +curl -X PUT "${ES_URL}/${ES_INDEX_NAME}/_mapping?pretty" -k \ +-u "${ES_USER}:${ES_PASSWORD}" \ +-H 'Content-Type: application/json' \ +-d' +{ + "properties": { + "passages": { + "type": "nested", + "properties": { + "sparse.tokens": { + "type": "sparse_vector" + } + } + } + } +}' +``` +The above command will update the index mappings to specify `passages` to be `nested` type and `passages.sparse.tokens` to be `sparse_vector` type, because the default dynamic index mappings of the web crawler cannot recognize these types correctly during document ingestion. + +NOTE: `sparse_vector` type is for the ELSER v2 model. For ELSER v1, please use `rank_features` type. ELSER v2 has become available since Elasticsearch 8.11. It is preferred to use ELSER v2 if it is avaiable. Learn more about ELSER v2 from [here](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html) + +### Build a custom ingest pipeline with two processors +Now you can build a custom ingest pipeline for your web crawler index on Kibana, following these steps: + +* Open http://localhost:5601 or https://localhost:5601 (for Kibana port-forwarded from CloudPak) and log into Kibana with your Elasticsearch credentials. Navigate to the indices page + from the left-side menu via `Content` under `Search`, find your web crawler index, and click on it to go to the index page. + + +* Under `Pipelines` tab, click on `Copy and customize` to create a custom ingest pipeline, and you will see a new ingest pipeline named `@custom`. + For example, + + + +* Click on `edit pipeline` and then `Manage` -> `Edit`, it will take you to the ingest pipeline `Edit` page + where you can add processors to the pipeline. + + +* Add a `Script` processor for chunking + In the ingest pipeline page, click on `Add a processor`, choose `Script` processor, and then add [a painless script](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html) to the `Source` field. + For example, + + + ```Groovy + String[] envSplit = /((? + ```json + { + "inference": { + "field_map": { + "_ingest._value.text": "text_field" + }, + "model_id": ".elser_model_2_linux-x86_64", + "target_field": "_ingest._value.sparse", + "inference_config": { + "text_expansion": { + "results_field": "tokens" + } + }, + "on_failure": [ + { + "append": { + "field": "_source._ingest.inference_errors", + "value": [ + { + "message": "Processor 'inference' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'", + "pipeline": "search-crawler-with-chunking@custom", + "timestamp": "{{{ _ingest.timestamp }}}" + } + ] + } + } + ] + } + } + ``` + NOTES: + * `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. + * `inference_config.text_expansion` is required in the config to tell the Foreach processor to use `text_expansion` + and store the results in `tokens` field for each chunked text. + * `_ingest._value.sparse` expects a `sparse` field for each chunk object as the target field. + * `_ingest._value.text` expects a `text` field for each chunk object as the input field. + * `"_ingest._value.text": "text_field"` means ELSER uses `text_field` as the input field. You may need to update it if your ELSER input field is different. + * `search-crawler-with-chunking@custom` is the name of the ingest pipeline. You need to update it with your ingest pipeline name. + + + > ⛔️ + > **Caution** + > Don't forget to click on `Save pipeline` to save your changes! + + +* Start your web crawler and monitor its progress + Once you have added the processors to your ingest pipeline, you can kick off your web crawler to crawl the website URLs + you have configured at earlier steps, following these steps: + * Go to your web crawler index page, and click on `Crawl` in the upper right corner to start it. + + * You will see new crawl requests on the overview page, and you can click on the request ids to see more details and to monitor the progress of your crawl requests. + + + * If you see your crawler is running and the number of documents is increasing, you can now inspect the documents to + see if they have the expected fields with content. For example, you should see chunked `passages`, each passage with + `sparse.tokens` and a chunked `text`, + + Learn more about the web crawler from [Elastic documentation](https://www.elastic.co/guide/en/enterprise-search/current/crawler.html) to improve or customize your web crawler. + + * If you don't see expected documents with chunked passages, or you want to update any processors in the ingest + pipeline to customize your web crawler, you may need to delete the existing documents first before starting the crawl again. + You can delete all the documents in an index using the following cURL command: + ```shell + curl -k -X POST "${ES_URL}/${ES_INDEX_NAME}/_delete_by_query" \ + -u "${ES_USER}:${ES_PASSWORD}" \ + -H 'Content-Type: application/json' -d' + { + "query":{ + "match_all":{} + } + }' + ``` + + +* Run a nested `text_expansion` query using cURL + ```shell + curl -k -X GET "${ES_URL}/${ES_INDEX_NAME}/_search?pretty" \ + -u "${ES_USER}:${ES_PASSWORD}" \ + -H 'Content-Type: application/json' -d' + { + "query": { + "nested": { + "path": "passages", + "query": { + "text_expansion": { + "passages.sparse.tokens": { + "model_id": ".elser_model_2_linux-x86_64", + "model_text": "Tell me about Acadia" + } + } + }, + "inner_hits": {"_source": {"excludes": ["passages.sparse"]}} + } + } + }' + ``` + The above command sends a nested query to the Elasticsearch index. `text_expansion` is used in this query on the ELSER tokens + generated for each chunked text by the ingest pipeline. So, the search happens on the chunked texts. Learn more about nested + query from the [Elastic documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html). + + If you see successful results from the above query, you have successfully created a web crawler index with an ELSER ingest pipeline with a chunking processor. + + + NOTES: + * If you run this query while the crawling is taking place, you might get a timeout error, because the ELSER model + is busy indexing and thus might not respond quickly enough to your query. If that happens, you should just wait until the crawl finishes. + * `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Otherwise, use `.elser_model_2` for the regular ELSER v2 model or `.elser_model_1` for ELSER v1. + +## Step 4: Connect a web crawler index to Agent Knowledge in watsonx Orchestrate + +To configure your web crawler index for Agent Knowledge in watsonx Orchestrate, you need to follow the documentation for [Connecting to an Elasticsearch content repository](https://www.ibm.com/docs/en/watsonx/watson-orchestrate/base?topic=agents-connecting-elasticsearch-content-repository). + +Importantly, you need to use the right fields to configure your result content (In this guide, use `title` for Title and `text` for Body). You also need to use the right query body to make the search integration work with your web crawler index. Here is an screenshot of the configuration: + + + +Here is the query body you need in the `Advanced Elasticsearch Settings` to search over the chunked passages: +```json +{ + "query": { + "nested": { + "path": "passages", + "query": { + "text_expansion": { + "passages.sparse.tokens": { + "model_id": ".elser_model_2_linux-x86_64", + "model_text": "$QUERY" + } + } + }, + "inner_hits": {"_source": {"excludes": ["passages.sparse"]}} + } + }, + "_source": false +} +``` +Notes: +* `passages` is the nested field that stores nested documents. You may need to update it if you use a different nested field in your index. +* `passages.sparse.tokens` refers to the field that stores the ELSER tokens for the nested documents. +* `"inner_hits": {"_source": {"excludes": ["passages.sparse"]}}` is to exclude the ELSER tokens from the nested documents in the search results. +* `"_source": false` is to exclude the unnecessary top-level fields in the search results. +* Learn more about nested queries and fields from [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html) + diff --git a/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/.gitignore b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/.gitignore new file mode 100644 index 0000000..87620ac --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/.gitignore @@ -0,0 +1 @@ +.ipynb_checkpoints/ diff --git a/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/README.md b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/README.md new file mode 100644 index 0000000..3c2def8 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/README.md @@ -0,0 +1,177 @@ +# Working with PDF and Office Documents in Elasticsearch + +## Table of contents: +* [Step 1: Create a python virtual environment and upgrade pip](#step-1-create-a-python-virtual-environment-and-upgrade-pip) +* [Step 2: Create all the necessary environment variables](#step-2-create-all-the-necessary-environment-variables) +* [Step 3: Create an ELSER ingest pipeline with an inference processor](#step-3-create-an-elser-ingest-pipeline-with-an-inference-processor) +* [Step 4: Create an index for storing tokens as rank features](#step-4-create-an-index-for-storing-tokens-as-rank-features) +* [Step 5: Running the python notebook](#step-5-running-the-python-notebook) + +## Pre-requisites: + +1. The following tutorial assumes there exists a folder of documents that you would like to index into Elasticsearch. These can be PDF, Microsoft Office, OpenOffice, HTML, Text files etc... See full list of supported types [here](https://tika.apache.org/2.9.1/formats.html) + + Optional: If using IBM COS or other Cloud Object Storage for your files, you can follow instructions to use [s3fs](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-s3fs) or [rclone](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-rclone) to synchronize your data to a local mounted filesystem + +2. You will need a working install of [python](https://www.python.org/downloads/) and Java 8+ on your machine. + +## Introduction + +Before you start, ensure you have set up your Elasticsearch cluster: + +* For Elasticsearch on IBM Cloud, please refer to [ICD-elasticsearch-install-and-setup](./ICD_Elasticsearch_install_and_setup.md) for more details. +* For Elasticsearch (watsonx Discovery) on CloudPak, please refer to [watsonx-discovery-install-and-setup](./watsonx_discovery_install_and_setup.md) for more details. + + +### Step 1: Create a python virtual environment and upgrade pip + +`venv` allows you to manage separate package installations for different projects. It creates a “virtual” isolated Python installation. When you switch projects, you can create a new virtual environment which is isolated from other virtual environments. You benefit from the virtual environment since packages can be installed confidently and will not interfere with another project’s environment. + +```bash +python3 -m venv .venv +source .venv/bin/activate +python3 -m pip install --upgrade pip +``` + + +### Step 2: Create all the necessary environment variables + +The `ES_INDEX_NAME` and `ES_PIPELINE_NAME` variables can be whatever you would like to name your index and ingestion pipeline for use throughout this process/guide as references in various steps below. + + ```bash + export ES_URL=https:// + export ES_USER= + export ES_PASSWORD= + export ES_CACERT= + export ES_INDEX_NAME= + export ES_PIPELINE_NAME= + export DOCS_DIR= + ``` + +### Step 3: Create an ELSER ingest pipeline with an inference processor + +If you already have an existing pipeline in your Elasticsearch instance that uses an inference processor with ELSER against the "text" field, you can choose to reuse that and skip creation of a new pipeline. + +If not, please continue reading below regarding the creation of a new pipeline. + +To use ELSER for text expansion queries on chunked texts, you need to build a pipeline with an inference processor that uses the ELSER model. + +NOTE: ELSER model is not enabled by default, and you can enable it in Kibana, following the [download-deploy-elser instructions](https://www.elastic.co/guide/en/machine-learning/8.11/ml-nlp-elser.html#download-deploy-elser). + +Depending on your Elasticsearch version, you can choose to deploy either ELSER v1 or v2 model. The following steps and commands are based on ELSER v1 model, but you can find what change is needed for ELSER v2 in the notes of each step. + +You will be able to reference this pipeline in the next few steps as a part of indexing the documents of choice. It transforms the "text" field using the ELSER model and produces the terms along with weights as a sparse vector in the "ml" field at index time. + +Learn more about [inference-ingest-pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/semantic-search-elser.html#inference-ingest-pipeline) from the tutorial + +Create the pipeline using the command below: + +```bash +curl -X PUT "${ES_URL}/_ingest/pipeline/${ES_PIPELINE_NAME}?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "processors": [ + { + "inference": { + "model_id": ".elser_model_1", + "target_field": "ml", + "field_map": { + "text": "text_field" + }, + "inference_config": { + "text_expansion": { + "results_field": "tokens" + } + } + } + } + ] +}' +``` + +NOTES: + +* `.elser_model_1` is the `model_id` for ELSER v1 model, and the `model_id` can be `.elser_model_2` or `.elser_model_2_linux-x86_64` + for ELSER v2 model depending on which one you want to use and have deployed in your Elasticsearch cluster. +* `inference_config.text_expansion` is required in the config to tell the processor to use `text_expansion` and store the results in `tokens` field for each text. +* `target_field` will contain the field name where the ELSER tokens should be stored +* `field_map` expects a key pointing to the input "text" field. In this example, the source `text` field is being read for processing by ELSER + + + +### Step 4: Create an index for storing tokens as rank features + +```bash +curl -X PUT "${ES_URL}/${ES_INDEX_NAME}?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ +-H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' +{ + "settings":{ + "index":{ + "default_pipeline":"'"${ES_PIPELINE_NAME}"'" + } +}, + "mappings": { + "properties": { + "ml.tokens": { + "type": "rank_features" + }, + "text": { + "type": "text" + } + } + } +}' +``` + +NOTE: `rank_features` only works for ELSER v1 model. ELSER v2 requires `sparse_vector` type. ELSER v2 has only been available since Elastic 8.11. Learn more about ELSER v2 from [here](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html) + + +### Step 5: Running the python notebook + +### Step 5a. Install the required libraries + +To get started, navigate to the `python-document-ingestion` directory and install the requirements provided in the `requirements.txt` directory: + +```bash +cd python-document-ingestion +pip3 install -r requirements.txt +``` + +### Step 5b. Run the provided notebook to ingest documents + +Start the notebook environment using the below command from within your created `venv` from [Step 1](#step-1-create-a-python-virtual-environment-and-upgrade-pip). Then, open the `doc-ingestion.ipynb` notebook on your web browser. + +```bash +jupyter notebook +``` + +Run each cell in order using the "Shift + Enter" shortcut or by clicking the ![play](../assets/jupyter_play_button.png) button. + +After the last cell has completed execution, you will see a message like "**Indexed n/n documents**" that indicates that your documents were successfully chunked and ingested into Elasticsearch. +If you see an error and only a few of your documents got indexed, you may want to delete the index through the [Kibana dashboard](../ICD_Elasticsearch_install_and_setup.md#step-2-set-up-kibana-to-connect-to-elasticsearch) or using the [delete index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html) +and restart again by attempting with smaller number of documents at a time. + +If you'd like to update the number of documents to send per request in the Elasticsearch API, you can update the +`chunk_size` parameter in the call to `helpers.streaming_bulk` method in the final cell of the notebook. + +Your documents are now available in the index, ready for searching and querying. Follow the steps outlined below to use this index for Agent Knowledge in watsonx Orchestrate. + +**NOTE**: There are some example documents available [here](../assets/sample_pdf_docs), if you would like to test the setup. + +### Step 5: Connecting to Agent Knowledge in watsonx Orchestrate + +Follow instructions in [Elasticsearch integration with Agent Knowledge in watsonx Orchestrate](README.md#elasticsearch-integration-with-agent-knowledge-in-watsonx-orchestrate) to set up Agent Knowledge using your Elasticsearch index. + +Here is an example Elasticsearch query body: +``` +{ + "query": { + "text_expansion": { + "ml.tokens": { + "model_id": ".elser_model_1", + "model_text": "$QUERY" + } + } + } +} +``` diff --git a/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/doc-ingestion.ipynb b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/doc-ingestion.ipynb new file mode 100644 index 0000000..fe334fd --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/doc-ingestion.ipynb @@ -0,0 +1,224 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c6a64f3d-1fae-4c63-bf90-bbc7b66b27f9", + "metadata": {}, + "source": [ + "## Ingest PDF or Office documents using Tika, LangChain and Elasticsearch" + ] + }, + { + "cell_type": "markdown", + "id": "8894730b-fddf-4ad9-96d3-cb32a4f53de6", + "metadata": {}, + "source": [ + "First, we will import all the necessary libraries and fetch variable values that we have previously setup using the previous defined environment variables" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9d7f6e1d-f89a-4aa3-9f47-0be01f0e7729", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "import os\n", + "import hashlib\n", + "import fnmatch\n", + "import tqdm\n", + "from tika import parser\n", + "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", + "from ssl import create_default_context\n", + "from elasticsearch import Elasticsearch, helpers" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "327c3f18-484a-439e-86a0-4023769f206f", + "metadata": {}, + "outputs": [], + "source": [ + "es_url = os.getenv('ES_URL')\n", + "es_user = os.getenv('ES_USER')\n", + "es_password = os.getenv('ES_PASSWORD')\n", + "es_cacert = os.getenv('ES_CACERT')\n", + "es_index_name = os.getenv('ES_INDEX_NAME')\n", + "docs_dir = os.getenv('DOCS_DIR')" + ] + }, + { + "cell_type": "markdown", + "id": "0d7692cd-734c-46e6-bdeb-343f79e0ad0c", + "metadata": {}, + "source": [ + "## Define helper functions" + ] + }, + { + "cell_type": "markdown", + "id": "4cf85840-7cb1-4185-8b19-43cd5d10d58f", + "metadata": {}, + "source": [ + "Next, we define the utility functions below that we will use later on to initialize Elasticsearch and generate a dictionary of documents to ingest into an Elasticsearch index" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1a2efda-ad13-41f1-805f-386519bd0013", + "metadata": {}, + "outputs": [], + "source": [ + "def init_elasticsearch(es_cacert, es_url, es_user, es_password):\n", + " context = create_default_context(cafile=es_cacert)\n", + " client = Elasticsearch(\n", + " es_url,\n", + " basic_auth=(es_user, es_password),\n", + " ssl_context=context,\n", + " request_timeout=480\n", + " )\n", + " return client\n", + "\n", + "def generate_documents(docs_dir, es_index_name, text_splitter):\n", + " doc_count = 0\n", + " documents = []\n", + " for filename in os.listdir(docs_dir):\n", + " f = os.path.join(docs_dir, filename)\n", + " if os.path.isfile(f):\n", + " with open(f) as file:\n", + " print(\"Processing \" + f)\n", + " doc_count += 1\n", + " parsed_document = parser.from_file(f)\n", + " document_text = parsed_document['content']\n", + " chunks = text_splitter.split_text(document_text)\n", + " for chunk_id, chunk_text in enumerate(chunks):\n", + " source_dict = dict()\n", + " document_dict = {\"_index\": es_index_name}\n", + " source_dict['doc_id'] = doc_count\n", + " source_dict['chunk_id'] = chunk_id\n", + " source_dict['title'] = str(filename)\n", + " source_dict['text'] = chunk_text\n", + " document_dict['_id'] = hashlib.md5(chunk_text.encode('utf-8')).hexdigest()\n", + " document_dict['_source'] = source_dict\n", + " documents.append(document_dict)\n", + " return documents\n", + "\n", + "def yield_docs(documents):\n", + " for doc in documents:\n", + " yield doc" + ] + }, + { + "cell_type": "markdown", + "id": "76c9b8e5-8620-4e1d-b679-892e4932f45b", + "metadata": {}, + "source": [ + "## Using LangChain text splitting for chunking" + ] + }, + { + "cell_type": "markdown", + "id": "5af516b4-e181-47d3-b9d2-bbc4f235c7fd", + "metadata": {}, + "source": [ + "We use the LangChain library to split our document text into smaller chunks. This is needed so as to fit text within our model's context window. For example, ELSER in Elasticsearch has a 512 token limit, so we need to ensure every chunk remains within that size so we can obtain embedded vectors on each of the 512 token chunks of the document text. \n", + "\n", + "You can update the code below and use different values of \"chunk_size\" and \"chunk_overlap\" depending on the downstream model that you are working with or try out other text splitters as well. You can read more about it in the [LangChain documentation](https://python.langchain.com/docs/modules/data_connection/document_transformers/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dd94fa98-4bd0-42d0-98bb-95b6177be422", + "metadata": {}, + "outputs": [], + "source": [ + "text_splitter =RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n", + " chunk_size=512,\n", + " chunk_overlap=124,\n", + " is_separator_regex=False,\n", + " disallowed_special=()\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "5cec98b1-1008-4c52-8d4b-c228caffad2e", + "metadata": {}, + "source": [ + "## Index into Elasticsearch" + ] + }, + { + "cell_type": "markdown", + "id": "ec850f62-4235-425e-9fc3-f26754a0f851", + "metadata": {}, + "source": [ + "We will now use the helpers to initialize an Elasticsearch client and ingest documents into an index upon breaking them into chunks as defined by our chunking process above" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a024d2c0-a35e-4114-b50e-2266ab15a0c3", + "metadata": {}, + "outputs": [], + "source": [ + "client = init_elasticsearch(es_cacert, es_url, es_user, es_password)\n", + "documents = generate_documents(docs_dir, es_index_name, text_splitter)\n", + "number_of_docs = len(documents)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13d94524-69e1-4a93-9fe7-877a1845fd01", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Indexing documents...\")\n", + "\n", + "progress = tqdm.tqdm(unit=\"docs\", total=number_of_docs)\n", + "successes = 0\n", + "\n", + "try:\n", + " for ok, action in helpers.streaming_bulk(\n", + " client=client,\n", + " chunk_size=50,\n", + " actions=yield_docs(documents),\n", + " ):\n", + " progress.update(50)\n", + " successes += ok\n", + "except Exception as e:\n", + " print(e.errors)\n", + "\n", + "print(\"Indexed %d/%d documents\" % (successes, number_of_docs))" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/requirements.txt b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/requirements.txt new file mode 100644 index 0000000..1b4fdb8 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/python-document-ingestion/requirements.txt @@ -0,0 +1,6 @@ +elasticsearch==8.11.1 +langchain==0.1.8 +tika==2.6.0 +notebook==7.0.6 +tqdm==4.66.2 +tiktoken==0.6.0 \ No newline at end of file diff --git a/agent_knowledge/elasticsearch-install-and-setup/text_embedding_deploy_and_use.md b/agent_knowledge/elasticsearch-install-and-setup/text_embedding_deploy_and_use.md new file mode 100644 index 0000000..22071f4 --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/text_embedding_deploy_and_use.md @@ -0,0 +1,248 @@ +# How to set up and use 3rd-party text embeddings for dense vector search in Elasticsearch +This guide demonstrates how to deploy and use a text embedding model in Elasticsearch. The model will generate vector representations for text, enabling vector similarity (k-nearest neighbours) search. + +## Set up Elasticsearch + +### Elasticsearch from IBM Cloud +If you are using Elasticsearch from IBM Cloud, please refer to [this guide](./ICD_Elasticsearch_install_and_setup.md) first to create an Elasticsearch instance and set up Kibana if you haven't already. + +### Elasticsearch on CloudPak +Alternatively, if you want to install Elasticsearch on Kubernetes (ECK) in CloudPak, you need to follow [this guide](./watsonx_discovery_install_and_setup.md) first to set up Elasticsearch and Kibana. You can skip [Enable ELSER model (v2)](./watsonx_discovery_install_and_setup.md#enable-elser-model-v2) and any section beyond that in the guide. + +## Install the eland library +Run the command below to install the [eland](https://github.com/elastic/eland) library. +```bash +python -m pip install "eland[pytorch]" +``` +This library allows us to pull and deploy a 3rd-party text embedding model to our Elasticsearch instance. + +> CAUTION: Open source and 3rd party models are not in scope of IBM or Elastic indemnity clauses. Customers must accept relevant terms and conditions to choose or bring their own models. Additionally, IBM has not assessed Elastic's supported multi-lingual models so any use of Elastic-supported models should be understood thoroughly both with respect to the terms of use for those models and the terms of use of all of the data that was used to train those models. + +NOTE: +* As of the time this documentation was written, `eland` only supports Python 3.8, 3.9, and 3.10. Please refer to the eland library [compatibility section](https://github.com/elastic/eland?tab=readme-ov-file#compatibility) to make sure you're using compatible Python and Elasticsearch versions. +* If you run into compatibility issues during installation, try to specify a version of `eland[pytorch]` that is compatible with your Elasticsearch version. + +NOTE: You can also use eland without installing the library in case you run into any issues with the library. This can be done by using the docker image provided [here](https://github.com/elastic/eland?tab=readme-ov-file#docker). + +## Create environment variables for ES credentials +Feel free to customize the names of the `ES_SOURCE_INDEX_NAME`, `ES_EMBEDDING_INDEX_NAME` and `ES_PIPELINE_NAME` variables below. These names will serve as references for your source index, embedding index, and ingestion pipeline throughout this guide. + ```bash + export ES_URL=https:// + export ES_USER= + export ES_PASSWORD= + export ES_CACERT= + export ES_SOURCE_INDEX_NAME= + export ES_EMBEDDING_INDEX_NAME= + export ES_PIPELINE_NAME= + ``` +You can find the credentials from the service credentials of your Elasticsearch instance. +## Pull and deploy an embedding model +Run the command below to pull your desired model from the [Huggingface Models Hub](https://huggingface.co/models) and deploy it on your Elasticsearch instance: +```bash +eland_import_hub_model \ + --url $ES_URL \ + -u $ES_USER -p $ES_PASSWORD --insecure \ + --hub-model-id intfloat/multilingual-e5-small \ + --task-type text_embedding \ + --start +``` + +In this example, we are using the `multilingual-e5-small` model which is a multi-lingual model that supports text embeddings in 100 languages. You can read more about this model [here](https://huggingface.co/intfloat/multilingual-e5-small) + +## Synchronize your deployed model +Go to the **Machine Learning > Trained Models** page http://localhost:5601/app/ml/trained_models and synchronize your trained models. A warning message is displayed at the top of the page that says "ML job and trained model synchronization required". Follow the link to "Synchronize your jobs and trained models." Then click Synchronize. + + + +Once you synchronize your model you should see your deployed model on the **Machine Learning > Model Management** page in Kibana. + + + +## Test your deployed model +Run the command below to test the model using the _infer API +```bash +curl -X POST "${ES_URL}/_ml/trained_models/intfloat__multilingual-e5-small/_infer" -u "${ES_USER}:${ES_PASSWORD}" -H "Content-Type: application/json" --cacert $ES_CACERT -d '{ + "docs": { + "text_field": "how to set up custom extension?" + } +}' +``` +You should see a response containing the predicted embedding vector. + +```bash +{ + "inference_results": [ + { + "predicted_value": [ + 0.016921168193221092, + -0.035475824028253555, + -0.0497407428920269, + ... +``` + +## Load sample data +Refer to the [Load data into Elasticsearch](./ICD_Elasticsearch_install_and_setup.md#load-data-into-elasticsearch) section in the Elasticsearch setup guide to upload a sample data to Elasticsearch using Kabana. + +## Add your embedding model to an inference ingest pipeline +Create an ingest pipeline using the command below: +```bash +curl -X PUT "${ES_URL}/_ingest/pipeline/${ES_PIPELINE_NAME}" \ + -u "${ES_USER}:${ES_PASSWORD}" --cacert "${ES_CACERT}"\ + -H 'Content-Type: application/json' -d '{ + "description": "Text embedding pipeline", + "processors": [ + { + "inference": { + "model_id": "intfloat__multilingual-e5-small", + "target_field": "text_embedding", + "field_map": { + "text": "text_field" + } + } + } + ], + "on_failure": [ + { + "set": { + "description": "Index document to '\''failed-'\''", + "field": "_index", + "value": "failed-{{{_index}}}" + } + }, + { + "set": { + "description": "Set error message", + "field": "ingest.failure", + "value": "{{_ingest.on_failure_message}}" + } + } + ] +}' +``` + +You can verify that the ingest pipeline was created by locating it in the list of your ingest pipelines on Kibana http://localhost:5601/app/management/ingest/ingest_pipelines + +## Create a mapping for the destination index containing the embeddings +Then run the command below to create the mappings of the destination index called `ES_EMBEDDING_INDEX_NAME`: +```bash +curl -X PUT "${ES_URL}/${ES_EMBEDDING_INDEX_NAME}" \ + -u "${ES_USER}:${ES_PASSWORD}" --cacert $ES_CACERT \ + -H 'Content-Type: application/json' -d '{ + "mappings": { + "properties": { + "text_embedding.predicted_value": { + "type": "dense_vector", + "dims": 384, + "index": true, + "similarity": "cosine" + }, + "text": { + "type": "text" + } + } + } +}' +``` + +* `text_embedding.predicted_value` is the field where the ingest processor stores the embeddings +* `dims` is the embedding size of the deployed model which is 384 for the `intfloat/multilingual-e5-small` model we are using here + +## Create the text embeddings +Run the ingest pipeline to reindex the data to the `ES_EMBEDDING_INDEX_NAME` index +```bash +curl -X POST "${ES_URL}/_reindex?wait_for_completion=false" \ + -u "${ES_USER}:${ES_PASSWORD}" --cacert "$ES_CACERT" \ + -H 'Content-Type: application/json' -d "{ + \"source\": { + \"index\": \"${ES_SOURCE_INDEX_NAME}\", + \"size\": 50 + }, + \"dest\": { + \"index\": \"${ES_EMBEDDING_INDEX_NAME}\", + \"pipeline\": \"${ES_PIPELINE_NAME}\" + } +}" +``` + +This command will return a task id that looks like this: +```json +{"task":} +``` + +The reindexing process can take around 10 minutes. You can use the task id that is returned above to check the status of the process. + +```bash +curl -X GET "${ES_URL}/_tasks/" \ + -u "${ES_USER}:${ES_PASSWORD}" --cacert $ES_CACERT +``` + +You can check the completion status by monitoring the `"completed"` field in the response: + +```bash +{ + "completed": true, + ... +} +``` + +Once the process is completed, you should see `ES_EMBEDDING_INDEX_NAME` in the list of your indices http://localhost:5601/app/enterprise_search/content/search_indices + +You can confirm the successful completion of this step by checking the `ES_EMBEDDING_INDEX_NAME` index. If you find the `text_embedding` column filled with embedding vectors as shown below, it indicates that the process was successful: +```bash +{ + "predicted_value": [ + -0.016909973695874214, + -0.05246243625879288, + -0.02864678204059601, + ... + ], + "model_id": "intfloat__multilingual-e5-small" +} +``` +## Run semantic search +After the dataset has been enriched with vector embeddings, you can query the data using semantic search. +```bash +curl -X GET "${ES_URL}/${ES_EMBEDDING_INDEX_NAME}/_search" \ + -u "${ES_USER}:${ES_PASSWORD}" --cacert $ES_CACERT \ + -H 'Content-Type: application/json' -d '{ + "knn": { + "field": "text_embedding.predicted_value", + "query_vector_builder": { + "text_embedding": { + "model_id": "intfloat__multilingual-e5-small", + "model_text": "how to set up custom extension?" + } + }, + "k": 10, + "num_candidates": 100 + }, + "_source": [ + "id", + "text" + ] +}' +``` + +## What's next? +Now that you've successfully deployed your text embedding model in Elasticsearch, see [Elasticsearch integration with Agent Knowledge in watsonx Orchestrate](README.md#elasticsearch-integration-with-agent-knowledge-in-watsonx-orchestrate) to set up Agent Knowledge using your Elasticsearch index. + +Here is an example Elasticsearch query body: +``` +{ + "knn": { + "field": "text_embedding.predicted_value", + "query_vector_builder": { + "text_embedding": { + "model_id": "intfloat__multilingual-e5-small", + "model_text": "$QUERY" + } + }, + "k": 10, + "num_candidates": 100 + }, + "_source": [ + "id", + "text" + ] +} +``` diff --git a/agent_knowledge/elasticsearch-install-and-setup/watsonx_discovery_install_and_setup.md b/agent_knowledge/elasticsearch-install-and-setup/watsonx_discovery_install_and_setup.md new file mode 100644 index 0000000..6a26aaa --- /dev/null +++ b/agent_knowledge/elasticsearch-install-and-setup/watsonx_discovery_install_and_setup.md @@ -0,0 +1,312 @@ +# How to set up watsonx Discovery (Elasticsearch) and integrate it with watsonx Orchestrate in CloudPak +This is a documentation about how to set up watsonx Discovery (aka Elasticsearch on-prem) and integrate it with watsonx Orchestrate in CloudPak. + +## Table of contents: +* [Step 1: Install Elastic Cloud on Kubernetes(ECK) on CloudPak](#step-1-install-elastic-cloud-on-kubernetes-eck-in-cloudpak) +* [Step 2: Enable semantic search with ELSER in Elasticsearch](#step-2-enable-semantic-search-with-elser-in-elasticsearch) + + +## Step 1: Install Elastic Cloud on Kubernetes(ECK) on CloudPak +This step is about installing Elastic Cloud on Kubernetes (ECK) in CloudPak. + +Before you begin, you will need: +* Access to a CloudPak cluster +* An ECK Enterprise orchestration license + +### Log in to your CloudPak cluster +* Log in to your Redhat OpenShift console using your admin credentials. +* Find your login API token by clicking on `Copy login command` from the top right dropdown menu. +* After displaying the token, copy the `Log in with this token` command, and run it from your terminal window. + NOTE: If you don't have the OpenShift CLI (`oc`) installed, you can find the instructions to install it by clicking on + the `Command line tools` from the top right :grey_question: dropdown menu. +* If you see logs like below, you have successfully logged into your CloudPak cluster + ``` + Logged into "https://api.wa24.cp.fyre.ibm.com:6443" as "kube:admin" using the token provided. + + You have access to 78 projects, the list has been suppressed. You can list all projects with 'oc projects' + + Using project "default". + ``` + +### Install ECK with Elasticsearch cluster and Kibana +* Create environment variables for the installation, for example, + ```shell + export ES_NAMESPACE="elastic" + export STORAGECLASS="ocs-storagecluster-ceph-rbd" + export ES_CLUSTER="wxd" + ``` + NOTE: you need to find a valid StorageClass name. `oc get sc` will show you all the StorageClass objects. +* Create a new namespace for ECK installation + ```shell + oc create ns ${ES_NAMESPACE} + ``` +* Create ECK crds + ```shell + oc create -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml + ``` + NOTE: `2.9.0` is the latest supported version by watsonx Discovery +* Create ECK operator in a different namespace `elastic-system` + ```shell + oc apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml + ``` +* Create a Elasticsearch cluster + ```shell + cat < tls.crt + + export ES_CACERT="tls.crt" + ``` + NOTE: you can skip the cert if you are okay with insecure connections. Instead of providing `--cacert` to your curl commands, + you can use `-k` or `--insecure` for insecure connections. +* Create other environment variables for Elasticsearch credentials + ```shell + export ES_URL=https://localhost:9200 + export ES_USER= + export ES_PASSWORD= + ``` + You have obtained `ES_USER` and `ES_PASSWORD` from [obtain-the-elasticsearch-credentials](#verify-the-installation) step. + +### Enable ELSER model (v2) +ELSER model is not enabled by default, but you can enable it in Kibana. Please follow the [download-deploy-elser instructions](https://www.elastic.co/guide/en/machine-learning/8.11/ml-nlp-elser.html#download-deploy-elser) to do it. + +Note: `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Learn more about [inference-ingest-pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/semantic-search-elser.html#inference-ingest-pipeline) from the tutorial. + +### Load data into Elasticsearch +In Kibana, you can upload a data file to Elasticsearch cluster using the Data Visualizer in the Machine Learning UI https://localhost:5601/app/ml/filedatavisualizer. +(You need to use `port-forward` to allow local access to Kibana service and login using Elasticsearch credentials as described in +[verify-the-installation](#verify-the-installation) step) + +As an example, you can download [wa-docs-100](./assets/wa_docs_100.tsv) TSV data and upload it to Elasticsearch. +This dataset contains documents processed from the watsonx Assistant product documents. There are three columns in this TSV file, +`title`, `section_title` and `text`. The columns are extracted from the original documents. Specifically, +each `text` value is a small chunk of text split from the original document. + +In Kibana, +* Select your downloaded file to upload + +* Click `Override settings` and then check `Has header row` checkbox because the example dataset has header row + +* Import the data to a new Elasticsearch index and name it `wa-docs` + + Once finished, you have created an index for the data you just uploaded. + +### Create an index with mappings for ELSER output + ```bash + curl -X PUT "${ES_URL}/search-wa-docs?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "mappings": { + "_source": { + "excludes": [ + "text_embedding" + ] + }, + "properties": { + "text_embedding": { + "type": "sparse_vector" + }, + "text": { + "type": "text" + } + } + } + }' + ``` +Notes: +* `search-wa-docs` will be your index name +* `text_embedding` is the field that will keep ELSER output when data is ingested, and `sparse_vector` type is required for ELSER output field +* `text` is the input filed for the inference processor. In the example dataset, the name of the input field is `text` which will be used by ELSER model to process. +* Learn more about [elser-mappings](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/semantic-search-elser.html#elser-mappings) from the tutorial + +### Create an ingest pipeline with an inference processor +Create an ingest pipeline with an inference processor to use ELSER to infer against the data that will be ingested in the pipeline. + ```bash + curl -X PUT "${ES_URL}/_ingest/pipeline/elser-v2-test?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "processors": [ + { + "inference": { + "model_id": ".elser_model_2_linux-x86_64", + "input_output": [ + { + "input_field": "text", + "output_field": "text_embedding" + } + ] + } + } + ] + }' + ``` +Note: `.elser_model_2_linux-x86_64` is an optimized version of the ELSER v2 model and is preferred to use if it is available. Learn more about [inference-ingest-pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/semantic-search-elser.html#inference-ingest-pipeline) from the tutorial. + +### Ingest the data through the inference ingest pipeline +Create the tokens from the text by reindexing the data through the inference pipeline that uses ELSER as the inference model. + ```bash + curl -X POST "${ES_URL}/_reindex?wait_for_completion=false&pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "source": { + "index": "wa-docs" + }, + "dest": { + "index": "search-wa-docs", + "pipeline": "elser-v2-test" + } + }' + ``` +* `wa-docs` is the index you created when uploading the example file to Elasticsearch cluster. It contains the text data +* `search_wa-docs` is the search index that has ELSER output field +* `elser-v2-test` is the ingest pipeline with an inference processor using ELSER model + +### Semantic search by using the text_expansion query +To perform semantic search, use the `text_expansion` query, and provide the query text and the ELSER model ID. +The example below uses the query text "How to set up custom extension?", the `text_embedding` field contains +the generated ELSER output: + ```bash + curl -X GET "${ES_URL}/search-wa-docs/_search?pretty" -u "${ES_USER}:${ES_PASSWORD}" \ + -H "Content-Type: application/json" --cacert "${ES_CACERT}" -d' + { + "query":{ + "text_expansion":{ + "text_embedding":{ + "model_id":".elser_model_2_linux-x86_64", + "model_text":"how to set up custom extension?" + } + } + } + }' + ``` +Note: Learn more about [text-expansion-query](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/semantic-search-elser.html#text-expansion-query) from the tutorial. + + +## What's next? +Now you have successfully created your Elasticsearch index with semantic search enabled, see [Elasticsearch integration with Agent Knowledge in watsonx Orchestrate](README.md#elasticsearch-integration-with-agent-knowledge-in-watsonx-orchestrate) to set up Agent Knowledge using your Elasticsearch index.