Skip to content

Commit da0e41b

Browse files
kderussokderusso
committed
Update to use exclude rules, match queries
1 parent d5dff87 commit da0e41b

File tree

1 file changed

+76
-63
lines changed

1 file changed

+76
-63
lines changed

‎notebooks/enterprise-search/app-search-engine-exporter.ipynb‎

Lines changed: 76 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@
6666
},
6767
{
6868
"cell_type": "code",
69-
"execution_count": 2,
69+
"execution_count": null,
7070
"metadata": {},
7171
"outputs": [],
7272
"source": [
@@ -95,12 +95,12 @@
9595
"\n",
9696
"You can find your App Search endpoint and your search private key from the `Credentials` menu inside your App Search instance in Kibana.\n",
9797
"\n",
98-
"Also note here, we define our `ENGINE_NAME`. For this examplem we are using the `national-parks-demo` sample engine that is available within App Search."
98+
"Also note here, we define our `ENGINE_NAME`. For this example, we are using the `national-parks-demo` sample engine that is available within App Search."
9999
]
100100
},
101101
{
102102
"cell_type": "code",
103-
"execution_count": 3,
103+
"execution_count": null,
104104
"metadata": {},
105105
"outputs": [],
106106
"source": [
@@ -129,7 +129,7 @@
129129
},
130130
{
131131
"cell_type": "code",
132-
"execution_count": 4,
132+
"execution_count": null,
133133
"metadata": {
134134
"id": "kpV8K5jHvRK6"
135135
},
@@ -173,9 +173,9 @@
173173
"\n",
174174
"Next, we will export any curations that may be in our App Search engine.\n",
175175
"\n",
176-
"To export App Search curations we will use Elasticsearch [query rules](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-using-query-rules.html).\n",
177-
"At the moment of writing this notebook Elasticsearch query rules only allow for pinning results unlike App Search curations that also allow excluding results.\n",
178-
"For this reason we will only export pinned results. The code below will create the necessary `query_rules` to achieve this. Note that there is a default soft limit of 100 curations for `query_rules` that can be configured up to a hard limit of 1,000."
176+
"To export App Search curations we will use Elasticsearch [query rules](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-using-query-rules.html). The code below will create the necessary `query_rules` to achieve this. Note that there is a default soft limit of 100 curations for `query_rules` that can be configured up to a hard limit of 1,000.\n",
177+
"\n",
178+
"NOTE: This example outputs query rules requiring `exact` matches, which are case-sensitive. If you need typo tolerance, consider using `fuzzy`. If you need different case values consider adding multiple values to your criteria. "
179179
]
180180
},
181181
{
@@ -187,24 +187,56 @@
187187
"query_rules = []\n",
188188
"\n",
189189
"for curation in app_search.list_curations(engine_name=ENGINE_NAME).body[\"results\"]:\n",
190-
" query_rules.append(\n",
191-
" {\n",
192-
" \"rule_id\": curation[\"id\"],\n",
193-
" \"type\": \"pinned\",\n",
194-
" \"criteria\": [\n",
195-
" {\n",
196-
" \"type\": \"exact\",\n",
197-
" \"metadata\": \"user_query\",\n",
198-
" \"values\": curation[\"queries\"],\n",
199-
" }\n",
200-
" ],\n",
201-
" \"actions\": {\"ids\": curation[\"promoted\"]},\n",
202-
" }\n",
203-
" )\n",
190+
" if (curation[\"promoted\"]):\n",
191+
" query_rules.append(\n",
192+
" {\n",
193+
" \"rule_id\": curation[\"id\"] + \"-pinned\",\n",
194+
" \"type\": \"pinned\",\n",
195+
" \"criteria\": [\n",
196+
" {\n",
197+
" \"type\": \"exact\",\n",
198+
" \"metadata\": \"user_query\",\n",
199+
" \"values\": curation[\"queries\"],\n",
200+
" }\n",
201+
" ],\n",
202+
" \"actions\": {\"ids\": curation[\"promoted\"]},\n",
203+
" }\n",
204+
" )\n",
205+
" if(curation[\"hidden\"]):\n",
206+
" query_rules.append(\n",
207+
" {\n",
208+
" \"rule_id\": curation[\"id\"] + \"-exclude\",\n",
209+
" \"type\": \"exclude\",\n",
210+
" \"criteria\": [\n",
211+
" {\n",
212+
" \"type\": \"exact\",\n",
213+
" \"metadata\": \"user_query\",\n",
214+
" \"values\": curation[\"queries\"],\n",
215+
" }\n",
216+
" ],\n",
217+
" \"actions\": {\"ids\": curation[\"hidden\"]},\n",
218+
" }\n",
219+
" )\n",
204220
"\n",
205221
"elasticsearch.query_rules.put_ruleset(ruleset_id=ENGINE_NAME, rules=query_rules)"
206222
]
207223
},
224+
{
225+
"cell_type": "markdown",
226+
"metadata": {},
227+
"source": [
228+
"Let's take a quick look at the query rules we've migrated. We'll do this via the `GET _query_rules/ENGINE_NAME` endpoint. Note that curations with both pinned and hidden documents will be represented as two rules in the ruleset."
229+
]
230+
},
231+
{
232+
"cell_type": "code",
233+
"execution_count": null,
234+
"metadata": {},
235+
"outputs": [],
236+
"source": [
237+
"print(json.dumps(elasticsearch.query_rules.get_ruleset(ruleset_id=ENGINE_NAME).body, indent=2))"
238+
]
239+
},
208240
{
209241
"cell_type": "markdown",
210242
"metadata": {
@@ -215,7 +247,15 @@
215247
"\n",
216248
"We recommend reindexing your App Search engine data into a new Elasticsearch index instead of reusing the existing one. This allows you to update the index mapping to take advantage of modern features like semantic search and the newly created Elasticsearch synonym set.\n",
217249
"\n",
218-
"App Search has the following data types: `text`, `number`, `date` and `geolocation`. Each of these types is mapped to Elasticsearch field types.\n",
250+
"App Search has the following data types:\n",
251+
"\n",
252+
"- `text`\n",
253+
"- `number`\n",
254+
"- `date`\n",
255+
"- `geolocation`\n",
256+
" \n",
257+
"Each of these types is mapped to Elasticsearch field types.\n",
258+
"\n",
219259
"We can take a closer look at how App Search field types are mapped to Elasticsearch fields, by using the [`GET mapping API`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html).\n",
220260
"For App Search engines, the associated Elasticsearch index name is `.ent-search-engine-documents-[ENGINE_NAME]`, e.g. `.ent-search-engine-documents-national-parks-demo` for the App Search sample engine `national-parks-demo`.\n",
221261
"One thing to notice is how App Search uses [multi-fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) in Elasticsearch that allow for quickly changing the field type in App Search without requiring reindexing by creating subfields for each type of supported field:\n",
@@ -578,38 +618,11 @@
578618
"source": [
579619
"# Add semantic text fields for semantic search (optional)\n",
580620
"\n",
581-
"One of the advantages of exporting our index directly to Elasticsearch is that we can easily perform semantic search with ELSER. To do this, we'll need to add an inference endpoint using ELSER, and a `semantic_text` field to our index to use it.\n",
621+
"One of the advantages of exporting our index directly to Elasticsearch is that we can easily perform semantic search with ELSER. To do this, we'll need to add a `semantic_text` field to our index to use it. We will set up a `semantic_text` field using our default ELSER endpoint.\n",
582622
"\n",
583623
"Note that to use this feature, your cluster must have at least one ML node set up with enough resources allocated to it.\n",
584624
"\n",
585-
"If you have not already, be sure that your ELSER v2 model is [setup and deployed](https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-elser.html).\n",
586-
"\n",
587-
"Let's first start by creating our inference endpoint using the [Create inference API]](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html)."
588-
]
589-
},
590-
{
591-
"cell_type": "code",
592-
"execution_count": null,
593-
"metadata": {},
594-
"outputs": [],
595-
"source": [
596-
"# delete our inference endpoint if it is already created\n",
597-
"if elasticsearch.inference.get(inference_id=\"elser_inference_endpoint\"):\n",
598-
" elasticsearch.inference.delete(inference_id=\"elser_inference_endpoint\")\n",
599-
"\n",
600-
"# and create our endpoint using the ELSER v2 model\n",
601-
"elasticsearch.inference.put(\n",
602-
" inference_id=\"elser_inference_endpoint\",\n",
603-
" inference_config={\n",
604-
" \"service\": \"elasticsearch\",\n",
605-
" \"service_settings\": {\n",
606-
" \"model_id\": \".elser_model_2_linux-x86_64\",\n",
607-
" \"num_allocations\": 1,\n",
608-
" \"num_threads\": 1,\n",
609-
" },\n",
610-
" },\n",
611-
" task_type=\"sparse_embedding\",\n",
612-
")"
625+
"If you do not have an ELSER endpoint running, it will be automatically downloaded, deployed and started for you when you use `semantic_text`. This means the first few commands may take a while as the model loads."
613626
]
614627
},
615628
{
@@ -637,8 +650,7 @@
637650
"for field_name in SEMANTIC_TEXT_FIELDS:\n",
638651
" semantic_field_name = field_name + \"_semantic\"\n",
639652
" mapping[semantic_field_name] = {\n",
640-
" \"type\": \"semantic_text\",\n",
641-
" \"inference_id\": \"elser_inference_endpoint\",\n",
653+
" \"type\": \"semantic_text\"\n",
642654
" }\n",
643655
"\n",
644656
"# and for our text fields, add a \"copy_to\" directive to copy the text to the semantic_text field\n",
@@ -778,7 +790,7 @@
778790
"\n",
779791
"For the results, we sort on our score descending as the primary sort, with the document id as the secondary.\n",
780792
"\n",
781-
"We apply highlights to our results, request a return size of the top 10 hits, and for each hit, return the result fields."
793+
"We apply highlights to returned text search descriptions, request a return size of the top 10 hits, and for each hit, return the result fields."
782794
]
783795
},
784796
{
@@ -826,7 +838,7 @@
826838
" \"order\": \"score\",\n",
827839
" \"encoder\": \"html\",\n",
828840
" \"require_field_match\": False,\n",
829-
" \"fields\": {},\n",
841+
" \"fields\": { \"description\" : { \"pre_tags\" : [\"<em>\"], \"post_tags\" : [\"</em>\"] } },\n",
830842
" },\n",
831843
" \"size\": 10,\n",
832844
" \"_source\": result_fields,\n",
@@ -849,7 +861,7 @@
849861
"outputs": [],
850862
"source": [
851863
"results = elasticsearch.search(\n",
852-
" index=SOURCE_INDEX,\n",
864+
" index=DEST_INDEX,\n",
853865
" query=app_search_query_payload[\"query\"],\n",
854866
" highlight=app_search_query_payload[\"highlight\"],\n",
855867
" source=app_search_query_payload[\"_source\"],\n",
@@ -866,7 +878,9 @@
866878
"### How to do semantic search using ELSER with semantic text fields\n",
867879
"\n",
868880
"If you [enabled and reindexed your data with ELSER](#add-sparse_vector-fields-for-semantic-search-optional), we can now use this to do semantic search.\n",
869-
"For each `semantic_text` field type, we can define a [semantic query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-semantic-query.html) to easily perform a semantic search on these fields.\n"
881+
"For each `semantic_text` field type, we can define a [match query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html) to easily perform a semantic search on these fields.\n",
882+
"\n",
883+
"NOTE: For Elasticsearch versions prior to 8.18, we should update this to use a [semantic query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-semantic-query.html) to easily perform a semantic search on these fields.\n"
870884
]
871885
},
872886
{
@@ -883,9 +897,8 @@
883897
" semantic_field_name = field_name + \"_semantic\"\n",
884898
" semantic_text_queries.append(\n",
885899
" {\n",
886-
" \"semantic\": {\n",
887-
" \"field\": semantic_field_name,\n",
888-
" \"query\": QUERY_STRING,\n",
900+
" \"match\": {\n",
901+
" semantic_field_name: QUERY_STRING\n",
889902
" }\n",
890903
" }\n",
891904
" )\n",
@@ -926,7 +939,7 @@
926939
" \"should\": [\n",
927940
" // multi_match query with best_fields from App Search generated query\n",
928941
" // multi_match query with cross_fields from App Search generated query\n",
929-
" // text_expansion queries for sparse_vector fields\n",
942+
" // match queries for semantic_text fields\n",
930943
" ]\n",
931944
" }\n",
932945
" } \n",
@@ -960,7 +973,7 @@
960973
"outputs": [],
961974
"source": [
962975
"results = elasticsearch.search(\n",
963-
" index=SOURCE_INDEX,\n",
976+
" index=DEST_INDEX,\n",
964977
" query=payload[\"query\"],\n",
965978
" highlight=payload[\"highlight\"],\n",
966979
" source=payload[\"_source\"],\n",
@@ -969,7 +982,7 @@
969982
" min_score=1,\n",
970983
")\n",
971984
"\n",
972-
"print(f\"Text expansion query results:\\n{json.dumps(results.body, indent=2)}\\n\")"
985+
"print(f\"Semantic query results:\\n{json.dumps(results.body, indent=2)}\\n\")"
973986
]
974987
}
975988
],

0 commit comments

Comments
 (0)