Skip to content

Commit 5545043

Browse files
HeidiSteenHeidiSteen
committed
Updated python quickstart and semantic quickstart
1 parent 7560810 commit 5545043

File tree

3 files changed

+168
-246
lines changed

3 files changed

+168
-246
lines changed

‎Quickstart-Semantic-Search/semantic-search-quickstart.ipynb‎

Lines changed: 75 additions & 125 deletions
Original file line numberDiff line numberDiff line change
@@ -5,52 +5,22 @@
55
"cell_type": "markdown",
66
"metadata": {},
77
"source": [
8-
"# Semantic search using the azure.search.documents library in the Azure SDK for Python"
8+
"# Semantic ranking using the azure.search.documents library in the Azure SDK for Python"
99
]
1010
},
1111
{
1212
"attachments": {},
1313
"cell_type": "markdown",
1414
"metadata": {},
1515
"source": [
16-
"This Jupyter Notebook adds semantic search, using pre-trained models from Microsoft to re-rank results based on a semantic match to the query. "
16+
"This notebook demonstrates a semantic configuration in a search index and the semanti query syntax for reranking search results."
1717
]
1818
},
1919
{
20-
"cell_type": "code",
21-
"execution_count": null,
22-
"metadata": {},
23-
"outputs": [],
24-
"source": [
25-
"# Import the libraries needed to create a search index, upload documents, and query the index\n",
26-
"%pip install azure-search-documents\n",
27-
"%pip show azure-search-documents\n",
28-
"%pip install python-dotenv\n",
29-
"\n",
30-
"import os\n",
31-
"from azure.core.credentials import AzureKeyCredential\n",
32-
"from azure.search.documents.indexes import SearchIndexClient \n",
33-
"from azure.search.documents import SearchClient\n",
34-
"from azure.search.documents.indexes.models import ( \n",
35-
" SearchIndex, \n",
36-
" SearchFieldDataType, \n",
37-
" SimpleField, \n",
38-
" SearchableField,\n",
39-
" ComplexField,\n",
40-
" SearchIndex, \n",
41-
" SemanticConfiguration, \n",
42-
" PrioritizedFields, \n",
43-
" SemanticField, \n",
44-
" SemanticSettings, \n",
45-
")"
46-
]
47-
},
48-
{
49-
"attachments": {},
5020
"cell_type": "markdown",
5121
"metadata": {},
5222
"source": [
53-
"In this step, initialize the search client used to make each request. Provide the name and admin API key of your search service. If you get ConnectionError \"Failed to establish a new connection\", verify that the api-key is a primary or secondary admin key, and not a query key."
23+
"## Install packages and set variables"
5424
]
5525
},
5626
{
@@ -59,30 +29,9 @@
5929
"metadata": {},
6030
"outputs": [],
6131
"source": [
62-
"# Set the service endpoint and API key from the environment\n",
63-
"\n",
64-
"service_name = \"<YOUR-SEARCH-SERVICE-NAME>\"\n",
65-
"admin_key = \"<YOUR-SEARCH-SERVICE-ADMIN-KEY>\"\n",
66-
"\n",
67-
"index_name = \"hotels-quickstart\"\n",
68-
"\n",
69-
"# Create an SDK client\n",
70-
"endpoint = \"https://{}.search.windows.net/\".format(service_name)\n",
71-
"admin_client = SearchIndexClient(endpoint=endpoint,\n",
72-
" index_name=index_name,\n",
73-
" credential=AzureKeyCredential(admin_key))\n",
74-
"\n",
75-
"search_client = SearchClient(endpoint=endpoint,\n",
76-
" index_name=index_name,\n",
77-
" credential=AzureKeyCredential(admin_key))\n"
78-
]
79-
},
80-
{
81-
"attachments": {},
82-
"cell_type": "markdown",
83-
"metadata": {},
84-
"source": [
85-
"In the next cell, the index \"hotels-quickstart\" will be deleted if it previously existed. This step allows you to reuse the index name."
32+
"! pip install azure-search-documents==11.6.0b1 --quiet\n",
33+
"! pip install azure-identity --quiet\n",
34+
"! pip install python-dotenv --quiet"
8635
]
8736
},
8837
{
@@ -91,20 +40,18 @@
9140
"metadata": {},
9241
"outputs": [],
9342
"source": [
94-
"# Delete the index if it exists\n",
95-
"try:\n",
96-
" result = admin_client.delete_index(index_name)\n",
97-
" print ('Index', index_name, 'Deleted')\n",
98-
"except Exception as ex:\n",
99-
" print (ex)\n"
43+
"# Provide variables\n",
44+
"search_endpoint: str = \"PUT-YOUR-SEARCH-ENDPOINT-HERE\"\n",
45+
"search_api_key: str = \"PUT-YOUR-SEARCH-API-KEY-HERE\"\n",
46+
"index_name: str = \"hotels-quickstart\""
10047
]
10148
},
10249
{
10350
"attachments": {},
10451
"cell_type": "markdown",
10552
"metadata": {},
10653
"source": [
107-
"Specify the index definition, including the fields that define each search document. This schema adds a semantic configuration that specifies how to use search fields during semantic ranking."
54+
"## Create an index"
10855
]
10956
},
11057
{
@@ -113,15 +60,34 @@
11360
"metadata": {},
11461
"outputs": [],
11562
"source": [
116-
"# Specify the index schema\n",
117-
"name = index_name\n",
63+
"from azure.core.credentials import AzureKeyCredential\n",
64+
"\n",
65+
"credential = AzureKeyCredential(search_api_key)\n",
66+
"\n",
67+
"from azure.search.documents.indexes import SearchIndexClient\n",
68+
"from azure.search.documents import SearchClient\n",
69+
"from azure.search.documents.indexes.models import (\n",
70+
" ComplexField,\n",
71+
" SimpleField,\n",
72+
" SearchFieldDataType,\n",
73+
" SearchableField,\n",
74+
" SearchIndex,\n",
75+
" SemanticConfiguration,\n",
76+
" SemanticField,\n",
77+
" SemanticPrioritizedFields,\n",
78+
" SemanticSearch\n",
79+
")\n",
80+
"\n",
81+
"# Create a search schema\n",
82+
"index_client = SearchIndexClient(\n",
83+
" endpoint=search_endpoint, credential=credential)\n",
11884
"fields = [\n",
11985
" SimpleField(name=\"HotelId\", type=SearchFieldDataType.String, key=True),\n",
12086
" SearchableField(name=\"HotelName\", type=SearchFieldDataType.String, sortable=True),\n",
12187
" SearchableField(name=\"Description\", type=SearchFieldDataType.String, analyzer_name=\"en.lucene\"),\n",
12288
" SearchableField(name=\"Description_fr\", type=SearchFieldDataType.String, analyzer_name=\"fr.lucene\"),\n",
12389
" SearchableField(name=\"Category\", type=SearchFieldDataType.String, facetable=True, filterable=True, sortable=True),\n",
124-
" \n",
90+
"\n",
12591
" SearchableField(name=\"Tags\", collection=True, type=SearchFieldDataType.String, facetable=True, filterable=True),\n",
12692
"\n",
12793
" SimpleField(name=\"ParkingIncluded\", type=SearchFieldDataType.Boolean, facetable=True, filterable=True, sortable=True),\n",
@@ -136,54 +102,35 @@
136102
" SearchableField(name=\"Country\", type=SearchFieldDataType.String, facetable=True, filterable=True, sortable=True),\n",
137103
" ])\n",
138104
" ]\n",
105+
"\n",
139106
"semantic_config = SemanticConfiguration(\n",
140107
" name=\"my-semantic-config\",\n",
141-
" prioritized_fields=PrioritizedFields(\n",
108+
" prioritized_fields=SemanticPrioritizedFields(\n",
142109
" title_field=SemanticField(field_name=\"HotelName\"),\n",
143-
" prioritized_keywords_fields=[SemanticField(field_name=\"Category\")],\n",
144-
" prioritized_content_fields=[SemanticField(field_name=\"Description\")]\n",
110+
" keywords_fields=[SemanticField(field_name=\"Category\")],\n",
111+
" content_fields=[SemanticField(field_name=\"Description\")]\n",
145112
" )\n",
146113
")\n",
147114
"\n",
148-
"semantic_settings = SemanticSettings(configurations=[semantic_config])\n",
115+
"# Create the semantic settings with the configuration\n",
116+
"semantic_search = SemanticSearch(configurations=[semantic_config])\n",
117+
"\n",
118+
"semantic_settings = SemanticSearch(configurations=[semantic_config])\n",
149119
"scoring_profiles = []\n",
150-
"suggester = [{'name': 'sg', 'source_fields': ['Tags', 'Address/City', 'Address/Country']}]\n"
151-
]
152-
},
153-
{
154-
"attachments": {},
155-
"cell_type": "markdown",
156-
"metadata": {},
157-
"source": [
158-
"Formulate the create_index request. This request targets the indexes collection of your search service and creates an index using the index schema from the previous cell."
159-
]
160-
},
161-
{
162-
"cell_type": "code",
163-
"execution_count": null,
164-
"metadata": {},
165-
"outputs": [],
166-
"source": [
167-
"index = SearchIndex(\n",
168-
" name=name,\n",
169-
" fields=fields,\n",
170-
" semantic_settings=semantic_settings,\n",
171-
" scoring_profiles=scoring_profiles,\n",
172-
" suggesters = suggester)\n",
120+
"suggester = [{'name': 'sg', 'source_fields': ['Tags', 'Address/City', 'Address/Country']}]\n",
173121
"\n",
174-
"try:\n",
175-
" result = admin_client.create_index(index)\n",
176-
" print ('Index', result.name, 'created')\n",
177-
"except Exception as ex:\n",
178-
" print (ex)"
122+
"# Create the search index with the semantic settings\n",
123+
"index = SearchIndex(name=index_name, fields=fields, suggesters=suggester, scoring_profiles=scoring_profiles, semantic_search=semantic_search)\n",
124+
"result = index_client.create_or_update_index(index)\n",
125+
"print(f' {result.name} created')"
179126
]
180127
},
181128
{
182129
"attachments": {},
183130
"cell_type": "markdown",
184131
"metadata": {},
185132
"source": [
186-
"Next, set up documents to include four hotel documents conforming to the schema."
133+
"## Create a documents payload"
187134
]
188135
},
189136
{
@@ -192,6 +139,7 @@
192139
"metadata": {},
193140
"outputs": [],
194141
"source": [
142+
"# Create a documents payload\n",
195143
"documents = [\n",
196144
" {\n",
197145
" \"@search.action\": \"upload\",\n",
@@ -277,7 +225,7 @@
277225
"cell_type": "markdown",
278226
"metadata": {},
279227
"source": [
280-
"Formulate the request. This upload_documents request targets the docs collection of the hotels-quickstart index and pushes the documents from the previous step into the search index."
228+
"## Upload documents"
281229
]
282230
},
283231
{
@@ -286,23 +234,27 @@
286234
"metadata": {},
287235
"outputs": [],
288236
"source": [
237+
"# Upload documents to the index\n",
238+
"search_client = SearchClient(endpoint=search_endpoint,\n",
239+
" index_name=index_name,\n",
240+
" credential=credential)\n",
289241
"try:\n",
290242
" result = search_client.upload_documents(documents=documents)\n",
291243
" print(\"Upload of new document succeeded: {}\".format(result[0].succeeded))\n",
292244
"except Exception as ex:\n",
293-
" print (ex.message)"
245+
" print (ex.message)\n",
246+
"\n",
247+
"\n",
248+
" index_client = SearchIndexClient(\n",
249+
" endpoint=search_endpoint, credential=credential)"
294250
]
295251
},
296252
{
297253
"attachments": {},
298254
"cell_type": "markdown",
299255
"metadata": {},
300256
"source": [
301-
"You're now ready to run some queries. For this operation, use search_client. \n",
302-
"\n",
303-
"### Empty query with unscored results\n",
304-
"\n",
305-
"The next cell contains a query expression that executes an empty search (`search=*`), returning an unranked list (search score = 1.0) of arbitrary documents. Because there is no criteria, all documents are included in results. This query prints fields from each document. It also adds `include_total_count=True` to get a count of all documents (4) in the results."
257+
"## Run your first query"
306258
]
307259
},
308260
{
@@ -311,6 +263,7 @@
311263
"metadata": {},
312264
"outputs": [],
313265
"source": [
266+
"# Run an empty query (returns selected fields, all documents)\n",
314267
"results = search_client.search(query_type='simple',\n",
315268
" search_text=\"*\" ,\n",
316269
" select='HotelName,Description',\n",
@@ -328,9 +281,7 @@
328281
"cell_type": "markdown",
329282
"metadata": {},
330283
"source": [
331-
"### Full text search with BM25 ranking\n",
332-
"\n",
333-
"The previous query used an empty search string, which bypasses the search engine. In this query, search for \"what hotel has a good restaurant on site\". The query string undergoes lexical analysis and tokenization. The search engine scans for matches and assigns a search score based on term frequency and proximity. Higher scoring matches are returned first. In this query for \"what hotel has a good restaurant on site\", Sublime Cliff Hotel comes out on top because it's description includes \"site\". Terms that occur infrequently raise the search score of the document."
284+
"## Run a term query"
334285
]
335286
},
336287
{
@@ -339,10 +290,12 @@
339290
"metadata": {},
340291
"outputs": [],
341292
"source": [
293+
"# Run a text query (returns a BM25-scored result set)\n",
342294
"results = search_client.search(query_type='simple',\n",
343295
" search_text=\"what hotel has a good restaurant on site\" ,\n",
344-
" select='HotelName,HotelId,Description')\n",
345-
"\n",
296+
" select='HotelName,HotelId,Description',\n",
297+
" include_total_count=True)\n",
298+
" \n",
346299
"for result in results:\n",
347300
" print(result[\"@search.score\"])\n",
348301
" print(result[\"HotelName\"])\n",
@@ -354,9 +307,7 @@
354307
"cell_type": "markdown",
355308
"metadata": {},
356309
"source": [
357-
"### Semantic search with captions\n",
358-
"\n",
359-
"Here's the same query, but with semantic ranking. Notice that the semantic ranker correctly identifies Triple Landscape Hotel as a more relevant result given the initial query. This query also returns captions generated by the models. The inputs are too minimal in this sample to create interesting captions, but the example succeeds in demonstrating the syntax."
310+
"## Run a semantic query"
360311
]
361312
},
362313
{
@@ -365,6 +316,7 @@
365316
"metadata": {},
366317
"outputs": [],
367318
"source": [
319+
"# Runs a semantic query (runs a BM25-ranked query and promotes the most relevant matches to the top)\n",
368320
"results = search_client.search(query_type='semantic', semantic_configuration_name='my-semantic-config',\n",
369321
" search_text=\"what hotel has a good restaurant on site\", \n",
370322
" select='HotelName,Description,Category', query_caption='extractive')\n",
@@ -373,7 +325,7 @@
373325
" print(result[\"@search.reranker_score\"])\n",
374326
" print(result[\"HotelName\"])\n",
375327
" print(f\"Description: {result['Description']}\")\n",
376-
" \n",
328+
"\n",
377329
" captions = result[\"@search.captions\"]\n",
378330
" if captions:\n",
379331
" caption = captions[0]\n",
@@ -384,13 +336,10 @@
384336
]
385337
},
386338
{
387-
"attachments": {},
388339
"cell_type": "markdown",
389340
"metadata": {},
390341
"source": [
391-
"### Add semantic answers\n",
392-
"\n",
393-
"Semantic search can generate answers to a query string that has the characteristics of a question. The generated answer is extracted verbatim from your content. To get a semantic answer, the question and answer must be closely aligned, and the model must find content that clearly answers the question. If potential answers don't have a high enough confidence score, the model won't return an answer. For demonstration purposes, the question in this example is designed to get a response so that you can see the syntax."
342+
"## Return semantic answers"
394343
]
395344
},
396345
{
@@ -399,9 +348,10 @@
399348
"metadata": {},
400349
"outputs": [],
401350
"source": [
351+
"# Run a semantic query that returns semantic answers \n",
402352
"results = search_client.search(query_type='semantic', semantic_configuration_name='my-semantic-config',\n",
403-
" search_text=\"what hotel stands out for its gastronomic excellence\", \n",
404-
" select='HotelName,Description,Category', query_caption='extractive', query_answer=\"extractive\",)\n",
353+
" search_text=\"what hotel is in a historic building\",\n",
354+
" select='HotelName,Description,Category', query_caption='extractive', query_answer=\"extractive\",)\n",
405355
"\n",
406356
"semantic_answers = results.get_answers()\n",
407357
"for answer in semantic_answers:\n",
@@ -415,7 +365,7 @@
415365
" print(result[\"@search.reranker_score\"])\n",
416366
" print(result[\"HotelName\"])\n",
417367
" print(f\"Description: {result['Description']}\")\n",
418-
" \n",
368+
"\n",
419369
" captions = result[\"@search.captions\"]\n",
420370
" if captions:\n",
421371
" caption = captions[0]\n",
@@ -448,7 +398,7 @@
448398
"outputs": [],
449399
"source": [
450400
"try:\n",
451-
" result = admin_client.delete_index(index_name)\n",
401+
" result = index_client.delete_index(index_name)\n",
452402
" print ('Index', index_name, 'Deleted')\n",
453403
"except Exception as ex:\n",
454404
" print (ex)"
@@ -469,7 +419,7 @@
469419
"outputs": [],
470420
"source": [
471421
"try:\n",
472-
" result = admin_client.get_index(index_name)\n",
422+
" result = index_client.get_index(index_name)\n",
473423
" print (result)\n",
474424
"except Exception as ex:\n",
475425
" print (ex)\n"

0 commit comments

Comments
 (0)