Merge pull request #109742 from HeidiSteen/heidist-search

[Azure Cognitive Search] paginated queries update

Author: megvanhuygen

articles/search/TOC.yml

@@ -261,7 +261,7 @@
       href: search-example-adventureworks-multilevel-faceting.md
   - name: Results
     items:
-    - name: Page-related features
+    - name: Work with results
       href: search-pagination-page-layout.md
     - name: Relevance tuning (scoring profiles)    
       href: index-add-scoring-profiles.md

articles/search/media/search-sku-tier/tiers.png

@@ -8,7 +8,7 @@ author: HeidiSteen
 ms.author: heidist
 ms.service: cognitive-search
 ms.topic: conceptual
-ms.date: 02/14/2020
+ms.date: 03/30/2020
 ---
 
 # Adjust capacity in Azure Cognitive Search
@@ -34,7 +34,8 @@ A single service must have sufficient resources to handle all workloads (indexin
 
 As a general rule, search applications tend to need more replicas than partitions, particularly when the service operations are biased toward query workloads. The section on [high availability](#HA) explains why.
 
-Adding more replicas or partitions increases your cost of running the service. Be sure to check the [pricing calculator](https://azure.microsoft.com/pricing/calculator/) to understand the billing implications of adding more nodes. The [chart below](#chart) can help you cross-reference the number of search units required for a specific configuration.
+> [!NOTE]
+> Adding more replicas or partitions increases the cost of running the service, and can introduce slight variations in how results are ordered. Be sure to check the [pricing calculator](https://azure.microsoft.com/pricing/calculator/) to understand the billing implications of adding more nodes. The [chart below](#chart) can help you cross-reference the number of search units required for a specific configuration. For more information on how additional replicas impact query processing, see [Ordering results](search-pagination-page-layout.md#ordering-results).
 
 ## How to allocate replicas and partitions
 

articles/search/search-capacity-planning.md

@@ -8,98 +8,95 @@ author: HeidiSteen
 ms.author: heidist
 ms.service: cognitive-search
 ms.topic: conceptual
-ms.date: 01/24/2020
+ms.date: 04/01/2020
 ---
 
 # How to work with search results in Azure Cognitive Search
-This article provides guidance on how to implement standard elements of a search results page, such as total counts, document retrieval, sort orders, and navigation. Page-related options that contribute data or information to your search results are specified through the [Search Document](https://docs.microsoft.com/rest/api/searchservice/Search-Documents) requests sent to your Azure Cognitive Search service. 
 
-In the REST API, requests include a GET command, path, and query parameters that inform the service what is being requested, and how to formulate the response. In the .NET SDK, the equivalent API is [DocumentSearchResult Class](https://docs.microsoft.com/dotnet/api/microsoft.azure.search.models.documentsearchresult-1).
+This article explains how to get a query response that comes back with a total count of matching documents, paginated results, sorted results, and hit-highlighted terms.
 
-To quickly generate a search page for your client, explore these options:
+The structure of a response is determined by parameters in the query: [Search Document](https://docs.microsoft.com/rest/api/searchservice/Search-Documents) in the REST API, or [DocumentSearchResult Class](https://docs.microsoft.com/dotnet/api/microsoft.azure.search.models.documentsearchresult-1) in the .NET SDK.
 
-+ Use the [application generator](search-create-app-portal.md) in the portal to create an HTML page with a search bar, faceted navigation, and results area.
-+ Follow the [Create your first app in C#](tutorial-csharp-create-first-app.md) tutorial to create a functional client.
+## Result composition
 
-Several code samples include a web front-end interface, which you can find here: [New York City Jobs demo app](https://aka.ms/azjobsdemo), [JavaScript sample code with a live demo site](https://github.com/liamca/azure-search-javascript-samples), and [CognitiveSearchFrontEnd](https://github.com/LuisCabrer/CognitiveSearchFrontEnd).
+While a search document might consist of a large number of fields, typically only a few are needed to represent each document in the result set. On a query request, append `$select=<field list>` to specify which fields show up in the response. A field must be attributed as **Retrievable** in the index to be included in a result. 
 
-> [!NOTE]
-> A valid request includes a number of elements, such as a service URL and path, HTTP verb, `api-version`, and so on. For brevity, we trimmed the examples to highlight just the syntax that is relevant to pagination. For more information about request syntax, see [Azure Cognitive Search REST APIs](https://docs.microsoft.com/rest/api/searchservice).
->
-
-## Total hits and Page Counts
-
-Showing the total number of results returned from a query, and then returning those results in smaller chunks, is fundamental to virtually all search pages.
-
-![][1]
-
-In Azure Cognitive Search, you use the `$count`, `$top`, and `$skip` parameters to return these values. The following example shows a sample request for total hits on an index called "online-catalog", returned as `@odata.count`:
-
-    GET /indexes/online-catalog/docs?$count=true
-
-Retrieve documents in groups of 15, and also show the total hits, starting at the first page:
-
-    GET /indexes/online-catalog/docs?search=*&$top=15&$skip=0&$count=true
-
-Paginating results requires both `$top` and `$skip`, where `$top` specifies how many items to return in a batch, and `$skip` specifies how many items to skip. In the following example, each page shows the next 15 items, indicated by the incremental jumps in the `$skip` parameter.
-
-    GET /indexes/online-catalog/docs?search=*&$top=15&$skip=0&$count=true
-
-    GET /indexes/online-catalog/docs?search=*&$top=15&$skip=15&$count=true
-
-    GET /indexes/online-catalog/docs?search=*&$top=15&$skip=30&$count=true
+Fields that work best include those that contrast and differentiate among documents, providing sufficient information to invite a click-through response on the part of the user. On an e-commerce site, it might be a product name, description, brand, color, size, price, and rating. For the hotels-sample-index built-in sample, it might be fields in the following example:
 
-## Layout
+```http
+POST /indexes/hotels-sample-index/docs/search?api-version=2019-05-06 
+    {  
+      "search": "sandy beaches",
+      "select": "HotelId, HotelName, Description, Rating, Address/City"
+      "count": true
+    }
+```
 
-On a search results page, you might want to show a thumbnail image, a subset of fields, and a link to a full product page.
+> [!NOTE]
+> If want to include image files in a result, such as a product photo or logo, store them outside of Azure Cognitive Search, but include a field in your index to reference the image URL in the search document. Sample indexes that support images in the results include the **realestate-sample-us** demo, featured in this [quickstart](search-create-app-portal.md), and the [New York City Jobs demo app](https://aka.ms/azjobsdemo).
 
- ![][2]
+## Results returned
 
-In Azure Cognitive Search, you would use `$select` and a [Search API request](https://docs.microsoft.com/rest/api/searchservice/search-documents) to implement this experience.
+By default, the search engine returns up to the first 50 matches, as determined by search score if the query is full text search, or in an arbitrary order for exact match queries.
 
-To return a subset of fields for a tiled layout:
+To return a different number of matching documents, add `$top` and `$skip` parameters to the query request. The following list explains the logic.
 
-    GET /indexes/online-catalog/docs?search=*&$select=productName,imageFile,description,price,rating
++ Add `$count=true` to get a count of the total number of matching documents within an index.
 
-Images and media files are not directly searchable and should be stored in another storage platform, such as Azure Blob storage, to reduce costs. In the index and documents, define a field that stores the URL address of the external content. You can then use the field as an image reference. The URL to the image should be in the document.
++ Return the first set of 15 matching documents plus a count of total matches: `GET /indexes/<INDEX-NAME>/docs?search=<QUERY STRING>&$top=15&$skip=0&$count=true`
 
-To retrieve a product description page for an **onClick** event, use [Lookup Document](https://docs.microsoft.com/rest/api/searchservice/Lookup-Document) to pass in the key of the document to retrieve. The data type of the key is `Edm.String`. In this example, it is *246810*.
++ Return the second set, skipping the first 15 to get the next 15: `$top=15&$skip=15`. Do the same for the third set of 15: `$top=15&$skip=30`
 
-    GET /indexes/online-catalog/docs/246810
+The results of paginated queries are not guaranteed to be stable if the underlying index is changing. Paging changes the value of `$skip` for each page, but each query is independent and operates on the current view of the data as it exists in the index at query time (in other words, there is no caching or snapshot of results, such as those found in a general purpose database).
+ 
+Following is an example of how you might get duplicates. Assume an index with four documents:
 
-## Sort by relevance, rating, or price
+    { "id": "1", "rating": 5 }
+    { "id": "2", "rating": 3 }
+    { "id": "3", "rating": 2 }
+    { "id": "4", "rating": 1 }
+ 
+Now assume you want results returned two at a time, ordered by rating. You would execute this query to get the first page of results: `$top=2&$skip=0&$orderby=rating desc`, producing the following results:
 
-Sort orders often default to relevance, but it's common to make alternative sort orders readily available so that customers can quickly reshuffle existing results into a different rank order.
+    { "id": "1", "rating": 5 }
+    { "id": "2", "rating": 3 }
+ 
+On the service, assume a fifth document is added to the index in between query calls: `{ "id": "5", "rating": 4 }`.  Shortly thereafter, you execute a query to fetch the second page: `$top=2&$skip=2&$orderby=rating desc`, and get these results:
 
- ![][3]
+    { "id": "2", "rating": 3 }
+    { "id": "3", "rating": 2 }
+ 
+Notice that document 2 is fetched twice. This is because the new document 5 has a greater value for rating, so it sorts before document 2 and lands on the first page. While this behavior might be unexpected, it's typical of how a search engine behaves.
 
-In Azure Cognitive Search, sorting is based on the `$orderby` expression, for all fields that are indexed as `"Sortable": true.` An `$orderby` clause is an OData expression. For information about syntax, see [OData expression syntax for filters and order-by clauses](query-odata-filter-orderby-syntax.md).
+## Ordering results
 
-Relevance is strongly associated with scoring profiles. You can use the default scoring, which relies on text analysis and statistics to rank order all results, with higher scores going to documents with more or stronger matches on a search term.
+For full text search queries, results are automatically ranked by a search score, calculated based on term frequency and proximity in a document, with higher scores going to documents having more or stronger matches on a search term. Search scores convey general sense of relevance, relative to other documents in the same results set, and are not guaranteed to be consistent from one query to the next.
 
-Alternative sort orders are typically associated with **onClick** events that call back to a method that builds the sort order. For example, given this page element:
+As you work with queries, you might notice small discrepancies in ordered results. There are several explanations for why this might occur.
 
- ![][4]
+| Condition | Description |
+|-----------|-------------|
+| Data volatility | The contents of an index varies as you add, modify, or delete documents. Term frequencies will change as index updates are processed over time, affecting the search scores of matching documents. |
+| Query execution location | For services using multiple replicas, queries are issued against each replica in parallel. The index statistics used to calculate a search score are calculated on a per-replica basis, with results merged and ordered in the query response. Replicas are mostly mirrors of each other, but statistics can differ due to small differences in state. For example, one replica might have deleted documents contributing to their statistics, which were merged out of other replicas. Generally, differences in per-replica statistics are more noticeable in smaller indexes. |
+| Breaking a tie between identical search scores | Discrepancies in ordered results can also occur when search documents have identical scores. In this case, as you rerun the same query, there is no guarantee which document will appear first. |
 
-You would create a method that accepts the selected sort option as input, and returns an ordered list for the criteria associated with that option.
+### Consistent ordering
 
- ![][5]
+Given the flex in search scoring, you might want to explore other options if consistency in result orders is an application requirement. The easiest approach is sorting by a field value, such as rating or date. For scenarios where you want to sort by a specific field, such as a rating or date, you can explicitly define an [`$orderby` expression](query-odata-filter-orderby-syntax.md), which can be applied to any field that is indexed as **Sortable**.
 
-> [!NOTE]
-> While the default scoring is sufficient for many scenarios, we recommend basing relevance on a custom scoring profile instead. A custom scoring profile gives you a way to boost items that are more beneficial to your business. See [Add scoring profiles](index-add-scoring-profiles.md) for more information.
->
+Another option is using a [custom scoring profile](index-add-scoring-profiles.md). Scoring profiles give you more control over the ranking of items in search results, with the ability to boost matches found in specific fields. The additional scoring logic can help override minor differences among replicas because the search scores for each document are farther apart. We recommend the [ranking algorithm](index-ranking-similarity.md) for this approach.
 
 ## Hit highlighting
 
-You can apply formatting to matching terms in search results, making it easy to spot the match. Hit highlighting instructions are provided on the [query request](https://docs.microsoft.com/rest/api/searchservice/search-documents). 
+Hit highlighting refers to text formatting (such as bold or yellow highlights) applied to matching term in a result, making it easy to spot the match. Hit highlighting instructions are provided on the [query request](https://docs.microsoft.com/rest/api/searchservice/search-documents). The search engine encloses the matching term in tags, `highlightPreTag` and `highlightPostTag`, and your code handles the response (for example, applying a bold font).
 
-Formatting is applied to whole term queries. Queries on partial terms, such as fuzzy search or wildcard search that result in query expansion in the engine, cannot use hit highlighting.
+Formatting is applied to whole term queries. In the following example, the terms "sandy", "sand", "beaches", "beach" found within the Description field are tagged for highlighting. Queries on partial terms, such as fuzzy search or wildcard search that result in query expansion in the engine, cannot use hit highlighting.
 
 ```http
-POST /indexes/hotels/docs/search?api-version=2019-05-06 
+POST /indexes/hotels-sample-index/docs/search?api-version=2019-05-06 
     {  
-      "search": "something",  
-      "highlight": "Description"  
+      "search": "sandy beaches",  
+      "highlight": "Description"
     }
 ```
 
@@ -108,30 +105,11 @@ POST /indexes/hotels/docs/search?api-version=2019-05-06
 >
 > When you are writing client code that implements hit highlighting, be aware of this change. Note that this will not impact you unless you create a completely new search service.
 
-## Faceted navigation
-
-Search navigation is common on a results page, often located at the side or top of a page. In Azure Cognitive Search, faceted navigation provides self-directed search based on predefined filters. See [Faceted navigation in Azure Cognitive Search](search-faceted-navigation.md) for details.
-
-## Filters at the page level
-
-If your solution design included dedicated search pages for specific types of content (for example, an online retail application that has departments listed at the top of the page), you can insert a [filter expression](search-filters.md) alongside an **onClick** event to open a page in a pre-filtered state.
-
-You can send a filter with or without a search expression. For example, the following request will filter on brand name, returning only those documents that match it.
-
-    GET /indexes/online-catalog/docs?$filter=brandname eq 'Microsoft' and category eq 'Games'
-
-See [Search Documents (Azure Cognitive Search API)](https://docs.microsoft.com/rest/api/searchservice/Search-Documents) for more information about `$filter` expressions.
+## Next steps
 
-## See Also
+To quickly generate a search page for your client, consider these options:
 
-- [Azure Cognitive Search REST API](https://docs.microsoft.com/rest/api/searchservice)
-- [Index Operations](https://docs.microsoft.com/rest/api/searchservice/Index-operations)
-- [Document Operations](https://docs.microsoft.com/rest/api/searchservice/Document-operations)
-- [Faceted Navigation in Azure Cognitive Search](search-faceted-navigation.md)
++ [Application Generator](search-create-app-portal.md), in the portal, creates an HTML page with a search bar, faceted navigation, and results area that includes images.
++ [Create your first app in C#](tutorial-csharp-create-first-app.md) is a tutorial that builds a functional client. Sample code demonstrates paginated queries, hit highlighting, and sorting.
 
-<!--Image references-->
-[1]: ./media/search-pagination-page-layout/Pages-1-Viewing1ofNResults.PNG
-[2]: ./media/search-pagination-page-layout/Pages-2-Tiled.PNG
-[3]: ./media/search-pagination-page-layout/Pages-3-SortBy.png
-[4]: ./media/search-pagination-page-layout/Pages-4-SortbyRelevance.png
-[5]: ./media/search-pagination-page-layout/Pages-5-BuildSort.png
+Several code samples include a web front-end interface, which you can find here: [New York City Jobs demo app](https://aka.ms/azjobsdemo), [JavaScript sample code with a live demo site](https://github.com/liamca/azure-search-javascript-samples), and [CognitiveSearchFrontEnd](https://github.com/LuisCabrer/CognitiveSearchFrontEnd).