|
| 1 | +--- |
| 2 | +title: Fuzzy search |
| 3 | +titleSuffix: Azure Cognitive Search |
| 4 | +description: Implement a "did you mean" search experience to auto-correct a misspelled term or typo. |
| 5 | + |
| 6 | +manager: nitinme |
| 7 | +author: HeidiSteen |
| 8 | +ms.author: heidist |
| 9 | +ms.service: cognitive-search |
| 10 | +ms.topic: conceptual |
| 11 | +ms.date: 04/08/2020 |
| 12 | +--- |
| 13 | +# Fuzzy search to correct misspellings and typos |
| 14 | + |
| 15 | +Azure Cognitive Search supports fuzzy search, a type of query that compensates for typos and misspelled terms in the input string. It does this by scanning for terms having a similar composition. Expanding search to cover near-matches has the effect of auto-correcting a typo when the discrepancy is just a few misplaced characters. |
| 16 | + |
| 17 | +## What is fuzzy search? |
| 18 | + |
| 19 | +It's an expansion exercise that produces a match on terms having a similar composition. When a fuzzy search is specified, the engine builds a graph of similarly composed terms, for all whole terms in the query. For example, if your query includes three terms "university of washington", a graph is created for each term (`search=university~ of~ washington~`). |
| 20 | + |
| 21 | +The graph consists of up to 50 expansions, or permutations, of each term, capturing both correct and incorrect variants in the process. The engine then returns the topmost relevant matches in the response. |
| 22 | + |
| 23 | +For a term like "university", the graph might have "unversty, universty, university, universe, inverse". Any documents that match on those in the graph are included in results. In contrast with language analyzers that can handle irregularities between singular and plural forms of the same word ("mice" and "mouse"), the comparisons in a fuzzy query are taken at face value with no attempt at reconciling the semantic differences. "Universe" and "inverse" will match because the character discrepancies are small. |
| 24 | + |
| 25 | +A match succeeds if the discrepancies are limited to two or fewer edits, where an edit is an inserted, deleted, substituted, or transposed character. The string correction algorithm that specifies the differential is the [Damerau-Levenshtein distance](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance) metric, described as the "minimum number of operations (insertions, deletions, substitutions, or transpositions of two adjacent characters) required to change one word into the other". |
| 26 | + |
| 27 | +In Azure Cognitive Search: |
| 28 | + |
| 29 | ++ Fuzzy query applies to whole terms, but you can support phrases through AND constructions. For example, "Unviersty~ of~ "Wshington~" would match on "University of Washington". |
| 30 | + |
| 31 | ++ The default distance of an edit is 2. A value of `~0` signifies no expansion (only the exact term is considered a match), but you could specify `~1` for one degree of difference, or one edit. |
| 32 | + |
| 33 | ++ A fuzzy query can expand a term up to 50 additional permutations. This limit is not configurable, but you can effectively reduce the number of expansions by decreasing the edit distance to 1. |
| 34 | + |
| 35 | ++ Responses consist of documents containing a relevant match (up to 50). |
| 36 | + |
| 37 | +Collectively, the graphs are submitted as match criteria against tokens in the index. As you can imagine, fuzzy search is inherently slower than other query forms. The size and complexity of your index can determine whether the benefits are enough to offset the latency of the response. |
| 38 | + |
| 39 | +> [!NOTE] |
| 40 | +> Because fuzzy search tends to be slow, it might be worthwhile to investigate alternatives such as n-gram indexing, with its progression of short character sequences (two and three character sequences for bigram and trigram tokens). Depending on your language and query surface, n-gram might give you better performance. |
| 41 | +> |
| 42 | +> Another alternative, which you could consider if you want to handle just the most egregious cases, would be a [synonym map](search-synonyms.md). For example, mapping "search" to "serach, serch, sarch", or "retrieve" to "retreive". |
| 43 | +
|
| 44 | +## Indexing for fuzzy search |
| 45 | + |
| 46 | +Analyzers are not used during query processing to create an expansion graph, but that doesn't mean analyzers should be ignored in fuzzy search scenarios. After all, analyzers are used during indexing to create tokens against which matching is done, whether the query is free form, filtered search, or a fuzzy search with a graph as input. |
| 47 | + |
| 48 | +Generally, when assigning analyzers on a per-field basis, the decision to fine-tune the analysis chain is based on the primary use case (a filter or full text search) rather than specialized query forms like fuzzy search. For this reason, there is not a specific analyzer recommendation for fuzzy search. |
| 49 | + |
| 50 | +However, if test queries are not producing the matches you expect, you could try varying the indexing analyzer, setting it to a [language analyzer](index-add-language-analyzers.md), to see if you get better results. Some languages, particularly those with vowel mutations, can benefit from the inflection and irregular word forms generated by the Microsoft natural language processors. In some cases, using the right language analyzer can make a difference in whether a term is tokenized in a way that is compatible with the value provided by the user. |
| 51 | + |
| 52 | +## How to use fuzzy search |
| 53 | + |
| 54 | +Fuzzy queries are constructed using the full Lucene query syntax, invoking the [Lucene query parser](https://lucene.apache.org/core/6_6_1/queryparser/org/apache/lucene/queryparser/classic/package-summary.html). |
| 55 | + |
| 56 | +1. Set the full Lucene parser on the query (`queryType=full`). |
| 57 | + |
| 58 | +1. Optionally, scope the request to specific fields, using this parameter (`searchFields=<field1,field2>`). |
| 59 | + |
| 60 | +1. Append the tilde (`~`) operator at the end of the whole term (`search=<string>~`). |
| 61 | + |
| 62 | + Include an optional parameter, a number between 0 and 2 (default), if you want to specify the edit distance (`~1`). For example, "blue~" or "blue~1" would return "blue", "blues", and "glue". |
| 63 | + |
| 64 | +In Azure Cognitive Search, besides the term and distance (maximum of 2), there are no additional parameters to set on the query. |
| 65 | + |
| 66 | +> [!NOTE] |
| 67 | +> During query processing, fuzzy queries do not undergo [lexical analysis](search-lucene-query-architecture.md#stage-2-lexical-analysis). The query input is added directly to the query tree and expanded to create a graph of terms. The only transformation performed is lower casing. |
| 68 | +
|
| 69 | +## How to test fuzzy search |
| 70 | + |
| 71 | +For simple testing, we recommend [Search explorer](search-explorer.md) or [Postman](search-get-started-postman.md) for iterating over a query expression. Both tools are interactive, which means you can quickly step through multiple variants of a term and evaluate the responses that come back. |
| 72 | + |
| 73 | +When results are ambiguous, [hit highlighting](search-pagination-page-layout.md#hit-highlighting) can help you identify the match in the response. |
| 74 | + |
| 75 | +> [!Note] |
| 76 | +> The use of hit highlighting to identify fuzzy matches has limitations and only works for basic fuzzy search. If your index has scoring profiles, or if you layer the query with additional syntax, hit highlighting might fail to identify the match. |
| 77 | +
|
| 78 | +### Example 1: fuzzy search with the exact term |
| 79 | + |
| 80 | +Assume the following string exists in a `"Description"` field in a search document: `"Test queries with special characters, plus strings for MSFT, SQL and Java."` |
| 81 | + |
| 82 | +Start with a fuzzy search on "special" and add hit highlighting to the Description field: |
| 83 | + |
| 84 | + search=special~&highlight=Description |
| 85 | + |
| 86 | +In the response, because you added hit highlighting, formatting is applied to "special" as the matching term. |
| 87 | + |
| 88 | + "@search.highlights": { |
| 89 | + "Description": [ |
| 90 | + "Test queries with <em>special</em> characters, plus strings for MSFT, SQL and Java." |
| 91 | + ] |
| 92 | + |
| 93 | +Try the request again, misspelling "special" by taking out letters several letters ("pe"): |
| 94 | + |
| 95 | + search=scial~&highlight=Description |
| 96 | + |
| 97 | +So far, no change to the response. Using the default of 2 degrees distance, removing two characters "pe" from "special" still allows for a successful match on that term. |
| 98 | + |
| 99 | + "@search.highlights": { |
| 100 | + "Description": [ |
| 101 | + "Test queries with <em>special</em> characters, plus strings for MSFT, SQL and Java." |
| 102 | + ] |
| 103 | + |
| 104 | +Trying one more request, further modify the search term by taking out one last character for a total of three deletions (from "special" to "scal"): |
| 105 | + |
| 106 | + search=scal~&highlight=Description |
| 107 | + |
| 108 | +Notice that the same response is returned, but now instead of matching on "special", the fuzzy match is on "SQL". |
| 109 | + |
| 110 | + "@search.score": 0.4232868, |
| 111 | + "@search.highlights": { |
| 112 | + "Description": [ |
| 113 | + "Mix of special characters, plus strings for MSFT, <em>SQL</em>, 2019, Linux, Java." |
| 114 | + ] |
| 115 | + |
| 116 | +The point of this expanded example is to illustrate the clarity that hit highlighting can bring to ambiguous results. In all cases, the same document is returned. Had you relied on document IDs to verify a match, you might have missed the shift from "special" to "SQL". |
| 117 | + |
| 118 | +## See also |
| 119 | + |
| 120 | ++ [How full text search works in Azure Cognitive Search (query parsing architecture)](search-lucene-query-architecture.md) |
| 121 | ++ [Search explorer](search-explorer.md) |
| 122 | ++ [How to query in .NET](search-query-dotnet.md) |
| 123 | ++ [How to query in REST](search-create-index-rest-api.md) |
0 commit comments