Merge pull request #95599 from diberry/diberry/1111-luis-pattern-reference

[Cogsvcs] LUIS - pattern syntax as reference

Author: PMEds28

articles/cognitive-services/LUIS/luis-concept-patterns.md

@@ -9,7 +9,7 @@ ms.custom: seodec18
 ms.service: cognitive-services
 ms.subservice: language-understanding
 ms.topic: conceptual
-ms.date: 10/15/2019
+ms.date: 11/11/2019
 ms.author: diberry
 ---
 # Patterns improve prediction accuracy
@@ -52,128 +52,16 @@ Given enough example utterances, LUIS would be able to increase prediction confi
 ## Pattern matching
 A pattern is matched based on detecting the entities inside the pattern first, then validating the rest of the words and word order of the pattern. Entities are required in the pattern for a pattern to match. The pattern is applied at the token level, not the character level. 
 
-## Pattern syntax
-Pattern syntax is a template for an utterance. The template should contain words and entities you want to match as well as words and punctuation you want to ignore. It is **not** a regular expression. 
-
-Entities in patterns are surrounded by curly brackets, `{}`. Patterns can include entities, and entities with roles. [Pattern.any](luis-concept-entity-types.md#patternany-entity) is an entity only used in patterns. 
-
-Pattern syntax supports the following syntax:
-
-|Function|Syntax|Nesting level|Example|
-|--|--|--|--|
-|entity| {} - curly brackets|2|Where is form {entity-name}?|
-|optional|[] - square brackets<BR><BR>There is a limit of 3 on nesting levels of any combination of optional and grouping |2|The question mark is optional [?]|
-|grouping|() - parentheses|2|is (a \| b)|
-|or| \| - vertical bar (pipe)<br><br>There is a limit of 2 on the vertical bars (Or) in one group |-|Where is form ({form-name-short} &#x7c; {form-name-long} &#x7c; {form-number})| 
-|beginning and/or end of utterance|^ - caret|-|^begin the utterance<br>the utterance is done^<br>^strict literal match of entire utterance with {number} entity^|
-
-## Nesting syntax in patterns
-
-The **optional** syntax, with square brackets, can be nested two levels. For example: `[[this]is] a new form`. This example allows for the following utterances: 
-
-|Nested optional utterance example|Explanation|
-|--|--|
-|this is a new form|matches all words in pattern|
-|is a new form|matches outer optional word and non-optional words in pattern|
-|a new form|matches required words only|
-
-The **grouping** syntax, with parentheses, can be nested two levels. For example: `(({Entity1.RoleName1} | {Entity1.RoleName2} ) | {Entity2} )`. This feature allows any of the three entities to be matched. 
-
-If Entity1 is a Location with roles such as origin (Seattle) and destination (Cairo) and Entity 2 is a known building name from a list entity (RedWest-C), the following utterances would map to this pattern:
-
-|Nested grouping utterance example|Explanation|
-|--|--|
-|RedWest-C|matches outer grouping entity|
-|Seattle|matches one of the inner grouping entities|
-|Cairo|matches one of the inner grouping entities|
-
-## Nesting limits for groups with optional syntax
-
-A combination of **grouping** with **optional** syntax has a limit of 3 nesting levels.
-
-|Allowed|Example|
-|--|--|
-|Yes|( [ ( test1 &#x7c; test2 ) ] &#x7c; test3 )|
-|No|( [ ( [ test1 ] &#x7c; test2 ) ] &#x7c; test3 )|
-
-## Nesting limits for groups with or-ing syntax
-
-A combination of **grouping** with **or-ing** syntax has a limit of 2 vertical bars.
-
-|Allowed|Example|
-|--|--|
-|Yes|( test1 &#x7c; test2 &#x7c; ( test3 &#x7c; test4 ) )|
-|No|( test1 &#x7c; test2 &#x7c; test3 &#x7c; ( test4 &#x7c; test5 ) ) |
-
-## Syntax to add an entity to a pattern template
-To add an entity into the pattern template, surround the entity name with curly braces, such as `Who does {Employee} manage?`. 
-
-|Pattern with entity|
-|--|
-|`Who does {Employee} manage?`|
-
-## Syntax to add an entity and role to a pattern template
-An entity role is denoted as `{entity:role}` with the entity name followed by a colon, then the role name. To add an entity with a role into the pattern template, surround the entity name and role name with curly braces, such as `Book a ticket from {Location:Origin} to {Location:Destination}`. 
-
-|Pattern with entity roles|
-|--|
-|`Book a ticket from {Location:Origin} to {Location:Destination}`|
-
-## Syntax to add a pattern.any to pattern template
-The Pattern.any entity allows you to add an entity of varying length to the pattern. As long as the pattern template is followed, the pattern.any can be any length. 
-
-To add a **Pattern.any** entity into the pattern template, surround the Pattern.any entity with the curly braces, such as `How much does {Booktitle} cost and what format is it available in?`.  
-
-|Pattern with Pattern.any entity|
-|--|
-|`How much does {Booktitle} cost and what format is it available in?`|
-
-|Book titles in the pattern|
-|--|
-|How much does **steal this book** cost and what format is it available in?|
-|How much does **ask** cost and what format is it available in?|
-|How much does **The Curious Incident of the Dog in the Night-Time** cost and what format is it available in?| 
-
-The words of the book title are not confusing to LUIS because LUIS knows where the book title ends, based on the Pattern.any entity.
-
-## Explicit lists
-
-create an [Explicit List](https://westus.dev.cognitive.microsoft.com/docs/services/5890b47c39e2bb17b84a55ff/operations/5ade550bd5b81c209ce2e5a8) through the authoring API to allow the exception when:
-
-* Your pattern contains a [Pattern.any](luis-concept-entity-types.md#patternany-entity)
-* And that pattern syntax allows for the possibility of an incorrect entity extraction based on the utterance. 
-
-For example, suppose you have a pattern containing both optional syntax, `[]`, and entity syntax, `{}`, combined in a way to extract data incorrectly.
-
-Consider the pattern `[find] email about {subject} [from {person}]'.
-
-In the following utterances, the **subject** and **person** entity are extracted correctly and incorrectly:
-
-|Utterance|Entity|Correct extraction|
-|--|--|:--:|
-|email about dogs from Chris|subject=dogs<br>person=Chris|✔|
-|email about the man from La Mancha|subject=the man<br>person=La Mancha|X|
-
-In the preceding table, the subject should be `the man from La Mancha` (a book title) but because the subject includes the optional word `from`, the title is incorrectly predicted. 
-
-To fix this exception to the pattern, add `the man from la mancha` as an explicit list match for the {subject} entity using the [authoring API for explicit list](https://westus.dev.cognitive.microsoft.com/docs/services/5890b47c39e2bb17b84a55ff/operations/5ade550bd5b81c209ce2e5a8).
-
-## Syntax to mark optional text in a template utterance
-Mark optional text in the utterance using the regular expression square bracket syntax, `[]`. The optional text can nest square brackets up to two brackets only.
-
-|Pattern with optional text|Meaning|
-|--|--|
-|`[find] email about {subject} [from {person}]`|`find` and `from {person}` are optional|
-|`Can you help me[?]|The punctuation mark is optional|
-
-Punctuation marks (`?`, `!`, `.`) should be ignored and you need to ignore them using the square bracket syntax in patterns. 
-
 ## Pattern-only apps
 You can build an app with intents that have no example utterances, as long as there's a pattern for each intent. For a pattern-only app, the pattern shouldn't contain machine-learned entities because these do require example utterances. 
 
 ## Best practices
 Learn [best practices](luis-concept-best-practices.md).
 
+## Pattern syntax
+
+Learn pattern syntax from the [pattern syntax reference](reference-pattern-syntax.md). 
+
 ## Next steps
 
 > [!div class="nextstepaction"]

articles/cognitive-services/LUIS/luis-how-to-add-entities.md

@@ -9,7 +9,7 @@ ms.custom: seodec18
 ms.service: cognitive-services
 ms.subservice: language-understanding
 ms.topic: conceptual
-ms.date: 10/25/2019
+ms.date: 11/11/2019
 ms.author: diberry
 ---
 
@@ -103,7 +103,7 @@ In the utterance `Where is Request relocation from employee new to the company o
 
     To use the pattern.any entity, add a pattern on the **Patterns** page, in the **Improve app performance** section, with the correct curly brace syntax, such as `Where is **{HumanResourcesFormTitle}** on the server?`.
 
-    If you find that your pattern, when it includes a Pattern.any, extracts entities incorrectly, use an [explicit list](luis-concept-patterns.md#explicit-lists) to correct this problem. 
+    If you find that your pattern, when it includes a Pattern.any, extracts entities incorrectly, use an [explicit list](reference-pattern-syntax.md#explicit-lists) to correct this problem. 
 
 <a name="add-a-role-to-pattern-based-entity"></a>
 

articles/cognitive-services/LUIS/luis-quickstart-intent-and-list-entity.md

@@ -9,7 +9,7 @@ ms.custom: seodec18
 ms.service: cognitive-services
 ms.subservice: language-understanding
 ms.topic: tutorial
-ms.date: 10/14/2019
+ms.date: 11/11/2019
 ms.author: diberry
 #Customer intent: As a new user, I want to understand how and why to use the list entity. 
 ---
@@ -45,7 +45,7 @@ A list entity is a good choice for this type of data when:
 
 * The data values are a known set.
 * The set doesn't exceed the maximum LUIS [boundaries](luis-boundaries.md) for this entity type.
-* The text in the utterance is an exact match with a synonym or the canonical name. LUIS doesn't use the list beyond exact text matches. Stemming, plurals, and other variations are not resolved with just a list entity. To manage variations, consider using a [pattern](luis-concept-patterns.md#syntax-to-mark-optional-text-in-a-template-utterance) with the optional text syntax. 
+* The text in the utterance is an exact match with a synonym or the canonical name. LUIS doesn't use the list beyond exact text matches. Stemming, plurals, and other variations are not resolved with just a list entity. To manage variations, consider using a [pattern](reference-pattern-syntax.md#syntax-to-mark-optional-text-in-a-template-utterance) with the optional text syntax. 
 
 ## Create a new app
 

articles/cognitive-services/LUIS/luis-tutorial-pattern-any.md

@@ -9,7 +9,7 @@ manager: nitinme
 ms.service: cognitive-services
 ms.subservice: language-understanding
 ms.topic: tutorial
-ms.date: 10/14/2019
+ms.date: 11/11/2019
 ms.author: diberry
 #Customer intent: As a new user, I want to understand how and why to use pattern.any entity to improve predictions. 
 ---
@@ -134,7 +134,7 @@ The Pattern.any entity extracts entities of varying length. It only works in a p
 
 ## Using an explicit list
 
-If you find that your pattern, when it includes a Pattern.any, extracts entities incorrectly, use an [explicit list](luis-concept-patterns.md#explicit-lists) to correct this problem.
+If you find that your pattern, when it includes a Pattern.any, extracts entities incorrectly, use an [explicit list](reference-pattern-syntax.md#explicit-lists) to correct this problem.
 
 
 ## Clean up resources

articles/cognitive-services/LUIS/reference-entity-list.md

@@ -8,7 +8,7 @@ manager: nitinme
 ms.service: cognitive-services
 ms.subservice: language-understanding
 ms.topic: reference
-ms.date: 09/29/2019
+ms.date: 11/11/2019
 ms.author: diberry
 ---
 # List entity 
@@ -22,7 +22,7 @@ A list entity isn't machine-learned. It is an exact text match. LUIS marks any m
 * Are a known set.
 * Doesn't change often. If you need to change the list often or want the list to self-expand, a simple entity boosted with a phrase list is a better choice. 
 * The set doesn't exceed the maximum LUIS [boundaries](luis-boundaries.md) for this entity type.
-* The text in the utterance is an exact match with a synonym or the canonical name. LUIS doesn't use the list beyond exact text matches. Fuzzy matching, case-insensitivity, stemming, plurals, and other variations are not resolved with a list entity. To manage variations, consider using a [pattern](luis-concept-patterns.md#syntax-to-mark-optional-text-in-a-template-utterance) with the optional text syntax.
+* The text in the utterance is an exact match with a synonym or the canonical name. LUIS doesn't use the list beyond exact text matches. Fuzzy matching, case-insensitivity, stemming, plurals, and other variations are not resolved with a list entity. To manage variations, consider using a [pattern](reference-pattern-syntax.md#syntax-to-mark-optional-text-in-a-template-utterance) with the optional text syntax.
 
 ![list entity](./media/luis-concept-entities/list-entity.png)
 

articles/cognitive-services/LUIS/reference-pattern-syntax.md

@@ -0,0 +1,134 @@
+---
+title: Pattern syntax reference - LUIS
+titleSuffix: Azure Cognitive Services
+description: Create entities to extract key data from user utterances in Language Understanding (LUIS) apps. Extracted data is used by the client application.     
+services: cognitive-services
+author: diberry
+manager: nitinme
+ms.service: cognitive-services
+ms.subservice: language-understanding
+ms.topic: reference
+ms.date: 11/11/2019
+ms.author: diberry
+---
+
+# Pattern syntax
+
+Pattern syntax is a template for an utterance. The template should contain words and entities you want to match as well as words and punctuation you want to ignore. It is **not** a regular expression. 
+
+Entities in patterns are surrounded by curly brackets, `{}`. Patterns can include entities, and entities with roles. [Pattern.any](luis-concept-entity-types.md#patternany-entity) is an entity only used in patterns. 
+
+Pattern syntax supports the following syntax:
+
+|Function|Syntax|Nesting level|Example|
+|--|--|--|--|
+|entity| {} - curly brackets|2|Where is form {entity-name}?|
+|optional|[] - square brackets<BR><BR>There is a limit of 3 on nesting levels of any combination of optional and grouping |2|The question mark is optional [?]|
+|grouping|() - parentheses|2|is (a \| b)|
+|or| \| - vertical bar (pipe)<br><br>There is a limit of 2 on the vertical bars (Or) in one group |-|Where is form ({form-name-short} &#x7c; {form-name-long} &#x7c; {form-number})| 
+|beginning and/or end of utterance|^ - caret|-|^begin the utterance<br>the utterance is done^<br>^strict literal match of entire utterance with {number} entity^|
+
+## Nesting syntax in patterns
+
+The **optional** syntax, with square brackets, can be nested two levels. For example: `[[this]is] a new form`. This example allows for the following utterances: 
+
+|Nested optional utterance example|Explanation|
+|--|--|
+|this is a new form|matches all words in pattern|
+|is a new form|matches outer optional word and non-optional words in pattern|
+|a new form|matches required words only|
+
+The **grouping** syntax, with parentheses, can be nested two levels. For example: `(({Entity1.RoleName1} | {Entity1.RoleName2} ) | {Entity2} )`. This feature allows any of the three entities to be matched. 
+
+If Entity1 is a Location with roles such as origin (Seattle) and destination (Cairo) and Entity 2 is a known building name from a list entity (RedWest-C), the following utterances would map to this pattern:
+
+|Nested grouping utterance example|Explanation|
+|--|--|
+|RedWest-C|matches outer grouping entity|
+|Seattle|matches one of the inner grouping entities|
+|Cairo|matches one of the inner grouping entities|
+
+## Nesting limits for groups with optional syntax
+
+A combination of **grouping** with **optional** syntax has a limit of 3 nesting levels.
+
+|Allowed|Example|
+|--|--|
+|Yes|( [ ( test1 &#x7c; test2 ) ] &#x7c; test3 )|
+|No|( [ ( [ test1 ] &#x7c; test2 ) ] &#x7c; test3 )|
+
+## Nesting limits for groups with or-ing syntax
+
+A combination of **grouping** with **or-ing** syntax has a limit of 2 vertical bars.
+
+|Allowed|Example|
+|--|--|
+|Yes|( test1 &#x7c; test2 &#x7c; ( test3 &#x7c; test4 ) )|
+|No|( test1 &#x7c; test2 &#x7c; test3 &#x7c; ( test4 &#x7c; test5 ) ) |
+
+## Syntax to add an entity to a pattern template
+To add an entity into the pattern template, surround the entity name with curly braces, such as `Who does {Employee} manage?`. 
+
+|Pattern with entity|
+|--|
+|`Who does {Employee} manage?`|
+
+## Syntax to add an entity and role to a pattern template
+An entity role is denoted as `{entity:role}` with the entity name followed by a colon, then the role name. To add an entity with a role into the pattern template, surround the entity name and role name with curly braces, such as `Book a ticket from {Location:Origin} to {Location:Destination}`. 
+
+|Pattern with entity roles|
+|--|
+|`Book a ticket from {Location:Origin} to {Location:Destination}`|
+
+## Syntax to add a pattern.any to pattern template
+The Pattern.any entity allows you to add an entity of varying length to the pattern. As long as the pattern template is followed, the pattern.any can be any length. 
+
+To add a **Pattern.any** entity into the pattern template, surround the Pattern.any entity with the curly braces, such as `How much does {Booktitle} cost and what format is it available in?`.  
+
+|Pattern with Pattern.any entity|
+|--|
+|`How much does {Booktitle} cost and what format is it available in?`|
+
+|Book titles in the pattern|
+|--|
+|How much does **steal this book** cost and what format is it available in?|
+|How much does **ask** cost and what format is it available in?|
+|How much does **The Curious Incident of the Dog in the Night-Time** cost and what format is it available in?| 
+
+The words of the book title are not confusing to LUIS because LUIS knows where the book title ends, based on the Pattern.any entity.
+
+## Explicit lists
+
+create an [Explicit List](https://westus.dev.cognitive.microsoft.com/docs/services/5890b47c39e2bb17b84a55ff/operations/5ade550bd5b81c209ce2e5a8) through the authoring API to allow the exception when:
+
+* Your pattern contains a [Pattern.any](luis-concept-entity-types.md#patternany-entity)
+* And that pattern syntax allows for the possibility of an incorrect entity extraction based on the utterance. 
+
+For example, suppose you have a pattern containing both optional syntax, `[]`, and entity syntax, `{}`, combined in a way to extract data incorrectly.
+
+Consider the pattern `[find] email about {subject} [from {person}]'.
+
+In the following utterances, the **subject** and **person** entity are extracted correctly and incorrectly:
+
+|Utterance|Entity|Correct extraction|
+|--|--|:--:|
+|email about dogs from Chris|subject=dogs<br>person=Chris|✔|
+|email about the man from La Mancha|subject=the man<br>person=La Mancha|X|
+
+In the preceding table, the subject should be `the man from La Mancha` (a book title) but because the subject includes the optional word `from`, the title is incorrectly predicted. 
+
+To fix this exception to the pattern, add `the man from la mancha` as an explicit list match for the {subject} entity using the [authoring API for explicit list](https://westus.dev.cognitive.microsoft.com/docs/services/5890b47c39e2bb17b84a55ff/operations/5ade550bd5b81c209ce2e5a8).
+
+## Syntax to mark optional text in a template utterance
+Mark optional text in the utterance using the regular expression square bracket syntax, `[]`. The optional text can nest square brackets up to two brackets only.
+
+|Pattern with optional text|Meaning|
+|--|--|
+|`[find] email about {subject} [from {person}]`|`find` and `from {person}` are optional|
+|`Can you help me[?]|The punctuation mark is optional|
+
+Punctuation marks (`?`, `!`, `.`) should be ignored and you need to ignore them using the square bracket syntax in patterns. 
+
+## Next steps
+
+Understand how [sentiment](luis-reference-prebuilt-sentiment.md) is returned in the .json response.

articles/cognitive-services/LUIS/toc.yml

@@ -317,6 +317,8 @@
       href: reference-entity-regular-expression.md
     - name: Simple entity
       href: reference-entity-simple.md
+  - name: Pattern syntax
+    href: reference-pattern-syntax.md
   - name: Sentiment 
     href: luis-reference-prebuilt-sentiment.md
   - name: Developer resources