Merge pull request #266082 from MicrosoftDocs/main

02/12 PM Publishing

Author: v-alje

articles/ai-services/language-service/language-detection/how-to/call-api.md

@@ -7,7 +7,7 @@ author: jboback
 manager: nitinme
 ms.service: azure-ai-language
 ms.topic: how-to
-ms.date: 12/19/2023
+ms.date: 01/16/2024
 ms.author: jboback
 ms.custom: language-service-language-detection
 ---
@@ -50,14 +50,19 @@ Analysis is performed upon receipt of the request. Using the language detection
 
 When you get results from language detection, you can stream the results to an application or save the output to a file on the local system.
 
-Language detection will return one predominant language for each document you submit, along with it's [ISO 639-1](https://www.iso.org/standard/22109.html) name, a human-readable name, and a confidence score. A positive score of 1 indicates the highest possible confidence level of the analysis.
+Language detection will return one predominant language for each document you submit, along with it's [ISO 639-1](https://www.iso.org/standard/22109.html) name, a human-readable name, a confidence score, script name and script code according to the [ISO 15924 standard](https://wikipedia.org/wiki/ISO_15924). A positive score of 1 indicates the highest possible confidence level of the analysis.
+
 
 ### Ambiguous content
 
 In some cases it may be hard to disambiguate languages based on the input. You can use the `countryHint` parameter to specify an [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) country/region code. By default the API uses "US" as the default country hint. To remove this behavior, you can reset this parameter by setting this value to empty string `countryHint = ""` .
 
 For example, "communication" is common to both English and French and if given with limited context the response will be based on the "US" country/region hint. If the origin of the text is known to be coming from France that can be given as a hint.
 
+> [!NOTE] 
+> Ambiguous content can cause confidence scores to be lower.
+> The `countryHint` in the response is only applicable if the confidence score is less than 0.8.
+
 **Input**
 
 ```json
@@ -76,7 +81,8 @@ For example, "communication" is common to both English and French and if given w
 }
 ```
 
-The language detection model now has additional context to make a better judgment: 
+With the second document, the language detection model has additional context to make a better judgment because it contains the `countryHint` property in the input above. This will return the following output.
+ 
 
 **Output**
 
@@ -129,7 +135,7 @@ If the analyzer can't parse the input, it returns `(Unknown)`. An example is if
         }
     ],
     "errors": [],
-    "modelVersion": "2021-01-05"
+    "modelVersion": "2023-12-01"
 }
 ```
 
@@ -156,22 +162,107 @@ The resulting output consists of the predominant language, with a score of less
 
 ```json
 {
-    "documents": [
-        {
-            "id": "1",
-            "detectedLanguage": {
-                "name": "Spanish",
-                "iso6391Name": "es",
-                "confidenceScore": 0.88
-            },
-            "warnings": []
-        }
-    ],
-    "errors": [],
-    "modelVersion": "2021-01-05"
+    "kind": "LanguageDetectionResults",
+    "results": {
+        "documents": [
+            {
+                "id": "1",
+                "detectedLanguage": {
+                    "name": "Spanish",
+                    "iso6391Name": "es",
+                    "confidenceScore": 0.97,
+                    "script": "Latin",
+                    "scriptCode": "Latn"
+                },
+                "warnings": []
+            }
+        ],
+        "errors": [],
+        "modelVersion": "2023-12-01"
+    }
+}
+```
+
+## Script name and script code
+
+> [!NOTE]
+> * Script detection is currently limited to [select languages](../language-support.md#script-detection).  
+> * The script detection is only available for textual input which is greater than 12 characters in length.
+
+Language detection offers the ability to detect more than one script per language according to the [ISO 15924 standard](https://wikipedia.org/wiki/ISO_15924). Specifically, Language Detection returns two script-related properties:
+
+* `script`: The human-readable name of the identified script
+* `scriptCode`: The ISO 15924 code for the identified script
+
+The output of the API includes the value of the `scriptCode` property for documents that are at least 12 characters or greater in length and matches the list of supported languages and scripts. Script detection is designed to benefit users whose language can be transliterated or written in more than one script, such as Kazakh or Hindi language.
+
+Previously, language detection was designed to detect the language of documents in a wide variety of languages, dialects, and regional variants, but was limited by "Romanization". Romanization refers to conversion of text from one writing system to the Roman (Latin) script, and is necessary to detect many Indo-European languages. However, there are other languages which are written in multiple scripts, such as Kazakh, which can be written in Cyrillic, Perso-Arabic, and Latin scripts. There are also other cases in which users may either choose or are required to transliterate their language in more than one script, such as Hindi transliterated in Latin script, due to the limited availability of keyboards which support its Devanagari script.  
+
+Consequently, language detection's expanded support for script detection behaves as follows:
+
+**Input**
+
+```json
+{ 
+    "kind": "LanguageDetection", 
+    "parameters": { 
+        "modelVersion": "latest" 
+    }, 
+    "analysisInput": { 
+        "documents": [ 
+            { 
+                "id": "1", 
+                "text": "आप कहाँ जा रहे हैं?" 
+            }, 
+            { 
+                "id": "2", 
+                "text": "Туған жерім менің - Қазақстаным" 
+            } 
+        ] 
+    } 
+} 
+```
+
+**Output**
+
+The resulting output consists of the predominant language, along with a script name, script code, and confidence score.
+
+```json
+{ 
+    "kind": "LanguageDetectionResults", 
+    "results": { 
+        "documents": [ 
+            { 
+                "id": "1", 
+                "detectedLanguage": { 
+                    "name": "Hindi", 
+                    "iso6391Name": "hi", 
+                    "confidenceScore": 1.0, 
+                    "script": "Devanagari", 
+                    "scriptCode": "Deva" 
+                }, 
+                "warnings": [] 
+            }, 
+            { 
+                "id": "2", 
+                "detectedLanguage": { 
+                    "name": "Kazakh", 
+                    "iso6391Name": "kk", 
+                    "confidenceScore": 1.0, 
+                    "script": "Cyrillic",  
+                    "scriptCode": "Cyrl" 
+                }, 
+                "warnings": [] 
+            } 
+        ], 
+        "errors": [], 
+        "modelVersion": "2023-12-01" 
+    } 
 }
 ```
 
+
+
 ## Service and data limits
 
 [!INCLUDE [service limits article](../../includes/service-limits-link.md)]

articles/ai-services/language-service/language-detection/how-to/use-containers.md

@@ -7,7 +7,7 @@ author: jboback
 manager: nitinme
 ms.service: azure-ai-language
 ms.topic: how-to
-ms.date: 12/19/2023
+ms.date: 02/12/2024
 ms.author: jboback
 ms.custom: language-service-language-detection
 keywords: on-premises, Docker, container
@@ -35,7 +35,7 @@ The following table describes the minimum and recommended specifications for the
 
 |  | Minimum host specs | Recommended host specs | Minimum TPS | Maximum TPS|
 |---|---------|-------------|--|--|
-| **Language detection**   | 1 core, 2GB memory | 1 core, 4GB memory |15 | 30| 
+| **Language detection**   | 1 core, 5GB memory | 1 core, 8GB memory |15 | 30| 
 
 CPU core and memory correspond to the `--cpus` and `--memory` settings, which are used as part of the `docker run` command.
 

articles/ai-services/language-service/language-detection/includes/quickstarts/rest-api.md

@@ -10,7 +10,7 @@ ms.author: jboback
 
 [Reference documentation](https://go.microsoft.com/fwlink/?linkid=2239169)
 
-Use this quickstart to send language detection requests using the REST API. In the following example, you will use cURL to identify the language that a text sample was written in.
+Use this quickstart to send language detection requests using the REST API. In the following example, you'll use cURL to identify the language that a text sample was written in.
 
 [!INCLUDE [Use Language Studio](../../../includes/use-language-studio.md)]
 
@@ -20,7 +20,7 @@ Use this quickstart to send language detection requests using the REST API. In t
 * Azure subscription - [Create one for free](https://azure.microsoft.com/free/cognitive-services)
 * The current version of [cURL](https://curl.haxx.se/).
 * Once you have your Azure subscription, <a href="https://portal.azure.com/#create/Microsoft.CognitiveServicesTextAnalytics"  title="Create a Language resource"  target="_blank">create a Language resource </a> in the Azure portal to get your key and endpoint. After it deploys, select **Go to resource**.
-    * You will need the key and endpoint from the resource you create to connect your application to the API. You'll paste your key and endpoint into the code below later in the quickstart.
+    * You'll need the key and endpoint from the resource you create to connect your application to the API. You'll paste your key and endpoint into the code below later in the quickstart.
     * You can use the free pricing tier (`Free F0`) to try the service, and upgrade later to a paid tier for production.
 
 > [!NOTE]
@@ -47,7 +47,7 @@ The following cURL commands are executed from a BASH shell. Edit these commands
 [!INCLUDE [REST API quickstart instructions](../../../includes/rest-api-instructions.md)]
 
 ```bash
-curl -i -X POST https://<your-language-resource-endpoint>/language/:analyze-text?api-version=2022-05-01 \
+curl -i -X POST https://<your-language-resource-endpoint>/language/:analyze-text?api-version=2023-11-15-preview \
 -H "Content-Type: application/json" \
 -H "Ocp-Apim-Subscription-Key:<your-language-resource-key>" \
 -d \
@@ -76,19 +76,23 @@ curl -i -X POST https://<your-language-resource-endpoint>/language/:analyze-text
 
 ```json
 {
-	"kind": "LanguageDetectionResults",
-	"results": {
-		"documents": [{
-			"id": "1",
-			"detectedLanguage": {
-				"name": "English",
-				"iso6391Name": "en",
-				"confidenceScore": 1.0
-			},
-			"warnings": []
-		}],
-		"errors": [],
-		"modelVersion": "2022-10-01"
-	}
+    "kind": "LanguageDetectionResults",
+    "results": {
+        "documents": [
+            {
+                "id": "1",
+                "detectedLanguage": {
+                    "name": "English",
+                    "iso6391Name": "en",
+                    "confidenceScore": 1.0,
+                    "script": "Latin",
+                    "scriptCode": "Latn"
+                },
+                "warnings": []
+            }
+        ],
+        "errors": [],
+        "modelVersion": "2023-12-01"
+    }
 }
 ```

articles/ai-services/language-service/language-detection/language-support.md

@@ -162,6 +162,26 @@ If you have content expressed in a less frequently used language, you can try La
 | Telugu              | `te`          |
 | Urdu                | `ur`          |
 
+## Script detection
+
+| Language |Script code	| Scripts |
+| --- | ---	| --- |
+| Bengali (Bengali-Assamese) | `as` | `Latn`, `Beng` |
+| Bengali (Bangla) | `bn` | `Latn`, `Beng` |
+| Gujarati | `gu` | `Latn`, `Gujr` |
+| Hindi | `hi` | `Latn`, `Deva` |
+| Kannada | `kn` | `Latn`, `Knda` |
+| Malayalam | `ml` | `Latn`, `Mlym` |
+| Marathi	| `mr` | `Latn`, `Deva` |
+| Oriya | `or` | `Latn`, `Orya` |
+| Gurmukhi | `pa` | `Latn`, `Guru` |
+| Tamil | `ta` | `Latn`, `Taml` |
+| Telugu | `te` | `Latn`, `Telu` |
+| Arabic | `ur` | `Latn`, `Arab` |
+| Cyrillic | `tt` | `Latn`, `Cyrl` |
+| Serbian `sr` | `Latn`, `Cyrl` |
+| Unified Canadian Aboriginal Syllabics	| `iu` | `Latn`, `Cans` |
+
 ## Next steps
 
 [Language detection overview](overview.md)

articles/ai-services/language-service/language-detection/overview.md

@@ -14,13 +14,21 @@ ms.custom: language-service-language-detection
 
 # What is language detection in Azure AI Language?
 
-Language detection is one of the features offered by [Azure AI Language](../overview.md), a collection of machine learning and AI algorithms in the cloud for developing intelligent applications that involve written language. Language detection can detect the language a document is written in, and returns a language code for a wide range of languages, variants, dialects, and some regional/cultural languages. 
+Language detection is one of the features offered by [Azure AI Language](../overview.md), a collection of machine learning and AI algorithms in the cloud for developing intelligent applications that involve written language. Language detection is able to detect more than 100 languages in their primary script. In addition, it offers [script detection](./how-to/call-api.md#script-name-and-script-code) to detect multiple scripts per language according to the [ISO 15924 standard](https://wikipedia.org/wiki/ISO_15924) for a [select number of languages](./language-support.md#script-detection).
 
 This documentation contains the following types of articles:
 
 * [**Quickstarts**](quickstart.md) are getting-started instructions to guide you through making requests to the service.
 * [**How-to guides**](how-to/call-api.md) contain instructions for using the service in more specific or customized ways.
 
+## Language detection features
+
+* Language detection: Returns one predominant language for each document you submit, along with its ISO 639-1 name, a human-readable name, confidence score, script name and script code according to ISO 15924 standard.
+
+* Script detection: To distinguish between multiple scripts used to write certain languages, such as Kazakh, language detection returns a script name and script code according to the ISO 15924 standard.  
+
+* Ambiguous content handling: To help disambiguate language based on the input, you can specify an ISO 3166-1 alpha-2 country/region code. For example, the word "communication" is common to both English and French. Specifying the origin of the text as France can help the language detection model determine the correct language.
+
 [!INCLUDE [Typical workflow for pre-configured language features](../includes/overview-typical-workflow.md)]
 
 
@@ -30,7 +38,7 @@ This documentation contains the following types of articles:
 
 ## Responsible AI 
 
-An AI system includes not only the technology, but also the people who will use it, the people who will be affected by it, and the environment in which it is deployed. Read the [transparency note for language detection](/legal/cognitive-services/language-service/transparency-note-language-detection?context=/azure/ai-services/language-service/context/context) to learn about responsible AI use and deployment in your systems. You can also see the following articles for more information:
+An AI system includes not only the technology, but also the people who will use it, the people who will be affected by it, and the environment in which it's deployed. Read the [transparency note for language detection](/legal/cognitive-services/language-service/transparency-note-language-detection?context=/azure/ai-services/language-service/context/context) to learn about responsible AI use and deployment in your systems. You can also see the following articles for more information:
 
 [!INCLUDE [Responsible AI links](../includes/overview-responsible-ai-links.md)]
 

articles/ai-services/language-service/language-detection/quickstart.md

@@ -7,7 +7,7 @@ author: jboback
 manager: nitinme
 ms.service: azure-ai-language
 ms.topic: quickstart
-ms.date: 12/19/2023
+ms.date: 01/16/2024
 ms.author: jboback
 ms.devlang: csharp
 # ms.devlang: csharp, java, javascript, python

articles/ai-services/language-service/whats-new.md

@@ -15,6 +15,10 @@ ms.author: aahi
 
 Azure AI Language is updated on an ongoing basis. To stay up-to-date with recent developments, this article provides you with information about new releases and features.
 
+## February 2024
+
+* Expanded [language detection](./language-detection/how-to/call-api.md#script-name-and-script-code) support for additional scripts according to the [ISO 15924 standard](https://wikipedia.org/wiki/ISO_15924) is now available starting in API version `2023-11-15-preview`.
+
 ## January 2024
 
 * [Native document support](native-document-support/use-native-documents.md) is now available in `2023-11-15-preview` public preview.