Skip to content

Commit 4189658

Browse files
authored
Merge pull request #1602 from MicrosoftDocs/main
11/19 Publish
2 parents 51e566e + a4a9977 commit 4189658

File tree

481 files changed

+11712
-3848
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

481 files changed

+11712
-3848
lines changed

.openpublishing.publish.config.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,12 @@
104104
"branch": "master",
105105
"branch_mapping": {}
106106
},
107+
{
108+
"path_to_root": "azureai-samples-nov2024",
109+
"url": "https://github.com/Azure-Samples/azureai-samples",
110+
"branch": "dantaylo/nov2024",
111+
"branch_mapping": {}
112+
},
107113
{
108114
"path_to_root": "azureml-examples-main",
109115
"url": "https://github.com/azure/azureml-examples",

articles/ai-services/document-intelligence/quickstarts/try-document-intelligence-studio.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,43 @@ CORS should now be configured to use the storage account from Document Intellige
153153
> [!NOTE]
154154
> By default, the Studio will use documents that are located at the root of your container. However, you can use data organized in folders by specifying the folder path in the Custom form project creation steps. *See* [**Organize your data in subfolders**](../how-to-guides/build-a-custom-model.md?view=doc-intel-2.1.0&preserve-view=true#organize-your-data-in-subfolders-optional)
155155
156+
## Use Document Intelligence Studio features
157+
158+
### Auto label documents with prebuilt models or one of your own models
159+
160+
* In custom extraction model labeling page, you can now auto label your documents using one of Document Intelligent Service prebuilt models or your trained models.
161+
162+
:::image type="content" source="../media/studio/auto-label.gif" alt-text="Animated screenshot showing auto labeling in Studio.":::
163+
164+
* For some documents, duplicate labels after running autolabel are possible. Make sure to modify the labels so that there are no duplicate labels in the labeling page afterwards.
165+
166+
:::image type="content" source="../media/studio/duplicate-labels.png" alt-text="Screenshot showing duplicate label warning after auto labeling.":::
167+
168+
### Auto label tables
169+
170+
* In custom extraction model labeling page, you can now auto label the tables in the document without having to label the tables manually.
171+
172+
:::image type="content" source="../media/studio/auto-table-label.gif" alt-text="Animated screenshot showing auto table labeling in Studio.":::
173+
174+
### Add test files directly to your training dataset
175+
176+
* Once you train a custom extraction model, make use of the test page to improve your model quality by uploading test documents to training dataset if needed.
177+
178+
* If a low confidence score is returned for some labels, make sure to correctly label your content. If not, add them to the training dataset and relabel to improve the model quality.
179+
180+
:::image type="content" source="../media/studio/add-from-test.gif" alt-text="Animated screenshot showing how to add test files to training dataset.":::
181+
182+
### Make use of the document list options and filters in custom projects
183+
184+
* Use the custom extraction model labeling page to navigate through your training documents with ease by making use of the search, filter, and sort by feature.
185+
186+
* Utilize the grid view to preview documents or use the list view to scroll through the documents more easily.
187+
188+
:::image type="content" source="../media/studio/document-options.png" alt-text="Screenshot of document list view options and filters.":::
189+
190+
### Project sharing
191+
192+
Share custom extraction projects with ease. For more information, see [Project sharing with custom models](../how-to-guides/project-share-custom-models.md).
156193

157194
## Next steps
158195

articles/ai-services/document-intelligence/studio-overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,7 @@ Share custom extraction projects with ease. For more information, see [Project s
206206

207207
Document Intelligence is part of the Azure AI services offerings in the Azure AI Studio. Each of the Azure AI services helps developers and organizations rapidly create intelligent, cutting-edge, market-ready, and responsible applications with out-of-the-box and prebuilt and customizable APIs and models.
208208

209-
Learn how to [connect your AI services hub](../../ai-studio/ai-services/connect-ai-services.md) with AI services and get started using Document Intelligence.
209+
Learn how to [connect your AI services hub](../../ai-studio/ai-services/how-to/connect-ai-services.md) with AI services and get started using Document Intelligence.
210210

211211
## Next steps
212212

articles/ai-services/includes/quickstarts/ai-studio-prerequisites.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,4 @@ ms.author: eur
1010

1111
> [!div class="checklist"]
1212
> - Azure subscription - [Create one for free](https://azure.microsoft.com/free/cognitive-services).
13-
> - Some AI services features are free to try in AI Studio. For access to all capabilities described in this article, you need to [connect AI services to your hub in AI Studio](../../../ai-studio/ai-services/connect-ai-services.md#connect-to-ai-services).
13+
> - Some Azure AI services features are free to try in Azure AI Studio. For access to all capabilities described in this article, you need to [connect AI services in AI Studio](../../../ai-studio/ai-services/how-to/connect-ai-services.md).

articles/ai-services/language-service/concepts/configure-containers.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ ms.custom:
99
- ignite-2023
1010
ms.service: azure-ai-language
1111
ms.topic: conceptual
12-
ms.date: 12/19/2023
12+
ms.date: 11/04/2024
1313
ms.author: jboback
1414
---
1515

@@ -24,6 +24,7 @@ Language service provides each container with a common configuration framework,
2424
* Summarization
2525
* Named Entity Recognition (NER)
2626
* Personally Identifiable (PII) detection
27+
* Conversational Language Understanding (CLU)
2728

2829
## Configuration settings
2930

@@ -49,7 +50,7 @@ The `Billing` setting specifies the endpoint URI of the _Language_ resource on A
4950
|Yes| `Billing` | String | Billing endpoint URI. |
5051

5152

52-
## Eula setting
53+
## EULA setting
5354

5455
[!INCLUDE [Container shared configuration eula settings](../../includes/cognitive-services-containers-configuration-shared-settings-eula.md)]
5556

@@ -71,11 +72,11 @@ Use bind mounts to read and write data to and from the container. You can specif
7172

7273
The Language service containers don't use input or output mounts to store training or service data.
7374

74-
The exact syntax of the host mount location varies depending on the host operating system. Additionally, the host computer's mount location may not be accessible due to a conflict between permissions used by the docker service account and the host mount location permissions.
75+
The exact syntax of the host mount location varies depending on the host operating system. The host computer's mount location may not be accessible due to a conflict between the docker service account permissions and the host mount location permissions.
7576

7677
|Optional| Name | Data type | Description |
7778
|-------|------|-----------|-------------|
78-
|Not allowed| `Input` | String | Language service containers do not use this.|
79+
|Not allowed| `Input` | String | Language service containers don't use this.|
7980
|Optional| `Output` | String | The target of the output mount. The default value is `/output`. This is the location of the logs. This includes container logs. <br><br>Example:<br>`--mount type=bind,src=c:\output,target=/output`|
8081

8182
## Next steps

articles/ai-services/language-service/concepts/custom-features/multi-region-deployment.md

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: jboback
77
manager: nitinme
88
ms.service: azure-ai-language
99
ms.topic: conceptual
10-
ms.date: 12/19/2023
10+
ms.date: 11/04/2024
1111
ms.author: jboback
1212
ms.custom: language-service-clu
1313
---
@@ -22,7 +22,7 @@ ms.custom: language-service-clu
2222
> * [Custom named entity recognition (NER)](../../custom-named-entity-recognition/overview.md)
2323
> * [Orchestration workflow](../../orchestration-workflow/overview.md)
2424
25-
Custom language service features enable you to deploy your project to more than one region. This capability makes it much easier to access your project globally while you manage only one instance of your project in one place.
25+
Custom language service features enable you to deploy your project to more than one region. This capability makes it much easier to access your project globally while you manage only one instance of your project in one place. As of November 2024, custom language service features also enable you to deploy your project to multiple resources within a single region via the API, so that you can use your custom model wherever you need.
2626

2727
Before you deploy a project, you can assign *deployment resources* in other regions. Each deployment resource is a different Language resource from the one that you use to author your project. You deploy to those resources and then target your prediction requests to that resource in their respective regions and your queries are served directly from that region.
2828

@@ -54,13 +54,6 @@ You can only swap deployments that are available in the exact same regions. Othe
5454

5555
If you remove an assigned resource from your project, all of the project deployments to that resource are deleted.
5656

57-
> [!NOTE]
58-
> Orchestration workflow only:
59-
>
60-
> You *can't* assign deployment resources to orchestration workflow projects with custom question answering or LUIS connections. Subsequently, you can't add custom question answering or LUIS connections to projects that have assigned resources.
61-
>
62-
> For multiregion deployment to work as expected, the connected CLU projects *must also be deployed* to the same regional resources to which you deployed the orchestration workflow project. Otherwise, the orchestration workflow project attempts to route a request to a deployment in its region that doesn't exist.
63-
6457
Some regions are only available for deployment and not for authoring projects.
6558

6659
## Related content
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
title: Use conversational language understanding (CLU) Docker containers on-premises
3+
titleSuffix: Azure AI services
4+
description: Use Docker containers for the conversational language understanding (CLU) API to determine the language of written text, on-premises.
5+
#services: cognitive-services
6+
author: jboback
7+
manager: nitinme
8+
ms.service: azure-ai-language
9+
ms.custom:
10+
ms.topic: how-to
11+
ms.date: 10/07/2024
12+
ms.author: jboback
13+
keywords: on-premises, Docker, container
14+
---
15+
16+
# Install and run Conversational Language Understanding (CLU) containers
17+
18+
> [!NOTE]
19+
> The data limits in a single synchronous API call for the CLU container are 5120 characters per document and up to 10 documents per call.
20+
21+
Containers enable you to host the CLU API on your own infrastructure. If you have security or data governance requirements that can't be fulfilled by calling CLU remotely, then containers might be a good option.
22+
23+
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/cognitive-services/) before you begin.
24+
25+
## Prerequisites
26+
27+
You must meet the following prerequisites before using CLU containers.
28+
29+
* If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/cognitive-services/).
30+
* [Docker](https://docs.docker.com/) installed on a host computer. Docker must be configured to allow the containers to connect with and send billing data to Azure.
31+
* On Windows, Docker must also be configured to support Linux containers.
32+
* You should have a basic understanding of [Docker concepts](https://docs.docker.com/get-started/overview/).
33+
* A <a href="https://portal.azure.com/#create/Microsoft.CognitiveServicesTextAnalytics" title="Create a Language resource" target="_blank">Language resource </a>
34+
35+
[!INCLUDE [Gathering required parameters](../../../containers/includes/container-gathering-required-parameters.md)]
36+
37+
## Host computer requirements and recommendations
38+
39+
[!INCLUDE [Host Computer requirements](../../../includes/cognitive-services-containers-host-computer.md)]
40+
41+
The following table describes the minimum and recommended specifications for the available container. Each CPU core must be at least 2.6 gigahertz (GHz) or faster.
42+
43+
It is recommended to have a CPU with AVX-512 instruction set, for the best experience (performance and accuracy).
44+
45+
| | Minimum host specs | Recommended host specs |
46+
|---------------------|------------------------|------------------------|
47+
| **CLU** | 1 core, 2GB memory | 4 cores, 8GB memory |
48+
49+
CPU core and memory correspond to the `--cpus` and `--memory` settings, which are used as part of the `docker run` command.
50+
51+
## Get the container image with `docker pull`
52+
53+
The CLU container image can be found on the `mcr.microsoft.com` container registry syndicate. It resides within the `azure-cognitive-services/textanalytics/` repository and is named `clu`. The fully qualified container image name is, `mcr.microsoft.com/azure-cognitive-services/textanalytics/clu`
54+
55+
To use the latest version of the container, you can use the `latest` tag, which is for English. You can also find a full list of containers for supported languages using the [tags on the MCR](https://mcr.microsoft.com/product/azure-cognitive-services/textanalytics/clu/tags).
56+
57+
The latest CLU container is available in several languages. To download the container for the English container, use the command below.
58+
59+
```
60+
docker pull mcr.microsoft.com/azure-cognitive-services/textanalytics/clu:latest
61+
```
62+
63+
[!INCLUDE [Tip for using docker list](../../../includes/cognitive-services-containers-docker-list-tip.md)]
64+
65+
## Run the container with `docker run`
66+
67+
Once the container is on the host computer, use the [docker run](https://docs.docker.com/engine/reference/commandline/run/) command to run the containers. The container will continue to run until you stop it. Replace the placeholders below with your own values:
68+
69+
70+
> [!IMPORTANT]
71+
> * The docker commands in the following sections use the back slash, `\`, as a line continuation character. Replace or remove this based on your host operating system's requirements.
72+
> * The `Eula`, `Billing`, and `ApiKey` options must be specified to run the container; otherwise, the container won't start. For more information, see [Billing](#billing).
73+
74+
To run the CLU container, execute the following `docker run` command. Replace the placeholders below with your own values:
75+
76+
| Placeholder | Value | Format or example |
77+
|-------------|-------|---|
78+
| **{API_KEY}** | The key for your Language resource. You can find it on your resource's **Key and endpoint** page, on the Azure portal. |`xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`|
79+
| **{ENDPOINT_URI}** | The endpoint for accessing the API. You can find it on your resource's **Key and endpoint** page, on the Azure portal. | `https://<your-custom-subdomain>.cognitiveservices.azure.com` |
80+
| **{IMAGE_TAG}** | The image tag representing the language of the container you want to run. Make sure this matches the `docker pull` command you used. | `latest` |
81+
82+
```bash
83+
docker run --rm -it -p 5000:5000 --memory 8g --cpus 1 \
84+
mcr.microsoft.com/azure-cognitive-services/textanalytics/clu:{IMAGE_TAG} \
85+
Eula=accept \
86+
Billing={ENDPOINT_URI} \
87+
ApiKey={API_KEY}
88+
```
89+
90+
This command:
91+
92+
* Runs a *CLU* container from the container image
93+
* Allocates one CPU core and 8 gigabytes (GB) of memory
94+
* Exposes TCP port 5000 and allocates a pseudo-TTY for the container
95+
* Automatically removes the container after it exits. The container image is still available on the host computer.
96+
97+
[!INCLUDE [Running multiple containers on the same host](../../../includes/cognitive-services-containers-run-multiple-same-host.md)]
98+
99+
## Query the container's prediction endpoint
100+
101+
The container provides REST-based query prediction endpoint APIs.
102+
103+
Use the host, `http://localhost:5000`, for container APIs.
104+
105+
<!-- ## Validate container is running -->
106+
107+
[!INCLUDE [Container's API documentation](../../../includes/cognitive-services-containers-api-documentation.md)]
108+
109+
For information on how to call CLU see [our guide](call-api.md).
110+
111+
## Run the container disconnected from the internet
112+
113+
[!INCLUDE [configure-disconnected-container](../../../containers/includes/configure-disconnected-container.md)]
114+
115+
## Stop the container
116+
117+
[!INCLUDE [How to stop the container](../../../includes/cognitive-services-containers-stop.md)]
118+
119+
## Troubleshooting
120+
121+
If you run the container with an output [mount](../../concepts/configure-containers.md#mount-settings) and logging enabled, the container generates log files that are helpful to troubleshoot issues that happen while starting or running the container.
122+
123+
[!INCLUDE [Azure AI services FAQ note](../../../containers/includes/cognitive-services-faq-note.md)]
124+
125+
## Billing
126+
127+
The CLU containers send billing information to Azure, using a _Language_ resource on your Azure account.
128+
129+
[!INCLUDE [Container's Billing Settings](../../../includes/cognitive-services-containers-how-to-billing-info.md)]
130+
131+
For more information about these options, see [Configure containers](../../concepts/configure-containers.md).
132+
133+
## Summary
134+
135+
In this article, you learned concepts and workflow for downloading, installing, and running CLU containers. In summary:
136+
137+
* CLU provides Linux containers for Docker
138+
* Container images are downloaded from the Microsoft Container Registry (MCR).
139+
* Container images run in Docker.
140+
* You must specify billing information when instantiating a container.
141+
142+
> [!IMPORTANT]
143+
> Azure AI containers are not licensed to run without being connected to Azure for metering. Customers need to enable the containers to communicate billing information with the metering service at all times. Azure AI containers do not send customer data (e.g. text that is being analyzed) to Microsoft.
144+
145+
## Next steps
146+
147+
* See [Configure containers](../../concepts/configure-containers.md) for configuration settings.

articles/ai-services/language-service/includes/use-language-studio.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,4 @@
1111
---
1212

1313
> [!TIP]
14-
> You can use [**Language Studio**](../language-studio.md) to try Language service features without needing to write code.
14+
> You can use [**AI Studio**](../../../ai-studio/what-is-ai-studio.md) to try summarization without needing to write code.

0 commit comments

Comments
 (0)