Merge pull request #225419 from sdgilley/sdg-updates

data labeling updates

Author: v-stsavell

articles/machine-learning/how-to-create-image-labeling-projects.md

@@ -42,7 +42,7 @@ Image data can be files with any of these types: ".jpg", ".jpeg", ".png", ".jpe"
 
 [!INCLUDE [start](../../includes/machine-learning-data-labeling-start.md)]
 
-1. To create a project, select **Add project**. Give the project an appropriate name. The project name cannot be reused, even if the project is deleted in future.
+1. To create a project, select **Add project**. Give the project an appropriate name. The project name can't be reused, even if the project is deleted in future.
 
 1. Select **Image** to create an image labeling project.
 
@@ -117,8 +117,8 @@ For bounding boxes, important questions include:
 * What should the labelers do if the object is tiny? Should it be labeled as an object or should it be ignored as background?
 * How to label the object that is partially shown in the image? 
 * How to label the object that partially covered by other object?
-* How to label the object if there is no clear boundary of the object?
-* How to label the object which is not object class of interest but visually similar to an interested object type?
+* How to label the object if there's no clear boundary of the object?
+* How to label the object that isn't the object class of interest but visually similar to an interested object type?
 
 > [!NOTE]
 > Be sure to note that the labelers will be able to select the first 9 labels by using number keys 1-9.
@@ -132,7 +132,7 @@ For bounding boxes, important questions include:
 
 ## Use ML-assisted data labeling
 
-The **ML-assisted labeling** page lets you trigger automatic machine learning models to accelerate labeling tasks. Medical images (".dcm") are not included in assisted labeling.
+The **ML-assisted labeling** page lets you trigger automatic machine learning models to accelerate labeling tasks. Medical images (".dcm") aren't included in assisted labeling.
 
 At the beginning of your labeling project, the items are shuffled into a random order to reduce potential bias. However, any biases that are present in the dataset will be reflected in the trained model. For example, if 80% of your items are of a single class, then approximately 80% of the data used to train the model will be of that class.
 
@@ -143,7 +143,7 @@ ML-assisted labeling consists of two phases:
 * Clustering
 * Prelabeling
 
-The exact number of labeled data necessary to start assisted labeling is not a fixed number.  This can vary significantly from one labeling project to another. For some projects, is sometimes possible to see prelabel or cluster tasks after 300 items have been manually labeled. ML Assisted Labeling uses a technique called *Transfer Learning*, which uses a pre-trained model to jump-start the training process. If your dataset's classes are similar to those in the pre-trained model, pre-labels may be available after only a few hundred manually labeled items. If your dataset is significantly different from the data used to pre-train the model, it may take much longer.
+The exact number of labeled data necessary to start assisted labeling isn't a fixed number.  This number can vary significantly from one labeling project to another. For some projects, is sometimes possible to see prelabel or cluster tasks after 300 items have been manually labeled. ML Assisted Labeling uses a technique called *Transfer Learning*, which uses a pre-trained model to jump-start the training process. If your dataset's classes are similar to those in the pre-trained model, pre-labels may be available after only a few hundred manually labeled items. If your dataset is significantly different from the data used to pre-train the model, it may take much longer.
 
 When you're using consensus labeling, the consensus label is used for training.
 
@@ -154,11 +154,11 @@ Since the final labels still rely on input from the labeler, this technology is
 
 ### Clustering
 
-After a certain number of labels are submitted, the machine learning model for classification starts to group together similar items.  These similar images are presented to the labelers on the same screen to speed up manual tagging. Clustering is especially useful when the labeler is viewing a grid of 4, 6, or 9 images.
+After some labels are submitted, the machine learning model for classification starts to group together similar items.  These similar images are presented to the labelers on the same screen to speed up manual tagging. Clustering is especially useful when the labeler is viewing a grid of 4, 6, or 9 images.
 
-Once a machine learning model has been trained on your manually labeled data, the model is truncated to its last fully-connected layer. Unlabeled images are then passed through the truncated model in a process commonly known as "embedding" or "featurization." This embeds each image in a high-dimensional space defined by this model layer. Images that are nearest neighbors in the space are used for clustering tasks. 
+Once a machine learning model has been trained on your manually labeled data, the model is truncated to its last fully connected layer. Unlabeled images are then passed through the truncated model in a process commonly known as "embedding" or "featurization." This process embeds each image in a high-dimensional space defined by this model layer. Images that are nearest neighbors in the space are used for clustering tasks. 
 
-The clustering phase does not appear for object detection models, or for text classification.
+The clustering phase doesn't appear for object detection models, or for text classification.
 
 ### Prelabeling
 
@@ -180,9 +180,9 @@ The **Dashboard** tab shows the progress of the labeling task.
 
 :::image type="content" source="./media/how-to-create-labeling-projects/labeling-dashboard.png" alt-text="Data labeling dashboard":::
 
-The progress chart shows how many items have been labeled, skipped, in need of review, or not yet done.  Hover over the chart to see the number of item in each section.
+The progress chart shows how many items have been labeled, skipped, in need of review, or not yet done.  Hover over the chart to see the number of items in each section.
 
-The middle section shows the queue of tasks yet to be assigned. When ML assisted labeling is off, this section shows the number of manual tasks to be assigned. When ML assisted labeling is on, this will also show:
+The middle section shows the queue of tasks yet to be assigned. When ML assisted labeling is off, this section shows the number of manual tasks to be assigned. When ML assisted labeling is on, this section will also show:
 
 * Tasks containing clustered items in the queue
 * Tasks containing prelabeled items in the queue
@@ -208,7 +208,7 @@ If your project uses consensus labeling, you'll also want to review those images
 
     :::image type="content" source="media/how-to-create-labeling-projects/select-filters.png" alt-text="Screenshot: select filters to review consensus label problems." lightbox="media/how-to-create-labeling-projects/select-filters.png":::
 
-1. Under **Labeled datapoints**, select **Consensus labels in need of review**.  This shows only those images where a consensus was not achieved among the labelers.
+1. Under **Labeled datapoints**, select **Consensus labels in need of review**.  This shows only those images where a consensus wasn't achieved among the labelers.
 
     :::image type="content" source="media/how-to-create-labeling-projects/select-need-review.png" alt-text="Screenshot: Select labels in need of review.":::
 
@@ -227,7 +227,8 @@ View and change details of your project.  In this tab you can:
 * View details of the storage container used to store labeled outputs in your project
 * Add labels to your project
 * Edit instructions you give to your labels
-* Edit details of ML assisted labeling, including enable/disable
+* Change settings for ML assisted labeling, and kick off a labeling task
+
 
 ### Access for labelers
 
@@ -237,19 +238,23 @@ View and change details of your project.  In this tab you can:
 
 [!INCLUDE [add-label](../../includes/machine-learning-data-labeling-add-label.md)]
 
+## Start an ML assisted labeling task
+
+[!INCLUDE [start-ml-assist](../../includes/machine-learning-data-labeling-start-ml-assist.md)]
+
 ## Export the labels
 
 Use the **Export** button on the **Project details** page of your labeling project. You can export the label data for Machine Learning experimentation at any time.
 
 * Image labels can be exported as:
-    * [COCO format](http://cocodataset.org/#format-data).The COCO file is created in the default blob store of the Azure Machine Learning workspace in a folder within *Labeling/export/coco*. 
+    * [COCO format](http://cocodataset.org/#format-data). The COCO file is created in the default blob store of the Azure Machine Learning workspace in a folder within *Labeling/export/coco*. 
     * An [Azure Machine Learning dataset with labels](v1/how-to-use-labeled-dataset.md). 
 
 Access exported Azure Machine Learning datasets in the **Datasets** section of Machine Learning. The dataset details page also provides sample code to access your labels from Python.
 
 ![Exported dataset](./media/how-to-create-labeling-projects/exported-dataset.png)
 
-Once you have exported your labeled data to an Azure Machine Learning dataset, you can use AutoML to build computer vision models trained on your labeled data. Learn more at [Set up AutoML to train computer vision models with Python](how-to-auto-train-image-models.md)
+Once you've exported your labeled data to an Azure Machine Learning dataset, you can use AutoML to build computer vision models trained on your labeled data. Learn more at [Set up AutoML to train computer vision models with Python](how-to-auto-train-image-models.md)
 
 ## Troubleshooting
 

articles/machine-learning/how-to-create-text-labeling-projects.md

@@ -114,6 +114,7 @@ To directly upload your data:
 > [!NOTE]
 > Incremental refresh is available for projects that use tabular (.csv or .tsv) dataset input. However, only new tabular files are added.  Changes to existing tabular files will not be recognized from the refresh.
 
+
 ## Specify label categories
 
 [!INCLUDE [classes](../../includes/machine-learning-data-labeling-classes.md)]
@@ -142,7 +143,7 @@ To use **ML-assisted labeling**:
 
 At the beginning of your labeling project, the items are shuffled into a random order to reduce potential bias. However, any biases that are present in the dataset will be reflected in the trained model. For example, if 80% of your items are of a single class, then approximately 80% of the data used to train the model will be of that class. 
 
-For training the text DNN model used by ML-assist, the input text per training example will be limited to approximately the first 128 words in the document.  For tabular input, all text columns are first concatenated before applying this limit. This is a practical limit imposed to allow for the model training to complete in a timely manner. The actual text in a document (for file input) or set of text columns (for tabular input) can exceed 128 words.  The limit only pertains to what is internally leveraged by the model during the training process.
+For training the text DNN model used by ML-assist, the input text per training example will be limited to approximately the first 128 words in the document.  For tabular input, all text columns are first concatenated before applying this limit. This is a practical limit imposed to allow for the model training to complete in a timely manner. The actual text in a document (for file input) or set of text columns (for tabular input) can exceed 128 words.  The limit only pertains to what is internally used by the model during the training process.
 
 The exact number of labeled items necessary to start assisted labeling isn't a fixed number. This can vary significantly from one labeling project to another, depending on many factors, including the number of labels classes and label distribution. 
 
@@ -193,7 +194,7 @@ If your project uses consensus labeling, you'll also want to review those images
 
     :::image type="content" source="media/how-to-create-text-labeling-projects/text-labeling-select-filter.png" alt-text="Screenshot: select filters to review consensus label problems." lightbox="media/how-to-create-text-labeling-projects/text-labeling-select-filter.png":::
 
-1. Under **Labeled datapoints**, select **Consensus labels in need of review**.  This shows only those images where a consensus was not achieved among the labelers.
+1. Under **Labeled datapoints**, select **Consensus labels in need of review**.  This shows only those images where a consensus wasn't achieved among the labelers.
 
     :::image type="content" source="media/how-to-create-labeling-projects/select-need-review.png" alt-text="Screenshot: Select labels in need of review.":::
 
@@ -212,6 +213,7 @@ View and change details of your project.  In this tab you can:
 * View details of the storage container used to store labeled outputs in your project
 * Add labels to your project
 * Edit instructions you give to your labels
+* Change settings for ML assisted labeling, and kick off a labeling task
 
 ### Access for labelers
 
@@ -221,6 +223,10 @@ View and change details of your project.  In this tab you can:
 
 [!INCLUDE [add-label](../../includes/machine-learning-data-labeling-add-label.md)]
 
+## Start an ML assisted labeling task
+
+[!INCLUDE [start-ml-assist](../../includes/machine-learning-data-labeling-start-ml-assist.md)]
+
 ## Export the labels
 
 Use the **Export** button on the **Project details** page of your labeling project. You can export the label data for Machine Learning experimentation at any time.

includes/machine-learning-data-labeling-refresh.md

@@ -15,6 +15,7 @@ Select **Enable incremental refresh at regular intervals** when you want your pr
 Unselect if you don't want new files in the datastore to automatically be added to your project.
 
 > [!IMPORTANT]
-> Don't create a new version for the dataset you want to update.  If you do, the updates will not be seen, as the data labeling project is pinned to the initial version.  Instead, use [Azure Storage Explorer](https://azure.microsoft.com/features/storage-explorer/) to modify your data in the appropriate folder in the blob storage.  
+> Don't create a new version for the dataset you want to update.  If you do, the updates will not be seen, as the data labeling project is pinned to the initial version.  Instead, use [Azure Storage Explorer](https://azure.microsoft.com/features/storage-explorer/) to modify your data in the appropriate folder in the blob storage. 
+> Also, don't remove data. Removing data from the dataset your project uses will cause an error in the project. 
 
 After the project is created, use the [**Details**](#details-tab) tab to change **incremental refresh**, view the timestamp for the last refresh, and request an immediate refresh of data.

includes/machine-learning-data-labeling-start-ml-assist.md

@@ -0,0 +1,18 @@
+---
+author: sgilley
+ms.service: machine-learning
+ms.topic: include
+ms.date: 01/27/2023
+ms.author: sdgilley
+---
+
+ML assisted labeling starts automatically after some items have been labeled.  This automatic threshold varies by project.  However, you can manually start an ML assisted training run, as long as your project contains at least some labeled data.
+
+> [!NOTE]
+> On-demand training is not available for projects created before December, 2022.  Create a new project to use this feature.
+
+Use the **Details** section to start a new ML assisted training run.
+
+1. At the top of your project, select **Details**.
+1. On the side navigation for **Details**, select **ML assisted labeling**
+1. Scroll to the bottom if necessary and select **Start** for **On-demand training**