Merge pull request #218472 from sdgilley/sdg-labeling

consensus labeling

Author: Court72

articles/machine-learning/how-to-create-image-labeling-projects.md

@@ -120,9 +120,16 @@ For bounding boxes, important questions include:
 * How to label the object if there is no clear boundary of the object?
 * How to label the object which is not object class of interest but visually similar to an interested object type?
 
->[!NOTE]
+> [!NOTE]
 > Be sure to note that the labelers will be able to select the first 9 labels by using number keys 1-9.
 
+## Quality control (preview)
+
+[!INCLUDE [describe](../../includes/machine-learning-data-labeling-quality-control.md)]
+
+> [!NOTE]
+> **Instance Segmentation** projects cannot use consensus labeling.
+
 ## Use ML-assisted data labeling
 
 The **ML-assisted labeling** page lets you trigger automatic machine learning models to accelerate labeling tasks. Medical images (".dcm") are not included in assisted labeling.
@@ -131,14 +138,15 @@ At the beginning of your labeling project, the items are shuffled into a random
 
 Select *Enable ML assisted labeling* and specify a GPU to enable assisted labeling. If you don't have one in your workspace, a GPU cluster will be created for you and added to your workspace.   The cluster is created with a minimum of 0 nodes, which means it doesn't cost anything when it's not in use.
 
-
 ML-assisted labeling consists of two phases:
 
 * Clustering
 * Prelabeling
 
 The exact number of labeled data necessary to start assisted labeling is not a fixed number.  This can vary significantly from one labeling project to another. For some projects, is sometimes possible to see prelabel or cluster tasks after 300 items have been manually labeled. ML Assisted Labeling uses a technique called *Transfer Learning*, which uses a pre-trained model to jump-start the training process. If your dataset's classes are similar to those in the pre-trained model, pre-labels may be available after only a few hundred manually labeled items. If your dataset is significantly different from the data used to pre-train the model, it may take much longer.
 
+When you're using consensus labeling, the consensus label is used for training.
+
 Since the final labels still rely on input from the labeler, this technology is sometimes called *human in the loop* labeling.
 
 > [!NOTE]
@@ -192,6 +200,24 @@ On the right side is a distribution of the labels for those tasks that are compl
 
 On the **Data** tab, you can see your dataset and review labeled data. Scroll through the labeled data to see the labels. If you see incorrectly labeled data, select it and choose **Reject**, which will remove the labels and put the data back into the unlabeled queue.
 
+If your project uses consensus labeling, you'll also want to review those images without a consensus.  To do so:
+
+1. Select the **Data** tab.
+1. On the left, select  **Review labels**.
+1. On the top right, select **All filters**.
+
+    :::image type="content" source="media/how-to-create-labeling-projects/select-filters.png" alt-text="Screenshot: select filters to review consensus label problems." lightbox="media/how-to-create-labeling-projects/select-filters.png":::
+
+1. Under **Labeled datapoints**, select **Consensus labels in need of review**.  This shows only those images where a consensus was not achieved among the labelers.
+
+    :::image type="content" source="media/how-to-create-labeling-projects/select-need-review.png" alt-text="Screenshot: Select labels in need of review.":::
+
+1. For each image in need of review, select the **Consensus label** dropdown to view the conflicting labels. 
+
+    :::image type="content" source="media/how-to-create-labeling-projects/consensus-dropdown.png" alt-text="Screenshot: Select Consensus label dropdown to review conflicting labels." lightbox="media/how-to-create-labeling-projects/consensus-dropdown.png":::
+
+1. While you can select an individual to see just their label(s), you can only update or reject the labels from the top choice, **Consensus label (preview)**.
+
 ### Details tab
 
 View and change details of your project.  In this tab you can:
@@ -213,7 +239,7 @@ View and change details of your project.  In this tab you can:
 
 ## Export the labels
 
-Use the **Export** button on the **Project details** page of your labeling project. You can export the label data for Machine Learning experimentation at any time. 
+Use the **Export** button on the **Project details** page of your labeling project. You can export the label data for Machine Learning experimentation at any time.
 
 * Image labels can be exported as:
     * [COCO format](http://cocodataset.org/#format-data).The COCO file is created in the default blob store of the Azure Machine Learning workspace in a folder within *Labeling/export/coco*. 

articles/machine-learning/how-to-create-text-labeling-projects.md

@@ -125,6 +125,10 @@ To directly upload your data:
 >[!NOTE]
 > Be sure to note that the labelers will be able to select the first 9 labels by using number keys 1-9.
 
+## Quality control (preview)
+
+[!INCLUDE [describe](../../includes/machine-learning-data-labeling-quality-control.md)]
+
 ## Use ML-assisted data labeling
 
 The **ML-assisted labeling** page lets you trigger automatic machine learning models to accelerate labeling tasks. ML-assisted labeling is available for both file (.txt) and tabular (.csv) text data inputs.
@@ -140,7 +144,9 @@ At the beginning of your labeling project, the items are shuffled into a random
 
 For training the text DNN model used by ML-assist, the input text per training example will be limited to approximately the first 128 words in the document.  For tabular input, all text columns are first concatenated before applying this limit. This is a practical limit imposed to allow for the model training to complete in a timely manner. The actual text in a document (for file input) or set of text columns (for tabular input) can exceed 128 words.  The limit only pertains to what is internally leveraged by the model during the training process.
 
-The exact number of labeled items necessary to start assisted labeling isn't a fixed number. This can vary significantly from one labeling project to another, depending on many factors, including the number of labels classes and label distribution.
+The exact number of labeled items necessary to start assisted labeling isn't a fixed number. This can vary significantly from one labeling project to another, depending on many factors, including the number of labels classes and label distribution. 
+
+When you're using consensus labeling, the consensus label is used for training.
 
 Since the final labels still rely on input from the labeler, this technology is sometimes called *human in the loop* labeling.
 
@@ -179,6 +185,24 @@ On the right side is a distribution of the labels for those tasks that are compl
 
 On the **Data** tab, you can see your dataset and review labeled data. Scroll through the labeled data to see the labels. If you see incorrectly labeled data, select it and choose **Reject**, which will remove the labels and put the data back into the unlabeled queue.
 
+If your project uses consensus labeling, you'll also want to review those images without a consensus.  To do so:
+
+1. Select the **Data** tab.
+1. On the left, select  **Review labels**.
+1. On the top right, select **All filters**.
+
+    :::image type="content" source="media/how-to-create-text-labeling-projects/text-labeling-select-filter.png" alt-text="Screenshot: select filters to review consensus label problems." lightbox="media/how-to-create-text-labeling-projects/text-labeling-select-filter.png":::
+
+1. Under **Labeled datapoints**, select **Consensus labels in need of review**.  This shows only those images where a consensus was not achieved among the labelers.
+
+    :::image type="content" source="media/how-to-create-labeling-projects/select-need-review.png" alt-text="Screenshot: Select labels in need of review.":::
+
+1. For each item in need of review, select the **Consensus label** dropdown to view the conflicting labels.
+
+    :::image type="content" source="media/how-to-create-text-labeling-projects/text-labeling-consensus-dropdown.png" alt-text="Screenshot: Select Consensus label dropdown to review conflicting labels." lightbox="media/how-to-create-text-labeling-projects/text-labeling-consensus-dropdown.png":::
+
+1. While you can select an individual to see just their label(s), you can only update or reject the labels from the top choice, **Consensus label (preview)**.
+
 ### Details tab
 
 View and change details of your project.  In this tab you can:

articles/machine-learning/media/how-to-create-labeling-projects/consensus-dropdown.png

@@ -0,0 +1,18 @@
+---
+author: sgilley
+ms.service: machine-learning
+ms.topic: include
+ms.date: 10/21/2021
+ms.author: sdgilley
+---
+
+To get more accurate labels, use the **Quality control** page to send each item to multiple labelers.
+
+> [!IMPORTANT]
+> Consensus labeling is currently in public preview.
+> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
+
+Select **Enable consensus labeling (preview)** to have each item sent to multiple labelers.  Then set the **Minimum labelers** and **Maximum labelers** to specify how many labelers to use.  Make sure you have as many labelers available as your maximum number. You can't later change these settings once the project has started.
+
+If a consensus is reached from the minimum number of labelers, the item is labeled.  If a consensus isn't reached, the item will be sent to more labelers.  If there's no consensus after the item goes to the maximum number of labelers, its status will be `Needs Review`, and the project owner will be responsible for labeling the item.