Add human sampling & standardize SNS messages

Author: leschan

0-Prerequisites/README.md

@@ -19,6 +19,6 @@
 
 - You need to initialize Rekognition Custom Labels for your selected Region before you can use Rekognition Custom Labels. On the Rekognition Custom Labels console, choose **Projects** in the navigation pane and take one of the following actions:
  - If you see the **Projects** page with or without projects, then your Region is initialized.
- - If you’re prompted with a message for first-time setup, follow the instructions to create the default [Amazon Simple Storage Service](http://aws.amazon.com/s3) (Amazon S3) bucket. We don’t use the default S3 bucket with this solution, but it’s part of the Rekognition Custom Labels initialization process.
+ - If you’re prompted with a message for first-time setup, follow the instructions to create the default [Amazon Simple Storage Service](https://aws.amazon.com/s3) (Amazon S3) bucket. We don’t use the default S3 bucket with this solution, but it’s part of the Rekognition Custom Labels initialization process.
 
 Next Step: [Deployment](../1-Deployment/)

2-Parameter-Store/README.md

@@ -2,15 +2,18 @@
 
 Because model performance can change over inference results, you need to implement an automated ML workflow that can continuously retrain a model with newly captured and human-labeled images. In addition, you need to consider adding flexibility into the ML workflow to allow for change without requiring development rework as business objectives evolve over time. Developing a customizable ML workflow that behaves similar to a business rule engine requires significant upfront investment, which can be a resource challenge.
 
-In this demo, we provide a set of configurable parameters for you to use to customize the ML workflow. Go to [AWS Systems Manager Parameter Store](https://console.aws.amazon.com/systems-manager/parameters). You should see a list of 8 parameters as shown below:
+In this demo, we provide a set of configurable parameters for you to use to customize the ML workflow. Go to [AWS Systems Manager Parameter Store](https://console.aws.amazon.com/systems-manager/parameters). You should see a list of 11 parameters as shown below:
 
-- **Automatic-Training-Poll-Frequency** - This is the polling frequency (in integer minutes) on how often the [Amazon EventBridge](https://aws.amazon.com/eventbridge/) Schedule rule is triggered to initiate the check for automatic model training. As you update this value, the value is applied to the rule by an [AWS Lambda](http://aws.amazon.com/lambda) function.
+- **Automatic-Human-Sampling-Frequency** - This is the polling frequency (in integer minutes) on how often the [Amazon EventBridge](https://aws.amazon.com/eventbridge/) Schedule rule is triggered to initiate the check for human sampling. As you update this value, the value is applied to the rule by an [AWS Lambda](https://aws.amazon.com/lambda) function.
+- **Automatic-Training-Poll-Frequency** - This is the polling frequency (in integer minutes) on how often the [Amazon EventBridge](https://aws.amazon.com/eventbridge/) Schedule rule is triggered to initiate the check for automatic model training. As you update this value, the value is applied to the rule by an [AWS Lambda](https://aws.amazon.com/lambda) function.
 - **Enable-A2I-Workflow** - This value (`true/false`) determines whether an [Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-start-human-loop.html) Human Loop is created when the inference confidence level from the [Amazon Rekognition Custom Labels](https://aws.amazon.com/rekognition/custom-labels-features/) detection result is less than the [minimum confidence](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/detecting-custom-labels.html) as set in the **Minimum-Label-Detection-Confidence** parameter.
+- **Enable-Automatic-Human-Sampling** - This value (`true/false`) determines whether automatic human sampling is `enabled/disabled`. When this value is updated to `true`, a Lambda function enables the EventBride Schedule rule. When the value is updated to `false`, a Lambda function disables the EventBride Schedule rule.
 - **Enable-Automatic-Training** - This value (`true/false`) determines whether automatic model training is `enabled/disabled`. When this value is updated to `true`, a Lambda function enables the EventBride Schedule rule. When the value is updated to `false`, a Lambda function disables the EventBride Schedule rule.
 - **For-System-Use-Only** - Do **NOT** modify this value. This is a reserved parameter consisting of environmental variables and operation data. The value is used and updated by Lambda functions.
-- **Minimum-Label-Detection-Confidence** - Rekognition Custom Labels returns a **Confidence** between 0-100 on each detection. The **Minimum Confidence** (Float 0.00-100.00) determines whether a detection result is acceptable. If the detection **Confidence** is greater than or equal to the **Minimum Confidence**, then the detection is accepted. If not, the image is sent to [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/) (Amazon A2I) human labeling process, provided that **Enable-A2I-Workflow** is `enabled`.
+- **Human-Sampling-Interval** - This value (Int) represents the minimum number of new Rekognition Custom Labelsdetections since the last human sampled detection before a new human sample is needed. If a new detection meets the interval requirement, it is marked for human sampling. An Amazon A2I human workflow is created, if a human workflow does not exist already.
 - **Minimum-F1-Score** - This value is the minimum [F1 score](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/gs-step-evaluate-model.html) (Float 0.00-1.00) that determines whether a newly trained model is acceptable for deployment. If the **F1 score** of the trained Model is greater or equal to the **Minimum-F1-Score** value, the model will be deployed. If not, the model is marked as failed training.
-- **Minimum-Inference-Units** - This value (Int), with a minimum of `1`, is number of [inference unit](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/rm-run-model.html) to use for the running Rekognition Customs Label model. A single inference unit represents 1 hour of processing and can support up to 5 Transaction Pers Second (TPS). Use a higher number to increase the TPS throughput of your model. You are charged for the number of inference units that you use.
+- **Minimum-Inference-Units** - This value (Int), with a minimum of `1`, is number of [inference unit](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/rm-run-model.html) to use for the running Rekognition Customs Labelsmodel. A single inference unit represents 1 hour of processing and can support up to 5 Transaction Pers Second (TPS). Use a higher number to increase the TPS throughput of your model. You are charged for the number of inference units that you use.
+- **Minimum-Label-Detection-Confidence** - Rekognition Custom Labels returns a **Confidence** between 0-100 on each detection. The **Minimum Confidence** (Float 0.00-100.00) determines whether a detection result is acceptable. If the detection **Confidence** is greater than or equal to the **Minimum Confidence**, then the detection is accepted. If not, the image is sent to [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/) (Amazon A2I) human labeling process, provided that **Enable-A2I-Workflow** is `enabled`.
 - **Minimum-Untrained-Images** - This value (Int) represents the minimum number of untrained or newly added images that will qualify the state machine to start a new model training process. At the instance the polled event is triggered, a Lambda function first determines the total number of training images in the designated S3 folder. Next it retrieves the value `previous_trained_images` from the parameter **For-System-Use-Only**. If the difference between the total number of images in S3 and the previous_trained_images is greater than or equal to minimum untrained images, then it will trigger a new model training. On a successful training, a Lambda function updates the `previous_trained_images` with the current total number of images trained.
 
 

3-Model-Training/README.md

@@ -6,7 +6,7 @@ Rekognition Custom Labels supports two methods of labeling datasets: [object-lev
 
 We use Amazon and AWS logos for our initial training images. The images are located in the `sample-images` folder. We use the folder name as the label such that images in the **amazon_logo** and **aws_logo** folders are labeled `amazon_log` and `aws_logo`, respectively.
 
-In this section, we walk you through the steps to add training images to [Amazon Simple Storage Service](http://aws.amazon.com/s3) (Amazon S3)  and explain how the automatic model training process works.
+In this section, we walk you through the steps to add training images to [Amazon Simple Storage Service](https://aws.amazon.com/s3) (Amazon S3)  and explain how the automatic model training process works.
 
 1. On the [Amazon S3](https://s3.console.aws.amazon.com/s3/home) console, choose the bucket with the name similar to `yourstackname-s3bucket-esxhty0wyhqu`.  and select the bucket with name similar to "yourstackname-s3bucket-esxhty0wyhqu". You should see 5 folders:
   - a2i-human-loop-data/

4-Custom-Labels-Detection/README.md

@@ -5,13 +5,20 @@ In this section, we walk you through the steps to detect custom logos and explai
 1.	Navigate to the [Amazon S3](https://s3.console.aws.amazon.com/s3/home) bucket and choose the `images_for_detection` folder.
 2.	Upload an image that you used previously for training.
 
-The upload process triggers an S3 `PutObject` event, which invokes an [AWS Lambda](http://aws.amazon.com/lambda) function to run the Rekognition Custom Labels detection process. You should receive an email with a detection result indicating a confidence score of at least 70, which is as expected because you’re using the same image you used for training.
+The upload process triggers an S3 `PutObject` event, which invokes an [AWS Lambda](https://aws.amazon.com/lambda) function to run the [Amazon Rekognition Custom Labels](https://aws.amazon.com/rekognition/custom-labels-features/) detection process. Upon successful completion of the detection event, the Lambda function performs additional tasks:
+- Log the image detection results to [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) table
+- Create an Amazon A2I human labeling task if the detection confidence level is less then the minimum confidence level and A2I workflow is enabled
+- Log the Amazon A2I request information to the DynamoDB table
+
+You should receive an email with a detection result indicating a confidence score of at least 70, which is as expected because you’re using the same image you used for training.
 
 3.	Upload all the images in the `inference_images` folder.
 
 You should receive emails with detection results with varying confidence scores, and some should indicate that Amazon A2I human loops have been created. When a detection result has a confidence score less than the acceptance level, which is currently set at 70, a human labeling task is created to label that image.
 
 4.	Optionally, go to the [AWS Step Functions](https://console.aws.amazon.com/states/) console to review the state machine runs of the custom label detection events.
 
+5.	Optionally, go to the [Amazon DynamoDB](https://console.aws.amazon.com/dynamodb/) console to review the custom label detection and A2I event logs.
+
 
 Next Step: [5-A2I-Human-Loop](../5-A2I-Human-Loop/)

5-A2I-Human-Loop/README.md

@@ -29,8 +29,10 @@ If the private team was deployed for you, the username and temporary password ar
 9.	For each image, review the image and choose the correct Amazon company logo or **None of the Above** to use as label.
 10.	Keep a tally of the options you choose in each task.
 11. Choose **Submit** to advance to the next image until you have completed all the labeling tasks.
-12. Return to the i`mages_labeled_by_folder` folder and review the changes.
+12. Return to the `images_labeled_by_folder` folder and review the changes.
 
-For each image you labeled, an [AWS Lambda](http://aws.amazon.com/lambda) function makes a copy of the original image, appends the prefix `humanLoopName` and an UUID to the original S3 object key, and adds it to the corresponding labeled folder. If the folder doesn’t exist, the Lambda function creates a corresponding labeled folder. For the `None of the Above` labels, nothing is done. This is how newly captured and labeled images are added to the training dataset.
+For each image you labeled, an [AWS Lambda](https://aws.amazon.com/lambda) function performs two tasks:
+- Make a copy of the original image, appends the prefix `humanLoopName` and an UUID to the original S3 object key, and adds it to the corresponding labeled folder. If the folder doesn’t exist, the Lambda function creates a corresponding labeled folder. For the `None of the Above` labels, nothing is done. This is how newly captured and labeled images are added to the training dataset.
+- Log the A2I results to the corresponding detection event for other downstream processes
 
-Next Step: [6-Conclusions-Cleanup](../6-Conclusions-Cleanup/)
+Next Step: [6-Human-Sampling](../6-Human-Sampling/)

6-Human-Sampling/README.md

@@ -0,0 +1,28 @@
+## 6 Automatic human sampling
+
+In this section, we explain the design principle and the processes behind the human sampling workflow.
+
+**Design principle**  
+The principle for the human-in-the-loop workflow is intended for model improvement by capturing inference images with [Amazon Rekognition Custom Labels](https://aws.amazon.com/rekognition/custom-labels-features/) with low confidence detection result and add them to the training dataset for new training. The principle for human sampling workflow is intended for business improvement. In human sampling, for every `nth` detection, the detection is flagged for human review and the human-labeled result is compared against the detected one, regardless of the original detection confidence level. The sampled detections can then used for qualify control, audit, analytics, etc. Although both workflows use the same [Amazon A2I](https://aws.amazon.com/augmented-ai/) process, inference images flagged as `sampled only` are **not** add to the training dataset.
+
+**Workflow**  
+The human sampling workflow is configurable by three [Amazon Parameter Store](https://console.aws.amazon.com/systems-manager/parameters) parameters as explained in [Section 2-Parameter Store](../2-Parameter-Store/). When **Enable-Automatic-Human-Sampling** is enabled, an [Amazon EventBridge](https://aws.amazon.com/eventbridge/) schedule rule is triggered every `nth` minutes as set in **Automatic-Human-Sampling-Frequency** to invoke the processes orchestrated by [AWS Step Functions](https://aws.amazon.com/step-functions/). The processes include:
+
+1. Initialize the starting `query date` with the `Last modified date` of **Enable-Automatic-Human-Sampling**. The purpose is to sample forward instead retroactively finding historic samples.
+2. Query the minimum number (`limit`) of detection events as set in **Human-Sampling-Interval** forward from the `query date`
+3. If the query result count is less than the minimum number, then the process ends. Otherwise the last (`limit`) detection event is marked as sampled and the new `query date` is set to the detection event date.
+4. Next an A2I human labeling task is created if none exists from another process and the originating source is marked as `Human Sampling`.
+5. Repeat Steps 2 through 4.
+
+Finally the labeling task is completed as explained in [Section 5-A2I Human Loop](../5-A2I-Human-Loop/).
+
+
+
+
+
+
+
+
+
+
+Next Step: [7-Conclusions-Cleanup](../7-Conclusions-Cleanup/)

6-Conclusions-Cleanup/README.md renamed to 7-Conclusions-Cleanup/README.md

@@ -1,7 +1,7 @@
-## 6 Conclusion and cleanup resources
+## 7 Conclusion and cleanup resources
 
 **Conclusions**  
-In this demo we explained how to build an automated and configurable continuous model improvement workflow for [Amazon Rekognition Custom Labels](https://aws.amazon.com/rekognition/custom-labels-features/) and [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/) (Amazon A2I). By incorporating [Amazon Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html), we demonstrated how to use parameters as variables for [AWS Step Functions](https://aws.amazon.com/step-functions/) to customize the behaviors of the ML workflow without needing development rework. We walked through the steps to deploy the solution with [AWS CloudFormation](https://aws.amazon.com/cloudformation/) and completed an end-to-end process to train and deploy a Rekognition Custom Labels model, perform custom label detection, and create and complete Amazon A2I human labeling tasks.
+In this demo we explained how to build an automated and configurable continuous model improvement workflow for [Amazon Rekognition Custom Labels](https://aws.amazon.com/rekognition/custom-labels-features/) and [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/) (Amazon A2I). By incorporating [Amazon Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html), we demonstrated how to use parameters as variables for [AWS Step Functions](https://aws.amazon.com/step-functions/) to customize the behaviors of the ML workflow without needing development rework. We walked through the steps to deploy the solution with [AWS CloudFormation](https://aws.amazon.com/cloudformation/) and completed an end-to-end process to train and deploy a Rekognition Custom Labels model, perform custom label detection, create and complete Amazon A2I human labeling tasks, and generate human samples.
 
 Things to consider to enhance this worfklow:
 - Add additional parameters to control the behaviors of this workflow.
@@ -18,16 +18,19 @@ Complete the following steps to clean up the AWS resources that we created as pa
 
 The following resources are not deleted by CloudFormation:
   - S3 bucket
-  - Rekognition Custom Labels project
+  - Rekognition Custom Labels project & models
+  - DynamoDB table
   - [Amazon SageMaker Ground Truth](https://aws.amazon.com/sagemaker/groundtruth/) private workforce
 
 3. On the S3 console, choose the bucket you used for this post.
 4. Choose **Empty**.
 5. Choose **Delete**
-6. On the Rekognition Custom Label console, choose the running model.
+6. On the Rekognition Custom Labelsconsole, choose the running model.
 7. On the **Evaluate and use model** tab, expand the **API Code** section.
-8. Run the [AWS Command Line Interface](http://aws.amazon.com/cli) (AWS CLI) stop model command.
+8. Run the [AWS Command Line Interface](https://aws.amazon.com/cli) (AWS CLI) stop model command.
 9.	On the **Projects** page, select the project and choose **Delete**.
-10.	To delete the private workforce, enter the following code (leave the workforce name as `default` and provide your Region):  
+10. On the DyanmoDB console, select the table you used for this post.
+11. Choose **Delete**
+12.	To delete the private workforce, enter the following code (leave the workforce name as `default` and provide your Region):  
 
   `aws sagemaker delete-workforce --workforce-name default --region your-region-x`

CONTRIBUTING.md

@@ -49,7 +49,7 @@ This project has adopted the [Amazon Open Source Code of Conduct](https://aws.gi
 
 ## Security issue notifications
 
-If you discover a potential security issue, please notify AWS Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public issue on GitHub.
+If you discover a potential security issue, please notify AWS Security via our [vulnerability reporting page](https://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public issue on GitHub.
 
 ## Licensing