|
1 | | -## 2-Parameter-Store |
| 1 | +## 2 Understanding the parameters |
2 | 2 |
|
3 | | -**AWS System Manager Parameter Store** |
4 | | -We provide a set of configurable parameters for the **Model Operator** to use to control the MLOps process. Go to [AWS Systems Manager Parameter Store](https://console.aws.amazon.com/systems-manager/parameters). You should see list of parameters similar the ones below. |
| 3 | +Because model performance can change over inference results, you need to implement an automated ML workflow that can continuously retrain a model with newly captured and human-labeled images. In addition, you need to consider adding flexibility into the ML workflow to allow for change without requiring development rework as business objectives evolve over time. Developing a customizable ML workflow that behaves similar to a business rule engine requires significant upfront investment, which can be a resource challenge. |
5 | 4 |
|
6 | | -**Note:** The parameter store uses string to store value regardless of type. Do NOT use comma separator for numeric values. |
| 5 | +In this demo, we provide a set of configurable parameters for you to use to customize the ML workflow. Go to [AWS Systems Manager Parameter Store](https://console.aws.amazon.com/systems-manager/parameters). You should see a list of 8 parameters as shown below: |
7 | 6 |
|
8 | | -- **Automatic-Training-Poll-Frequency** |
9 | | -This is the polling frequency (in integer minutes) on how often the Amazon EventBridge Scheduled Rule is triggered to initiate the check for automatic model training. As you update this value, the value is applied to the rule by a Lambda function. |
10 | | -- **Enable-A2I-Workflow** |
11 | | -This value (true/false) determines whether an A2I Human Loop is created regardless of the confidence level from the custom label detection result. |
12 | | -- **Enable-Automatic-Training** |
13 | | -This value (true/false) determines whether automatic model training is enabled/disabled. When this value is updated to true, a Lambda function enables the EventBride Scheduled Rule. When the value is updated to false, a Lambda function disables the EventBride Scheduled Rule. |
14 | | -- **For-System-Use-Only** |
15 | | -Do NOT modify this value. This is a reserved parameter consisting of environmental variables and operation data. The value is used and updated by Lambda functions. |
16 | | -- **Minimum-Label-Detection-Confidence** |
17 | | -Rekognition Custom Labels returns a Confidence between 0-100 on each detection. The Minimum Confidence (Float 0.00-100.00) determines whether a detection result is acceptable. If the detection Confidence is greater than or equal to the Minimum Confidence, then the detection is accepted. If not, the image is sent to A2I process, provided that Enable-A2I-Workflow is enabled. |
18 | | -- **Minimum-F1-Score** |
19 | | -This value is the minimum F1 Score (Float 0.00-1.00) that evaluate a newly trained model with a status of TRAINING-COMPLETE to accept for deployment. If the F1 Score is greater or equal to the minimum F1 Score, the model will be deployed. If not, the model is marked as failed training. |
20 | | -- **Minimum-Inference-Units** |
21 | | -This value (Int), with a minimum of 1, inference unit to use for the running Rekognition Customs Label model. A single inference unit represents 1 hour of processing and can support up to 5 Transaction Pers Second (TPS). Use a higher number to increase the TPS throughput of your model. You are charged for the number of inference units that you use. |
22 | | -- **Minimum-Untrained-Images** |
23 | | -The value (Int) represents the minimum number of untrained or newly added images that will qualify the state machine to start a new model training process on the scheduled polled event. At the instance the polled event is triggered, a Lambda function first determines the total number of training images in the designated S3 folder. Next it retrieves the value "previous_trained_images" from the parameter "For-System-Use-Only". If the difference between the total number of images in S3 and the previous_trained_images is greater than or equal to minimum untrained images, then it will trigger a new model training. On a successful training, a Lambda updates the previous_trained_images with the current total number of images trained. |
| 7 | +- **Automatic-Training-Poll-Frequency** - This is the polling frequency (in integer minutes) on how often the [Amazon EventBridge](https://aws.amazon.com/eventbridge/) Schedule rule is triggered to initiate the check for automatic model training. As you update this value, the value is applied to the rule by an [AWS Lambda](http://aws.amazon.com/lambda) function. |
| 8 | +- **Enable-A2I-Workflow** - This value (`true/false`) determines whether an [Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-start-human-loop.html) Human Loop is created when the inference confidence level from the [Amazon Rekognition Custom Labels](https://aws.amazon.com/rekognition/custom-labels-features/) detection result is less than the [minimum confidence](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/detecting-custom-labels.html) as set in the **Minimum-Label-Detection-Confidence** parameter. |
| 9 | +- **Enable-Automatic-Training** - This value (`true/false`) determines whether automatic model training is `enabled/disabled`. When this value is updated to `true`, a Lambda function enables the EventBride Schedule rule. When the value is updated to `false`, a Lambda function disables the EventBride Schedule rule. |
| 10 | +- **For-System-Use-Only** - Do **NOT** modify this value. This is a reserved parameter consisting of environmental variables and operation data. The value is used and updated by Lambda functions. |
| 11 | +- **Minimum-Label-Detection-Confidence** - Rekognition Custom Labels returns a **Confidence** between 0-100 on each detection. The **Minimum Confidence** (Float 0.00-100.00) determines whether a detection result is acceptable. If the detection **Confidence** is greater than or equal to the **Minimum Confidence**, then the detection is accepted. If not, the image is sent to [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/) (Amazon A2I) human labeling process, provided that **Enable-A2I-Workflow** is `enabled`. |
| 12 | +- **Minimum-F1-Score** - This value is the minimum [F1 score](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/gs-step-evaluate-model.html) (Float 0.00-1.00) that determines whether a newly trained model is acceptable for deployment. If the **F1 score** of the trained Model is greater or equal to the **Minimum-F1-Score** value, the model will be deployed. If not, the model is marked as failed training. |
| 13 | +- **Minimum-Inference-Units** - This value (Int), with a minimum of `1`, is number of [inference unit](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/rm-run-model.html) to use for the running Rekognition Customs Label model. A single inference unit represents 1 hour of processing and can support up to 5 Transaction Pers Second (TPS). Use a higher number to increase the TPS throughput of your model. You are charged for the number of inference units that you use. |
| 14 | +- **Minimum-Untrained-Images** - This value (Int) represents the minimum number of untrained or newly added images that will qualify the state machine to start a new model training process. At the instance the polled event is triggered, a Lambda function first determines the total number of training images in the designated S3 folder. Next it retrieves the value `previous_trained_images` from the parameter **For-System-Use-Only**. If the difference between the total number of images in S3 and the previous_trained_images is greater than or equal to minimum untrained images, then it will trigger a new model training. On a successful training, a Lambda function updates the `previous_trained_images` with the current total number of images trained. |
| 15 | + |
| 16 | + |
| 17 | +**Note:** The Parameter store uses string to store value regardless of type. Do **NOT** use comma separator for numeric values. |
24 | 18 |
|
25 | 19 | Next Step: [3-Model-Training](../3-Model-Training/) |
0 commit comments