Skip to content

Commit 15b86f8

Browse files
committed
Merge from upstream main
2 parents 7ce34e7 + 3eb7af7 commit 15b86f8

39 files changed

+3038
-557
lines changed

BUILD.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Amazon SageMaker Drift Detection Pipeline
1+
# Amazon SageMaker Drift Detection
22

33
This page has details on how to build a custom SageMaker MLOps template from source.
44

README.md

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Amazon SageMaker Drift Detection Pipeline
1+
# Amazon SageMaker Drift Detection
22

33
This sample demonstrates how to setup an Amazon SageMaker MLOps deployment pipeline for Drift detection
44

@@ -26,9 +26,9 @@ Follow are the list of the parameters.
2626
| PortfolioOwner | The owner of the portfolio |
2727
| ProductVersion | The product version to deploy |
2828

29-
You can copy the the required `ExecutionRoleArn` role from the Studio dashboard.
29+
You can copy the the required `ExecutionRoleArn` role from your **User Details** in the SageMaker Studio dashboard.
3030

31-
![Execution Role](docs/drift-execution-role.png)
31+
![Execution Role](docs/studio-execution-role.png)
3232

3333
Alternatively see [BUILD.md](BUILD.md) for instructions on how to build the MLOps template from source.
3434

@@ -39,21 +39,21 @@ Once your MLOps project template is registered in **AWS Service Catalog** you ca
3939
1. Switch back to the Launcher
4040
2. Click **New Project** from the **ML tasks and components** section.
4141

42-
On the Create project page, SageMaker templates is chosen by default. This option lists the built-in templates. However, you want to use the template you published for the Amazon SageMaker Drift Detection Pipeline.
42+
On the Create project page, SageMaker templates is chosen by default. This option lists the built-in templates. However, you want to use the template you published for Amazon SageMaker drift detection.
4343

44-
6. Choose **Organization templates**.
45-
7. Choose **Amazon SageMaker Drift Detection Pipeline**.
46-
8. Choose **Select project template**.
44+
3. Choose **Organization templates**.
45+
4. Choose **Amazon SageMaker drift detection template for real-time deployment**.
46+
5. Choose **Select project template**.
4747

4848
![Select Template](docs/drift-select-template.png)
4949

5050
`NOTE`: If you have recently updated your AWS Service Catalog Project, you may need to refresh SageMaker Studio to ensure it picks up the latest version of your template.
5151

52-
9. In the **Project details** section, for **Name**, enter **drift-pipeline**.
52+
6. In the **Project details** section, for **Name**, enter **drift-pipeline**.
5353
- The project name must have 32 characters or fewer.
54-
10. In the Project template parameters
55-
- For **RetrainSchedule**, input a validate [Cron Schedule](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-schedule-expression.html) which defaults to `cron(0 12 1 * ? *)` - the first day of every month.
56-
11. Choose **Create project**.
54+
7. In the Project template parameter, for **RetrainSchedule**, input a validate [Cron Schedule](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-schedule-expression.html)
55+
- This defaults to `cron(0 12 1 * ? *)` which is the first day of every month.
56+
8. Choose **Create project**.
5757

5858
![Create Project](docs/drift-create-project.png)
5959

@@ -66,10 +66,10 @@ The MLOps Drift Detection template will create the following AWS services and re
6666
1. An [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3) bucket is created for output model artifacts generated from the pipeline.
6767

6868
2. Two repositories are added to [AWS CodeCommit](https://aws.amazon.com/codecommit/):
69-
- The first repository provides code to create a multi-step model building pipeline using [AWS CloudFormation](https://aws.amazon.com/cloudformation/). The pipeline includes the following steps: data processing, model baseline, model training, model evaluation, and conditional model registration based on accuracy. The pipeline trains a linear regression model using the XGBoost algorithm on trip data from the [NYC Taxi Dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/). This repository also includes the [drift-detection.ipynb](build_pipeline/drift-detection.ipynb) notebook to [Run the Pipeline](#run-the-pipeline) (see below)
69+
- The first repository provides code to create a multi-step model building pipeline using [AWS CloudFormation](https://aws.amazon.com/cloudformation/). The pipeline includes the following steps: data processing, model baseline, model training, model evaluation, and conditional model registration based on accuracy. The pipeline trains a linear regression model using the XGBoost algorithm on trip data from the [NYC Taxi Dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/). This repository also includes the [build-pipeline.ipynb](build_pipeline/build-pipeline.ipynb) notebook to [Run the Pipeline](#run-the-pipeline) (see below)
7070
- The second repository contains code and configuration files for model deployment and monitoring. This repo also uses [AWS CodePipeline](https://aws.amazon.com/codepipeline/) and [CodeBuild](https://aws.amazon.com/codebuild/), which run an [AWS CloudFormation](https://aws.amazon.com/cloudformation/) template to create model endpoints for staging and production. This repository includes the [prod-config.json](deployment_pipeline/prod-config.json) configure to set metrics and threshold for drift detection.
7171

72-
3. Two CodePipeline pipelines:
72+
3. Two AWS CodePipeline pipelines:
7373
- The [model build pipeline](build_pipeline) creates or updates the pipeline definition and then starts a new execution with a custom [AWS Lambda](https://aws.amazon.com/lambda/) function whenever a new commit is made to the ModelBuild CodeCommit repository. The first time the CodePipeline is started, it will fail to complete expects input data to be uploaded to the Amazon S3 artifact bucket.
7474
- The [deployment pipeline](deployment_pipeline/README.md) automatically triggers whenever a new model version is added to the model registry and the status is marked as Approved. Models that are registered with Pending or Rejected statuses aren’t deployed.
7575

@@ -96,7 +96,7 @@ Once your project is created, following the instructions to [Clone the Code Repo
9696
1. Choose **Repositories**, and in the **Local path** column for the repository that ends with *build*, choose **clone repo....**
9797
2. In the dialog box that appears, accept the defaults and choose **Clone repository**
9898
3. When clone of the repository is complete, the local path appears in the **Local path** column. Click on the path to open the local folder that contains the repository code in SageMaker Studio.
99-
4. Click on the [drift-detection.ipynb](build_pipeline/drift-detection.ipynb) file to open the notebook.
99+
4. Click on the [build-pipeline.ipynb](build_pipeline/build-pipeline.ipynb) file to open the notebook.
100100

101101
In the notebook, provide the **Project Name** in the first cell to get started:
102102

@@ -141,7 +141,7 @@ This section outlines cost considerations for running the Drift Detection Pipeli
141141

142142
## Cleaning Up
143143

144-
The [drift-detection.ipynb](build_pipeline/drift-detection.ipynb) notebook includes cells that you can run to cleanup the resources.
144+
The [build-pipeline.ipynb](build_pipeline/build-pipeline.ipynb) notebook includes cells that you can run to cleanup the resources.
145145

146146
1. SageMaker prod endpoint
147147
2. SageMaker staging endpoint

app.py

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
import logging
44

55
from aws_cdk import core
6-
from infra.pipeline_stack import PipelineStack
6+
from infra.pipeline_stack import BatchPipelineStack, DeployPipelineStack
77
from infra.service_catalog_stack import ServiceCatalogStack
88

99
# Configure the logger
@@ -17,13 +17,27 @@
1717
artifact_bucket = app.node.try_get_context("drift:ArtifactBucket")
1818
artifact_bucket_prefix = app.node.try_get_context("drift:ArtifactBucketPrefix")
1919

20-
# Create the pipeline stack
21-
synth = core.DefaultStackSynthesizer(
22-
file_assets_bucket_name=artifact_bucket,
23-
generate_bootstrap_version_rule=False,
24-
bucket_prefix=artifact_bucket_prefix,
20+
# Create the batch pipeline stack
21+
BatchPipelineStack(
22+
app,
23+
"drift-batch-pipeline",
24+
synthesizer=core.DefaultStackSynthesizer(
25+
file_assets_bucket_name=artifact_bucket,
26+
bucket_prefix=artifact_bucket_prefix,
27+
generate_bootstrap_version_rule=False,
28+
),
29+
)
30+
31+
# Create the real-time deploy stack
32+
DeployPipelineStack(
33+
app,
34+
"drift-deploy-pipeline",
35+
synthesizer=core.DefaultStackSynthesizer(
36+
file_assets_bucket_name=artifact_bucket,
37+
bucket_prefix=artifact_bucket_prefix,
38+
generate_bootstrap_version_rule=False,
39+
),
2540
)
26-
PipelineStack(app, "drift-pipeline", synthesizer=synth)
2741

2842
# Create the SC stack
2943
synth = core.DefaultStackSynthesizer(

batch_pipeline/README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
2+
# Amazon SageMaker Drift Detection
3+
4+
This folder contains the code to create a batch pipeline that includes a SageMaker Transform Job and [Model Monitor](https://aws.amazon.com/sagemaker/model-monitor/) Processing Job.
5+
6+
## Build Pipeline
7+
8+
The model build pipeline contains three stages:
9+
1. Source: This stage pulls the latest code from the **AWS CodeCommit** repository.
10+
2. Build: The **AWS CodeBuild** action creates an Amazon SageMaker Pipeline definition and stores this definition as a JSON on S3. Take a look at the pipeline definition in the CodeCommit repository `pipelines/pipeline.py`. The build also creates an **AWS CloudFormation** template using the AWS CDK - take a look at the respective CDK App `app.py`.
11+
3. BatchStaging: This stage executes the staging CloudFormation template to create/update a **SageMaker Pipeline** based on the latest approved model. The pipeline includes a manual approval gate, which triggers the deployment of the model to production.
12+
4. BatchProd: This stage creates or updates a **SageMaker Pipelines** which includes a **SageMaker Model Monitor** and **Evaluate Drift Lambda** that will emit [CloudWatch Metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-interpreting-cloudwatch.html) (see below) that will trigger a **CloudWatch Alarm** for drift detection against the previously queried data quality baseline.
13+
14+
![Batch Pipeline](../docs/drift-batch-pipeline.png)
15+
16+
### Metrics Published
17+
18+
CloudWatch Metrics are emitted with the following:
19+
* Namespace `aws/sagemaker/ModelBuildingPipeline/data-metrics`
20+
* MetricName `feature_baseline_drift_<<feature_name>>`
21+
* MetricValue `distance` from the baseline
22+
23+
### Starting the Batch Pipeline
24+
25+
The batch pipeline outlined above will be started when code is committed to the **AWS CodeCommit** repository or when a model is approved in the **SageMaker Model Registry**.
26+
27+
## Testing
28+
29+
Once you have created a SageMaker Project, you can test the **Build** stage.
30+
31+
### Build Stage
32+
33+
Export the environment variables for the `SAGEMAKER_PROJECT_NAME` and `SAGEMAKER_PROJECT_ID` created by your SageMaker Project cloud formation.
34+
35+
Then run the `python` command:
36+
37+
```
38+
export SAGEMAKER_PROJECT_NAME="<<project_name>>"
39+
export SAGEMAKER_PROJECT_ID="<<project_id>>"
40+
export AWS_REGION="<<region>>"
41+
export ARTIFACT_BUCKET="sagemaker-project-<<project_id>>-build-<<region>>"
42+
export SAGEMAKER_PIPELINE_ROLE_ARN="<<service_catalog_product_use_role>>"
43+
export EVALUATE_DRIFT_FUNCTION_ARN="sagemaker-<<project_name>-evaluate-drift"
44+
cdk synth
45+
```

batch_pipeline/app.py

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
#!/usr/bin/env python3
2+
import argparse
3+
import json
4+
import logging
5+
import os
6+
7+
# Import the pipeline
8+
from pipelines.pipeline import get_pipeline, upload_pipeline
9+
10+
from aws_cdk import core
11+
from infra.batch_config import BatchConfig
12+
from infra.sagemaker_pipeline_stack import SageMakerPipelineStack
13+
from infra.model_registry import ModelRegistry
14+
15+
16+
# Configure the logger
17+
logger = logging.getLogger(__name__)
18+
logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"))
19+
20+
21+
registry = ModelRegistry()
22+
23+
24+
def create_pipeline(
25+
app: core.App,
26+
project_name: str,
27+
project_id: str,
28+
region: str,
29+
sagemaker_pipeline_role_arn: str,
30+
artifact_bucket: str,
31+
evaluate_drift_function_arn: str,
32+
stage_name: str,
33+
):
34+
# Get the stage specific deployment config for sagemaker
35+
with open(f"{stage_name}-config.json", "r") as f:
36+
j = json.load(f)
37+
batch_config = BatchConfig(**j)
38+
39+
# Set the model package group to project name
40+
package_group_name = project_name
41+
42+
# If we don't have a specific champion variant defined, get the latest approved
43+
if batch_config.model_package_version is None:
44+
logger.info("Selecting latest approved")
45+
p = registry.get_latest_approved_packages(package_group_name, max_results=1)[0]
46+
batch_config.model_package_version = p["ModelPackageVersion"]
47+
batch_config.model_package_arn = p["ModelPackageArn"]
48+
else:
49+
# Get the versioned package and update ARN
50+
logger.info(f"Selecting variant version {batch_config.model_package_version}")
51+
p = registry.get_versioned_approved_packages(
52+
package_group_name,
53+
model_package_versions=[batch_config.model_package_version],
54+
)[0]
55+
batch_config.model_package_arn = p["ModelPackageArn"]
56+
57+
# Set the default input data uri
58+
data_uri = f"s3://{artifact_bucket}/{project_id}/batch/{stage_name}"
59+
60+
# set the output transform uri
61+
transform_uri = f"s3://{artifact_bucket}/{project_id}/transform/{stage_name}"
62+
63+
# Get the pipeline execution to get the baseline uri
64+
pipeline_execution_arn = registry.get_pipeline_execution_arn(
65+
batch_config.model_package_arn
66+
)
67+
logger.info(f"Got pipeline exection arn: {pipeline_execution_arn}")
68+
model_uri = registry.get_model_artifact(pipeline_execution_arn)
69+
logger.info(f"Got model uri: {model_uri}")
70+
71+
# Set the sagemaker pipeline name and descrption with model version
72+
sagemaker_pipeline_name = f"{project_name}-batch-{stage_name}"
73+
sagemaker_pipeline_description = f"Batch Pipeline for {stage_name} model version: {batch_config.model_package_version}"
74+
75+
# If we have drift configuration then get the baseline uri
76+
baseline_uri = None
77+
if batch_config.drift_config is not None:
78+
baseline_uri = registry.get_processing_output(pipeline_execution_arn)
79+
logger.info(f"Got baseline uri: {baseline_uri}")
80+
81+
# Create batch pipeline
82+
pipeline = get_pipeline(
83+
region=region,
84+
role=sagemaker_pipeline_role_arn,
85+
pipeline_name=sagemaker_pipeline_name,
86+
default_bucket=artifact_bucket,
87+
base_job_prefix=project_id,
88+
evaluate_drift_function_arn=evaluate_drift_function_arn,
89+
data_uri=data_uri,
90+
model_uri=model_uri,
91+
transform_uri=transform_uri,
92+
baseline_uri=baseline_uri,
93+
)
94+
95+
# Create the pipeline definition
96+
logger.info("Creating/updating a SageMaker Pipeline for batch transform")
97+
pipeline_definition_body = pipeline.definition()
98+
parsed = json.loads(pipeline_definition_body)
99+
logger.info(json.dumps(parsed, indent=2, sort_keys=True))
100+
101+
# Upload the pipeline to S3 bucket/key and return JSON with key/value for for Cfn Stack parameters.
102+
# see: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sagemaker-pipeline.html
103+
logger.info(f"Uploading {stage_name} pipeline to {artifact_bucket}")
104+
pipeline_definition_key = upload_pipeline(
105+
pipeline,
106+
default_bucket=artifact_bucket,
107+
base_job_prefix=f"{project_id}/batch-{stage_name}",
108+
)
109+
110+
tags = [
111+
core.CfnTag(key="sagemaker:deployment-stage", value=stage_name),
112+
core.CfnTag(key="sagemaker:project-id", value=project_id),
113+
core.CfnTag(key="sagemaker:project-name", value=project_name),
114+
]
115+
116+
SageMakerPipelineStack(
117+
app,
118+
f"drift-batch-{stage_name}",
119+
pipeline_name=sagemaker_pipeline_name,
120+
pipeline_description=sagemaker_pipeline_description,
121+
pipeline_definition_bucket=artifact_bucket,
122+
pipeline_definition_key=pipeline_definition_key,
123+
sagemaker_role_arn=sagemaker_pipeline_role_arn,
124+
tags=tags,
125+
drift_config=batch_config.drift_config,
126+
)
127+
128+
129+
def main(
130+
project_name: str,
131+
project_id: str,
132+
region: str,
133+
sagemaker_pipeline_role_arn: str,
134+
artifact_bucket: str,
135+
evaluate_drift_function_arn: str,
136+
):
137+
# Create App and stacks
138+
app = core.App()
139+
140+
create_pipeline(
141+
app=app,
142+
project_name=project_name,
143+
project_id=project_id,
144+
region=region,
145+
sagemaker_pipeline_role_arn=sagemaker_pipeline_role_arn,
146+
artifact_bucket=artifact_bucket,
147+
evaluate_drift_function_arn=evaluate_drift_function_arn,
148+
stage_name="staging",
149+
)
150+
151+
create_pipeline(
152+
app=app,
153+
project_name=project_name,
154+
project_id=project_id,
155+
region=region,
156+
sagemaker_pipeline_role_arn=sagemaker_pipeline_role_arn,
157+
artifact_bucket=artifact_bucket,
158+
evaluate_drift_function_arn=evaluate_drift_function_arn,
159+
stage_name="prod",
160+
)
161+
162+
app.synth()
163+
164+
165+
if __name__ == "__main__":
166+
parser = argparse.ArgumentParser(description="Load parameters")
167+
parser.add_argument("--region", default=os.environ.get("AWS_REGION"))
168+
parser.add_argument(
169+
"--project-name",
170+
default=os.environ.get("SAGEMAKER_PROJECT_NAME"),
171+
)
172+
parser.add_argument("--project-id", default=os.environ.get("SAGEMAKER_PROJECT_ID"))
173+
parser.add_argument(
174+
"--sagemaker-pipeline-role-arn",
175+
default=os.environ.get("SAGEMAKER_PIPELINE_ROLE_ARN"),
176+
)
177+
parser.add_argument(
178+
"--evaluate-drift-function-arn",
179+
default=os.environ.get("EVALUATE_DRIFT_FUNCTION_ARN"),
180+
)
181+
parser.add_argument(
182+
"--artifact-bucket",
183+
default=os.environ.get("ARTIFACT_BUCKET"),
184+
)
185+
args = vars(parser.parse_args())
186+
logger.info("args: {}".format(args))
187+
main(**args)

batch_pipeline/cdk.json

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"app": "python3 app.py",
3+
"context": {
4+
"@aws-cdk/core:enableStackNameDuplicates": "true",
5+
"aws-cdk:enableDiffNoFail": "true",
6+
"@aws-cdk/core:stackRelativeExports": "true",
7+
"@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
8+
"@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
9+
"@aws-cdk/aws-kms:defaultKeyPolicies": true,
10+
"@aws-cdk/aws-s3:grantWriteWithoutAcl": true
11+
}
12+
}
13+

0 commit comments

Comments
 (0)