Skip to content

Commit f7c9538

Browse files
brightsparcbrightsparc
authored
Add batch pipeline to GitHub actions workflow (#3)
* Create lambda on fly in batch pipline, and change actions to pass role for SM product use role * Update github actions to allow read-only sagemaker permissions. Added batch stages * Default env at workflow instead of secret, set role based on creds * Update to pass function arn that is created in CFN * Adding lambda permissions, and enable batch prod, change build pipeline name * Update SC stack to add create lambda permissions for SM functions * Move lambda inline to SM pipeline * Update to run batch prod in main branch only * Update to only do on push or pull request for main * Renaming pipeline name to get around hung SM pipeline
1 parent 15b86f8 commit f7c9538

15 files changed

+266
-142
lines changed

‎.github/workflows/build-deploy-pipeline.yml

Lines changed: 172 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,22 @@
44

55
name: Build and Deploy
66

7-
on: [push, pull_request] # Optionally filter on branch
7+
on:
8+
# Trigger the workflow on push or pull request,
9+
# but only for the main branch
10+
push:
11+
branches:
12+
- main
13+
pull_request:
14+
branches:
15+
- main
16+
release:
17+
types:
18+
- created
19+
20+
env:
21+
AWS_REGION: us-east-1
22+
PROJECT_NAME: ${{ github.event.repository.name }}
823

924
jobs:
1025
build:
@@ -20,25 +35,15 @@ jobs:
2035
- name: Checkout
2136
uses: actions/checkout@v2
2237

23-
- name: Config Environment
24-
id: env-name
25-
env:
26-
PROJECT_NAME: ${{ github.event.repository.name }}
27-
run: |
28-
echo "Project name: $PROJECT_NAME"
29-
echo "::set-output name=project_name::$PROJECT_NAME"
30-
3138
- name: Setup Python
3239
uses: actions/setup-python@v2
3340
with:
3441
python-version: "3.8"
35-
architecture: "x64"
3642

3743
- name: Setup Node
3844
uses: actions/setup-node@v2
3945
with:
4046
node-version: "12"
41-
architecture: "x64"
4247
cache: npm
4348

4449
- name: Install Requirements
@@ -52,18 +57,16 @@ jobs:
5257
with:
5358
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
5459
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
55-
aws-region: ${{ secrets.AWS_REGION }}
56-
role-to-assume: ${{ secrets.AWS_SAGEMAKER_ROLE }}
60+
aws-region: ${{ env.AWS_REGION }}
5761
role-duration-seconds: 1200
5862

5963
- name: Build Pipeline
6064
id: build-pipeline
6165
env:
62-
SAGEMAKER_PROJECT_NAME: ${{ steps.env-name.outputs.project_name }}
63-
SAGEMAKER_PIPELINE_NAME: ${{ steps.env-name.outputs.project_name }}-pipeline
64-
SAGEMAKER_PIPELINE_DESCRIPTION: "SageMaker pipeline created from GitHub actions"
65-
SAGEMAKER_PIPELINE_ROLE_ARN: ${{ secrets.AWS_SAGEMAKER_ROLE }}
66-
AWS_REGION: ${{ secrets.AWS_REGION }}
66+
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
67+
SAGEMAKER_PIPELINE_NAME: ${{ env.PROJECT_NAME }}-pipeline
68+
SAGEMAKER_PIPELINE_DESCRIPTION: "Drift detection model build pipeline created from GitHub actions"
69+
SAGEMAKER_PIPELINE_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
6770
run: |
6871
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
6972
echo "Project id: $SAGEMAKER_PROJECT_ID"
@@ -76,15 +79,14 @@ jobs:
7679
run: cat drift-pipeline.yml
7780

7881
- name: Create CFN Pipeline
79-
id: deploy-pipeline
8082
uses: aws-actions/aws-cloudformation-github-deploy@v1
8183
with:
82-
name: sagemaker-${{ steps.build-pipeline.outputs.pipeline_name }}
84+
name: sagemaker-${{ env.PROJECT_NAME }}-pipeline
8385
template: ./build_pipeline/drift-pipeline.yml # Need to specify working-directory
86+
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
8487
no-fail-on-empty-changeset: "1"
8588

8689
- name: Start Pipeline
87-
id: start-pipeline # TODO: Run python code that waits for pipeline to complete
8890
run: aws sagemaker start-pipeline-execution --pipeline-name ${{ steps.build-pipeline.outputs.pipeline_name }} --pipeline-parameters Name=InputSource,Value=GitHubAction#${{ github.run_number }}
8991

9092
- name: Upload template
@@ -93,6 +95,145 @@ jobs:
9395
name: drift-pipeline
9496
path: ./build_pipeline/drift-pipeline.yml
9597

98+
batch_staging:
99+
needs: build
100+
name: Batch to staging
101+
runs-on: ubuntu-latest
102+
environment:
103+
name: batch-staging
104+
defaults:
105+
run:
106+
shell: bash
107+
working-directory: ./batch_pipeline
108+
steps:
109+
- name: Checkout
110+
uses: actions/checkout@v2
111+
112+
- name: Setup Python
113+
uses: actions/setup-python@v2
114+
with:
115+
python-version: "3.8"
116+
117+
- name: Setup Node
118+
uses: actions/setup-node@v2
119+
with:
120+
node-version: "12"
121+
cache: npm
122+
123+
- name: Install Requirements
124+
run: |
125+
npm install -g aws-cdk # Install cdk
126+
pip install --requirement requirements.txt
127+
128+
- name: Configure AWS Credentials
129+
uses: aws-actions/configure-aws-credentials@v1
130+
id: creds
131+
with:
132+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
133+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
134+
aws-region: ${{ env.AWS_REGION }}
135+
role-duration-seconds: 1200
136+
137+
- name: Build Templates
138+
id: build-templates
139+
env:
140+
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
141+
SAGEMAKER_PIPELINE_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
142+
LAMBDA_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
143+
run: |
144+
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
145+
echo "Project id: $SAGEMAKER_PROJECT_ID"
146+
export ARTIFACT_BUCKET=sagemaker-project-$SAGEMAKER_PROJECT_ID-$AWS_REGION
147+
echo "Artifact Bucket: $ARTIFACT_BUCKET"
148+
npx cdk synth drift-batch-staging --path-metadata false --asset-metadata=false > drift-batch-staging.yml
149+
150+
- name: Print template
151+
run: cat drift-batch-staging.yml
152+
153+
- name: Deploy Staging
154+
uses: aws-actions/aws-cloudformation-github-deploy@v1
155+
with:
156+
name: sagemaker-${{ env.PROJECT_NAME }}-batch-staging
157+
template: ./batch_pipeline/drift-batch-staging.yml # Need to specify working-directory
158+
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
159+
no-fail-on-empty-changeset: "1"
160+
161+
- name: Upload template
162+
uses: actions/upload-artifact@v2
163+
with:
164+
name: drift-batch-staging
165+
path: ./batch_pipeline/drift-batch-staging.yml
166+
167+
batch_prod:
168+
needs: batch_staging
169+
name: Batch to prod
170+
if: ${{ github.ref == 'refs/heads/main' }} # Filter to only run on main branch
171+
runs-on: ubuntu-latest
172+
environment:
173+
name: batch-prod
174+
defaults:
175+
run:
176+
shell: bash
177+
working-directory: ./batch_pipeline
178+
steps:
179+
- name: Checkout
180+
uses: actions/checkout@v2
181+
182+
- name: Setup Python
183+
uses: actions/setup-python@v2
184+
with:
185+
python-version: "3.8"
186+
187+
- name: Setup Node
188+
uses: actions/setup-node@v2
189+
with:
190+
node-version: "12"
191+
cache: npm
192+
193+
- name: Install Requirements
194+
run: |
195+
npm install -g aws-cdk # Install cdk
196+
pip install --requirement requirements.txt
197+
198+
- name: Configure AWS Credentials
199+
id: creds
200+
uses: aws-actions/configure-aws-credentials@v1
201+
with:
202+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
203+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
204+
aws-region: ${{ env.AWS_REGION }}
205+
role-duration-seconds: 1200
206+
207+
- name: Build Templates
208+
id: build-templates
209+
env:
210+
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
211+
SAGEMAKER_PIPELINE_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
212+
LAMBDA_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
213+
run: |
214+
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
215+
echo "Project id: $SAGEMAKER_PROJECT_ID"
216+
export ARTIFACT_BUCKET=sagemaker-project-$SAGEMAKER_PROJECT_ID-$AWS_REGION
217+
echo "Artifact Bucket: $ARTIFACT_BUCKET"
218+
npx cdk synth drift-batch-prod --path-metadata false --asset-metadata=false > drift-batch-prod.yml
219+
220+
- name: Print Template
221+
run: cat drift-batch-prod.yml
222+
223+
- name: Deploy Prod
224+
uses: aws-actions/aws-cloudformation-github-deploy@v1
225+
with:
226+
name: sagemaker-${{ env.PROJECT_NAME }}-batch-prod
227+
template: ./batch_pipeline/drift-batch-prod.yml # Need to specify working-directory
228+
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
229+
no-fail-on-empty-changeset: "1"
230+
231+
- name: Upload template
232+
uses: actions/upload-artifact@v2
233+
with:
234+
name: drift-batch-prod
235+
path: ./batch_pipeline/drift-batch-prod.yml
236+
96237
deploy_staging:
97238
needs: build
98239
name: Deploy to staging
@@ -107,25 +248,15 @@ jobs:
107248
- name: Checkout
108249
uses: actions/checkout@v2
109250

110-
- name: Config Environment
111-
id: env-name
112-
env:
113-
PROJECT_NAME: ${{ github.event.repository.name }}
114-
run: |
115-
echo "Project name: $PROJECT_NAME"
116-
echo "::set-output name=project_name::$PROJECT_NAME"
117-
118251
- name: Setup Python
119252
uses: actions/setup-python@v2
120253
with:
121254
python-version: "3.8"
122-
architecture: "x64"
123255

124256
- name: Setup Node
125257
uses: actions/setup-node@v2
126258
with:
127259
node-version: "12"
128-
architecture: "x64"
129260
cache: npm
130261

131262
- name: Install Requirements
@@ -139,16 +270,14 @@ jobs:
139270
with:
140271
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
141272
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
142-
aws-region: ${{ secrets.AWS_REGION }}
143-
role-to-assume: ${{ secrets.AWS_SAGEMAKER_ROLE }}
273+
aws-region: ${{ env.AWS_REGION }}
144274
role-duration-seconds: 1200
145275

146276
- name: Build Templates
147277
id: build-templates
148278
env:
149-
SAGEMAKER_PROJECT_NAME: ${{ steps.env-name.outputs.project_name }}
150-
SAGEMAKER_EXECUTION_ROLE_ARN: ${{ secrets.AWS_SAGEMAKER_ROLE }}
151-
AWS_REGION: ${{ secrets.AWS_REGION }}
279+
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
280+
SAGEMAKER_EXECUTION_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
152281
run: |
153282
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
154283
echo "Project id: $SAGEMAKER_PROJECT_ID"
@@ -160,11 +289,11 @@ jobs:
160289
run: cat drift-deploy-staging.yml
161290

162291
- name: Deploy Staging
163-
id: deploy-pipeline
164292
uses: aws-actions/aws-cloudformation-github-deploy@v1
165293
with:
166-
name: sagemaker-${{ steps.env-name.outputs.project_name }}-deploy-staging
294+
name: sagemaker-${{ env.PROJECT_NAME }}-deploy-staging
167295
template: ./deployment_pipeline/drift-deploy-staging.yml # Need to specify working-directory
296+
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
168297
no-fail-on-empty-changeset: "1"
169298

170299
- name: Upload template
@@ -188,25 +317,15 @@ jobs:
188317
- name: Checkout
189318
uses: actions/checkout@v2
190319

191-
- name: Config Environment
192-
id: env-name
193-
env:
194-
PROJECT_NAME: ${{ github.event.repository.name }}
195-
run: |
196-
echo "Project name: $PROJECT_NAME"
197-
echo "::set-output name=project_name::$PROJECT_NAME"
198-
199320
- name: Setup Python
200321
uses: actions/setup-python@v2
201322
with:
202323
python-version: "3.8"
203-
architecture: "x64"
204324

205325
- name: Setup Node
206326
uses: actions/setup-node@v2
207327
with:
208328
node-version: "12"
209-
architecture: "x64"
210329
cache: npm
211330

212331
- name: Install Requirements
@@ -220,16 +339,14 @@ jobs:
220339
with:
221340
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
222341
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
223-
aws-region: ${{ secrets.AWS_REGION }}
224-
role-to-assume: ${{ secrets.AWS_SAGEMAKER_ROLE }}
342+
aws-region: ${{ env.AWS_REGION }}
225343
role-duration-seconds: 1200
226344

227345
- name: Build Templates
228346
id: build-templates
229347
env:
230-
SAGEMAKER_PROJECT_NAME: ${{ steps.env-name.outputs.project_name }}
231-
SAGEMAKER_EXECUTION_ROLE_ARN: ${{ secrets.AWS_SAGEMAKER_ROLE }}
232-
AWS_REGION: ${{ secrets.AWS_REGION }}
348+
SAGEMAKER_PROJECT_NAME: ${{ env.PROJECT_NAME }}
349+
SAGEMAKER_EXECUTION_ROLE_ARN: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
233350
run: |
234351
export SAGEMAKER_PROJECT_ID=`aws sagemaker describe-project --project-name $SAGEMAKER_PROJECT_NAME --query ProjectId --output text`
235352
echo "Project id: $SAGEMAKER_PROJECT_ID"
@@ -244,8 +361,9 @@ jobs:
244361
id: deploy-pipeline
245362
uses: aws-actions/aws-cloudformation-github-deploy@v1
246363
with:
247-
name: sagemaker-${{ steps.env-name.outputs.project_name }}-deploy-prod
364+
name: sagemaker-${{ env.PROJECT_NAME }}-deploy-prod
248365
template: ./deployment_pipeline/drift-deploy-prod.yml # Need to specify working-directory
366+
role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole
249367
no-fail-on-empty-changeset: "1"
250368

251369
- name: Upload template

‎ACTIONS.md

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -36,19 +36,28 @@ aws secretsmanager get-secret-value \
3636
--output text
3737
```
3838

39-
The GitHub Actions workflow has three stages: Model build, Deploy to staging, and Deploy to production. Each has their on [environment](https://docs.github.com/en/actions/reference/environments) which secrets and optional protection rules.
40-
1. `development` This is the environment in which runs your Model Build and starts the SageMaker pipeline execution, and on completion it will publish a model to the Registry. It is recommend you run this on `pull_request` and `push`.
41-
1. `staging` This second stage deploys your Staging endpoint. It is recommend you run this on commit to the `development` branch and configure a *protection rule* to continue after you have approved the model in the SageMaker Model Registry.
42-
1. `prod` This final stage deploys you Production Endpoint. It is recommend you this on commit the `main` branch with a *protection rule* to require approval after Staging endpoint has been tested.
39+
### Jobs
4340

44-
For each of the environments you will require setting up the following secrets.
45-
1. Create a secret named `AWS_REGION` defaulted to region `us-east-1`
46-
1. Create a secret named `AWS_ACCESS_KEY_ID` containing the `AccessKeyId` value returned above.
47-
1. Create a secret named `AWS_SECRET_ACCESS_KEY` containing in the `SecretAccessKey` value returned above.
48-
1. Create a secret named `AWS_SAGEMAKER_ROLE` containing the `SageMakerRoleArn` output in the setup stack.
41+
This sample includes a `Build and Deploy` [GitHub Actions](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) workflow with contains the following jobs
42+
1. The `Build Model` job will create or update a SageMaker Model Build Pipeline using AWS CloudFormation.
43+
2. The `Batch` jobs will create or update a SageMaker Pipeline to run Batch Scoring for Staging and Production environments.
44+
3. The `Deploy` jobs will deploy SageMaker Endpoints to Staging and Production environments.
4945

50-
If you configure *protection rules* for you environments, you will need to click **Review deployments** to approve the next stage as show below:
46+
These jobs are configured to run against a specific [environment](https://docs.github.com/en/actions/reference/environments) which contains both secrets and optional *protection rules*.
5147

5248
![Execution Role](docs/github-actions-workflow.png)
5349

54-
When the workflow successfully completes, you will have both Endpoints deployed in staging and production, with drift detection enable which will trigger re-training on drift.
50+
### Environments
51+
52+
1. `development` environment in which runs your `Build Model` job and starts the SageMaker pipeline execution. On completion this pipeline will publish a model to the Registry. It is recommend you run this on `pull_request` and `push` events.
53+
2. `staging` and `batch-staging` environments will enable you to run the `Batch` and `Deploy` jobs respectively in staging.
54+
* You should configure a *protection rule* for data science team so this job is delayed until the latest model has been approved in the SageMaker model registry.
55+
3. `prod` and `batch-prod` environments will enable you to run the `Batch` and `Deploy` jobs respectively in production.
56+
* You should configure a *protection rule* for your operations team which will approve this only once they are happy that the staging environment has been tested.
57+
58+
For each of the environments you will require setting up the following secrets.
59+
1. Create a secret named `AWS_ACCESS_KEY_ID` containing the `AccessKeyId` value returned above.
60+
1. Create a secret named `AWS_SECRET_ACCESS_KEY` containing in the `SecretAccessKey` value returned above.
61+
1. Create a secret named `AWS_SAGEMAKER_ROLE` containing the ARN for the `AmazonSageMakerServiceCatalogProductsUseRole` in your account.
62+
63+
When the workflow successfully completes, drift detection is configured to trigger re-training on drift detection in the production batch pipeline or real-time endpoint.

0 commit comments

Comments
 (0)