this portfolio will contain the coorporate Sagemaker Projects products to be used by the ML teams to accelerate their ML models development while compliying with the organization´s best practices.
- We assume a role with required permissions and appropiate access has been set up to access the ML Shared Services Infra account from the AWS CLI. More information on this can be found on Configure the AWS CLI to use AWS IAM Identity Center and Use an IAM Role in the AWS CLI
For cdk deployment you will need the "target account id" and "target region" where you want to deploy the portoflios, we recommend this to be your ML Shared Services Account.
To find the "target account id", click the account information on the top right corner of the console page. The 12-digit number after "Account ID" is the target account id.
On the left side of the account id, click the region name, and copy the region, which should have the format of --, such as "us-east-1".
Use the following command to find "target account profile". For example, mine is called "default".
aws configure list-profiles
You can also get temporary programatic access by going into the Iam Identity Center sign in portal, looking for the account where you need AWS CLI access, click on the Command line or programmatic access link and follow the instructions.
For more information, read AWS CDK: Bootstrapping.
The AWS Cloud Development Kit (CDK) is written in python. Below is a list of packages requierd to deploy the code in this workshop. It is preferred to use a linux OS to be able to run all cli commands and avoid path issues.
We will create the GitHub repository that will host the CDK code for the Organization SageMaker Projects' templates and the AWS CodePipeline pipeline that will convert this code into Service Catalog products to be shared with Sandbox accounts.
In this step, we will bootstrap the infrastructure for the Sagemaker Projects portfolio in the ML Shared Services account.
Clone the code Git Repository to a local directory.
Clone the code Git Repository to a local directory.
git clone https://github.com/aws-samples/data-and-ml-governance-workshop.gitChange directory to ml-platform-shared-services/module-3/sagemaker-projects-portfolio directory.
cd data-and-ml-governance-workshop/module-3/sagemaker-projects-portfolioInstall dependencies in a separate python environment using your favourite python packages manager.
python3 -m venv env
source env/bin/activate
pip install -r requirements.txtBootstrap your deployment target account using the following command:
cdk bootstrap aws://<target account id>/<target region> --profile <target account profile>or if you make sure that you have the right role: aws sts get-caller-identity and the right region aws configure get region simply run:
cdk bootstrapNow we are going to set up the required resources in our ML Shared Services Account. For that follow this steps:
Deploy the stack with the CodeConnection and the corresponding pipeline.
cdk deploy --all --require-approval neverThis may take a few minutes. Once it's finished, you should see the message containing the ARN of the deployed stack.
Let's check the stack deployed.
First, navigate to the AWS CloudFormation console.
Then click "Stacks" on the CloudFormation page.
You should see a stack named "SmProjectsServiceCatalogPipeline". This is the stack that created resources such as CodeConnections Connection, CodePipeline, S3 buckets, and etc.
Let's check out the resources created. Take the CodeConnection connection as an example.
Type "Service Catalog" in the search bar, and then click "Service Catalog" in the dropdown menu. Then select "AWS CodeStar Connections" from the left sidebar.
You can see there's a connection named "codeconnection-service-catalog". If you click the connection, you will notice that we need to connect it to our GitHub to allow us to integrate it with our pipelines and start pushing code. Click the 'Update pending connection' to integrate with your GitHub account.
Once that is done, you need to create empty GitHub repositories to start pushing code to. For example, you can create a repository called "sm-projects-service-catalog-repo". Every project you deploy will need a repository created in GitHub beforehand.
We recommend to create a separate folder for the differnt repositories that will be created in the platform. To do that, get out of the cloned repository and create a parallel folder called platform-repositories
cd ../../.. # (as many .. as directories you have moved in)
mkdir platform-repositoriesLet´s clone and fill the empty created repository
cd platform-repositories
git clone https://github.com/example-org/sm-projects-service-catalog-repo.git
cd sm-projects-service-catalog-repo
cp -aR ../../ml-platform-shared-services/module-3/sagemaker-projects-portfolio/. .Let's push the code to the GitHub Repository to create the Service Catalog portfolio. Run the code below.
git add .
git commit -m "Initial commit"
git push -u origin mainOnce it is pushed, let's go back to the GitHub repository we created earlier. Now it's no longer empty. Once the code is pushed to the code repository, it triggers the CodePipeline run to build and deploy artifacts to the Service Catalog. Click Pipelines -> Pipeline to check it out. You will see a pipeline named "cdk-service-catalog-pipeline". Click on the pipeline name to check out the steps of it. For more information, read AWS CodePipeline.
It takes about 10 minutes for the pipeline to finish running. Once it's finished, let's check out the Service Catalog Portfolios.
Type "Service Catalog" in the search bar and click on "Service Catalog"
On the Service Catalog page, click "Portfolio" under "Administration". You will see a portfolio named "SM Projects Portfolio".
A product is a set of AWS cloud resources that you want to make available for deployment on AWS. Click on one of the products, and then click on the version name, you can see what's inside the product is mainly a CloudFormation template, which allows you to deploy infrastructure as code. For more information about CloudFormation templates, read AWS CloudFormation.
Now that we have set up and configure the ML Shared Services Account and the portfolios lets find out how we can make this products available to all the ML accounts in our next section
- CDK Version:
❌ Error: This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version. (Cloud assembly schema version mismatch: Maximum schema version supported is 34.0.0, but found 35.0.0)
This error happens when the CDK Cli version and the Virtual Environment aws-cdk-lib package version is not the same.
To check both of them run cdk --version for the CDK Cli and pip list for the aws-cdk-lib python package.
How to solve?: You can either modify the cdk cli version or the pip version.
- CodeBuild concurrent runs
❌ Error:
Error calling startBuild: Cannot have more than 1 builds in queue for the account (Service: AWSCodeBuild; Status Code: 400; Error Code: AccountLimitExceededException; Request ID: xxxxx; Proxy: null)This error happens because the given quota for our CodeBuild Environments is lower than the one required for the concurrent build of the Service Catalog Portfolio Products.
How to solve?: See the following Article and request a quota increase as specified in: Requesting a quota increase
