diff --git a/rls/README.md b/rls/README.md index d4e8956d..62ba9897 100644 --- a/rls/README.md +++ b/rls/README.md @@ -1,29 +1,16 @@ -# RLS generator for QuickSight +# QuickSight RLS -## About QS RLS generator -Generate RLS csv file for QuickSight based on AWS Organizational Units. - -[About QuickSight RLS](https://docs.aws.amazon.com/quicksight/latest/user/restrict-access-to-a-data-set-using-row-level-security.html) -[About AWS Organizational Unit ](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html) +This cloudformation will deploy Lambda code that will generate RLS rules based on OU tags. +Will create a Athena table based on the CSV rules in S3, create QuickSight DataSource and QuikcSight DataSet -## Getting Started +## RLS Generator and RLS dataset management +Generate RLS csv file for QuickSight based on AWS Organizational Units. -Code can be executed locally or as Lambda. [AWS Credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) are managed standard way. -To run the lambda define following `ENV_VARS` with following DEFAULTS if ENV_VAR is not set. - -[Using AWS Lambda environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) +[About QuickSight RLS](https://docs.aws.amazon.com/quicksight/latest/user/restrict-access-to-a-data-set-using-row-level-security.html) +[About AWS Organizational Unit ](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html) -List of Variables to preconfigure -``` -OWNER_TAG = 'cid_users' -BUCKET_NAME = 'NO DEFAULT' # Bucket where to upload the code -QS_REGION = 'QS region' -export MANAGEMENT_ACCOUNT_IDS='coma seaprated value of account_ids, format ACC_ID:REGION' -export MANAGMENTROLENAME=WA-Lambda-Assume-Role-Management-Account # Role to Assume in every payer/management account -TMP_RLS_FILE = '/tmp/cid_rls.csv' -``` ## Defining TAGS 1) Tags at root OU level, Give full access to all data and overwrite any other rules for user at other levels. @@ -32,93 +19,15 @@ TMP_RLS_FILE = '/tmp/cid_rls.csv' ## Output - -Output is writen to `TMP_RLS_FILE` location and uploaded to `BUCKET_NAME`. +Output is uploaded to `BUCKET_NAME`. ## Example Output - - -``` -UserName,account_id,payer_id -vmindru@megacorp.corp,, -vmindru_has_it_all,, -Admin/vmindru-Isengard,, -cross_ou_user,"0140000000,7200000,74700000,853000000", -foo_inherit,74700000000, -student1,"853000000,126000000", -student2,"853678200000,126600000", -other@company_foo.com,"363700000,1675000000", -other@company.com,"36370000000,16750000000", -vmindru@amazon.com,363000000000, -``` - - - -## Create Lambda - -### Create a new Lambda in same region with your QS Dashboards - -1) Create new Lambda -2) Select Python 3.8 - -### Configure Lambda - -1) Create and assign new Execution Role LambdaS3Org Role -2) Create and Add 2 Permission Policies to above LambdaS3Org Role - -`LambdaOrgS3ListTags` - -``` -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": [ - "organizations:ListAccountsForParent", - "organizations:ListAccounts", - "organizations:ListTagsForResource", - "organizations:ListOrganizationalUnitsForParent" - ], - "Resource": "*" - } - ] -} -``` - -`AWSLambdaS3ExecutionRole` - ``` -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": "s3:GetObject", - "Resource": "arn:aws:s3:::*" - }, - { - "Sid": "VisualEditor1", - "Effect": "Allow", - "Action": "s3:PutObject", - "Resource": "arn:aws:s3:::vmindru-cid-fr/cid_rls.csv" - } - ] -} +UserName,GroupName,account_id,payer_id +user_with_wildcard_access,,, +user_with_acces_of_all_acounts_of_single_payer,,000111222333 +cross_ou_user,,"0140000000,7200000,74700000,853000000", +,group1,"0140000000,7200000", +,group_accessa_all,, ``` - -### Add ENV Variables - -Go to function settings and add ENV VARS - -`BUCKET_NAME` - Bucket where to upload RLS file -`ROOT_OU` - ID of your root OU - -### Increase execution time to 120s - - - - diff --git a/rls/deploy/deploy_cid_rls.yaml b/rls/deploy/deploy_cid_rls.yaml index 6c40e075..6d2658f7 100644 --- a/rls/deploy/deploy_cid_rls.yaml +++ b/rls/deploy/deploy_cid_rls.yaml @@ -1,11 +1,64 @@ #https://github.com/awslabs/cid-data-collection-framework/blob/main/rls/deploy/deploy_cid_rls.yaml AWSTemplateFormatVersion: '2010-09-09' Description: Lambda to collect AWS Organization and Amazon QuickSight data and store in S3 for RLS implementation v0.2.0 - AWS Solution SO9011 +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: + default: "RLS common configuration parameters" + Parameters: + - ManagementAccountID + - DestinationBucket + - QuickSightUser + - Label: + default: "Default RLS System Configuration. Do not change if not needed" + Parameters: + - DeployLambdaResources + - ManagementAccountRole + - ResourcePrefix + - DatabaseName + - AthenaWorkGroup + - CodeKey + - CidRlsLocation + - CodeBucket + - Schedule + + ParameterLabels: + Parameters: + ManagementAccountRole: + Default: Lambda-Assume-Role-Management-Account + ResourcePrefix: + Default: CID-DC- + DatabaseName: + Default: "cid_cur" + AthenaWorkGroup: + Default: "CID" + Schedule: + Default: "rate(1 hour)" + CodeKey: + Default: cfn/rls/create_rls.zip # RLS Folder to be updated, once the LAB will be created + CidRlsLocation: + Default: "cid_rls" + CodeBucket: + Default: "aws-managed-cost-intelligence-dashboards" + DeployLambdaResources: + Default: Yes Parameters: + DeployLambdaResources: + Type: String + Description: "Yes: Deploy RLS Lambdaor. No: only create RLS Athena Table and DataSet" + AllowedValues: + - Yes + - No + Default: Yes DestinationBucket: Type: String Description: Name of the S3 Bucket that is created to hold org data AllowedPattern: (?=^.{3,63}$)(?!^(\d+\.)+\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$) + QuickSightUser: + Type: String + Description: QuickSight user that will own DataSet and DataSource resources, use the same as in CID deployment. + MinLength: 1 ManagementAccountRole: Type: String Description: The name of the IAM role that will be deployed in the management account which can retrieve AWS Organization data. KEEP THE SAME AS WHAT IS DEPLOYED INTO MANAGEMENT ACCOUNT @@ -18,34 +71,143 @@ Parameters: Type: String AllowedPattern: ([a-z0-9\-, ]*?$) Description: "(Ex: 123456789,098654321,789054312) List of Payer IDs you wish to collect data for. Can just be one Accounts" + DatabaseName: + Type: String + Description: Existing Athena Database where to deploy RLS table, should be same as CID database + Default: "cid_cur" + AthenaWorkGroup: + Type: String + Description: Existing Athena Workgroup, keep the same as for CID Dashboards. + Default: "CID" Schedule: Type: String Description: Cron job to trigger the lambda using cloudwatch event Default: "rate(1 hour)" CodeBucket: - Type: String - Description: S3 Bucket with RLS code,this coverts to CodeBucket-Region e.g. for us-east-1 this will be aws-managed-cost-intelligence-dashboards-us-east-1 - Default: aws-managed-cost-intelligence-dashboards + Type: String + Description: S3 Bucket with RLS code,this coverts to CodeBucket-Region e.g. for us-east-1 this will be aws-managed-cost-intelligence-dashboards-us-east-1 + Default: "aws-managed-cost-intelligence-dashboards" CodeKey: - Type: String - Description: file name of ZipFile with data code - Default: cfn/rls/create_rls.zip # RLS Folder to be updated, once the LAB will be created + Type: String + Description: file name of ZipFile with data code + Default: cfn/rls/create_rls.zip # RLS Folder to be updated, once the LAB will be created + CidRlsLocation: + Type: String + Description: location of the RLS files in the S3 bucket + Default: "cid_rls" Outputs: LambdaFunctionName: + Condition: DeployLambda Value: - Ref: CIDRLS + Ref: LambdaCIDRLS LambdaFunctionARN: + Condition: DeployLambda Description: Lambda function ARN. Value: Fn::GetAtt: - - CIDRLS + - LambdaCIDRLS - Arn +Conditions: + DeployLambda: !Equals [ !Ref DeployLambdaResources, Yes ] Resources: - CIDRLS: + CidRlsTable: + Type: AWS::Glue::Table + Properties: + DatabaseName: !Ref DatabaseName + CatalogId: !Ref AWS::AccountId + TableInput: + Name: cid_rls + TableType: EXTERNAL_TABLE + StorageDescriptor: + Columns: + - Name: UserName + Type: string + - Name: GroupName + Type: string + - Name: account_id + Type: string + - Name: payer_account_id + Type: string + Location: !Sub "s3://${DestinationBucket}/${CidRlsLocation}/" + InputFormat: org.apache.hadoop.mapred.TextInputFormat + OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + SerdeInfo: + SerializationLibrary: org.apache.hadoop.hive.serde2.OpenCSVSerde + Parameters: + escapeChar: \ + quoteChar: '"' + separatorChar: ',' + skip.header.line.count: 1 + CidRlsDataSource: + Type: AWS::QuickSight::DataSource + Properties: + AwsAccountId: !Ref AWS::AccountId + DataSourceId: cid-rls-datasource + Name: cid-rls-datasource + Type: ATHENA + DataSourceParameters: + AthenaParameters: + WorkGroup: !Ref AthenaWorkGroup + Permissions: + - Principal: !Sub 'arn:${AWS::Partition}:quicksight:${AWS::Region}:${AWS::AccountId}:user/default/${QuickSightUser}' + Actions: + - quicksight:PassDataSource + - quicksight:DescribeDataSourcePermissions + - quicksight:UpdateDataSource + - quicksight:UpdateDataSourcePermissions + - quicksight:DescribeDataSource + - quicksight:DeleteDataSource + + CidRlsDataSet: + Type: AWS::QuickSight::DataSet + DependsOn: CidRlsDataSource + Properties: + AwsAccountId: !Ref AWS::AccountId + DataSetId: cid_rls + Name: cid_rls + ImportMode: DIRECT_QUERY + UseAs: RLS_RULES + LogicalTableMap: + A-Map-ID: + Alias: cid_rls + Source: + PhysicalTableId: RlsAthenaTable + PhysicalTableMap: + RlsAthenaTable: + RelationalTable: + Catalog: AwsDataCatalog + DataSourceArn: !GetAtt CidRlsDataSource.Arn + Name: cid_rls + Schema: default + InputColumns: + - Name: username + Type: STRING + - Name: groupname + Type: STRING + - Name: account_id + Type: STRING + - Name: payer_account_id + Type: STRING + Permissions: + - Principal: !Sub 'arn:${AWS::Partition}:quicksight:${AWS::Region}:${AWS::AccountId}:user/default/${QuickSightUser}' + Actions: + - quicksight:PassDataSet + - quicksight:DescribeIngestion + - quicksight:CreateIngestion + - quicksight:UpdateDataSet + - quicksight:DeleteDataSet + - quicksight:DescribeDataSet + - quicksight:CancelIngestion + - quicksight:DescribeDataSetPermissions + - quicksight:ListIngestions + - quicksight:UpdateDataSetPermissions + + LambdaCIDRLS: Type: AWS::Lambda::Function + Condition: DeployLambda Properties: FunctionName: !Sub - - 'CIDRLS_${Id}' + - 'LambdaCIDRLS_${Id}' - Id: !Select [0, !Split ['-', !Ref 'AWS::StackName']] Description: LambdaFunction of python3.8. Runtime: python3.9 @@ -64,6 +226,7 @@ Resources: QS_REGION: !Ref AWS::Region LambdaRole: Type: AWS::IAM::Role + Condition: DeployLambda Properties: RoleName: !Sub "${ResourcePrefix}RLS-LambdaRole" AssumeRolePolicyDocument: @@ -96,7 +259,7 @@ Resources: - "logs:CreateLogStream" - "logs:PutLogEvents" - "logs:DescribeLogStreams" - Resource: !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/CID-RLS*" + Resource: !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/Lambda-CID-RLS*" - Effect: "Allow" Action: - "s3:PutObject" @@ -110,20 +273,22 @@ Resources: Resource: "*" # Cannot restrict this CloudWatchTrigger: Type: AWS::Events::Rule + Condition: DeployLambda Properties: Description: Scheduler Name: !Sub - - 'Scheduler_ForCIDRLS_${Id}' + - 'Scheduler_ForLambdaCIDRLS_${Id}' - Id: !Select [0, !Split ['-', !Ref 'AWS::StackName']] ScheduleExpression: !Ref Schedule State: ENABLED Targets: - - Arn: !GetAtt CIDRLS.Arn - Id: TriggerForCIDRLS + - Arn: !GetAtt LambdaCIDRLS.Arn + Id: TriggerForLambdaCIDRLS EventPermission: + Condition: DeployLambda Type: AWS::Lambda::Permission Properties: - FunctionName: !GetAtt CIDRLS.Arn + FunctionName: !GetAtt LambdaCIDRLS.Arn Action: lambda:InvokeFunction Principal: events.amazonaws.com SourceAccount: !Ref 'AWS::AccountId'