Add QuickSight data collection in Data Collection Account (#212)

Co-authored-by: SohamMajumder <[email protected]>

Author: iakov-aws

data-collection/README.md

@@ -9,7 +9,7 @@ This projects demonstrates usage of AWS API for collecting various types of usag
 ![Architecture](/data-collection/images/archi.png)
 
 1. Amazon EventBridge rule invokes Step Function of every every deployed data collection module. based on schedule.
-2. The Step Function launches a Lambda function Account Collector that assumes Read Role role in the Management account and retrieves linked accounts list via AWS Organizations API
+2. The Step Function launches a Lambda function Account Collector that assumes Read Role role in the Management accounts and retrieves linked accounts list via AWS Organizations API
 3. Step Functions launches Data Collection Lambda function for each collected Account.
 4. Each data collection module Lambda function assumes IAM role in linked accounts and retrieves respective optimization data via AWS SDK for Python. Retrieved data aggregated in Amazon S3 bucket
 5. Once data stored in S3 bucket, Step Functions triggers AWS Glue crawler which creates or updates the table in Glue Data Catalog
@@ -20,33 +20,34 @@ This projects demonstrates usage of AWS API for collecting various types of usag
 List of modules and objects collected:
 | Module Name                  | AWS Services          | Collected In        | Details  |
 | ---                          |  ---                  | ---                 | ---      |
-| `organization`               | AWS Organizations     | Management Account  |          |
-| `budgets`                    | AWS Budgest           | Linked Account      |          |
-| `compute-optimizer`          | AWS Compute Optimizer | Management Account  | Requires [Enablement of Compute Optimizer](https://aws.amazon.com/compute-optimizer/getting-started/#:~:text=Opt%20in%20for%20Compute%20Optimizer,created%20automatically%20in%20your%20account.) |
-| `trusted-advisor`            | AWS Trusted Advisor   | Linked Account      | Requires Enterpriso or OnRamp Support Level |
-| `support-cases`              | AWS Support           | Linked Account      | Requires Business, Enterprise On-Ramp, or Enterprise Support plan |
-| `cost-explorer-cost-anomaly` | AWS Anomalies         | Management Account  |          |
-| `cost-explorer-rightsizing`  | AWS Cost Explorer     | Management Account  | DEPRECATED. Please use `Data Exports` for `Cost Optimization Hub` |
-| `inventory`                  | Various services      | Linked Account      | Collects `Amazon OpenSearch Domains`, `Amazon ElastiCache Clusters`, `RDS DB Instances`, `EBS Volumes`, `AMI`, `EC2 Instances`, `EBS Snapshot`, `RDS Snapshot`, `Lambda`, `RDS DB Clusters`, `EKS Clusters` |
-| `pricing`                    | Various services      | N/A                 | Collects pricing for `Amazon RDS`, `Amazon EC2`, `Amazon ElastiCache`, `AWS Lambda`, `Amazon OpenSearch`, `AWS Compute Savings Plan` |
-| `rds-usage`                  |  Amazon RDS           | Linked Account      | Collects CloudWatch metrics for chargeback |
-| `transit-gateway`            |  AWS Transit Gateway  | Linked Account      | Collects CloudWatch metrics for chargeback |
-| `ecs-chargeback`             |  Amazon ECS           | Linked Account      |  |
-| `backup`                     |  AWS Backup           | Management Account  | Collects Backup Restore and Copy Jobs. Requires [activation of cross-account](https://docs.aws.amazon.com/aws-backup/latest/devguide/manage-cross-account.html#enable-cross-account) |
-| `health-events`      |  AWS Health | Management Accounts  | Collect AWS Health notificaitons via AWS Organizational view  |
-| `licence-manager`      |  AWS License Manager | Management Accounts  | Collect Licences and Grants |
-
+| `organization`               | AWS Organizations     | Management Accounts  |          |
+| `budgets`                    | AWS Budgest           | Linked Accounts      |          |
+| `compute-optimizer`          | AWS Compute Optimizer | Management Accounts  | Requires [Enablement of Compute Optimizer](https://aws.amazon.com/compute-optimizer/getting-started/#:~:text=Opt%20in%20for%20Compute%20Optimizer,created%20automatically%20in%20your%20account.) |
+| `trusted-advisor`            | AWS Trusted Advisor   | Linked Accounts      | Requires Enterpriso or OnRamp Support Level |
+| `support-cases`              | AWS Support           | Linked Accounts      | Requires Business, Enterprise On-Ramp, or Enterprise Support plan |
+| `cost-explorer-cost-anomaly` | AWS Anomalies         | Management Accounts  |          |
+| `cost-explorer-rightsizing`  | AWS Cost Explorer     | Management Accounts  | DEPRECATED. Please use `Data Exports` for `Cost Optimization Hub` |
+| `inventory`                  | Various services      | Linked Accounts      | Collects `Amazon OpenSearch Domains`, `Amazon ElastiCache Clusters`, `RDS DB Instances`, `EBS Volumes`, `AMI`, `EC2 Instances`, `EBS Snapshot`, `RDS Snapshot`, `Lambda`, `RDS DB Clusters`, `EKS Clusters` |
+| `pricing`                    | Various services      | Data Collection Account | Collects pricing for `Amazon RDS`, `Amazon EC2`, `Amazon ElastiCache`, `AWS Lambda`, `Amazon OpenSearch`, `AWS Compute Savings Plan` |
+| `rds-usage`                  |  Amazon RDS           | Linked Accounts      | Collects CloudWatch metrics for chargeback |
+| `transit-gateway`            |  AWS Transit Gateway  | Linked Accounts      | Collects CloudWatch metrics for chargeback |
+| `ecs-chargeback`             |  Amazon ECS           | Linked Accounts      |  |
+| `backup`                     |  AWS Backup           | Management Accounts  | Collects Backup Restore and Copy Jobs. Requires [activation of cross-account](https://docs.aws.amazon.com/aws-backup/latest/devguide/manage-cross-account.html#enable-cross-account) |
+| `health-events`              |  AWS Health | Management Accounts  | Collect AWS Health notificaitons via AWS Organizational view  |
+| `licence-manager`            |  AWS License Manager  | Management Accounts  | Collect Licences and Grants |
+| `aws-feeds`                  |  N/A                  | Data Collection Account |Collects Blog posts and News Feeds|
+| `quicksight`                 |  Amazon QuickSight    | Data Collection Account |Collects Quicksight User and Group information in the Data Collection Account only|
 
 
 ### Installation
 
 #### 1. In Management Account(s)
 
-The Management Account stack makes use of [stack sets](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/what-is-cfnstacksets.html) configured to use [service-managed permissions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-concepts.html#stacksets-concepts-stackset-permission-models) to deploy stack instances to linked accounts in the AWS Organization.
+The Management Accounts stack makes use of [stack sets](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/what-is-cfnstacksets.html) configured to use [service-managed permissions](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-concepts.html#stacksets-concepts-stackset-permission-models) to deploy stack instances to linked accounts in the AWS Organization.
 
-Before creating the Management Account stack, please make sure [trusted access with AWS Organizations](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-activate-trusted-access.html) is activated.
+Before creating the Management Accounts stack, please make sure [trusted access with AWS Organizations](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-orgs-activate-trusted-access.html) is activated.
 
-The Management Account Stack creates a read role in the Management Account and also a StackSet that will deploy another read role in each linked Account. Permissions depend on the set of modules you activate via parameters of the stack:
+The Management Accounts Stack creates a read role in the Management Accounts and also a StackSet that will deploy another read role in each linked Account. Permissions depend on the set of modules you activate via parameters of the stack:
 
    *  <kbd> <br> [Launch Stack >>](https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?&templateURL=https://aws-managed-cost-intelligence-dashboards-us-east-1.s3.amazonaws.com/cfn/data-collection/deploy-data-read-permissions.yaml&stackName=CidDataCollectionDataReadPermissionsStack&param_DataCollectionAccountID=REPLACE%20WITH%20DATA%20COLLECTION%20ACCOUNT%20ID&param_AllowModuleReadInMgmt=yes&param_OrganizationalUnitID=REPLACE%20WITH%20ORGANIZATIONAL%20UNIT%20ID&param_IncludeBudgetsModule=no&param_IncludeComputeOptimizerModule=no&param_IncludeCostAnomalyModule=no&param_IncludeECSChargebackModule=no&param_IncludeInventoryCollectorModule=no&param_IncludeRDSUtilizationModule=no&param_IncludeRightsizingModule=no&param_IncludeTAModule=no&param_IncludeTransitGatewayModule=no) <br> </kbd>
 

data-collection/deploy/deploy-data-collection.yaml

@@ -34,6 +34,7 @@ Metadata:
           - IncludeTransitGatewayModule
           - IncludeAWSFeedsModule
           - IncludeLicenseManagerModule
+          - IncludeQuickSightModule
     ParameterLabels:
       DestinationBucket:
         default: 'Destination S3 bucket prefix'
@@ -85,6 +86,8 @@ Metadata:
         default: 'Include AWS Health Events Module'
       IncludeLicenseManagerModule:
         default: 'Include Marketplace Licensing Collection'
+      IncludeQuickSightModule:
+        default: 'Include QuickSight User Collection Module'
 
 Mappings:
   RegionMap:
@@ -108,7 +111,7 @@ Mappings:
   StepFunctionCode:
     main-v2:        {TemplatePath: cfn/data-collection/source/step-functions/main-state-machine-v2.json}
     crawler-v1:     {TemplatePath: cfn/data-collection/source/step-functions/crawler-state-machine-v1.json}
-    awsfeeds-v1:    {TemplatePath: cfn/data-collection/source/step-functions/awsfeeds-state-machine-v1.json}
+    standalone-v1:  {TemplatePath: cfn/data-collection/source/step-functions/awsfeeds-state-machine-v1.json}
 
 Parameters:
   DestinationBucket:
@@ -229,6 +232,11 @@ Parameters:
     Description: Collects Marketplace Licenses and Grants
     AllowedValues: ['yes', 'no']
     Default: 'no'
+  IncludeQuickSightModule:
+    Type: String
+    Description: Collects Marketplace Licenses and Grants
+    AllowedValues: ['yes', 'no']
+    Default: 'no'
 
 Conditions:
   DeployTAModule: !Equals [ !Ref IncludeTAModule, "yes"]
@@ -246,6 +254,7 @@ Conditions:
   DeployAWSFeedsModule: !Equals [ !Ref IncludeAWSFeedsModule, "yes"]
   DeployHealthEventsModule: !Equals [ !Ref IncludeHealthEventsModule, "yes"]
   DeployLicenseManagerModule: !Equals [ !Ref IncludeLicenseManagerModule, "yes"]
+  DeployQuickSightModule: !Equals [ !Ref IncludeQuickSightModule, "yes"]
   DeployPricingModule: !Or
     - !Condition DeployInventoryCollectorModule
     - !Condition DeployRDSUtilizationModule
@@ -266,6 +275,7 @@ Conditions:
       - !Condition DeployTransitGatewayModule
       - !Condition DeployHealthEventsModule
       - !Condition DeployLicenseManagerModule
+      - !Condition DeployQuickSightModule
   RegionsInScopeIsEmpty: !Equals
     - !Join [ '', !Split [ ' ', !Ref RegionsInScope  ] ] # remove spaces
     - ""
@@ -1169,7 +1179,7 @@ Resources:
         ResourcePrefix: !Ref ResourcePrefix
         LambdaAnalyticsARN: !GetAtt LambdaAnalytics.Arn
         CodeBucket: !If [ ProdCFNTemplateUsed, !FindInMap [RegionMap, !Ref "AWS::Region", CodeBucket], !Ref CFNSourceBucket ]
-        StepFunctionTemplate: !FindInMap [StepFunctionCode, awsfeeds-v1, TemplatePath]
+        StepFunctionTemplate: !FindInMap [StepFunctionCode, standalone-v1, TemplatePath]
         StepFunctionExecutionRoleARN: !GetAtt StepFunctionExecutionRole.Arn
         SchedulerExecutionRoleARN: !GetAtt SchedulerExecutionRole.Arn
 
@@ -1213,6 +1223,24 @@ Resources:
         StepFunctionExecutionRoleARN: !GetAtt StepFunctionExecutionRole.Arn
         SchedulerExecutionRoleARN: !GetAtt SchedulerExecutionRole.Arn
 
+  QuickSightModule:
+    Type: AWS::CloudFormation::Stack
+    Condition: DeployQuickSightModule
+    Properties:
+      TemplateURL: !Sub "https://${CFNSourceBucket}.s3.amazonaws.com/cfn/data-collection/module-quicksight.yaml"
+      Parameters:
+        DatabaseName: !Ref DatabaseName
+        DestinationBucket: !Ref S3Bucket
+        DestinationBucketARN: !GetAtt S3Bucket.Arn
+        Schedule: !Ref ScheduleFrequent
+        GlueRoleARN: !GetAtt GlueRole.Arn
+        ResourcePrefix: !Ref ResourcePrefix
+        LambdaAnalyticsARN: !GetAtt LambdaAnalytics.Arn
+        CodeBucket: !If [ ProdCFNTemplateUsed, !FindInMap [RegionMap, !Ref "AWS::Region", CodeBucket], !Ref CFNSourceBucket ]
+        StepFunctionTemplate: !FindInMap [StepFunctionCode, standalone-v1, TemplatePath]
+        StepFunctionExecutionRoleARN: !GetAtt StepFunctionExecutionRole.Arn
+        SchedulerExecutionRoleARN: !GetAtt SchedulerExecutionRole.Arn
+
   AccountCollector:
     Type: AWS::CloudFormation::Stack
     Condition: DeployAccountCollector

data-collection/deploy/deploy-in-linked-account.yaml

@@ -40,6 +40,7 @@ Metadata:
         default: 'Include Budgets Collection Module'
       IncludeTransitGatewayModule:
         default: 'Include Transit Gateway Module'
+
 Parameters:
   DataCollectionAccountID:
     Type: String
@@ -301,4 +302,4 @@ Resources:
       cfn_nag:
         rules_to_suppress:
           - id: W12
-            reason: "Policy is used for scanning of a wide range of resources"
+            reason: "Policy is used for scanning of a wide range of resources"

data-collection/deploy/deploy-in-management-account.yaml

@@ -352,4 +352,4 @@ Resources:
       cfn_nag:
         rules_to_suppress:
           - id: W12
-            reason: "Policy is used for scanning of a wide range of resources"
+            reason: "Policy is used for scanning of a wide range of resources"