Skip to content

Commit 8a52ff2

Browse files
iakov-awsiakov-aws
authored
Enhance readme (#322)
1 parent f0d1a92 commit 8a52ff2

9 files changed

+225
-29
lines changed

‎data-collection/images/compute-optimizer-data-collection.png‎ renamed to ‎.images/architecture-data-collection-compute-optimizer.png‎

File renamed without changes.

‎data-collection/images/archi.png‎ renamed to ‎.images/architecture-data-collection-detailed.png‎

File renamed without changes.

‎.images/architecture-data-collection.png‎

560 KB
Loading

‎.images/architecture-data-exports-advanced.png‎

765 KB
Loading

‎.images/architecture-data-exports.png‎

401 KB
Loading

‎.images/architecture-legacy-cur.png‎

284 KB
Loading

‎README.md‎

Lines changed: 135 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,147 @@
1-
## Cloud Intelligence Dashboard Framework
2-
[Cloud Intelligence Dashboards](https://catalog.workshops.aws/awscid) is a project that provides AWS customers with a series of in-depth and customizable dashboards for the most comprehensive cost and usage details to help optimize cost, track usage goals, and achieve operational excellence.
1+
## CID Data Collection Framework
32

4-
This repostory contains following elements:
5-
* [data-collection](/data-collection) - a set of Cloud Formation Templates for collecting data from Management and Linked Accounts.
6-
* [case-summarization](/case-summarization) - a Cloud Formation Template for deploying the AWS Support Case Summarization plugin that offers the capability to summarize cases through Generative AI powered by Amazon Bedrock.
3+
## Table of Contents
4+
1. [Overview](#Overview)
5+
1. [Architecture of Data Exports](#Architecture-of-Data-Exports)
6+
1. [Architecture of Data Collection](#Architecture-of-Data-Collection)
7+
1. [Cost](#Cost)
8+
1. [Prerequisites](#Prerequisites)
9+
1. [Regions](#Regions)
10+
1. [Deployment and Cleanup Steps](#Deployment-and-Cleanup-Steps)
11+
1. [Changelogs](#Changelogs)
12+
1. [Feedback](#Feedback)
13+
1. [Security](#Security)
14+
1. [License](#License)
15+
1. [Notices](#Notices)
716

8-
## Support and Contribution
17+
## Overview
18+
This repository is a part of [Cloud Intelligence Dashboards](https://catalog.workshops.aws/awscid), a project that provides AWS customers with a series of in-depth and customizable dashboards for the most comprehensive cost and usage details to help optimize cost, track usage goals, and achieve operational excellence.
919

20+
This repository contains following elements:
21+
* [data-exports](/data-exports) - a Cloud Formation Templates for AWS Data Exports, such as Cost and Usage Report 2.0 and others. This allows a replication of Exports from your Management Account(s) to a Dedicated Data Collection Accounts as well as aggregation of multiple Exports from a set of Linked Accounts.
22+
* [data-collection](/data-collection) - a set of Cloud Formation Templates for collecting infrastructure operational data from Management and Linked Accounts. Such as data from AWS Trusted Advisor, AWS Compute Optimizer, Inventories, Pricing, AWS Health, AWS Support Cases etc. See more about types of data collected [here](/data-collection).
23+
* [case-summarization](/case-summarization) - an additional Cloud Formation Template for deploying the AWS Support Case Summarization plugin that offers the capability to summarize cases through Generative AI powered by Amazon Bedrock.
24+
* [rls](/rls) - a stack for managing Row Level Security for CID Dashboards.
25+
* [security-hub](/security-hub) - Collection of data from AWS Security Hub.
26+
27+
All Data Collections can be used independently from Dashboards. Typically data collections store data on [Amazon S3 Bucket](https://aws.amazon.com/s3/) and provide [AWS Glue](https://aws.amazon.com/glue/) tables and [Amazon Athena](https://aws.amazon.com/athena/) Views to explore and use these data.
28+
29+
### Other AWS Services
30+
* [Collection of AWS Config data](https://github.com/aws-samples/config-resource-compliance-dashboard)
31+
32+
### Multi-cloud data
33+
* [Collection of Azure Cost Data](https://github.com/aws-samples/aws-data-pipelines-for-azure-storage/)
34+
* [Collection of GCP Cost Data](https://github.com/awslabs/cid-gcp-cost-dashboard/)
35+
* [Collection of OCI Cost Data](https://github.com/awslabs/cid-oci-cost-dashboard/)
36+
37+
## Architecture of Data Exports
38+
![Architecture of Data Exports](.images/architecture-data-exports.png "Architecture of Data Exports")
39+
1. [AWS Data Exports](https://aws.amazon.com/aws-cost-management/aws-data-exports/) delivers daily the Cost & Usage Report (CUR2) to an [Amazon S3 Bucket](https://aws.amazon.com/s3/) in the Management Account.
40+
2. [Amazon S3](https://aws.amazon.com/s3/) replication rule copies Export data to a dedicated Data Collection Account S3 bucket automatically.
41+
3. [Amazon Athena](https://aws.amazon.com/athena/) allows querying data directly from the S3 bucket using an [AWS Glue](https://aws.amazon.com/glue/) table schema definition.
42+
4. [Amazon QuickSight](https://aws.amazon.com/quicksight/) datasets can read from [Amazon Athena](https://aws.amazon.com/athena/). Check Cloud Intelligence Dashboards.
43+
44+
See more in [data-exports](/data-exports).
45+
46+
47+
48+
## Architecture of Data Collection
49+
![Architecture of Advanced Data Collection](.images/architecture-data-collection.png "Architecture of Advanced Data Collection")
50+
1. The Advanced Data Collection can be deployed to enable advanced dashboards based on [AWS Trusted Advisor](https://aws.amazon.com/trustedadvisor/), [AWS Health Events](https://docs.aws.amazon.com/health/latest/ug/getting-started-phd.html) and other sources. Additional data is retrieved from [AWS Organization](https://aws.amazon.com/organizations/) or Linked Accounts. In this case [Amazon EventBridge](https://aws.amazon.com/eventbridge/) rule triggers an [AWS Step Functions](https://aws.amazon.com/step-functions/) for data collection modules on a configurable schedule.
51+
52+
2. The "Account Collector" [AWS Lambda](https://aws.amazon.com/lambda/) in AWS Step Functions retrieves linked account details using [AWS Organizations API](https://docs.aws.amazon.com/organizations/latest/APIReference/Welcome.html).
53+
54+
3. The "Data Collection" Lambda function in AWS Step Functions assumes role in each linked account to retrieve account-specific data via [AWS SDK](https://aws.amazon.com/sdk-for-python/).
55+
56+
4. Retrieved data is stored in a centralized [Amazon S3 Bucket](https://aws.amazon.com/s3/).
57+
58+
5. Advanced Cloud Intelligence Dashboards leverage [Amazon Athena](https://aws.amazon.com/athena/) and [Amazon QuickSight](https://aws.amazon.com/quicksight/) for comprehensive data analysis.
59+
60+
See more in [data-collection](/data-collection).
61+
62+
63+
64+
65+
## Cost
66+
The following table provides a sample cost breakdown for deploying of Foundational Dashboards with the default parameters in the US East (N. Virginia) Region for one month.
67+
68+
| AWS Service | Dimensions | Monthly Cost [USD] |
69+
|---------------------------------|-------------------------------|--------------------|
70+
| S3 | Monthly storage | $5-10* |
71+
| AWS Lambda | On the schedule 1/14 days | $<3* |
72+
| AWS Step Functions | On the schedule 1/14 days | $<3* |
73+
| AWS Glue Crawler | On schedule | $<3* |
74+
| AWS Athena | Data scanned monthly | $15* |
75+
| **Total Estimated Monthly Cost** | | **<$50** |
76+
77+
\* Costs are relative to the size of collected data (number of workloads, modules activated, AWS Accounts, Regions etc) and configured data collection frequency.
78+
79+
Pleas use AWS Pricing Calculator for precise estimation.
80+
81+
## Prerequisites
82+
You need access to AWS Accounts. We recommend deployment of the Data Collection in a dedicated Data Collection Account, other than your Management (Payer) Account. You can use it to aggregate data from multiple Management (Payer) Accounts or multiple Linked Accounts.
83+
84+
If you do not have access to the Management/Payer Account, you can still collect some types fo data across multiple Linked accounts.
85+
86+
## Regions
87+
Make sure you are installing data collection in the same region where you are going to use the data to avoid cross region charges.
88+
89+
| Region Name | Region Code | Available |
90+
|:------------ | :-------------| :-------------|
91+
| Africa (Cape Town) | af-south-1 | |
92+
| Asia Pacific (Tokyo) | ap-northeast-1 | :heavy_check_mark: |
93+
| Asia Pacific (Seoul) | ap-northeast-2 | :heavy_check_mark: |
94+
| Asia Pacific (Mumbai) | ap-south-1 | :heavy_check_mark: |
95+
| Asia Pacific (Singapore) | ap-southeast-1 | :heavy_check_mark: |
96+
| Asia Pacific (Sydney) | ap-southeast-2 | :heavy_check_mark: |
97+
| Asia Pacific (Jakarta) | ap-southeast-3 | |
98+
| Canada (Central) | ca-central-1 | :heavy_check_mark: |
99+
| China (Beijing) | cn-north-1 | |
100+
| Europe (Frankfurt) | eu-central-1 | :heavy_check_mark: |
101+
| Europe (Zurich) | eu-central-2 | |
102+
| Europe (Stockholm) | eu-north-1 | :heavy_check_mark: |
103+
| Europe (Milan) | eu-south-1 | |
104+
| Europe (Spain) | eu-south-2 | |
105+
| Europe (Ireland) | eu-west-1 | :heavy_check_mark: |
106+
| Europe (London) | eu-west-2 | :heavy_check_mark: |
107+
| Europe (Paris) | eu-west-3 | :heavy_check_mark: |
108+
| South America (São Paulo) | sa-east-1 | :heavy_check_mark: |
109+
| US East (N. Virginia) | us-east-1 | :heavy_check_mark: |
110+
| US East (Ohio) | us-east-2 | :heavy_check_mark: |
111+
| AWS GovCloud (US-East) | us-gov-east-1 | |
112+
| AWS GovCloud (US-West) | us-gov-west-1 | |
113+
| US West (Oregon) | us-west-2 | :heavy_check_mark: |
114+
115+
116+
## Deployment and Cleanup Steps
117+
Reference to folders.
118+
* [data-exports](/data-exports)
119+
* [data-collection](/data-collection)
120+
* [case-summarization](/case-summarization)
121+
* [rls](/rls)
122+
* [security-hub](/security-hub)
123+
124+
## Changelogs
125+
Check [Releases](/../../releases)
126+
127+
## Feedback
128+
Please reference to [this page](https://catalog.workshops.aws/awscid/en-US/feedback-support)
129+
130+
## Contribution
10131
See [CONTRIBUTING](CONTRIBUTING.md) for more information.
11132

12133
## Security
134+
When you build systems on AWS infrastructure, security responsibilities are shared between you and AWS. This [shared responsibility
135+
model](https://aws.amazon.com/compliance/shared-responsibility-model/) reduces your operational burden because AWS operates, manages, and
136+
controls the components including the host operating system, the virtualization layer, and the physical security of the facilities in
137+
which the services operate. For more information about AWS security, visit [AWS Cloud Security](http://aws.amazon.com/security/).
13138

14139
See [SECURITY](SECURITY.md) for more information.
15140

16141
## License
142+
This project is licensed under the Apache-2.0 License. See the [LICENSE](LICENSE) file.
143+
144+
## Notices
145+
Dashboards and their content: (a) are for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
17146

18-
This project is licensed under the Apache-2.0 License.
19147

‎data-collection/README.md‎

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -4,39 +4,41 @@
44

55
This projects demonstrates usage of AWS API for collecting various types of usage data.
66

7+
For deployment and additional information reference to the [documentation](https://catalog.workshops.aws/awscid/data-collection).
8+
79
### Architecture
810

9-
![Architecture](/data-collection/images/archi.png)
11+
![Architecture](/.images/architecture-data-collection-detailed.png)
1012

11-
1. Amazon EventBridge rule invokes Step Function of every every deployed data collection module. based on schedule.
12-
2. The Step Function launches a Lambda function Account Collector that assumes Read Role role in the Management accounts and retrieves linked accounts list via AWS Organizations API
13-
3. Step Functions launches Data Collection Lambda function for each collected Account.
14-
4. Each data collection module Lambda function assumes IAM role in linked accounts and retrieves respective optimization data via AWS SDK for Python. Retrieved data aggregated in Amazon S3 bucket
15-
5. Once data stored in S3 bucket, Step Functions triggers AWS Glue crawler which creates or updates the table in Glue Data Catalog
16-
6. Collected data visualized with the Cloud Intelligence Dashboards using Amazon QuickSight to get optimization recommendations and insights
13+
1. [Amazon EventBridge](https://aws.amazon.com/eventbridge/) rule invokes [AWS Step Functions](https://aws.amazon.com/step-functions/) for every deployed data collection module based on schedule.
14+
2. The Step Function launches a [AWS Lambda](https://aws.amazon.com/lambda/) function **Account Collector** that assumes **Read Role** in the Management accounts and retrieves linked accounts list via [AWS Organizations API](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_integrate_services.html).
15+
3. Step Functions launches **Data Collection Lambda** function for each collected Account.
16+
4. Each data collection module Lambda function assumes an [IAM](https://aws.amazon.com/iam/) role in linked accounts and retrieves respective optimization data via [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html). Retrieved data is aggregated in an [Amazon S3](https://aws.amazon.com/s3/) bucket.
17+
5. Once data is stored in the S3 bucket, Step Functions trigger an [AWS Glue](https://aws.amazon.com/glue/) crawler which creates or updates the table in the [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#data-catalog-intro).
18+
6. Collected data is visualized with the [Cloud Intelligence Dashboards](https://aws.amazon.com/solutions/implementations/cloud-intelligence-dashboards/) using [Amazon QuickSight](https://aws.amazon.com/quicksight/) to get optimization recommendations and insights.
1719

1820

1921
### Modules
2022
List of modules and objects collected:
2123
| Module Name | AWS Services | Collected In | Details |
2224
| --- | --- | --- | --- |
23-
| `organization` | AWS Organizations | Management Accounts | |
24-
| `budgets` | AWS Budgest | Linked Accounts | |
25-
| `compute-optimizer` | AWS Compute Optimizer | Management Accounts | Requires [Enablement of Compute Optimizer](https://aws.amazon.com/compute-optimizer/getting-started/#:~:text=Opt%20in%20for%20Compute%20Optimizer,created%20automatically%20in%20your%20account.) |
26-
| `trusted-advisor` | AWS Trusted Advisor | Linked Accounts | Requires Enterpriso or OnRamp Support Level |
27-
| `support-cases` | AWS Support | Linked Accounts | Requires Business, Enterprise On-Ramp, or Enterprise Support plan |
28-
| `cost-explorer-cost-anomaly` | AWS Anomalies | Management Accounts | |
29-
| `cost-explorer-rightsizing` | AWS Cost Explorer | Management Accounts | DEPRECATED. Please use `Data Exports` for `Cost Optimization Hub` |
25+
| `organization` | [AWS Organizations](https://aws.amazon.com/organizations/) | Management Accounts | |
26+
| `budgets` | [AWS Budgets](https://aws.amazon.com/aws-cost-management/aws-budgets/) | Linked Accounts | |
27+
| `compute-optimizer` | [AWS Compute Optimizer](https://aws.amazon.com/compute-optimizer/) | Management Accounts | Requires [Enablement of Compute Optimizer](https://aws.amazon.com/compute-optimizer/getting-started/#:~:text=Opt%20in%20for%20Compute%20Optimizer,created%20automatically%20in%20your%20account.) |
28+
| `trusted-advisor` | [AWS Trusted Advisor](https://aws.amazon.com/premiumsupport/technology/trusted-advisor/) | Linked Accounts | Requires Business, Enterprise or On-Ramp Support Level |
29+
| `support-cases` | [AWS Support](https://aws.amazon.com/premiumsupport/) | Linked Accounts | Requires Business, Enterprise On-Ramp, or Enterprise Support plan |
30+
| `cost-explorer-cost-anomaly` | [AWS Cost Anomaly Detection](https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-detection/) | Management Accounts | |
31+
| `cost-explorer-rightsizing` | [AWS Cost Explorer](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) | Management Accounts | DEPRECATED. Please use `Data Exports` for `Cost Optimization Hub` |
3032
| `inventory` | Various services | Linked Accounts | Collects `Amazon OpenSearch Domains`, `Amazon ElastiCache Clusters`, `RDS DB Instances`, `EBS Volumes`, `AMI`, `EC2 Instances`, `EBS Snapshot`, `RDS Snapshot`, `Lambda`, `RDS DB Clusters`, `EKS Clusters` |
3133
| `pricing` | Various services | Data Collection Account | Collects pricing for `Amazon RDS`, `Amazon EC2`, `Amazon ElastiCache`, `AWS Lambda`, `Amazon OpenSearch`, `AWS Compute Savings Plan` |
32-
| `rds-usage` | Amazon RDS | Linked Accounts | Collects CloudWatch metrics for chargeback |
33-
| `transit-gateway` | AWS Transit Gateway | Linked Accounts | Collects CloudWatch metrics for chargeback |
34-
| `ecs-chargeback` | Amazon ECS | Linked Accounts | |
35-
| `backup` | AWS Backup | Management Accounts | Collects Backup Restore and Copy Jobs. Requires [activation of cross-account](https://docs.aws.amazon.com/aws-backup/latest/devguide/manage-cross-account.html#enable-cross-account) |
36-
| `health-events` | AWS Health | Management Accounts | Collect AWS Health notificaitons via AWS Organizational view |
37-
| `licence-manager` | AWS License Manager | Management Accounts | Collect Licences and Grants |
38-
| `aws-feeds` | N/A | Data Collection Account |Collects Blog posts and News Feeds|
39-
| `quicksight` | Amazon QuickSight | Data Collection Account |Collects Quicksight User and Group information in the Data Collection Account only|
34+
| `rds-usage` | [Amazon RDS](https://aws.amazon.com/rds/) | Linked Accounts | Collects CloudWatch metrics for chargeback |
35+
| `transit-gateway` | [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/) | Linked Accounts | Collects CloudWatch metrics for chargeback |
36+
| `ecs-chargeback` | [Amazon ECS](https://aws.amazon.com/ecs/) | Linked Accounts | |
37+
| `backup` | [AWS Backup](https://aws.amazon.com/backup/) | Management Accounts | Collects Backup Restore and Copy Jobs. Requires [activation of cross-account](https://docs.aws.amazon.com/aws-backup/latest/devguide/manage-cross-account.html#enable-cross-account) |
38+
| `health-events` | [AWS Health](https://aws.amazon.com/health/) | Management Accounts | Collect AWS Health notifications via AWS Organizational view |
39+
| `licence-manager` | [AWS License Manager](https://aws.amazon.com/license-manager/) | Management Accounts | Collect Licenses and Grants |
40+
| `aws-feeds` | N/A | Data Collection Account | Collects Blog posts and News Feeds |
41+
| `quicksight` | [Amazon QuickSight](https://aws.amazon.com/quicksight/) | Data Collection Account | Collects QuickSight User and Group information in the Data Collection Account only |
4042

4143

4244
### Installation

0 commit comments

Comments
 (0)