Skip to content
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
f932232
added marketplace agreements
Aug 26, 2025
07cdd2e
removed quicksight dataset creation since this is data colelction only
Aug 27, 2025
d35fc4d
actual removed quicksight from before - forgot to git add .
Aug 27, 2025
6275c7b
removed ensure_and_trigger_craler that waits for crawler
Aug 27, 2025
7b39581
agreements module to the data-collection
Aug 27, 2025
08083a5
removed the Organizations policy from the account-
Aug 27, 2025
3f88a78
renamed from agreement to marketplace -in cluded relevant files
Aug 27, 2025
9ca8067
fixed crawler naming
Aug 27, 2025
e404304
Update data-collection/deploy/deploy-data-collection.yaml
Aug 27, 2025
617a9fb
updated version to 13.0 in marketplace module
Aug 27, 2025
0847180
added s3 permission
Aug 27, 2025
fe211b6
no need for s3 permissions or s3 add in marketplace module
Aug 28, 2025
db1a3d6
alphabetically ordered the marketplace actions in the linked
Aug 28, 2025
0fa4101
removed redudant manual replacement
Aug 28, 2025
b2f7eaa
updated python
Aug 28, 2025
1023a27
Marketplace-Discovery-GetProduct policy
Aug 28, 2025
89c2166
Fix marketplace module: add S3 permissions, CSV output, and collect
Aug 28, 2025
4b06f2d
removed the unnecessary organization permissions from
Aug 28, 2025
bf704de
fixed crawler name inconsistency in data collection
Aug 28, 2025
0659f34
removed the DELTE COMPLETE from utils.py and removed sts getcallerid…
Aug 28, 2025
54d37d7
Marketplace Agreements policy removed from management account, fixed …
Aug 31, 2025
b7ba110
fixed linting - added metadata supressions as necessary
Sep 1, 2025
8368f51
Merge branch 'main' into HEAD
iakov-gan Sep 1, 2025
ed3d0fe
lint fixes
iakov-gan Sep 1, 2025
48a2d85
Merge upstream changes from aws-solutions-library-samples
Sep 1, 2025
232ee6d
add note
iakov-gan Sep 1, 2025
04ab19e
release 3.14.0
iakov-gan Sep 1, 2025
0a68ef3
Merge branch 'release/3.14.0' into fix-mkpl-collector
iakov-gan Sep 1, 2025
f9f5b59
Merge branch 'main' into fix-mkpl-collector
iakov-gan Sep 1, 2025
357f9b7
Merge pull request #1 from aws-solutions-library-samples/fix-mkpl-col…
Sep 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This repository is a part of [Cloud Intelligence Dashboards](https://docs.aws.am

This repository contains following elements:
* [data-exports](/data-exports) - a Cloud Formation Templates for AWS Data Exports, such as Cost and Usage Report 2.0 and others. This allows a replication of Exports from your Management Account(s) to a Dedicated Data Collection Accounts as well as aggregation of multiple Exports from a set of Linked Accounts.
* [data-collection](/data-collection) - a set of Cloud Formation Templates for collecting infrastructure operational data from Management and Linked Accounts. Such as data from AWS Trusted Advisor, AWS Compute Optimizer, Inventories, Pricing, AWS Health, AWS Support Cases etc. See more about types of data collected [here](/data-collection).
* [data-collection](/data-collection) - a set of Cloud Formation Templates for collecting infrastructure operational data from Management and Linked Accounts. Such as data from AWS Trusted Advisor, AWS Compute Optimizer, Inventories, Pricing, AWS Health, AWS Support Cases, AWS Marketplace etc. See more about types of data collected [here](/data-collection).
* [case-summarization](/case-summarization) - an additional Cloud Formation Template for deploying the AWS Support Case Summarization plugin that offers the capability to summarize cases through Generative AI powered by Amazon Bedrock.
* [rls](/rls) - a stack for managing Row Level Security for CID Dashboards.
* [security-hub](/security-hub) - Collection of data from AWS Security Hub.
Expand Down
1 change: 1 addition & 0 deletions data-collection/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ List of modules and objects collected:
| `aws-feeds` | N/A | Data Collection Account | Collects Blog posts and News Feeds |
| `quicksight` | [Amazon QuickSight](https://aws.amazon.com/quicksight/) | Data Collection Account | Collects QuickSight User and Group information in the Data Collection Account only |
| `resilience-hub` | [AWS Resilince Hub](https://aws.amazon.com/resilience-hub/) | Linked Accounts | |
| `marketplace` | [AWS Marketplace](https://aws.amazon.com/marketplace/) | Linked Accounts | Collects AWS Marketplace data and terms |
| `reference` | Various services | Data Collection Account | Collects reference data for other modules and dashboard to function |

### Deployment Overview
Expand Down
98 changes: 67 additions & 31 deletions data-collection/deploy/deploy-data-collection.yaml

Large diffs are not rendered by default.

23 changes: 17 additions & 6 deletions data-collection/deploy/deploy-data-read-permissions.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# https://github.com/awslabs/cid-data-collection-framework/blob/main/data-collection/v3.12.1/deploy/deploy-data-read-permissions.yaml
# https://github.com/awslabs/cid-data-collection-framework/blob/main/data-collection/v3.13.1/deploy/deploy-data-read-permissions.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: CID Data Collection - All-in-One for Management Account v3.12.1 - AWS Solution SO9011
Description: CID Data Collection - All-in-One for Management Account v3.13.1 - AWS Solution SO9011
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
Expand Down Expand Up @@ -33,6 +33,7 @@ Metadata:
- IncludeLicenseManagerModule
- IncludeServiceQuotasModule
- IncludeResilienceHubModule
- IncludeMarketplaceModule
ParameterLabels:
ManagementAccountRole:
default: "Management account role"
Expand Down Expand Up @@ -80,6 +81,8 @@ Metadata:
default: "Include Service Quotas Module"
IncludeResilienceHubModule:
default: "Include ResilienceHub Module"
IncludeMarketplaceModule:
default: "Include Marketplace Agreements Module"

Parameters:
ManagementAccountRole:
Expand Down Expand Up @@ -191,14 +194,19 @@ Parameters:
Description: Collects Resilience Hub information
AllowedValues: ['yes', 'no']
Default: 'no'
IncludeMarketplaceModule:
Type: String
Description: Collects Marketplace Agreement information
AllowedValues: ['yes', 'no']
Default: 'no'
Conditions:
DeployModuleReadInMgmt: !Equals [!Ref AllowModuleReadInMgmt, "yes"]

Resources:
DataCollectorMgmtAccountReadStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub "https://${CFNSourceBucket}.s3.${AWS::URLSuffix}/cfn/data-collection/v3.12.1/deploy-in-management-account.yaml"
TemplateURL: !Sub "https://${CFNSourceBucket}.s3.${AWS::URLSuffix}/cfn/data-collection/v3.13.1/deploy-in-management-account.yaml"
Parameters:
DataCollectionAccountID: !Ref DataCollectionAccountID
ManagementAccountRole: !Ref ManagementAccountRole
Expand All @@ -215,7 +223,7 @@ Resources:
Type: AWS::CloudFormation::Stack
Condition: DeployModuleReadInMgmt
Properties:
TemplateURL: !Sub "https://${CFNSourceBucket}.s3.${AWS::URLSuffix}/cfn/data-collection/v3.12.1/deploy-in-linked-account.yaml"
TemplateURL: !Sub "https://${CFNSourceBucket}.s3.${AWS::URLSuffix}/cfn/data-collection/v3.13.1/deploy-in-linked-account.yaml"
Parameters:
DataCollectionAccountID: !Ref DataCollectionAccountID
MultiAccountRoleName: !Ref MultiAccountRoleName
Expand All @@ -230,11 +238,12 @@ Resources:
IncludeTransitGatewayModule: !Ref IncludeTransitGatewayModule
IncludeServiceQuotasModule: !Ref IncludeServiceQuotasModule
IncludeResilienceHubModule: !Ref IncludeResilienceHubModule
IncludeMarketplaceModule: !Ref IncludeMarketplaceModule

DataCollectorOrgAccountModulesReadStackSet:
Type: AWS::CloudFormation::StackSet
Properties:
Description: "StackSet in charge of deploying read roles across organization accounts v3.12.1"
Description: "StackSet in charge of deploying read roles across organization accounts v3.13.1"
PermissionModel: SERVICE_MANAGED
AutoDeployment:
Enabled: true
Expand Down Expand Up @@ -272,6 +281,8 @@ Resources:
ParameterValue: !Ref IncludeServiceQuotasModule
- ParameterKey: IncludeResilienceHubModule
ParameterValue: !Ref IncludeResilienceHubModule
- ParameterKey: IncludeMarketplaceModule
ParameterValue: !Ref IncludeMarketplaceModule
StackInstancesGroup:
- DeploymentTargets:
OrganizationalUnitIds: !Split [",", !Ref OrganizationalUnitIds]
Expand All @@ -281,4 +292,4 @@ Resources:
- CAPABILITY_IAM
- CAPABILITY_NAMED_IAM
StackSetName: !Sub "StackSet-${AWS::AccountId}-OptimizationDataRole"
TemplateURL: !Sub "https://${CFNSourceBucket}.s3.${AWS::URLSuffix}/cfn/data-collection/v3.12.1/deploy-in-linked-account.yaml"
TemplateURL: !Sub "https://${CFNSourceBucket}.s3.${AWS::URLSuffix}/cfn/data-collection/v3.13.1/deploy-in-linked-account.yaml"
50 changes: 48 additions & 2 deletions data-collection/deploy/deploy-in-linked-account.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
AWSTemplateFormatVersion: '2010-09-09'
Description: CID Data Collection - Role for Linked Account v3.12.1
Description: CID Data Collection - Role for Linked Account v3.13.1
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
Expand All @@ -22,6 +22,7 @@ Metadata:
- IncludeTransitGatewayModule
- IncludeServiceQuotasModule
- IncludeResilienceHubModule
- IncludeMarketplaceModule
ParameterLabels:
DataCollectionAccountID:
default: 'Data Collection Account ID'
Expand Down Expand Up @@ -49,6 +50,8 @@ Metadata:
default: 'Include Service Quotas Module'
IncludeResilienceHubModule:
default: 'Include Resilience Hub Module'
IncludeMarketplaceModule:
default: 'Include Marketplace Agreements Module'

Parameters:
DataCollectionAccountID:
Expand Down Expand Up @@ -112,6 +115,11 @@ Parameters:
Description: Collects Resilience Hub data from your accounts
AllowedValues: ['yes', 'no']
Default: 'no'
IncludeMarketplaceModule:
Type: String
Description: Collects Marketplace Agreement data from your accounts
AllowedValues: ['yes', 'no']
Default: 'no'

Conditions:
IncludeTAModulePolicy: !Equals [!Ref IncludeTAModule, "yes"]
Expand All @@ -124,6 +132,7 @@ Conditions:
IncludeTransitGatewayModulePolicy: !Equals [!Ref IncludeTransitGatewayModule, "yes"]
IncludeServiceQuotasModulePolicy: !Equals [!Ref IncludeServiceQuotasModule, "yes"]
IncludeResilienceHubModulePolicy: !Equals [!Ref IncludeResilienceHubModule, "yes"]
IncludeMarketplaceModulePolicy: !Equals [!Ref IncludeMarketplaceModule, "yes"]

Outputs:
LambdaRole:
Expand Down Expand Up @@ -155,6 +164,7 @@ Resources:
- !Sub "arn:${AWS::Partition}:iam::${DataCollectionAccountID}:role/${ResourcePrefix}support-cases-LambdaRole"
- !Sub "arn:${AWS::Partition}:iam::${DataCollectionAccountID}:role/${ResourcePrefix}service-quotas-LambdaRole"
- !Sub "arn:${AWS::Partition}:iam::${DataCollectionAccountID}:role/${ResourcePrefix}resilience-hub-LambdaRole"
- !Sub "arn:${AWS::Partition}:iam::${DataCollectionAccountID}:role/${ResourcePrefix}marketplace-LambdaRole"
Path: /
Metadata:
cfn_nag:
Expand Down Expand Up @@ -464,4 +474,40 @@ Resources:
cfn_nag:
rules_to_suppress:
- id: W12
reason: "Policy is used for scanning of a wide range of resources"
reason: "Policy is used for scanning of a wide range of resources"
# Marketplace Agreements policy
AgreementsPolicy:
Type: 'AWS::IAM::Policy'
Condition: IncludeMarketplaceModulePolicy
Properties:
PolicyName: AgreementsPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "organizations:ListAccounts"
- "organizations:DescribeOrganization"
Resource: "*"
- Effect: Allow
Action:
- "aws-marketplace:DescribeAgreement"
- "aws-marketplace:DescribeEntity"
- "aws-marketplace:GetAgreementTerms"
- "aws-marketplace:GetProduct"
- "aws-marketplace:ListAgreementCharges"
- "aws-marketplace:ListEntities"
- "aws-marketplace:SearchAgreements"
- "aws-marketplace:SearchEntities"
- "marketplace-agreement:DescribeAgreement"
- "marketplace-agreement:GetAgreementTerms"
- "marketplace-agreement:ListAgreementCharges"
- "marketplace-agreement:SearchAgreements"
Resource: "*"
Roles:
- Ref: LambdaRole
Metadata:
cfn_nag:
rules_to_suppress:
- id: W12
reason: "Policy is used for scanning of a wide range of marketplace resources"
11 changes: 10 additions & 1 deletion data-collection/deploy/deploy-in-management-account.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
AWSTemplateFormatVersion: '2010-09-09'
Description: CID Data Collection - Role for Management Account v3.12.1
Description: CID Data Collection - Role for Management Account v3.13.1
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
Expand All @@ -19,6 +19,7 @@ Metadata:
- IncludeRightsizingModule
- IncludeLicenseManagerModule
- IncludeServiceQuotasModule
- IncludeMarketplaceModule
ParameterLabels:
ManagementAccountRole:
default: "Management account role"
Expand All @@ -40,6 +41,8 @@ Metadata:
default: "Include Marketplace Licensing Module"
IncludeServiceQuotasModule:
default: "Include Service Quotas Module"
IncludeMarketplaceModule:
default: "Include Marketplace Agreements Module"
Parameters:
DataCollectionAccountID:
Type: String
Expand Down Expand Up @@ -87,6 +90,11 @@ Parameters:
Description: Collects Service Quotas Information from your accounts
AllowedValues: ['yes', 'no']
Default: 'no'
IncludeMarketplaceModule:
Type: String
Description: Collects Marketplace Agreement Information from your accounts
AllowedValues: ['yes', 'no']
Default: 'no'

Conditions:
EnableComputeOptimizerModule: !Equals [!Ref IncludeComputeOptimizerModule, "yes"]
Expand All @@ -96,6 +104,7 @@ Conditions:
EnableHealthEventsModule: !Equals [!Ref IncludeHealthEventsModule, "yes"]
EnableLicenseManagerModule: !Equals [!Ref IncludeLicenseManagerModule, "yes"]
EnableServiceQuotasModule: !Equals [!Ref IncludeServiceQuotasModule, "yes"]
EnableMarketplaceModule: !Equals [!Ref IncludeMarketplaceModule, "yes"]

Outputs:
LambdaRole:
Expand Down
Loading
Loading