Skip to content

Commit 7a9d2e2

Browse files
stepanclbstepanclb
committed
Add AWS cost reporting Terraform module
Multi-account CUR aggregation with source and target submodules: - modules/source: CUR report definition, S3 bucket, cross-account Lambda notification - modules/target: Lambda forwarder, aggregated S3 bucket, Athena/Glue analysis, IAM access
0 parents  commit 7a9d2e2

32 files changed

+2488
-0
lines changed

‎.editorconfig‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# EditorConfig is awesome: http://EditorConfig.org
2+
# Uses editorconfig to maintain consistent coding styles
3+
4+
# top-most EditorConfig file
5+
root = true
6+
7+
# Unix-style newlines with a newline ending every file
8+
[*]
9+
charset = utf-8
10+
end_of_line = lf
11+
indent_size = 2
12+
indent_style = space
13+
insert_final_newline = true
14+
max_line_length = 80
15+
trim_trailing_whitespace = true
16+
17+
[*.{tf,tfvars}]
18+
indent_size = 2
19+
indent_style = space
20+
21+
[*.md]
22+
max_line_length = 0
23+
trim_trailing_whitespace = false
24+

‎.github/workflows/check.yml‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
name: Check
2+
on:
3+
push:
4+
branches:
5+
- "**"
6+
7+
jobs:
8+
fmt:
9+
name: Terraform Format
10+
runs-on: ubuntu-latest
11+
container:
12+
image: ghcr.io/cookielab/container-image-terraform:1.14
13+
options: --user root
14+
credentials:
15+
username: ${{ github.actor }}
16+
password: ${{ secrets.GITHUB_TOKEN }}
17+
permissions:
18+
packages: read
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@v6
22+
- name: Check Terraform format
23+
run: terraform fmt -check=true -recursive
24+
lint:
25+
name: TF Lint
26+
runs-on: ubuntu-latest
27+
container:
28+
image: ghcr.io/cookielab/container-image-terraform:1.14
29+
options: --user root
30+
credentials:
31+
username: ${{ github.actor }}
32+
password: ${{ secrets.GITHUB_TOKEN }}
33+
steps:
34+
- name: Checkout repository
35+
uses: actions/checkout@v6
36+
- name: Check TF lint - source module
37+
run: tflint --chdir=modules/source
38+
- name: Check TF lint - target module
39+
run: tflint --chdir=modules/target
40+
validate-source:
41+
name: Validate Source Module
42+
runs-on: ubuntu-latest
43+
container:
44+
image: ghcr.io/cookielab/container-image-terraform:1.14
45+
options: --user root
46+
credentials:
47+
username: ${{ github.actor }}
48+
password: ${{ secrets.GITHUB_TOKEN }}
49+
steps:
50+
- name: Checkout repository
51+
uses: actions/checkout@v6
52+
- name: Inject additional provider
53+
uses: "finnp/create-file-action@master"
54+
env:
55+
FILE_NAME: "modules/source/test_injection.tf"
56+
FILE_DATA: |
57+
provider "aws" {
58+
alias = "us_east_1"
59+
}
60+
- name: Terraform init
61+
run: terraform -chdir=modules/source init
62+
- name: Terraform validate
63+
run: terraform -chdir=modules/source validate
64+
validate-target:
65+
name: Validate Target Module
66+
runs-on: ubuntu-latest
67+
container:
68+
image: ghcr.io/cookielab/container-image-terraform:1.14
69+
options: --user root
70+
credentials:
71+
username: ${{ github.actor }}
72+
password: ${{ secrets.GITHUB_TOKEN }}
73+
steps:
74+
- name: Checkout repository
75+
uses: actions/checkout@v6
76+
- name: Terraform init
77+
run: terraform -chdir=modules/target init
78+
- name: Terraform validate
79+
run: terraform -chdir=modules/target validate

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
.infracost
2+
.terraform
3+
.terraform.tfstate.lock.info
4+
terraform.tfstate
5+
terraform.tfstate.backup
6+
terraform.tfvars
7+
.terraform.lock.hcl
8+
.infracost
9+
.terraform-version

‎README.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Terraform module for AWS Cost Reporting
2+
3+
Multi-account AWS Cost and Usage Report (CUR) aggregation and analysis. This module provides two submodules:
4+
5+
- **`modules/source`** - Deployed in each source AWS account. Creates CUR report definition, S3 bucket, and event notification to forward reports to the target account.
6+
- **`modules/target`** - Deployed in the central/target AWS account. Aggregates CUR reports from multiple source accounts using a Lambda function, with optional Athena/Glue analysis and IAM access management.
7+
8+
## Architecture
9+
10+
```
11+
┌─────────────────────┐ ┌─────────────────────┐
12+
│ Source Account A │ │ Source Account B │
13+
│ │ │ │
14+
│ CUR Report → S3 │ │ CUR Report → S3 │
15+
│ │ │ │ │ │
16+
│ └─── S3 Event ─┼─────┼───── S3 Event ─────┤
17+
└─────────────────────┘ └─────────────────────┘
18+
│ │
19+
▼ ▼
20+
┌─────────────────────────────────┐
21+
│ Target Account │
22+
│ │
23+
│ Lambda (CUR Forwarder) │
24+
│ │ │
25+
│ ▼ │
26+
│ S3 Bucket (Aggregated CUR) │
27+
│ │ │
28+
│ ▼ (optional) │
29+
│ Athena + Glue (Analysis) │
30+
│ IAM Roles/Users (Access) │
31+
└─────────────────────────────────┘
32+
```
33+
34+
## Usage
35+
36+
### 1. Deploy the target module first (central account)
37+
38+
```terraform
39+
module "cur_target" {
40+
source = "cookielab/cost-reporting/aws//modules/target"
41+
42+
cur_reports_bucket_name = "my-aggregated-cur-reports"
43+
44+
source_accounts = {
45+
"prod" = {
46+
account_id = "111111111111"
47+
}
48+
"staging" = {
49+
account_id = "222222222222"
50+
}
51+
}
52+
53+
enable_athena = true
54+
}
55+
```
56+
57+
### 2. Deploy the source module in each source account
58+
59+
```terraform
60+
provider "aws" {
61+
alias = "us_east_1"
62+
region = "us-east-1"
63+
}
64+
65+
module "cur_source" {
66+
source = "cookielab/cost-reporting/aws//modules/source"
67+
68+
providers = {
69+
aws.us_east_1 = aws.us_east_1
70+
}
71+
72+
lambda_function_arn = "arn:aws:lambda:eu-west-1:000000000000:function:cur-forwarder"
73+
lambda_function_role_arn = "arn:aws:iam::000000000000:role/cur-forwarder-role"
74+
}
75+
```
76+
77+
## Source Module
78+
79+
Creates AWS CUR report definition and S3 bucket in a source account, with cross-account access for the target Lambda.
80+
81+
### Requirements
82+
83+
| Name | Version |
84+
|------|---------|
85+
| terraform | >= 1.5, < 2.0 |
86+
| aws | >= 5.27 |
87+
88+
### Inputs
89+
90+
| Name | Description | Type | Default | Required |
91+
|------|-------------|------|---------|:--------:|
92+
| lambda_function_arn | ARN of the Lambda function in the target account | `string` | n/a | yes |
93+
| lambda_function_role_arn | ARN of the IAM role for the Lambda function (for bucket policy) | `string` | n/a | yes |
94+
| s3_bucket_name | S3 bucket name (defaults to `cur-csv-{account_id}`) | `string` | `null` | no |
95+
| cur_time_unit | Time unit for CUR report (`HOURLY` or `DAILY`) | `string` | `"HOURLY"` | no |
96+
| cur_format | Report format (`textORcsv` or `Parquet`) | `string` | `"textORcsv"` | no |
97+
| cur_compression | Compression (`GZIP`, `ZIP`, or `Parquet`) | `string` | `"GZIP"` | no |
98+
| s3_bucket_lifecycle | S3 lifecycle transitions | `object` | `{transition_to_ia_days=90, transition_to_glacier_days=180}` | no |
99+
| tags | Tags to apply to resources | `map(string)` | `{}` | no |
100+
101+
### Outputs
102+
103+
| Name | Description |
104+
|------|-------------|
105+
| bucket_id | ID of the CUR S3 bucket |
106+
| bucket_arn | ARN of the CUR S3 bucket |
107+
| bucket_name | Name of the CUR S3 bucket |
108+
| cur_report_name | Name of the CUR report |
109+
| cur_prefix | S3 prefix where CUR reports are stored |
110+
| account_id | AWS Account ID |
111+
112+
## Target Module
113+
114+
Aggregates CUR reports from multiple source accounts into a central S3 bucket using Lambda, with optional Athena analysis.
115+
116+
### Requirements
117+
118+
| Name | Version |
119+
|------|---------|
120+
| terraform | >= 1.5, < 2.0 |
121+
| aws | >= 5.27 |
122+
123+
### Inputs
124+
125+
| Name | Description | Type | Default | Required |
126+
|------|-------------|------|---------|:--------:|
127+
| cur_reports_bucket_name | Name of the S3 bucket for aggregated CUR reports | `string` | n/a | yes |
128+
| source_accounts | Map of source account configurations | `map(object)` | `{}` | no |
129+
| create_bucket | Whether to create a new S3 bucket | `bool` | `true` | no |
130+
| enable_athena | Enable Athena/Glue for CUR analysis | `bool` | `true` | no |
131+
| create_reader_role | Create IAM role for read-only access | `bool` | `true` | no |
132+
| create_reader_user | Create IAM user with access keys | `bool` | `false` | no |
133+
| lambda_function_name | Name of the Lambda function | `string` | `"cur-forwarder"` | no |
134+
| glue_database_name | Glue database name for partition management | `string` | `""` | no |
135+
| glue_region | AWS region for Glue catalog | `string` | `"eu-west-1"` | no |
136+
| table_mapping | Map of destination_prefix to Glue table name | `map(string)` | `{}` | no |
137+
| tags | Tags to apply to resources | `map(string)` | `{}` | no |
138+
139+
### Outputs
140+
141+
| Name | Description |
142+
|------|-------------|
143+
| bucket_id | ID of the aggregated CUR S3 bucket |
144+
| bucket_arn | ARN of the aggregated CUR S3 bucket |
145+
| lambda_function_arn | ARN of the Lambda function |
146+
| lambda_function_name | Name of the Lambda function |
147+
| lambda_role_arn | ARN of the Lambda IAM role |
148+
| athena_workgroup_name | Athena workgroup name |
149+
| glue_database_name | Glue database name |
150+
| reader_role_arn | IAM role ARN for read-only access |
151+
152+
## Bootstrap Script
153+
154+
For existing CUR data, use `modules/target/scripts/bootstrap_partitions.py` to create initial Glue partitions. Configure the variables at the top of the script and run:
155+
156+
```bash
157+
export AWS_PROFILE=your-profile
158+
python3 modules/target/scripts/bootstrap_partitions.py
159+
```

0 commit comments

Comments
 (0)