Skip to content

Commit 9e2cc45

Browse files
goruhagoruha
committed
Initial commit
1 parent 5826bfc commit 9e2cc45

File tree

8 files changed

+652
-59
lines changed

8 files changed

+652
-59
lines changed

‎.github/settings.yml‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,7 @@
11
# Upstream changes from _extends are only recognized when modifications are made to this file in the default branch.
22
_extends: .github
33
repository:
4-
name: template
5-
description: Template for Terraform Components
4+
name: aws-datadog-logs-archive
5+
description: This component is responsible for provisioning Datadog Log Archives
66
homepage: https://cloudposse.com/accelerate
77
topics: terraform, terraform-component
8-
9-
10-
11-

‎README.yaml‎

Lines changed: 142 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,164 @@
1-
name: "template"
2-
1+
name: "aws-datadog-logs-archive"
32
# Canonical GitHub repo
4-
github_repo: "cloudposse-terraform-components/template"
5-
3+
github_repo: "cloudposse-terraform-components/aws-datadog-logs-archive"
64
# Short description of this project
75
description: |-
8-
Description of this component
6+
This component is responsible for provisioning Datadog Log Archives. It creates a single log archive pipeline for each
7+
AWS account. If the `catchall` flag is set, it creates a catchall archive within the same S3 bucket.
8+
9+
Each log archive filters for the tag `env:$env` where $env is the environment/account name (ie sbx, prd, tools, etc), as
10+
well as any tags identified in the additional_tags key. The `catchall` archive, as the name implies, filters for '\*'.
11+
12+
A second bucket is created for cloudtrail, and a cloudtrail is configured to monitor the log archive bucket and log
13+
activity to the cloudtrail bucket. To forward these cloudtrail logs to datadog, the cloudtrail bucket's id must be added
14+
to the s3_buckets key for our datadog-lambda-forwarder component.
15+
16+
Both buckets support object lock, with overridable defaults of COMPLIANCE mode with a duration of 7 days.
17+
18+
## Prerequisites
19+
20+
- Datadog integration set up in target environment
21+
- We rely on the datadog api and app keys added by our datadog integration component
22+
23+
## Issues, Gotchas, Good-to-Knows
924
10-
usage: |-
11-
**Stack Level**: Regional or Test47
25+
### Destroy/reprovision process
26+
27+
Because of the protections for S3 buckets, if we want to destroy/replace our bucket, we need to do so in two passes or
28+
destroy the bucket manually and then use terraform to clean up the rest. If reprovisioning a recently provisioned
29+
bucket, the two-pass process works well. If the bucket has a full day or more of logs, though, deleting it manually
30+
first will avoid terraform timeouts, and then the terraform process can be used to clean up everything else.
31+
32+
#### Two step process to destroy via terraform
33+
34+
- first set `s3_force_destroy` var to true and apply
35+
- next set `enabled` to false and apply or use tf destroy
36+
37+
## Usage
38+
39+
**Stack Level**: Global
40+
41+
Here's an example snippet for how to use this component. It's suggested to apply this component to all accounts from
42+
which Datadog receives logs.
1243
13-
Here's an example snippet for how to use this component.
14-
1544
```yaml
1645
components:
1746
terraform:
18-
foo:
47+
datadog-logs-archive:
48+
settings:
49+
spacelift:
50+
workspace_enabled: true
1951
vars:
2052
enabled: true
53+
# additional_query_tags:
54+
# - "forwardername:*-dev-datadog-lambda-forwarder-logs"
55+
# - "account:123456789012"
2156
```
2257
23-
include:
24-
- "docs/terraform.md"
58+
## Requirements
2559
26-
tags:
27-
- terraform
28-
- terraform-modules
29-
- aws
30-
- components
31-
- terraform-components
32-
- root
33-
- geodesic
34-
- reference-implementation
35-
- reference-architecture
60+
| Name | Version |
61+
| --------- | --------- |
62+
| terraform | >= 0.13.0 |
63+
| aws | >= 2.0 |
64+
| datadog | >= 3.3.0 |
65+
| local | >= 1.3 |
66+
67+
## Providers
3668
69+
| Name | Version |
70+
| ------- | -------- |
71+
| aws | >= 2.0 |
72+
| datadog | >= 3.7.0 |
73+
| http | >= 2.1.0 |
74+
75+
## Modules
76+
77+
| Name | Source | Version |
78+
| -------------------- | ----------------------------------- | ------- |
79+
| cloudtrail | cloudposse/cloudtrail/aws | 0.21.0 |
80+
| cloudtrail_s3_bucket | cloudposse/cloudtrail-s3-bucket/aws | 0.23.1 |
81+
| iam_roles | ../account-map/modules/iam-roles | n/a |
82+
| s3_bucket | cloudposse/s3-bucket/aws | 0.46.0 |
83+
| this | cloudposse/label/null | 0.25.0 |
84+
85+
## Resources
86+
87+
| Name | Type |
88+
| --------------------------------------- | ----------- |
89+
| aws_caller_identity.current | data source |
90+
| aws_partition.current | data source |
91+
| aws_ssm_parameter.datadog_api_key | data source |
92+
| aws_ssm_parameter.datadog_app_key | data source |
93+
| aws_ssm_parameter.datadog_aws_role_name | data source |
94+
| aws_ssm_parameter.datadog_external_id | data source |
95+
| datadog_logs_archive.catchall_archive | resource |
96+
| datadog_logs_archive.logs_archive | resource |
97+
| http.current_order | data source |
98+
99+
## Inputs
100+
101+
| Name | Description | Type | Default | Required |
102+
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | ---------------- |
103+
| additional_query_tags | Additional tags to include in query for logs for this archive | `list` | [] | no |
104+
| catchall | Set to true to enable a catchall for logs unmatched by any queries. This should only be used in one environment/account | `bool` | false | no |
105+
| datadog_aws_account_id | The AWS account ID Datadog's integration servers use for all integrations | `string` | 464622532012 | no |
106+
| enable_glacier_transition | Enable/disable transition to glacier. Has no effect unless `lifecycle_rules_enabled` set to true | `bool` | true | no |
107+
| glacier_transition_days | Number of days after which to transition objects to glacier storage | `number` | 365 | no |
108+
| lifecycle_rules_enabled | Enable/disable lifecycle management rules for s3 objects | `bool` | true | no |
109+
| object_lock_days_archive | Set duration of archive bucket object lock | `number` | 7 | yes |
110+
| object_lock_days_cloudtrail | Set duration of cloudtrail bucket object lock | `number` | 7 | yes |
111+
| object_lock_mode_archive | Set mode of archive bucket object lock | `string` | COMPLIANCE | yes |
112+
| object_lock_mode_cloudtrail | Set mode of cloudtrail bucket object lock | `string` | COMPLIANCE | yes |
113+
| s3_force_destroy | Set to true to delete non-empty buckets when `enabled` is set to false | `bool` | false | for destroy only |
114+
115+
## Outputs
116+
117+
| Name | Description |
118+
| ----------------------------- | ----------------------------------------------------------- |
119+
| archive_id | The ID of the environment-specific log archive |
120+
| bucket_arn | The ARN of the bucket used for log archive storage |
121+
| bucket_domain_name | The FQDN of the bucket used for log archive storage |
122+
| bucket_id | The ID (name) of the bucket used for log archive storage |
123+
| bucket_region | The region of the bucket used for log archive storage |
124+
| cloudtrail_bucket_arn | The ARN of the bucket used for cloudtrail log storage |
125+
| cloudtrail_bucket_domain_name | The FQDN of the bucket used for cloudtrail log storage |
126+
| cloudtrail_bucket_id | The ID (name) of the bucket used for cloudtrail log storage |
127+
| catchall_id | The ID of the catchall log archive |
128+
129+
## References
130+
131+
- [cloudposse/s3-bucket/aws](https://registry.terraform.io/modules/cloudposse/s3-bucket/aws/latest) - Cloud Posse's S3
132+
component
133+
- [datadog_logs_archive resource]
134+
(https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_archive) - Datadog's provider
135+
documentation for the datadog_logs_archive resource
136+
tags:
137+
- component/datadog-logs-archive
138+
- layer/datadog
139+
- provider/aws
140+
- provider/datadog
37141
# Categories of this project
38142
categories:
39-
- terraform-modules/root
40-
- terraform-components
41-
143+
- component/datadog-logs-archive
144+
- layer/datadog
145+
- provider/aws
146+
- provider/datadog
42147
# License of this project
43148
license: "APACHE2"
44-
45149
# Badges to display
46150
badges:
47-
- name: "Latest Release"
48-
image: "https://img.shields.io/github/release/cloudposse-terraform-components/template.svg?style=for-the-badge"
49-
url: "https://github.com/cloudposse-terraform-components/template/releases/latest"
50-
- name: "Slack Community"
51-
image: "https://slack.cloudposse.com/for-the-badge.svg"
52-
url: "https://slack.cloudposse.com"
53-
54-
references:
55-
- name: "Cloud Posse Documentation"
56-
description: "Complete documentation for the Cloud Posse solution"
57-
url: "https://docs.cloudposse.com"
58-
- name: "Reference Architectures"
59-
description: "Launch effortlessly with our turnkey reference architectures, built either by your team or ours."
60-
url: "https://cloudposse.com/"
61-
151+
- name: Latest Release
152+
image: https://img.shields.io/github/release/cloudposse-terraform-components/aws-datadog-logs-archive.svg?style=for-the-badge
153+
url: https://github.com/cloudposse-terraform-components/aws-datadog-logs-archive/releases/latest
154+
- name: Slack Community
155+
image: https://slack.cloudposse.com/for-the-badge.svg
156+
url: https://slack.cloudposse.com
62157
related:
63-
- name: "Cloud Posse Terraform Modules"
64-
description: Our collection of reusable Terraform modules used by our reference architectures.
65-
url: "https://docs.cloudposse.com/modules/"
66-
- name: "Atmos"
67-
description: "Atmos is like docker-compose but for your infrastructure"
68-
url: "https://atmos.tools"
69-
158+
- name: "Cloud Posse Terraform Modules"
159+
description: Our collection of reusable Terraform modules used by our reference architectures.
160+
url: "https://docs.cloudposse.com/modules/"
161+
- name: "Atmos"
162+
description: "Atmos is like docker-compose but for your infrastructure"
163+
url: "https://atmos.tools"
70164
contributors: [] # If included generates contribs

0 commit comments

Comments
 (0)