Skip to content

Commit c9ec9ae

Browse files
ViBiOhViBiOh
committed
feat(aws): AWSX-1566 Adding new variable in the cloudformation template
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
1 parent 5d82d44 commit c9ec9ae

14 files changed

+38
-306
lines changed

‎aws/logs_monitoring/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -568,16 +568,16 @@ To test different patterns against your logs, turn on [debug logs](#troubleshoot
568568
### Advanced (optional)
569569
570570
`DD_ENRICH_S3_TAGS`
571-
: Instruct Datadog Backend to enrich a log coming from a S3 bucket with the tag attached to this bucket. It's the equivalent behavior of `DD_FETCH_S3_TAG` but done after ingestion. This require Resource Collection to be enabled. Enabled by default.
571+
: Instruct Datadog backend to enrich a log coming from a S3 bucket with the tag attached to this bucket. It's the equivalent behavior of `DD_FETCH_S3_TAG` but done after ingestion. This require Resource Collection to be enabled. Flag is enabled by default.
572572

573573
`DD_ENRICH_CLOUDWATCH_TAGS`
574-
: Instruct Datadog Backend to enrich a log coming from a Cloudwatch logGroup with the tag attached to this logGroup. It's the equivalent behavior of `DD_FETCH_LOG_GROUP_TAGS` but done after ingestion. This require Resource Collection to be enabled.
574+
: Instruct Datadog backend to enrich a log coming from a Cloudwatch logGroup with the tag attached to this logGroup. It's the equivalent behavior of `DD_FETCH_LOG_GROUP_TAGS` but done after ingestion. This require Resource Collection to be enabled. Flag is enabled by default.
575575
576576
`DD_FETCH_LAMBDA_TAGS`
577577
: Let the Forwarder fetch Lambda tags using GetResources API calls and apply them to logs, metrics, and traces. If set to true, permission `tag:GetResources` will be automatically added to the Lambda execution IAM role.
578578
579579
`DD_FETCH_LOG_GROUP_TAGS`
580-
: [DEPRECATED, use DD_ENRICH_CLOUDWATCH_TAG] Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics, and traces. If set to true, permission `logs:ListTagsForResource` will be automatically added to the Lambda execution IAM role.
580+
: [DEPRECATED, use DD_ENRICH_CLOUDWATCH_TAGS] Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics, and traces. If set to true, permission `logs:ListTagsForResource` will be automatically added to the Lambda execution IAM role.
581581
582582
`DD_FETCH_STEP_FUNCTIONS_TAGS`
583583
: Let the Forwarder fetch Step Functions tags using GetResources API calls and apply them to logs and traces (if Step Functions tracing is enabled). If set to true, permission `tag:GetResources` will be automatically added to the Lambda execution IAM role.

‎aws/logs_monitoring/settings.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -253,7 +253,7 @@ def __init__(self, name, pattern, placeholder, enabled=True):
253253
DD_ENRICH_S3_TAGS = get_env_var("DD_ENRICH_S3_TAGS", default="true", boolean=True)
254254

255255
DD_ENRICH_CLOUDWATCH_TAGS = get_env_var(
256-
"DD_ENRICH_CLOUDWATCH_TAGS", default="false", boolean=True
256+
"DD_ENRICH_CLOUDWATCH_TAGS", default="true", boolean=True
257257
)
258258

259259

‎aws/logs_monitoring/template.yaml‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,20 @@ Parameters:
7575
Type: String
7676
Default: ""
7777
Description: Add custom tags to forwarded logs, comma-delimited string, no trailing comma, e.g., env:prod,stack:classic
78+
DdEnrichS3Tags:
79+
Type: String
80+
Default: true
81+
AllowedValues:
82+
- true
83+
- false
84+
Description: Instruct Datadog backend to enrich a log coming from a S3 bucket with the tag attached to this bucket. Datadog AWS Resource Collection needs to be enabled.
85+
DdEnrichCloudwatchTags:
86+
Type: String
87+
Default: true
88+
AllowedValues:
89+
- true
90+
- false
91+
Description: Instruct Datadog backend to enrich a log coming from a Cloudwatch logGroup with the tag attached to this logGroup. Datadog AWS Resource Collection needs to be enabled.
7892
DdFetchLambdaTags:
7993
Type: String
8094
Default: true
@@ -88,7 +102,7 @@ Parameters:
88102
AllowedValues:
89103
- true
90104
- false
91-
Description: Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics and traces. If set to true, permission logs:ListTagsLogGroup will be automatically added to the Lambda execution IAM role. The tags are cached in memory and S3 so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.list_tags_log_group_api_call metric for each API call made.
105+
Description: (DEPRECATED in favor of DdEnrichCloudwatchTags) Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics and traces. If set to true, permission logs:ListTagsLogGroup will be automatically added to the Lambda execution IAM role. The tags are cached in memory and S3 so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.list_tags_log_group_api_call metric for each API call made.
92106
DdFetchStepFunctionsTags:
93107
Type: String
94108
Default: true
@@ -98,11 +112,11 @@ Parameters:
98112
Description: Let the forwarder fetch Step Functions tags using GetResources API calls and apply them to logs, metrics and traces. If set to true, permission tag:GetResources will be automatically added to the Lambda execution IAM role. The tags are cached in memory and S3 so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.get_resources_api_calls metric for each API call made.
99113
DdFetchS3Tags:
100114
Type: String
101-
Default: true
115+
Default: false
102116
AllowedValues:
103117
- true
104118
- false
105-
Description: Let the forwarder fetch S3 buckets tags using GetResources API calls and apply them to S3 based logs. If set to true, permission tag:GetResources will be automatically added to the Lambda execution IAM role. The tags are cached in memory and S3 so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.get_resources_api_calls metric for each API call made.
119+
Description: (DEPRECATED in favor of DdEnrichS3Tags) Let the forwarder fetch S3 buckets tags using GetResources API calls and apply them to S3 based logs. If set to true, permission tag:GetResources will be automatically added to the Lambda execution IAM role. The tags are cached in memory and S3 so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.get_resources_api_calls metric for each API call made.
106120
DdNoSsl:
107121
Type: String
108122
Default: false
@@ -448,11 +462,13 @@ Resources:
448462
- !Ref DdTags
449463
- !Ref AWS::NoValue
450464
DD_TAGS_CACHE_TTL_SECONDS: !Ref TagsCacheTTLSeconds
465+
DD_ENRICH_S3_TAGS: !Ref DdEnrichS3Tags
466+
DD_ENRICH_CLOUDWATCH_TAGS: !Ref DdEnrichCloudwatchTags
467+
DD_FETCH_S3_TAGS: !Ref DdFetchS3Tags
451468
DD_FETCH_LAMBDA_TAGS: !If
452469
- SetDdFetchLambdaTags
453470
- !Ref DdFetchLambdaTags
454471
- !Ref AWS::NoValue
455-
DD_FETCH_S3_TAGS: !Ref DdFetchS3Tags
456472
DD_FETCH_LOG_GROUP_TAGS: !If
457473
- SetDdFetchLogGroupTags
458474
- !Ref DdFetchLogGroupTags
@@ -1018,6 +1034,8 @@ Metadata:
10181034
- Label:
10191035
default: Advanced (Optional)
10201036
Parameters:
1037+
- DdEnrichS3Tags
1038+
- DdEnrichCloudwatchTags
10211039
- DdFetchLambdaTags
10221040
- DdFetchLogGroupTags
10231041
- DdFetchStepFunctionsTags

‎aws/logs_monitoring/tools/integration_tests/docker-compose.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,9 @@ services:
4040
DD_USE_COMPRESSION: "false"
4141
DD_ADDITIONAL_TARGET_LAMBDAS: "${EXTERNAL_LAMBDAS}"
4242
DD_S3_BUCKET_NAME: "${DD_S3_BUCKET_NAME}"
43-
DD_FETCH_LAMBDA_TAGS: "true"
44-
DD_FETCH_LOG_GROUP_TAGS: "true"
45-
DD_FETCH_STEP_FUNCTIONS_TAGS: "false" # intentionally set false to allow integration test for step function logs to run without hitting aws
43+
DD_FETCH_LAMBDA_TAGS: "${DD_FETCH_LAMBDA_TAGS:-false}"
44+
DD_FETCH_LOG_GROUP_TAGS: "${DD_FETCH_LOG_GROUP_TAGS:-false}"
45+
DD_FETCH_STEP_FUNCTIONS_TAGS: "${DD_FETCH_STEP_FUNCTIONS_TAGS:-false}"
4646
DD_STORE_FAILED_EVENTS: "false"
4747
DD_TRACE_ENABLED: "true"
4848
expose:

‎aws/logs_monitoring/tools/integration_tests/integration_tests.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
# Unless explicitly stated otherwise all files in this repository are licensed
44
# under the Apache License Version 2.0.

‎aws/logs_monitoring/tools/integration_tests/snapshots/cloudwatch_customized_log_group_lambda_invocation.json~snapshot‎

Lines changed: 1 addition & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@
7575
"DD-API-KEY": "abcdefghijklmnopqrstuvwxyz012345",
7676
"DD-EVP-ORIGIN": "aws_forwarder",
7777
"DD-EVP-ORIGIN-VERSION": "<redacted from snapshot>",
78+
"DD-STORAGE-TAG": "s3,cloudwatch",
7879
"Host": "recorder:8080",
7980
"User-Agent": "<redacted from snapshot>",
8081
"traceparent": "<redacted from snapshot>",
@@ -90,48 +91,6 @@
9091
{
9192
"data": {
9293
"series": [
93-
{
94-
"device": null,
95-
"host": null,
96-
"interval": 10,
97-
"metric": "aws.dd_forwarder.loggroup_cache_fetch_failure",
98-
"points": "<redacted from snapshot>",
99-
"tags": [
100-
"forwardername:test_function",
101-
"forwarder_memorysize:3008",
102-
"forwarder_version:<redacted from snapshot>",
103-
"event_type:awslogs"
104-
],
105-
"type": "distribution"
106-
},
107-
{
108-
"device": null,
109-
"host": null,
110-
"interval": 10,
111-
"metric": "aws.dd_forwarder.list_tags_log_group_api_call",
112-
"points": "<redacted from snapshot>",
113-
"tags": [
114-
"forwardername:test_function",
115-
"forwarder_memorysize:3008",
116-
"forwarder_version:<redacted from snapshot>",
117-
"event_type:awslogs"
118-
],
119-
"type": "distribution"
120-
},
121-
{
122-
"device": null,
123-
"host": null,
124-
"interval": 10,
125-
"metric": "aws.dd_forwarder.loggroup_cache_write_failure",
126-
"points": "<redacted from snapshot>",
127-
"tags": [
128-
"forwardername:test_function",
129-
"forwarder_memorysize:3008",
130-
"forwarder_version:<redacted from snapshot>",
131-
"event_type:awslogs"
132-
],
133-
"type": "distribution"
134-
},
13594
{
13695
"device": null,
13796
"host": null,

‎aws/logs_monitoring/tools/integration_tests/snapshots/cloudwatch_log.json~snapshot‎

Lines changed: 1 addition & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
"DD-API-KEY": "abcdefghijklmnopqrstuvwxyz012345",
4949
"DD-EVP-ORIGIN": "aws_forwarder",
5050
"DD-EVP-ORIGIN-VERSION": "<redacted from snapshot>",
51+
"DD-STORAGE-TAG": "s3,cloudwatch",
5152
"Host": "recorder:8080",
5253
"User-Agent": "<redacted from snapshot>",
5354
"traceparent": "<redacted from snapshot>",
@@ -63,48 +64,6 @@
6364
{
6465
"data": {
6566
"series": [
66-
{
67-
"device": null,
68-
"host": null,
69-
"interval": 10,
70-
"metric": "aws.dd_forwarder.loggroup_cache_fetch_failure",
71-
"points": "<redacted from snapshot>",
72-
"tags": [
73-
"forwardername:test_function",
74-
"forwarder_memorysize:3008",
75-
"forwarder_version:<redacted from snapshot>",
76-
"event_type:awslogs"
77-
],
78-
"type": "distribution"
79-
},
80-
{
81-
"device": null,
82-
"host": null,
83-
"interval": 10,
84-
"metric": "aws.dd_forwarder.list_tags_log_group_api_call",
85-
"points": "<redacted from snapshot>",
86-
"tags": [
87-
"forwardername:test_function",
88-
"forwarder_memorysize:3008",
89-
"forwarder_version:<redacted from snapshot>",
90-
"event_type:awslogs"
91-
],
92-
"type": "distribution"
93-
},
94-
{
95-
"device": null,
96-
"host": null,
97-
"interval": 10,
98-
"metric": "aws.dd_forwarder.loggroup_cache_write_failure",
99-
"points": "<redacted from snapshot>",
100-
"tags": [
101-
"forwardername:test_function",
102-
"forwarder_memorysize:3008",
103-
"forwarder_version:<redacted from snapshot>",
104-
"event_type:awslogs"
105-
],
106-
"type": "distribution"
107-
},
10867
{
10968
"device": null,
11069
"host": null,

‎aws/logs_monitoring/tools/integration_tests/snapshots/cloudwatch_log_cloudtrail.json~snapshot‎

Lines changed: 1 addition & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@
8484
"DD-API-KEY": "abcdefghijklmnopqrstuvwxyz012345",
8585
"DD-EVP-ORIGIN": "aws_forwarder",
8686
"DD-EVP-ORIGIN-VERSION": "<redacted from snapshot>",
87+
"DD-STORAGE-TAG": "s3,cloudwatch",
8788
"Host": "recorder:8080",
8889
"User-Agent": "<redacted from snapshot>",
8990
"traceparent": "<redacted from snapshot>",
@@ -99,48 +100,6 @@
99100
{
100101
"data": {
101102
"series": [
102-
{
103-
"device": null,
104-
"host": null,
105-
"interval": 10,
106-
"metric": "aws.dd_forwarder.loggroup_cache_fetch_failure",
107-
"points": "<redacted from snapshot>",
108-
"tags": [
109-
"forwardername:test_function",
110-
"forwarder_memorysize:3008",
111-
"forwarder_version:<redacted from snapshot>",
112-
"event_type:awslogs"
113-
],
114-
"type": "distribution"
115-
},
116-
{
117-
"device": null,
118-
"host": null,
119-
"interval": 10,
120-
"metric": "aws.dd_forwarder.list_tags_log_group_api_call",
121-
"points": "<redacted from snapshot>",
122-
"tags": [
123-
"forwardername:test_function",
124-
"forwarder_memorysize:3008",
125-
"forwarder_version:<redacted from snapshot>",
126-
"event_type:awslogs"
127-
],
128-
"type": "distribution"
129-
},
130-
{
131-
"device": null,
132-
"host": null,
133-
"interval": 10,
134-
"metric": "aws.dd_forwarder.loggroup_cache_write_failure",
135-
"points": "<redacted from snapshot>",
136-
"tags": [
137-
"forwardername:test_function",
138-
"forwarder_memorysize:3008",
139-
"forwarder_version:<redacted from snapshot>",
140-
"event_type:awslogs"
141-
],
142-
"type": "distribution"
143-
},
144103
{
145104
"device": null,
146105
"host": null,

‎aws/logs_monitoring/tools/integration_tests/snapshots/cloudwatch_log_coldstart.json~snapshot‎

Lines changed: 0 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -3,48 +3,6 @@
33
{
44
"data": {
55
"series": [
6-
{
7-
"device": null,
8-
"host": null,
9-
"interval": 10,
10-
"metric": "aws.dd_forwarder.loggroup_cache_fetch_failure",
11-
"points": "<redacted from snapshot>",
12-
"tags": [
13-
"forwardername:test_function",
14-
"forwarder_memorysize:3008",
15-
"forwarder_version:<redacted from snapshot>",
16-
"event_type:awslogs"
17-
],
18-
"type": "distribution"
19-
},
20-
{
21-
"device": null,
22-
"host": null,
23-
"interval": 10,
24-
"metric": "aws.dd_forwarder.list_tags_log_group_api_call",
25-
"points": "<redacted from snapshot>",
26-
"tags": [
27-
"forwardername:test_function",
28-
"forwarder_memorysize:3008",
29-
"forwarder_version:<redacted from snapshot>",
30-
"event_type:awslogs"
31-
],
32-
"type": "distribution"
33-
},
34-
{
35-
"device": null,
36-
"host": null,
37-
"interval": 10,
38-
"metric": "aws.dd_forwarder.loggroup_cache_write_failure",
39-
"points": "<redacted from snapshot>",
40-
"tags": [
41-
"forwardername:test_function",
42-
"forwarder_memorysize:3008",
43-
"forwarder_version:<redacted from snapshot>",
44-
"event_type:awslogs"
45-
],
46-
"type": "distribution"
47-
},
486
{
497
"device": null,
508
"host": null,

‎aws/logs_monitoring/tools/integration_tests/snapshots/cloudwatch_log_custom_tags.json~snapshot‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
"DD-API-KEY": "abcdefghijklmnopqrstuvwxyz012345",
3131
"DD-EVP-ORIGIN": "aws_forwarder",
3232
"DD-EVP-ORIGIN-VERSION": "<redacted from snapshot>",
33+
"DD-STORAGE-TAG": "s3,cloudwatch",
3334
"Host": "recorder:8080",
3435
"User-Agent": "<redacted from snapshot>",
3536
"traceparent": "<redacted from snapshot>",
@@ -45,20 +46,6 @@
4546
{
4647
"data": {
4748
"series": [
48-
{
49-
"device": null,
50-
"host": null,
51-
"interval": 10,
52-
"metric": "aws.dd_forwarder.loggroup_local_cache_hit",
53-
"points": "<redacted from snapshot>",
54-
"tags": [
55-
"forwardername:test_function",
56-
"forwarder_memorysize:3008",
57-
"forwarder_version:<redacted from snapshot>",
58-
"event_type:awslogs"
59-
],
60-
"type": "distribution"
61-
},
6249
{
6350
"device": null,
6451
"host": null,

0 commit comments

Comments
 (0)