Skip to content

Commit d29a3f9

Browse files
author
sourabh
committed
adding cloudwatchlogs forwarder.
1 parent 187b2e7 commit d29a3f9

File tree

12 files changed

+558
-0
lines changed

12 files changed

+558
-0
lines changed

‎aws/cloudwatchlogsforwarder/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# SumoLogic-AWS-CloudWatchLogsForwarder
2+
3+
This module is used to create the SumoLogic AWS HTTP source to collect AWS CloudWatch logs. Features include
4+
- Create AWS resources to setup IAM Roles, SQS, SNS, Metric Alarm, Lambda functions.
5+
- Create Sumo Logic hosted collector. Existing collector can also be used.
6+
- Create Sumo Logic HTTP Source for logs.
7+
- Auto enable logs subscription for Existing and New log groups after installing the module.
8+
9+
## Requirements
10+
11+
| Name | Version |
12+
|------|---------|
13+
| terraform | >= 0.13.0 |
14+
| aws | >= 3.42.0 |
15+
| random | >= 3.1.0 |
16+
| sumologic | >= 2.9.0 |
17+
18+
## Providers
19+
20+
| Name | Version |
21+
|------|---------|
22+
| aws | >= 3.42.0 |
23+
| random | >= 3.1.0 |
24+
| sumologic | >= 2.9.0 |
25+
26+
## Inputs
27+
28+
| Name | Description | Type | Default | Required |
29+
|------|-------------|------|---------|:--------:|
30+
| auto\_enable\_logs\_subscription | New - Automatically subscribes new log groups to send logs to Sumo Logic.<br> Existing - Automatically subscribes existing log groups to send logs to Sumo Logic.<br> Both - Automatically subscribes new and existing log groups.<br> None - Skips Automatic subscription. | `string` | `"Both"` | no |
31+
| auto\_enable\_logs\_subscription\_options | filter - Enter regex for matching logGroups. Regex will check for the name. Visit https://help.sumologic.com/03Send-Data/Collect-from-Other-Data-Sources/Auto-Subscribe_AWS_Log_Groups_to_a_Lambda_Function#Configuring_parameters | <pre>object({<br> filter = string<br> })</pre> | <pre>{<br> "filter": "lambda"<br>}</pre> | no |
32+
| collector\_details | Provide details for the Sumo Logic collector. If not provided, then defaults will be used. | <pre>object({<br> collector_name = string<br> description = string<br> fields = map(string)<br> })</pre> | <pre>{<br> "collector_name": "SumoLogic CloudWatch Logs Collector <Random ID>",<br> "description": "This collector is created using Sumo Logic terraform AWS CloudWatch Logs forwarder to collect AWS cloudwatch logs.",<br> "fields": {}<br>}</pre> | no |
33+
| create\_collector | Provide "true" if you would like to create the Sumo Logic Collector. | `bool` | n/a | yes |
34+
| email\_id | Email for receiving alerts. A confirmation email is sent after the deployment is complete. It can be confirmed to subscribe for alerts. | `string` | `"[email protected]"` | no |
35+
| include\_log\_group\_info | Enable loggroup/logstream values in logs. | `bool` | `true` | no |
36+
| log\_format | Service for Cloudwatch logs source. | `string` | `"Others"` | no |
37+
| log\_stream\_prefix | LogStream name prefixes to filter by logStream. Please note this is separate from a logGroup. This is used only to send certain logStreams within a Cloudwatch logGroup(s). LogGroups still need to be subscribed to the created Lambda function regardless of this input value. | `list(string)` | `[]` | no |
38+
| source\_details | Provide details for the Sumo Logic HTTP source. If not provided, then defaults will be used. | <pre>object({<br> source_name = string<br> source_category = string<br> collector_id = string<br> description = string<br> fields = map(string)<br> })</pre> | <pre>{<br> "collector_id": "",<br> "description": "This source is created using Sumo Logic terraform AWS CloudWatch Logs forwarder to collect AWS cloudwatch logs.",<br> "fields": {},<br> "source_category": "Labs/aws/cloudwatch",<br> "source_name": "CloudWatch Logs Source"<br>}</pre> | no |
39+
| workers | Number of lambda function invocations for Cloudwatch logs source Dead Letter Queue processing. | `number` | `4` | no |
40+
41+
## Outputs
42+
43+
| Name | Description |
44+
|------|-------------|
45+
| aws\_cloudwatch\_log\_group | AWS Log group created to attach to the lambda function. |
46+
| aws\_cloudwatch\_metric\_alarm | AWS CLoudWatch metric alarm. |
47+
| aws\_cw\_lambda\_function | AWS Lambda fucntion to send logs to Sumo Logic. |
48+
| aws\_iam\_role | AWS IAM role with permission to setup lambda. |
49+
| aws\_serverlessapplicationrepository\_cloudformation\_stack | AWS CloudFormation stack for Auto Enable logs subscription. |
50+
| aws\_sns\_topic | AWS SNS topic |
51+
| aws\_sqs\_queue | AWS SQS queue to Store the Failed data. |
52+
| random\_string | Random String value created. |
53+
| sumologic\_collector | Sumo Logic hosted collector. |
54+
| sumologic\_source | Sumo Logic HTTP source. |

‎aws/cloudwatchlogsforwarder/cloudwatchlogsforwarder.tf‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# *************** Steps are as Below to create Sumo Logic Kinesis Firehose for Logs source *************** #
2+
# 1. Create AWS S3 Bucket and use an existing bucket as provided in the inputs.
3+
# 2. Create Log Groups and Log Streams to attach to the kinesis firehose delivery stream.
4+
# 3. Create IAM roles and IAM policies to attach to Kinesis Firehose and delivery stream.
5+
# 4. Create a Kinesis Firehose delivery stream.
6+
# 5. Create subscription for log group.
7+
8+
resource "random_string" "aws_random" {
9+
length = 10
10+
special = false
11+
upper = false
12+
}
13+
14+
resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
15+
name = "SumoCWLogGroup-${random_string.aws_random.id}"
16+
retention_in_days = 7
17+
}
18+
19+
resource "aws_cloudwatch_log_subscription_filter" "cloudwatch_log_subscription_filter" {
20+
destination_arn = aws_lambda_function.logs_lambda_function.arn
21+
filter_pattern = ""
22+
log_group_name = aws_cloudwatch_log_group.cloudwatch_log_group.name
23+
name = "SumoCWLogSubscriptionFilter"
24+
}
25+
26+
resource "aws_lambda_permission" "logs_lambda_invoke_permission" {
27+
action = "lambda:InvokeFunction"
28+
function_name = aws_lambda_function.logs_lambda_function.function_name
29+
principal = "logs.${local.aws_region}.amazonaws.com"
30+
source_account = local.aws_account_id
31+
}
32+
33+
resource "aws_sqs_queue" "sqs_queue" {
34+
name = "SumoCWDeadLetterQueue-${random_string.aws_random.id}"
35+
}
36+
37+
resource "aws_iam_role" "lambda_iam_role" {
38+
name = "SumoCWLambdaExecutionRole-${random_string.aws_random.id}"
39+
path = "/"
40+
41+
assume_role_policy = templatefile("${path.module}/templates/logs_assume_role.tmpl", {})
42+
}
43+
44+
resource "aws_iam_policy" "lambda_sqs_policy" {
45+
name = "SQSCreateLogs-${random_string.aws_random.id}"
46+
policy = templatefile("${path.module}/templates/lambda_sqs.tmpl", {
47+
DEAD_LETTER_QUEUE_ARN = aws_sqs_queue.sqs_queue.arn
48+
})
49+
}
50+
51+
resource "aws_iam_role_policy_attachment" "lambda_sqs_policy_attachment" {
52+
role = aws_iam_role.lambda_iam_role.name
53+
policy_arn = aws_iam_policy.lambda_sqs_policy.arn
54+
}
55+
56+
resource "aws_iam_policy" "create_logs_policy" {
57+
name = "CloudWatchCreateLogs-${random_string.aws_random.id}"
58+
policy = templatefile("${path.module}/templates/lambda_logs.tmpl", {
59+
LOG_GROUP_ARN = aws_cloudwatch_log_group.cloudwatch_log_group.arn
60+
})
61+
}
62+
63+
resource "aws_iam_role_policy_attachment" "create_logs_policy_attachment" {
64+
role = aws_iam_role.lambda_iam_role.name
65+
policy_arn = aws_iam_policy.create_logs_policy.arn
66+
}
67+
68+
resource "aws_iam_policy" "invoke_lambda_policy" {
69+
name = "InvokeLambda-${random_string.aws_random.id}"
70+
policy = templatefile("${path.module}/templates/invoke_lambda.tmpl", {
71+
LAMBDA_ARN = aws_lambda_function.process_dead_letter_queue_lambda.arn
72+
})
73+
}
74+
75+
resource "aws_iam_role_policy_attachment" "invoke_lambda_policy_attachment" {
76+
role = aws_iam_role.lambda_iam_role.name
77+
policy_arn = aws_iam_policy.invoke_lambda_policy.arn
78+
}
79+
80+
resource "aws_lambda_function" "logs_lambda_function" {
81+
function_name = "SumoCWLogsLambda-${random_string.aws_random.id}"
82+
handler = "cloudwatchlogs_lambda.handler"
83+
runtime = "nodejs14.x"
84+
role = aws_iam_role.lambda_iam_role.arn
85+
s3_bucket = "appdevzipfiles-${local.aws_region}"
86+
s3_key = "cloudwatchlogs-with-dlq.zip"
87+
timeout = 300
88+
memory_size = 128
89+
dead_letter_config {
90+
target_arn = aws_sqs_queue.sqs_queue.arn
91+
}
92+
93+
environment {
94+
variables = {
95+
"SUMO_ENDPOINT" = sumologic_http_source.source.url,
96+
"LOG_FORMAT" = var.log_format,
97+
"INCLUDE_LOG_INFO" = var.include_log_group_info,
98+
"LOG_STREAM_PREFIX" = join(",", var.log_stream_prefix)
99+
}
100+
}
101+
}
102+
103+
resource "aws_lambda_function" "process_dead_letter_queue_lambda" {
104+
function_name = "SumoCWProcessDLQLambda-${random_string.aws_random.id}"
105+
handler = "DLQProcessor.handler"
106+
runtime = "nodejs14.x"
107+
role = aws_iam_role.lambda_iam_role.arn
108+
s3_bucket = "appdevzipfiles-${local.aws_region}"
109+
s3_key = "cloudwatchlogs-with-dlq.zip"
110+
timeout = 300
111+
memory_size = 128
112+
dead_letter_config {
113+
target_arn = aws_sqs_queue.sqs_queue.arn
114+
}
115+
116+
environment {
117+
variables = {
118+
"SUMO_ENDPOINT" = sumologic_http_source.source.url,
119+
"LOG_FORMAT" = var.log_format,
120+
"INCLUDE_LOG_INFO" = var.include_log_group_info,
121+
"LOG_STREAM_PREFIX" = join(",", var.log_stream_prefix),
122+
"TASK_QUEUE_URL" = aws_sqs_queue.sqs_queue.arn,
123+
"NUM_OF_WORKERS" = var.workers
124+
}
125+
}
126+
}
127+
128+
resource "aws_lambda_permission" "process_dead_letter_queue_lambda_permission" {
129+
action = "lambda:InvokeFunction"
130+
function_name = aws_lambda_function.process_dead_letter_queue_lambda.function_name
131+
principal = "events.amazonaws.com"
132+
source_arn = aws_cloudwatch_event_rule.process_dead_letter_queue_event_rule.arn
133+
}
134+
135+
resource "aws_cloudwatch_event_rule" "process_dead_letter_queue_event_rule" {
136+
description = "Events rule for Cron"
137+
schedule_expression = "rate(5 minutes)"
138+
is_enabled = true
139+
}
140+
141+
resource "aws_cloudwatch_event_target" "process_dead_letter_queue_event_rule_target" {
142+
arn = aws_lambda_function.process_dead_letter_queue_lambda.arn
143+
rule = aws_cloudwatch_event_rule.process_dead_letter_queue_event_rule.name
144+
target_id = "TargetFunctionV1"
145+
}
146+
147+
resource "aws_sns_topic" "sns_topic" {
148+
name = "SumoCWEmailSNSTopic-${random_string.aws_random.id}"
149+
}
150+
151+
resource "aws_sns_topic_subscription" "sns_topic_subscription" {
152+
endpoint = var.email_id
153+
protocol = "email"
154+
topic_arn = aws_sns_topic.sns_topic.arn
155+
}
156+
157+
resource "aws_cloudwatch_metric_alarm" "metric_alarm" {
158+
alarm_actions = [aws_sns_topic.sns_topic.arn]
159+
alarm_description = "Notify via email if number of messages in DeadLetterQueue exceeds threshold"
160+
alarm_name = "SumoCWSpilloverAlarm-${random_string.aws_random.id}"
161+
comparison_operator = "GreaterThanThreshold"
162+
dimensions = { "QueueName" = aws_sqs_queue.sqs_queue.name }
163+
evaluation_periods = 1
164+
metric_name = "ApproximateNumberOfMessagesVisible"
165+
namespace = "AWS/SQS"
166+
period = 3600
167+
statistic = "Sum"
168+
threshold = 100000
169+
}
170+
171+
resource "sumologic_collector" "collector" {
172+
for_each = toset(var.create_collector ? ["collector"] : [])
173+
name = local.collector_name
174+
description = var.collector_details.description
175+
fields = var.collector_details.fields
176+
timezone = "UTC"
177+
}
178+
179+
resource "sumologic_http_source" "source" {
180+
name = var.source_details.source_name
181+
description = var.source_details.description
182+
category = var.source_details.source_category
183+
collector_id = var.create_collector ? sumologic_collector.collector["collector"].id : var.source_details.collector_id
184+
fields = var.source_details.fields
185+
}
186+
187+
# Reason to use the SAM app, is to have single source of truth for Auto Subscribe functionality.
188+
resource "aws_serverlessapplicationrepository_cloudformation_stack" "auto_enable_logs_subscription" {
189+
for_each = toset(local.auto_enable_logs_subscription ? ["auto_enable_logs_subscription"] : [])
190+
191+
name = "Auto-Enable-Logs-Subscription-${random_string.aws_random.id}"
192+
application_id = "arn:aws:serverlessrepo:us-east-1:956882708938:applications/sumologic-loggroup-connector"
193+
semantic_version = "1.0.5"
194+
capabilities = data.aws_serverlessapplicationrepository_application.app.required_capabilities
195+
parameters = {
196+
DestinationArnType = "Lambda"
197+
DestinationArnValue = aws_lambda_function.logs_lambda_function.arn
198+
LogGroupPattern = var.auto_enable_logs_subscription_options.filter
199+
UseExistingLogs = local.auto_enable_existing
200+
}
201+
}

‎aws/cloudwatchlogsforwarder/data.tf‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
data "aws_region" "current" {}
2+
3+
data "aws_caller_identity" "current" {}
4+
5+
data "sumologic_caller_identity" "current" {}
6+
7+
data "aws_serverlessapplicationrepository_application" "app" {
8+
application_id = "arn:aws:serverlessrepo:us-east-1:956882708938:applications/sumologic-loggroup-connector"
9+
semantic_version = "1.0.5"
10+
}

‎aws/cloudwatchlogsforwarder/locals.tf‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
locals {
2+
3+
aws_account_id = data.aws_caller_identity.current.account_id
4+
5+
aws_region = data.aws_region.current.id
6+
7+
# Get the default collector name if no collector name is provided.
8+
collector_name = var.collector_details.collector_name == "SumoLogic CloudWatch Logs Collector <Random ID>" ? "SumoLogic CloudWatch Logs Collector ${random_string.aws_random.id}" : var.collector_details.collector_name
9+
10+
# Auto enable should be called if input is anything other than None.
11+
auto_enable_logs_subscription = var.auto_enable_logs_subscription != "None" ? true : false
12+
13+
# Existing
14+
auto_enable_existing = var.auto_enable_logs_subscription == "Existing" || var.auto_enable_logs_subscription == "Both" ? true : false
15+
}

‎aws/cloudwatchlogsforwarder/outputs.tf‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
output "random_string" {
2+
value = random_string.aws_random
3+
description = "Random String value created."
4+
}
5+
6+
output "aws_sqs_queue" {
7+
value = aws_sqs_queue.sqs_queue
8+
description = "AWS SQS queue to Store the Failed data."
9+
}
10+
11+
output "aws_cloudwatch_log_group" {
12+
value = aws_cloudwatch_log_group.cloudwatch_log_group
13+
description = "AWS Log group created to attach to the lambda function."
14+
}
15+
16+
output "aws_iam_role" {
17+
value = aws_iam_role.lambda_iam_role
18+
description = "AWS IAM role with permission to setup lambda."
19+
}
20+
21+
output "aws_sns_topic" {
22+
value = aws_sns_topic.sns_topic
23+
description = "AWS SNS topic"
24+
}
25+
26+
output "aws_cloudwatch_metric_alarm" {
27+
value = aws_cloudwatch_metric_alarm.metric_alarm
28+
description = "AWS CLoudWatch metric alarm."
29+
}
30+
31+
output "aws_cw_lambda_function" {
32+
value = aws_lambda_function.logs_lambda_function
33+
description = "AWS Lambda fucntion to send logs to Sumo Logic."
34+
}
35+
36+
output "sumologic_collector" {
37+
value = var.create_collector ? sumologic_collector.collector : {}
38+
description = "Sumo Logic hosted collector."
39+
}
40+
41+
output "sumologic_source" {
42+
value = sumologic_http_source.source
43+
description = "Sumo Logic HTTP source."
44+
}
45+
46+
output "aws_serverlessapplicationrepository_cloudformation_stack" {
47+
value = local.auto_enable_logs_subscription ? aws_serverlessapplicationrepository_cloudformation_stack.auto_enable_logs_subscription : {}
48+
description = "AWS CloudFormation stack for Auto Enable logs subscription."
49+
}

‎aws/cloudwatchlogsforwarder/templates/invoke_lambda.tmpl‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [
4+
{
5+
"Effect": "Allow",
6+
"Action": "lambda:InvokeFunction",
7+
"Resource": "${LAMBDA_ARN}"
8+
}
9+
]
10+
}

‎aws/cloudwatchlogsforwarder/templates/lambda_logs.tmpl‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [
4+
{
5+
"Effect": "Allow",
6+
"Action": [
7+
"logs:CreateLogGroup",
8+
"logs:CreateLogStream",
9+
"logs:PutLogEvents",
10+
"logs:DescribeLogStreams"
11+
],
12+
"Resource": "${LOG_GROUP_ARN}"
13+
}
14+
]
15+
}

‎aws/cloudwatchlogsforwarder/templates/lambda_sqs.tmpl‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [
4+
{
5+
"Effect": "Allow",
6+
"Action": [
7+
"sqs:DeleteMessage",
8+
"sqs:GetQueueUrl",
9+
"sqs:ListQueues",
10+
"sqs:ChangeMessageVisibility",
11+
"sqs:SendMessageBatch",
12+
"sqs:ReceiveMessage",
13+
"sqs:SendMessage",
14+
"sqs:GetQueueAttributes",
15+
"sqs:ListQueueTags",
16+
"sqs:ListDeadLetterSourceQueues",
17+
"sqs:DeleteMessageBatch",
18+
"sqs:PurgeQueue",
19+
"sqs:DeleteQueue",
20+
"sqs:CreateQueue",
21+
"sqs:ChangeMessageVisibilityBatch",
22+
"sqs:SetQueueAttributes"
23+
],
24+
"Resource": "${DEAD_LETTER_QUEUE_ARN}"
25+
}
26+
]
27+
}

0 commit comments

Comments
 (0)