Skip to content

Commit f8f65a2

Browse files
MichaelKatsoulisKavindu-Dodantheletterfmlunadia
authored
Chore: Update edot cloud forwarder for AWS documentation (#455)
* Chore: Update edot cloud forwarder for AWS documentation * Add a section for dashboards and remove references to not supported log formats * refine documentation and add error replay section Signed-off-by: Kavindu Dodanduwa <[email protected]> * Set version of ga to v1 * Update format names * Update versions and comment out sar * Apply suggestions from code review Co-authored-by: Miguel Luna <[email protected]> * Update aws.md * Update aws.md --------- Signed-off-by: Kavindu Dodanduwa <[email protected]> Co-authored-by: Kavindu Dodanduwa <[email protected]> Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]> Co-authored-by: Miguel Luna <[email protected]>
1 parent 0df3349 commit f8f65a2

File tree

1 file changed

+187
-22
lines changed
  • docs/reference/edot-cloud-forwarder

1 file changed

+187
-22
lines changed

‎docs/reference/edot-cloud-forwarder/aws.md‎

Lines changed: 187 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,10 @@ products:
2020

2121
{{edot-cf}} for AWS supports the following log sources:
2222

23-
| Log source | Description |
23+
| AWS Service | Telemetry Description |
2424
| --- | --- |
25-
| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) |
26-
| ELB Access | Logs generated by an Elastic Load Balancer (ELB) |
25+
| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic |
26+
| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer |
2727
% | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch |
2828

2929
Read on to learn how to set up {{edot-cf}} for AWS.
@@ -64,6 +64,7 @@ To collect Elastic Load Balancer (ELB) Access logs, you need:
6464
- Access logging enabled, with the bucket as the destination
6565

6666
:::
67+
6768
<!--
6869
:::{tab-item} CloudWatch
6970
@@ -93,8 +94,9 @@ Trim the API key from `Authorization=ApiKey MYKEYVALUE...` to just `MYKEYVALUE..
9394

9495
Before deploying {{edot-cf}} for AWS, keep these points in mind:
9596

96-
- Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Logs. Each CloudFormation stack can only process one log source and format at a time.
97+
- Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
9798
- Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
99+
- The CloudFormation stack deployment region must match the region of the S3 bucket.
98100

99101
## Download the template [download-templates]
100102

@@ -134,12 +136,13 @@ For S3 logs, use the following settings:
134136

135137
| Setting | Description |
136138
| ------------------ | --- |
137-
| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:<br>- `vpc_flow_log`: VPC Flow logs<br>- `elb_access_log`: Elastic Load Balancer (ELB) Access logs<br>- `s3_access_log`: S3 Access logs<br>- `json`: JSON-formatted logs |
138-
| `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_<br>Defines how JSON logs are structured:<br>- `body` _(default)_: Stores logs in the request body<br>- `body_with_inline_attributes`: Logs include inline attributes |
139+
| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:<br>- `vpcflow`: VPC Flow Logs<br>- `elbaccess`: ELB Access logs|
139140
| `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. |
141+
% | `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_<br>Defines how JSON logs are structured:<br>- `body` _(default)_: Stores logs in the request body<br>- `body_with_inline_attributes`: Logs include inline attributes |
140142

141143
::::
142-
<!--
144+
145+
<!-- TODO: Enable when CloudWatch logs are supported
143146
::::{tab-item} CloudWatch
144147
145148
For CloudWatch logs, use the following settings:
@@ -152,30 +155,34 @@ For CloudWatch logs, use the following settings:
152155
The log group must already exist in your AWS account and region. If the ARN points to a non-existent log group, stack deployment or updates might fail.
153156
:::
154157
::::
158+
155159
-->
160+
156161
:::::
157162

158163
### Optional settings
159164

160165
These are optional settings you can set in the CloudFormation template:
161166

162-
| Setting | Description |
163-
| ------------------- | --- |
164-
| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. |
165-
| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.1.5`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. |
166-
| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `1024` MB. Minimum value is `128` MB. Maximum value is `10240` MB. |
167-
| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. |
167+
| Setting | Description |
168+
| ------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
169+
| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. |
170+
| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.2.4`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. |
171+
| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. |
172+
| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. |
173+
| `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size. Default is `50` MB. You may incease this based on the data volume. |
168174

175+
Default values of `EdotCloudForwarderMemorySize` and `EdotCloudForwarderConcurrentExecutions` are sufficient for most use cases. Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. Along with these, you might also need to adjust `EdotCloudForwarderExporterMaxQueueSize` to export higher data volumes.
169176

170177
## Deployment examples
171178

172-
The following examples use the CloudFormation template files hosted in the [public S3 bucket](#download-templates).
179+
The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation. Copy and paste these commands after replacing the placeholder values with your actual configuration.
173180

174181
- Use the `--template-url` flag to reference a template hosted on S3.
175182
- To always use the most recent stable templates, use the `latest` path. For example, `v0/latest`.
176-
- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v0.1.5`.
183+
- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v{{version.edot-cf-aws}}`.
177184

178-
Alternatively, if you have downloaded the template file, you can use the `--template-body file://<path>` option with a local template file.
185+
Alternatively, if you have downloaded the template file, use the `--template-body file://<path>` option with a local template file.
179186

180187
:::::{tab-set}
181188
::::{tab-item} VPC Flow logs
@@ -192,7 +199,7 @@ aws cloudformation create-stack \
192199
ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
193200
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
194201
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
195-
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpc_flow_log"
202+
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow"
196203
```
197204
::::
198205

@@ -210,10 +217,89 @@ aws cloudformation create-stack \
210217
ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \
211218
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
212219
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
213-
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elb_access_log"
220+
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess"
214221
```
215222
::::
216-
<!--
223+
224+
<!-- To be added post GA
225+
226+
::::{tab-item} S3 Access logs
227+
228+
This example deploys a CloudFormation stack to collect S3 Access logs stored in an S3 bucket.
229+
230+
```sh
231+
aws cloudformation create-stack \
232+
--stack-name edot-cloud-forwarder-s3-access \
233+
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
234+
--capabilities CAPABILITY_NAMED_IAM \
235+
--region eu-central-1 \
236+
--parameters \
237+
ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-access-logs-bucket-arn \
238+
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
239+
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
240+
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="s3_access_log"
241+
```
242+
::::
243+
244+
::::{tab-item} CloudTrail logs
245+
246+
This example deploys a CloudFormation stack to collect CloudTrail logs stored in an S3 bucket.
247+
248+
```sh
249+
aws cloudformation create-stack \
250+
--stack-name edot-cloud-forwarder-cloudtrail \
251+
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
252+
--capabilities CAPABILITY_NAMED_IAM \
253+
--region eu-central-1 \
254+
--parameters \
255+
ParameterKey=SourceS3BucketARN,ParameterValue=your-cloudtrail-bucket-arn \
256+
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
257+
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
258+
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="cloudtrail_log"
259+
```
260+
::::
261+
262+
::::{tab-item} WAF logs
263+
264+
This example deploys a CloudFormation stack to collect AWS WAF logs stored in an S3 bucket.
265+
266+
```sh
267+
aws cloudformation create-stack \
268+
--stack-name edot-cloud-forwarder-waf \
269+
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
270+
--capabilities CAPABILITY_NAMED_IAM \
271+
--region eu-central-1 \
272+
--parameters \
273+
ParameterKey=SourceS3BucketARN,ParameterValue=arn:aws:s3:::aws-waf-logs-your-bucket-arn \
274+
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
275+
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
276+
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="waf_log"
277+
```
278+
279+
:::{note}
280+
Replace `aws-waf-logs-your-bucket-name` with your actual WAF logging bucket ARN. Remember that the bucket name must start with `aws-waf-logs-` as required by AWS WAF.
281+
:::
282+
::::
283+
284+
::::{tab-item} JSON logs
285+
286+
This example deploys a CloudFormation stack to collect JSON-formatted logs stored in an S3 bucket.
287+
288+
```sh
289+
aws cloudformation create-stack \
290+
--stack-name edot-cloud-forwarder-json \
291+
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
292+
--capabilities CAPABILITY_NAMED_IAM \
293+
--region eu-central-1 \
294+
--parameters \
295+
ParameterKey=SourceS3BucketARN,ParameterValue=your-json-logs-bucket-arn \
296+
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
297+
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
298+
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="json" \
299+
ParameterKey=S3LogsJsonEncodingMode,ParameterValue="body"
300+
```
301+
::::
302+
217303
::::{tab-item} CloudWatch logs
218304
219305
This example deploys a CloudFormation stack to collect CloudWatch logs sent to a Log Group.
@@ -230,10 +316,12 @@ aws cloudformation create-stack \
230316
ParameterKey=SourceCloudWatchLogGroupARN,ParameterValue="<your-log-group-arn>"
231317
```
232318
319+
-->
320+
233321
:::{note}
234322
The `--capabilities CAPABILITY_NAMED_IAM` flag is required because this CloudFormation template creates AWS Identity and Access Management (IAM) resources. More specifically, it creates a named IAM role (`LambdaExecutionRole`) for the Lambda function. To acknowledge that AWS CloudFormation might create or modify IAM resources with custom names, you must specify the `CAPABILITY_NAMED_IAM` capability.
235323
:::
236-
-->
324+
237325
::::
238326
:::::
239327

@@ -315,6 +403,27 @@ You can deploy the stack manually using the AWS Management Console by following
315403
6. Review your configuration and select **Submit** to deploy the stack.
316404
7. Monitor the progress until the stack reaches the `CREATE_COMPLETE` state.
317405

406+
<!-- To be added at GA
407+
## Deployment using the AWS Serverless Application Repository
408+
409+
In addition to deploying via CloudFormation templates, you can deploy the EDOT Cloud Forwarder application directly from the AWS Serverless Application Repository (SAR).
410+
411+
:::{note}
412+
The same [deployment considerations](#deployment-considerations) apply to SAR deployments, including the requirement to deploy separate serverless applications for each log type and ensure the deployment region matches your S3 bucket region.
413+
:::
414+
415+
To deploy from SAR, follow these steps:
416+
417+
1. Navigate to **AWS Serverless Application Repository** in the AWS Management Console.
418+
2. Search for `edot-cloud-forwarder-s3-logs` and select the application.
419+
3. Select **Deploy**.
420+
4. **Configure the application settings**: You will be prompted to fill in the same parameters described in the [Required settings](#required-settings), [Log source settings](#log-source-settings), and [Optional settings](#optional-settings) sections. Refer to those sections for details on each parameter.
421+
5. **Acknowledge IAM role creation**: At the bottom of the page, check the box to acknowledge that the application will create custom IAM roles. This is required for the forwarder to access your S3 bucket and send data to Elastic Observability.
422+
6. Select **Deploy**.
423+
424+
The deployment process will start, and a CloudFormation stack will be created with all the necessary resources. You can monitor the progress in the AWS CloudFormation console under **Stacks**.
425+
-->
426+
318427
## CloudFormation stack resources
319428

320429
The CloudFormation templates create a number of resources to process logs from a specific log source.
@@ -339,7 +448,10 @@ This is a list of resources created by the stack when processing S3 logs.
339448
The main Lambda function, `LambdaFunction`, is the core component for processing S3 logs. S3 event notifications are handled dynamically using `CustomNotificationUpdater` and `NotificationUpdaterLambda`.
340449

341450
CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while `S3FailureBucketARN` prevents data loss by capturing unprocessed logs.
342-
<!--
451+
452+
453+
<!-- TODO: Enable when CloudWatch logs are supported
454+
343455
### Resources for CloudWatch Logs
344456
345457
This is a list of resources created by the stack when CloudWatch logs are the source.
@@ -357,8 +469,61 @@ This is a list of resources created by the stack when CloudWatch logs are the so
357469
The CloudWatch Log Subscription Filter, `CloudWatchLogSubscriptionFilter`, ensures logs are correctly forwarded to the Lambda function. The Lambda function, `LambdaFunction`, serves as the core processing unit for CloudWatch logs.
358470
359471
CloudWatch Log Groups help monitor execution performance and debug issues. IAM permissions (`LambdaExecutionRole`, `LambdaPermissionCloudWatch`) control interactions between CloudWatch and Lambda, while the failure bucket, `S3FailureBucketARN`, helps prevent data loss in case of processing errors.
472+
360473
-->
361-
## **Delete a CloudFormation stack**
474+
## Kibana integration setup
475+
476+
After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations.
477+
478+
To set up data visualization in {{kib}}:
479+
480+
1.Log into your Elastic Cloud deployment and open Kibana.
481+
2. Go to **Management** → **Integrations** in the Kibana navigation menu.
482+
3. Search for the appropriate integration based on your log type and install it:
483+
484+
| **AWS Log Type** | **Integration Name** | **Description** |
485+
|------------------|---------------------|-----------------|
486+
| ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs |
487+
| VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data |
488+
489+
4. Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data.
490+
491+
## Error handling and retrying
492+
493+
{{edot-cf}} store Lambda invocation events related to retryable errors at the S3 bucket specified by `S3FailureBucketARN` parameter.
494+
Retryable errors here include,
495+
- Network errors when attempting to forward to OTLPEndpoint
496+
- Invalid or expired ElasticApiKey
497+
- Lambda triggered by events that mismatch EdotCloudForwarderS3LogsType selection
498+
499+
These errors can be replayed manually to back-fill any gaps in your data.
500+
501+
### Replay failed events
502+
503+
To replay errors invoke the Lambda with manual trigger type `replayFailedEvents`. Replace `<LAMBDA_NAME>` with the name from your deployment.
504+
505+
```sh
506+
aws lambda invoke \
507+
--function-name <LAMBDA_NAME> \
508+
--payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \
509+
--cli-binary-format raw-in-base64-out /dev/null
510+
```
511+
512+
The following settings are available:
513+
514+
| Option | Description | Default |
515+
|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------|
516+
| dryrun | Run the command without processing actual backup events. Useful to understand details about replaying error files from Lambda CloudWatch logs. | false |
517+
| removeOnSuccess | Configure whether to remove error event from S3 error destination, if processing is successful. | true |
518+
519+
When successful, you should get `"StatusCode": 200` as the output. Check CloudWatch logs (resource `LambdaLogGroup`) for detailed logs.
520+
521+
:::{note}
522+
With AWS CLI, you can use `--timeout` to increase currently configured Lambda timeout for custom invocations.
523+
However, if a timeout occurs, you need to run the custom event multiple times to fully process all error events from the bucket.
524+
:::
525+
526+
## Delete a CloudFormation stack
362527

363528
If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command:
364529

0 commit comments

Comments
 (0)