Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 128 additions & 5 deletions docs/reference/edot-cloud-forwarder/aws.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check notice on line 1 in docs/reference/edot-cloud-forwarder/aws.md

View workflow job for this annotation

GitHub Actions / docs-preview / build

The 'planned' lifecycle is deprecated and will be removed in a future release.
navigation_title: AWS
description: Set up the EDOT Cloud Forwarder for AWS to bring your AWS logs to Elastic Observability.
applies_to:
Expand All @@ -18,7 +18,7 @@

| Log source | Description |
| --- | --- |
| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) |
| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) |
| ELB Access | Logs generated by an Elastic Load Balancer (ELB) |
% | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch |

Expand Down Expand Up @@ -58,6 +58,7 @@
- Access logging enabled, with the bucket as the destination

:::

<!--
:::{tab-item} CloudWatch

Expand Down Expand Up @@ -89,6 +90,7 @@

- Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Logs. Each CloudFormation stack can only process one log source and format at a time.
- Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
- The CloudFormation stack deployment region must match the region of the S3 bucket.

## Download the template [download-templates]

Expand Down Expand Up @@ -128,7 +130,7 @@

| Setting | Description |
| ------------------ | --- |
| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:<br>- `vpc_flow_log`: VPC Flow logs<br>- `elb_access_log`: Elastic Load Balancer (ELB) Access logs<br>- `s3_access_log`: S3 Access logs<br>- `json`: JSON-formatted logs |
| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:<br>- `vpc_flow_log`: VPC Flow logs<br>- `elb_access_log`: Elastic Load Balancer (ELB) Access logs<br>- `s3_access_log`: S3 Access logs<br>- `cloudtrail_log`: CloudTrail logs<br>- `waf_log`: AWS WAF logs<br>- `json`: JSON-formatted logs |
| `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_<br>Defines how JSON logs are structured:<br>- `body` _(default)_: Stores logs in the request body<br>- `body_with_inline_attributes`: Logs include inline attributes |
| `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. |

Expand Down Expand Up @@ -156,7 +158,7 @@
| Setting | Description |
| ------------------- | --- |
| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. |
| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.1.5`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. |
| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `1.0.0`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. |
| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `1024` MB. Minimum value is `128` MB. Maximum value is `10240` MB. |
| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `50`. Make sure this value doesn't exceed your AWS account's concurrency limit. |

Expand All @@ -166,8 +168,8 @@
The following examples use the CloudFormation template files hosted in the [public S3 bucket](#download-templates).

- Use the `--template-url` flag to reference a template hosted on S3.
- To always use the most recent stable templates, use the `latest` path. For example, `v0/latest`.
- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v0.1.5`.
- To always use the most recent stable templates, use the `latest` path. For example, `v1/latest`.
- To pin a specific version, replace `latest` with the desired version tag. For example, `v1/v1.0.0`.

Alternatively, if you have downloaded the template file, you can use the `--template-body file://<path>` option with a local template file.

Expand Down Expand Up @@ -207,6 +209,83 @@
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elb_access_log"
```
::::

::::{tab-item} S3 Access logs

This example deploys a CloudFormation stack to collect S3 Access logs stored in an S3 bucket.

```sh
aws cloudformation create-stack \
--stack-name edot-cloud-forwarder-s3-access \
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--region eu-central-1 \
--parameters \
ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-access-logs-bucket-arn \
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="s3_access_log"
```
::::

::::{tab-item} CloudTrail logs

This example deploys a CloudFormation stack to collect CloudTrail logs stored in an S3 bucket.

```sh
aws cloudformation create-stack \
--stack-name edot-cloud-forwarder-cloudtrail \
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--region eu-central-1 \
--parameters \
ParameterKey=SourceS3BucketARN,ParameterValue=your-cloudtrail-bucket-arn \
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="cloudtrail_log"
```
::::

::::{tab-item} WAF logs

This example deploys a CloudFormation stack to collect AWS WAF logs stored in an S3 bucket.

```sh
aws cloudformation create-stack \
--stack-name edot-cloud-forwarder-waf \
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--region eu-central-1 \
--parameters \
ParameterKey=SourceS3BucketARN,ParameterValue=arn:aws:s3:::aws-waf-logs-your-bucket-arn \
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="waf_log"
```

:::{note}
Replace `aws-waf-logs-your-bucket-name` with your actual WAF logging bucket ARN. Remember that the bucket name must start with `aws-waf-logs-` as required by AWS WAF.
:::
::::

::::{tab-item} JSON logs

This example deploys a CloudFormation stack to collect JSON-formatted logs stored in an S3 bucket.

```sh
aws cloudformation create-stack \
--stack-name edot-cloud-forwarder-json \
--template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--region eu-central-1 \
--parameters \
ParameterKey=SourceS3BucketARN,ParameterValue=your-json-logs-bucket-arn \
ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="json" \
ParameterKey=S3LogsJsonEncodingMode,ParameterValue="body"
```
::::
<!--
::::{tab-item} CloudWatch logs

Expand Down Expand Up @@ -309,6 +388,25 @@
6. Review your configuration and select **Submit** to deploy the stack.
7. Monitor the progress until the stack reaches the `CREATE_COMPLETE` state.

## Deployment using the AWS Serverless Application Repository

In addition to deploying via CloudFormation templates, you can deploy the EDOT Cloud Forwarder application directly from the AWS Serverless Application Repository (SAR).

:::{note}
The same [deployment considerations](#deployment-considerations) apply to SAR deployments, including the requirement to deploy separate serverless applications for each log type and ensure the deployment region matches your S3 bucket region.
:::

To deploy from SAR, follow these steps:

1. Navigate to **AWS Serverless Application Repository** in the AWS Management Console.
2. Search for `edot-cloud-forwarder-s3-logs` and select the application.
3. Select **Deploy**.
4. **Configure the application settings**: You will be prompted to fill in the same parameters described in the [Required settings](#required-settings), [Log source settings](#log-source-settings), and [Optional settings](#optional-settings) sections. Refer to those sections for details on each parameter.
5. **Acknowledge IAM role creation**: At the bottom of the page, check the box to acknowledge that the application will create custom IAM roles. This is required for the forwarder to access your S3 bucket and send data to Elastic Observability.
6. Select **Deploy**.

The deployment process will start, and a CloudFormation stack will be created with all the necessary resources. You can monitor the progress in the AWS CloudFormation console under **Stacks**.

## CloudFormation stack resources

The CloudFormation templates create a number of resources to process logs from a specific log source.
Expand Down Expand Up @@ -352,6 +450,31 @@

CloudWatch Log Groups help monitor execution performance and debug issues. IAM permissions (`LambdaExecutionRole`, `LambdaPermissionCloudWatch`) control interactions between CloudWatch and Lambda, while the failure bucket, `S3FailureBucketARN`, helps prevent data loss in case of processing errors.
-->
## Kibana integration setup

After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, you can install pre-built integrations in Kibana to visualize your data with out-of-the-box dashboards and visualizations.

### Install integrations

To set up data visualization for your AWS logs:

1. **Navigate to Kibana**: Log into your Elastic Cloud deployment and open Kibana.

2. **Access Integrations**: Go to **Management** → **Integrations** in the Kibana navigation menu.

3. **Search and install**: Search for the appropriate integration based on your log type and install it:

| **AWS Log Type** | **Integration Name** | **Description** |
|------------------|---------------------|-----------------|
| ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs |
| VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data |

4. **Access dashboards**: Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data.

### Benefits

This allows you to immediately start analyzing your AWS infrastructure without building dashboards from scratch.

## **Delete a CloudFormation stack**

If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command:
Expand Down
Loading