Skip to content

Commit 6d3d563

Browse files
AbhiPrasadjaffrepaulcoolguyzone
authored
feat: add guides for more otel collector use cases (#15680)
Add guides for leveraging the otel collector with common infra/applications like. - aws cloudwatch - kafka - nginx - syslog - windows events Most of these have been brought up before, so with some small test + help from claude wrote these up. --------- Co-authored-by: Paul Jaffre <[email protected]> Co-authored-by: Alex Krawiec <[email protected]>
1 parent 946ca08 commit 6d3d563

File tree

7 files changed

+1102
-0
lines changed

7 files changed

+1102
-0
lines changed

‎docs/concepts/otlp/index.mdx‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,3 +250,7 @@ The following SDKs have dedicated integrations that make OTLP setup easy:
250250
label="Python"
251251
url="/platforms/python/integrations/otlp"
252252
/>
253+
254+
## OpenTelemetry Collector Guides
255+
256+
View the [OpenTelemetry Collector Guides](/product/drains/integration/opentelemetry-collector/#open-telemetry-collector-guides) to learn how to leverage the OpenTelemetry Collector to send traces and logs to Sentry from different sources like Kafka or Nginx.

‎docs/product/drains/integration/opentelemetry-collector.mdx‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,13 @@ service:
137137
receivers: [routing]
138138
exporters: [otlphttp/project-b]
139139
```
140+
141+
## OpenTelemetry Collector Guides
142+
143+
These guides show you how to leverage the OpenTelemetry Collector to send traces and logs to Sentry from different sources.
144+
145+
- [Forwarding AWS CloudWatch Logs to Sentry](/product/drains/otlp-guides/aws-cloudwatch/)
146+
- [Forwarding Logs and Traces from Kafka to Sentry](/product/drains/otlp-guides/kafka/)
147+
- [Forwarding Nginx Logs to Sentry](/product/drains/otlp-guides/nginx/)
148+
- [Forwarding Syslog Messages to Sentry](/product/drains/otlp-guides/syslog/)
149+
- [Forwarding Windows Event Logs to Sentry](/product/drains/otlp-guides/windows-events/)

‎docs/product/drains/otlp-guides/aws-cloudwatch.mdx‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
---
2+
title: Forwarding AWS CloudWatch Logs to Sentry via the OpenTelemetry Protocol (OTLP)
3+
sidebar_order: 100
4+
description: "Learn how to forward AWS CloudWatch logs to Sentry via the OpenTelemetry Protocol (OTLP)."
5+
keywords:
6+
["otlp", "otel", "opentelemetry", "aws", "cloudwatch", "logs", "amazon"]
7+
---
8+
9+
This guide shows you how to collect AWS CloudWatch logs and forward them to Sentry using the OpenTelemetry Collector with the [AWS CloudWatch Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver).
10+
11+
## Prerequisites
12+
13+
Before you begin, ensure you have:
14+
15+
- AWS credentials configured with permissions to read CloudWatch logs
16+
- A Sentry project to send data to
17+
18+
## Step 1: Install the OpenTelemetry Collector
19+
20+
The AWS CloudWatch Receiver is included in the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib) distribution. You'll need to download and install this version, as the standard `otelcol` binary does not include the AWS CloudWatch Receiver.
21+
22+
Download the latest `otelcol-contrib` binary from the [OpenTelemetry Collector releases page](https://github.com/open-telemetry/opentelemetry-collector-releases/releases).
23+
24+
## Step 2: Configure AWS Credentials
25+
26+
The AWS CloudWatch Receiver uses the [AWS SDK](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for authentication, which supports multiple methods including credentials files and EC2 instance metadata (IMDS).
27+
28+
### Using an AWS Credentials File
29+
30+
Configure your AWS credentials using the AWS CLI:
31+
32+
```bash
33+
aws configure
34+
```
35+
36+
This creates a credentials file at `~/.aws/credentials` with your access key and secret.
37+
38+
### Using IAM Role (EC2)
39+
40+
For EC2 environments, attach an IAM role with the `CloudWatchLogsReadOnlyAccess` policy to your instance.
41+
42+
### Required IAM Permissions
43+
44+
Your AWS credentials need the following permissions:
45+
46+
- `logs:DescribeLogGroups`
47+
- `logs:DescribeLogStreams`
48+
- `logs:GetLogEvents`
49+
50+
## Step 3: Get Your Sentry OTLP Credentials
51+
52+
You'll need your Sentry OTLP endpoint and authentication header. These can be found in your [Sentry Project Settings](https://sentry.io/settings/projects/) under **Client Keys (DSN)** > **OpenTelemetry (OTLP)**.
53+
54+
### Logs Endpoint
55+
56+
```bash
57+
___OTLP_LOGS_URL___
58+
```
59+
60+
### Authentication Header
61+
62+
```
63+
x-sentry-auth: sentry sentry_key=___PUBLIC_KEY___
64+
```
65+
66+
## Step 4: Configure the Collector
67+
68+
Create a configuration file with the AWS CloudWatch Receiver and the OTLP HTTP exporter configured to send logs to Sentry.
69+
70+
For additional configuration options, see the [AWS CloudWatch Receiver Documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver).
71+
72+
### Collect All Log Groups (Autodiscover)
73+
74+
This configuration automatically discovers and collects logs from all CloudWatch log groups:
75+
76+
```yaml {filename:config.yaml}
77+
receivers:
78+
awscloudwatch:
79+
region: us-east-1
80+
logs:
81+
poll_interval: 1m
82+
83+
processors:
84+
batch:
85+
send_batch_size: 1024
86+
send_batch_max_size: 2048
87+
timeout: "1s"
88+
89+
exporters:
90+
otlphttp/sentry:
91+
logs_endpoint: ___OTLP_LOGS_URL___
92+
headers:
93+
x-sentry-auth: "sentry sentry_key=___PUBLIC_KEY___"
94+
compression: gzip
95+
encoding: proto
96+
97+
service:
98+
pipelines:
99+
logs:
100+
receivers:
101+
- awscloudwatch
102+
processors:
103+
- batch
104+
exporters:
105+
- otlphttp/sentry
106+
```
107+
108+
### Collect Specific Log Groups by Prefix
109+
110+
This configuration discovers log groups matching a specific prefix, useful for collecting logs from specific AWS services like EKS or Lambda:
111+
112+
```yaml {filename:config.yaml}
113+
receivers:
114+
awscloudwatch:
115+
region: us-east-1
116+
logs:
117+
poll_interval: 1m
118+
groups:
119+
autodiscover:
120+
limit: 100
121+
prefix: /aws/lambda/
122+
123+
processors:
124+
batch:
125+
send_batch_size: 1024
126+
send_batch_max_size: 2048
127+
timeout: "1s"
128+
129+
exporters:
130+
otlphttp/sentry:
131+
logs_endpoint: ___OTLP_LOGS_URL___
132+
headers:
133+
x-sentry-auth: "sentry sentry_key=___PUBLIC_KEY___"
134+
compression: gzip
135+
encoding: proto
136+
137+
service:
138+
pipelines:
139+
logs:
140+
receivers:
141+
- awscloudwatch
142+
processors:
143+
- batch
144+
exporters:
145+
- otlphttp/sentry
146+
```
147+
148+
### Collect Named Log Groups
149+
150+
This configuration collects logs from specific, named log groups:
151+
152+
```yaml {filename:config.yaml}
153+
receivers:
154+
awscloudwatch:
155+
region: us-east-1
156+
logs:
157+
poll_interval: 1m
158+
groups:
159+
named:
160+
/aws/lambda/my-function:
161+
/aws/eks/my-cluster/cluster:
162+
163+
processors:
164+
batch:
165+
send_batch_size: 1024
166+
send_batch_max_size: 2048
167+
timeout: "1s"
168+
169+
exporters:
170+
otlphttp/sentry:
171+
logs_endpoint: ___OTLP_LOGS_URL___
172+
headers:
173+
x-sentry-auth: "sentry sentry_key=___PUBLIC_KEY___"
174+
compression: gzip
175+
encoding: proto
176+
177+
service:
178+
pipelines:
179+
logs:
180+
receivers:
181+
- awscloudwatch
182+
processors:
183+
- batch
184+
exporters:
185+
- otlphttp/sentry
186+
```
187+
188+
## Troubleshooting
189+
190+
- Verify your AWS credentials are correctly configured and have the required permissions
191+
- Ensure the specified AWS region matches where your CloudWatch log groups are located
192+
- Check that the log group names or prefixes match existing CloudWatch log groups
193+
194+
## Additional Resources
195+
196+
- [AWS CloudWatch Receiver Documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver)
197+
- [AWS SDK Authentication](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html)
198+
- [Sentry OpenTelemetry Collector Configuration](/product/drains/integration/opentelemetry-collector/)
199+
- [Sentry Logs](/product/explore/logs/)

0 commit comments

Comments
 (0)