Skip to content

Commit e001e9e

Browse files
rapphilbryan-aguilar
rapphil
and
bryan-aguilar
authored
Chore: Add documentation for confmap providers and Kafka receiver and exporter (#521)
* Chore: Add documentation for confmap providers Signed-off-by: Raphael Silva <[email protected]> * Add documentation for Kakfa receiver/exporter Signed-off-by: Raphael Silva <[email protected]> * Fix typos and improve documentation Signed-off-by: Raphael Silva <[email protected]> * Add example of usage for the kafka components * Fix s3 urls --------- Signed-off-by: Raphael Silva <[email protected]> Co-authored-by: bryan-aguilar <[email protected]>
1 parent 77d440c commit e001e9e

File tree

3 files changed

+472
-7
lines changed

3 files changed

+472
-7
lines changed

‎src/config/sideBarData.js

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -98,18 +98,20 @@ export const sideBarData = [
9898
{
9999
node: {
100100
id: nextId(),
101-
label: "Components",
101+
label: "ADOT Collector Components",
102102
items: [
103+
{label: "CloudWatch EMF Exporter", link: "/docs/getting-started/cloudwatch-metrics#cloudwatch-emf-exporter-awsemf"},
104+
{label: "Confmap Providers", link: "/docs/components/confmap-providers"},
105+
{label: "ECS Container Metrics Receiver", link: "/docs/components/ecs-metrics-receiver"},
106+
{label: "File Exporter", link: "/docs/components/misc-exporters#file-exporter"},
107+
{label: "Jaeger Receiver", link: "/docs/components/jaeger-zipkin-receiver#jaeger-receiver"},
108+
{label: "Kafka Receiver/Expoter", link: "/docs/components/kafka-receiver-exporter"},
109+
{label: "Logging Exporter", link: "/docs/components/misc-exporters#logging-exporter"},
103110
{label: "OTLP Exporters", link: "/docs/components/otlp-exporter"},
104111
{label: "Prometheus Exporters", link: "/docs/components/prometheus-exporters"},
105-
{label: "CloudWatch EMF Exporter", link: "/docs/getting-started/cloudwatch-metrics#cloudwatch-emf-exporter-awsemf"},
112+
{label: "StatsD Receiver", link: "/docs/components/statsd-receiver"},
106113
{label: "X-Ray Exporter", link: "/docs/getting-started/x-ray#configuring-the-aws-x-ray-exporter"},
107-
{label: "Logging Exporter", link: "/docs/components/misc-exporters#logging-exporter"},
108-
{label: "File Exporter", link: "/docs/components/misc-exporters#file-exporter"},
109-
{label: "ECS Container Metrics Receiver", link: "/docs/components/ecs-metrics-receiver"},
110114
{label: "X-Ray Receiver", link: "/docs/components/x-ray-receiver"},
111-
{label: "StatsD Receiver", link: "/docs/components/statsd-receiver"},
112-
{label: "Jaeger Receiver", link: "/docs/components/jaeger-zipkin-receiver#jaeger-receiver"},
113115
{label: "Zipkin Receiver", link: "/docs/components/jaeger-zipkin-receiver#zipkin-receiver"},
114116
],
115117
link: "/docs/null"

‎src/docs/components/confmap-providers.mdx

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
---
2+
title: 'Confmap providers - Get configuration from different sources'
3+
description: |
4+
Confmap represents the raw configuration for the OpenTelemetry Collector. A Confmap provider
5+
is a Collector component that provides configuration retrieved from a URI to configure the collector.
6+
This section will describe all the confmap providers supported by the ADOT collector.
7+
path: '/docs/components/confmap-providers'
8+
---
9+
10+
import SectionSeparator from "components/MdxSectionSeparator/sectionSeparator.jsx"
11+
12+
The confmap providers are a type of OpenTelemetry collector component that is responsible by fetching configuration
13+
from a URI. This configuration is be used to configure the Collector and all the other components. The URI for the confmap
14+
providers is passed through the `-c/--config` command line parameter of the collector. The URI has the following format:
15+
`<scheme>:<opaque data>`.
16+
17+
More technical details about this component can be found [here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/confmap/README.md).
18+
19+
<SectionSeparator />
20+
21+
## Confmap providers supported by the ADOT collector
22+
23+
The ADOT collector support the following types of confmap providers: file, env, yaml, http, https and s3.
24+
25+
26+
### File provider
27+
28+
* scheme: `file`. Scheme is optional in the file provider and can be omitted when passing this as parameter.
29+
* description: Retrieves configuration from the local file system
30+
* examples:
31+
* `/path/to/configuration.yaml`
32+
* `file:/path/to/configuration.yaml`
33+
* `c:\path\to\configuration.yaml`
34+
* `file:c:\path\to\configuration.yaml`
35+
36+
### Env provider
37+
38+
* scheme: `env`. Scheme is optional.
39+
* description: Retrieves configuration from an environment variable.
40+
* examples:
41+
* `env:ENVIRONMENT_FILE_NAME`
42+
43+
### YAML provider
44+
45+
* scheme: `yaml`
46+
* description: Retrieves configuration directly from the command line, in yaml format.
47+
* examples:
48+
* `yaml:processors::batch::timeout: 2s`
49+
50+
### HTTP provider
51+
52+
* scheme: `http`
53+
* description: Retrieves configuration from an http server.
54+
* examples:
55+
* `http://server/path/to/config.yaml`
56+
57+
### HTTPS provider
58+
59+
* scheme: `https`
60+
* description: Retrieves configuration from an https server. It uses the certificate pool in the operating system to validate the hostname.
61+
* examples:
62+
* `https://server/path/to/config.yaml`
63+
64+
### S3 provider
65+
66+
* scheme: `s3`
67+
* description: Retrieves configuration from AWS s3. This component uses the [default credentials chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) to authenticate in s3 in order to fetch the configuration.
68+
* examples:
69+
* `https://<bucket_name>.s3.<region>.amazonaws.com/path/to/config.yaml`
70+
71+
<SectionSeparator />
72+
73+
## Passing command line parameters to the collector
74+
75+
When installing the ADOT collector in EC2 from a RPM or MSI, you can pass the configuration parameter to a control
76+
script that is responsible by managing the Operating system service that run the ADOT collector. Control scripts support
77+
the following confmap providers: file, http, https and s3.
78+
79+
### Linux
80+
81+
82+
On Linux, to use any of the supported confmap providers, you can use the `-c` parameter of the the `aws-otel-collector-ctl` control script.
83+
84+
```
85+
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c <configuration uri> -a start"
86+
```
87+
88+
Here are some examples:
89+
90+
```bash
91+
# File provider
92+
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c /path/to/file.yaml -a start
93+
# HTTP provider
94+
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c "http://server/configuration.yaml" -a start
95+
# HTTPS provider
96+
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c "https://server/configuration.yaml" -a start
97+
# S3 provider
98+
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c "s3://bucket-example.s3.us-west-2.amazonaws.com/object.yaml" -a start
99+
```
100+
101+
### Windows
102+
103+
On Windows you can use the `-ConfigLocation` command line parameter of the `aws-otel-collector-ctl.ps1` control script to setup the ADOT collector service.
104+
105+
Assuming you are in a Powershell session:
106+
```
107+
. C:\\Program Files\\Amazon\\AwsOtelCollector\\aws-otel-collector-ctl.ps1' -ConfigLocation "<uri>" -Action start
108+
```
109+
110+
Here are some examples:
111+
112+
```
113+
# File provider
114+
. 'C:\Program Files\Amazon\AwsOtelCollector\aws-otel-collector-ctl.ps1' -ConfigLocation 'C:\path\to\file.yaml' -a start
115+
# HTTP provider
116+
. 'C:\Program Files\Amazon\AwsOtelCollector\aws-otel-collector-ctl.ps1' -ConfigLocation 'http://server/configuration.yaml' -a start
117+
# HTTPS provider
118+
. 'C:\Program Files\Amazon\AwsOtelCollector\aws-otel-collector-ctl.ps1' -ConfigLocation 'https://server/configuration.yaml' -a start
119+
# S3 provider
120+
. 'C:\Program Files\Amazon\AwsOtelCollector\aws-otel-collector-ctl.ps1' -ConfigLocation 's3://bucket-example.s3.us-west-2.amazonaws.com/object.yaml' -a start
121+
```
122+
123+
### Container environments
124+
125+
In container environments, you can override the Docker [CMD instruction](https://docs.docker.com/engine/reference/builder/#cmd) to use the configuration from the URI that you want. This can be done
126+
because all the parameters passed in the CMD instruction are passed to the ADOT collector since the [entrypoint for the ADOT collector image](https://github.com/aws-observability/aws-otel-collector/blob/main/cmd/awscollector/Dockerfile#L79) is the
127+
Collector executable itself.
128+
129+
130+
In ECS you can use the `command` property of the [environment container definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_environment) to specify the
131+
parameters that will be passed to the collector.
132+
133+
Example:
134+
135+
```json
136+
{
137+
"name": "aoc-collector",
138+
"image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",
139+
"command": ["--config", "<configuration uri>"],
140+
"environment": [],
141+
"environmentFiles": [],
142+
"dependsOn": [],
143+
"logConfiguration": {
144+
"logDriver": "awslogs",
145+
"options": {
146+
"awslogs-group": "/ecs/ecs-cwagent-sidecar-collector",
147+
"awslogs-region": "us-west-2",
148+
"awslogs-stream-prefix": "ecs",
149+
"awslogs-create-group": "True"
150+
}
151+
}
152+
}
153+
```
154+
155+
Similarly, In Kubernetes you can use the `args` property of container definition to specify the command line parameters
156+
that should be passed to the ADOT collector.
157+
158+
Example:
159+
```yaml
160+
# create namespace
161+
apiVersion: v1
162+
kind: Namespace
163+
metadata:
164+
name: aws-otel-eks
165+
labels:
166+
name: aws-otel-eks
167+
---
168+
# create deployment
169+
apiVersion: apps/v1
170+
kind: Deployment
171+
metadata:
172+
name: aws-otel-eks-sidecar
173+
namespace: aws-otel-eks
174+
labels:
175+
name: aws-otel-eks-sidecar
176+
spec:
177+
replicas: 1
178+
selector:
179+
matchLabels:
180+
name: aws-otel-eks-sidecar
181+
template:
182+
metadata:
183+
labels:
184+
name: aws-otel-eks-sidecar
185+
spec:
186+
containers:
187+
- name: aws-otel-collector
188+
image: "public.ecr.aws/aws-observability/aws-otel-collector:latest"
189+
env:
190+
- name: AWS_REGION
191+
value: "us-west-2"
192+
imagePullPolicy: Always
193+
args: ["--config", "<configuration uri>"]
194+
resources:
195+
limits:
196+
cpu: 256m
197+
memory: 512Mi
198+
requests:
199+
cpu: 32m
200+
memory: 24Mi
201+
```
202+
203+
## Embedding URIs in the configuration
204+
205+
It is possible to embed configuration URIs in a configuration of the OpenTelemetry collector. These URIs will be expanded and replaced by the
206+
content of the URI that they point to. To use this feature, you need to add the placeholders in the collector configuration with the
207+
the following format: `${uri}`. You can provided URIs with any of the supported schemes.
208+
209+
210+
The following pieces will demonstrate how this feature works. Supposed you have the following configuration that is passed to the
211+
collector through the command line parameter.
212+
213+
```yaml
214+
extensions:
215+
health_check:
216+
217+
receivers:
218+
${s3://example-bucket.s3.us-west-2.amazonaws.com/receivers.yaml}
219+
220+
processors:
221+
${s3://example-bucket.s3.us-west-2.amazonaws.com/processors.yaml}
222+
223+
exporters:
224+
${s3://example-bucket.s3.us-west-2.amazonaws.com/exporters.yaml}
225+
226+
227+
service:
228+
pipelines:
229+
traces:
230+
receivers: [otlp,awsxray]
231+
processors: [batch/traces]
232+
exporters: [awsxray]
233+
metrics:
234+
receivers: [otlp]
235+
processors: [batch/metrics]
236+
exporters: [awsemf]
237+
238+
extensions: [health_check]
239+
```
240+
In the example presented above, we are embeeding three different URIs. These embedded URIs will be expanded when the collector load the configuration.
241+
242+
The following is the content of each URI:
243+
244+
`s3://example-bucket.s3.us-west-2.amazonaws.com/receivers.yaml`
245+
246+
```
247+
otlp:
248+
protocols:
249+
grpc:
250+
endpoint: 0.0.0.0:4317
251+
http:
252+
endpoint: 0.0.0.0:4318
253+
awsxray:
254+
endpoint: 0.0.0.0:2000
255+
transport: udp
256+
```
257+
258+
`s3://example-bucket.s3.us-west-2.amazonaws.com/processors.yaml`
259+
260+
```
261+
batch/traces:
262+
timeout: 1s
263+
send_batch_size: 50
264+
batch/metrics:
265+
timeout: 60s
266+
```
267+
268+
`s3://example-bucket.s3.us-west-2.amazonaws.com/exporters.yaml`
269+
270+
```
271+
awsxray:
272+
awsemf:
273+
```
274+
275+
The final configuration would look like:
276+
277+
```
278+
extensions:
279+
health_check:
280+
281+
receivers:
282+
otlp:
283+
protocols:
284+
grpc:
285+
endpoint: 0.0.0.0:4317
286+
http:
287+
endpoint: 0.0.0.0:4318
288+
awsxray:
289+
endpoint: 0.0.0.0:2000
290+
transport: udp
291+
292+
processors:
293+
batch/traces:
294+
timeout: 1s
295+
send_batch_size: 50
296+
batch/metrics:
297+
timeout: 60s
298+
299+
exporters:
300+
awsxray:
301+
awsemf:
302+
303+
service:
304+
pipelines:
305+
traces:
306+
receivers: [otlp,awsxray]
307+
processors: [batch/traces]
308+
exporters: [awsxray]
309+
metrics:
310+
receivers: [otlp]
311+
processors: [batch/metrics]
312+
exporters: [awsemf]
313+
314+
extensions: [health_check]
315+
```

0 commit comments

Comments
 (0)