Skip to content

Commit 579d9df

Browse files
theletterftheletterf
committed
Add EDOT tutorial
1 parent f1ff36d commit 579d9df

File tree

4 files changed

+470
-99
lines changed

4 files changed

+470
-99
lines changed

‎solutions/observability/get-started.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ Elastic provides a powerful LLM observability framework including key metrics, l
105105

106106
Refer to [LLM observability](/solutions/observability/applications/llm-observability.md) for more information.
107107
:::
108-
108+
::::
109109
:::::
110110
::::::
111111

@@ -178,5 +178,5 @@ Many [{{observability}} integrations](https://www.elastic.co/integrations/data-i
178178
### Other resources
179179

180180
* [What's Elastic {{observability}}](/solutions/observability/get-started/what-is-elastic-observability.md)
181-
* [What’s new in Elastic Stack](/release-notes/elastic-observability/index.md)
181+
* [What's new in Elastic Stack](/release-notes/elastic-observability/index.md)
182182
* [{{obs-serverless}} billing dimensions](/deploy-manage/cloud-organization/billing/elastic-observability-billing-dimensions.md)

‎solutions/observability/logs/stream-any-log-file-using-edot-collector.md‎

Lines changed: 374 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,374 @@
1+
---
2+
navigation_title: Stream any log file using OTel Collector
3+
mapped_pages:
4+
- https://www.elastic.co/guide/en/observability/current/logs-stream-edot.html
5+
- https://www.elastic.co/guide/en/serverless/current/observability-stream-log-files-edot.html
6+
applies_to:
7+
stack: all
8+
serverless: all
9+
products:
10+
- id: observability
11+
- id: cloud-serverless
12+
---
13+
14+
# Stream any log file using OTel Collector [logs-stream-edot]
15+
16+
This guide shows you how to manually configure the {{edot}} (EDOT) Collector to send your log data to {{es}} by configuring the `otel.yml` file. For an Elastic Agent equivalent, refer to [Stream any log file using {{agent}}](/solutions/observability/logs/stream-any-log-file.md).
17+
18+
For more OpenTelemetry quickstarts, refer to [EDOT quickstarts](opentelemetry://reference/quickstart/index.md).
19+
20+
## Prerequisites [logs-stream-edot-prereq]
21+
22+
::::{tab-set}
23+
:group: stack-serverless
24+
25+
:::{tab-item} Elastic Stack
26+
:sync: stack
27+
28+
To follow the steps in this guide, you need an {{stack}} deployment that includes:
29+
30+
* {{es}} for storing and searching data
31+
* {{kib}} for visualizing and managing data
32+
* Kibana user with `All` privileges on {{fleet}} and Integrations. Because many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.
33+
* Integrations Server (included by default in every {{ech}} deployment)
34+
35+
To get started quickly, create an {{ech}} deployment and host it on AWS, GCP, or Azure. [Try it out for free](https://cloud.elastic.co/registration?page=docs&placement=docs-body).
36+
37+
38+
:::
39+
40+
:::{tab-item} Serverless
41+
:sync: serverless
42+
43+
The **Admin** role or higher is required to onboard log data. To learn more, refer to [Assign user roles and privileges](/deploy-manage/users-roles/cloud-organization/user-roles.md#general-assign-user-roles).
44+
45+
:::
46+
47+
::::
48+
49+
## Install the EDOT Collector [logs-stream-edot-install-config]
50+
51+
Complete these steps to install and configure the EDOT Collector and send your log data to Elastic Observability.
52+
53+
::::::{stepper}
54+
55+
:::::{step} Download and install the EDOT Collector
56+
57+
On your host, download the EDOT Collector installation package that corresponds with your system:
58+
59+
::::{tab-set}
60+
61+
:::{tab-item} Linux
62+
63+
```shell subs=true
64+
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{{version.stack}}-linux-x86_64.tar.gz
65+
tar xzvf elastic-agent-{{version.stack}}-linux-x86_64.tar.gz
66+
```
67+
:::
68+
69+
:::{tab-item} macOS
70+
71+
```shell subs=true
72+
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{{version.stack}}-darwin-x86_64.tar.gz
73+
tar xzvf elastic-agent-{{version.stack}}-darwin-x86_64.tar.gz
74+
```
75+
:::
76+
77+
:::{tab-item} Windows
78+
79+
```powershell subs=true
80+
# PowerShell 5.0+
81+
wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{{version.stack}}-windows-x86_64.zip -OutFile elastic-agent-{{version.stack}}-windows-x86_64.zip
82+
Expand-Archive .\elastic-agent-{{version.stack}}-windows-x86_64.zip
83+
```
84+
:::
85+
86+
::::
87+
:::::
88+
89+
:::::{step} Configure the EDOT Collector
90+
91+
Follow these steps to retrieve the managed OTLP endpoint URL for your Serverless project:
92+
93+
1. In Elastic Cloud Serverless, open your Observability project.
94+
2. Go to **Add data** → **Application** → **OpenTelemetry**.
95+
3. Select **Managed OTLP Endpoint** in the second step.
96+
4. Copy the OTLP endpoint configuration value.
97+
5. Select **Create API Key** to generate an API key.
98+
99+
Replace `<ELASTIC_OTLP_ENDPOINT>` and `<ELASTIC_API_KEY>` before applying the following commands:
100+
101+
::::{tab-set}
102+
103+
:::{tab-item} Linux
104+
105+
```bash
106+
ELASTIC_OTLP_ENDPOINT=<ELASTIC_OTLP_ENDPOINT> && \
107+
ELASTIC_API_KEY=<ELASTIC_API_KEY> && \
108+
cp ./otel_samples/managed_otlp/logs_metrics_traces.yml ./otel.yml && \
109+
mkdir -p ./data/otelcol && \
110+
sed -i "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \
111+
sed -i "s#\${env:ELASTIC_OTLP_ENDPOINT}#${ELASTIC_OTLP_ENDPOINT}#g" ./otel.yml && \
112+
sed -i "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml
113+
```
114+
:::
115+
116+
:::{tab-item} macOS
117+
118+
```bash
119+
ELASTIC_OTLP_ENDPOINT=<ELASTIC_OTLP_ENDPOINT> && \
120+
ELASTIC_API_KEY=<ELASTIC_API_KEY> && \
121+
cp ./otel_samples/managed_otlp/logs_metrics_traces.yml ./otel.yml && \
122+
mkdir -p ./data/otelcol && \
123+
sed -i '' "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \
124+
sed -i '' "s#\${env:ELASTIC_OTLP_ENDPOINT}#${ELASTIC_OTLP_ENDPOINT}#g" ./otel.yml && \
125+
sed -i '' "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml
126+
```
127+
:::
128+
129+
:::{tab-item} Windows
130+
131+
```powershell
132+
Remove-Item -Path .\otel.yml -ErrorAction SilentlyContinue
133+
Copy-Item .\otel_samples\managed_otlp\logs_metrics_traces.yml .\otel.yml
134+
New-Item -ItemType Directory -Force -Path .\data\otelcol | Out-Null
135+
136+
$content = Get-Content .\otel.yml
137+
$content = $content -replace '\${env:STORAGE_DIR}', "$PWD\data\otelcol"
138+
$content = $content -replace '\${env:ELASTIC_OTLP_ENDPOINT}', "<ELASTIC_OTLP_ENDPOINT>"
139+
$content = $content -replace '\${env:ELASTIC_API_KEY}', "<ELASTIC_API_KEY>"
140+
$content | Set-Content .\otel.yml
141+
```
142+
:::
143+
::::
144+
:::::
145+
146+
:::::{step} Configure log file collection
147+
148+
To collect logs from specific log files, you need to modify the `otel.yml` configuration file. The configuration includes receivers, processors, and exporters that handle log data.
149+
150+
::::{tab-set}
151+
:group: stack-serverless
152+
153+
:::{tab-item} Elastic Stack
154+
:sync: stack
155+
156+
Here's an example configuration for collecting log files with Elastic Stack:
157+
158+
:::{dropdown} otel.yml for logs collection (Elastic Stack)
159+
160+
```yaml
161+
receivers:
162+
# Receiver for platform specific log files
163+
filelog/platformlogs:
164+
include: [ /var/log/*.log ]
165+
retry_on_failure:
166+
enabled: true
167+
start_at: end
168+
storage: file_storage
169+
# start_at: beginning
170+
171+
extensions:
172+
file_storage:
173+
directory: ${env:STORAGE_DIR}
174+
175+
processors:
176+
resourcedetection:
177+
detectors: ["system"]
178+
system:
179+
hostname_sources: ["os"]
180+
resource_attributes:
181+
host.name:
182+
enabled: true
183+
host.id:
184+
enabled: false
185+
host.arch:
186+
enabled: true
187+
host.ip:
188+
enabled: true
189+
host.mac:
190+
enabled: true
191+
host.cpu.vendor.id:
192+
enabled: true
193+
host.cpu.family:
194+
enabled: true
195+
host.cpu.model.id:
196+
enabled: true
197+
host.cpu.model.name:
198+
enabled: true
199+
host.cpu.stepping:
200+
enabled: true
201+
host.cpu.cache.l2.size:
202+
enabled: true
203+
os.description:
204+
enabled: true
205+
os.type:
206+
enabled: true
207+
208+
exporters:
209+
# Exporter to print the first 5 logs/metrics and then every 1000th
210+
debug:
211+
verbosity: detailed
212+
sampling_initial: 5
213+
sampling_thereafter: 1000
214+
215+
# Exporter to send logs and metrics to Elasticsearch
216+
elasticsearch/otel:
217+
endpoints: ["${env:ELASTIC_ENDPOINT}"]
218+
api_key: ${env:ELASTIC_API_KEY}
219+
mapping:
220+
mode: otel
221+
222+
service:
223+
extensions: [file_storage]
224+
pipelines:
225+
logs/platformlogs:
226+
receivers: [filelog/platformlogs]
227+
processors: [resourcedetection]
228+
exporters: [debug, elasticsearch/otel]
229+
```
230+
231+
:::
232+
:::
233+
234+
:::{tab-item} Serverless
235+
:sync: serverless
236+
237+
Here's an example configuration for collecting log files with Elastic Cloud Serverless:
238+
239+
:::{dropdown} otel.yml for logs collection (Serverless)
240+
241+
```yaml
242+
receivers:
243+
# Receiver for platform specific log files
244+
filelog/platformlogs:
245+
include: [/var/log/*.log]
246+
retry_on_failure:
247+
enabled: true
248+
start_at: end
249+
storage: file_storage
250+
# start_at: beginning
251+
252+
extensions:
253+
file_storage:
254+
directory: ${env:STORAGE_DIR}
255+
256+
processors:
257+
resourcedetection:
258+
detectors: ["system"]
259+
system:
260+
hostname_sources: ["os"]
261+
resource_attributes:
262+
host.name:
263+
enabled: true
264+
host.id:
265+
enabled: false
266+
host.arch:
267+
enabled: true
268+
host.ip:
269+
enabled: true
270+
host.mac:
271+
enabled: true
272+
host.cpu.vendor.id:
273+
enabled: true
274+
host.cpu.family:
275+
enabled: true
276+
host.cpu.model.id:
277+
enabled: true
278+
host.cpu.model.name:
279+
enabled: true
280+
host.cpu.stepping:
281+
enabled: true
282+
host.cpu.cache.l2.size:
283+
enabled: true
284+
os.description:
285+
enabled: true
286+
os.type:
287+
enabled: true
288+
289+
exporters:
290+
# Exporter to print the first 5 logs/metrics and then every 1000th
291+
debug:
292+
verbosity: detailed
293+
sampling_initial: 5
294+
sampling_thereafter: 1000
295+
296+
# Exporter to send logs and metrics to Elasticsearch Managed OTLP Input
297+
otlp/ingest:
298+
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
299+
headers:
300+
Authorization: ApiKey ${env:ELASTIC_API_KEY}
301+
302+
service:
303+
extensions: [file_storage]
304+
pipelines:
305+
logs/platformlogs:
306+
receivers: [filelog/platformlogs]
307+
processors: [resourcedetection]
308+
exporters: [debug, otlp/ingest]
309+
```
310+
:::
311+
:::
312+
::::
313+
314+
Key configuration elements:
315+
316+
* `receivers.filelog/platformlogs.include`: Specifies the path to your log files. You can use patterns like `/var/log/*.log`.
317+
* `processors.resourcedetection`: Automatically detects and adds host system information to your logs.
318+
* `extensions.file_storage`: Provides persistent storage for the collector's state.
319+
* `exporters`: Configures how data is sent to Elasticsearch (Elastic Stack) or OTLP endpoint (Serverless).
320+
:::::
321+
322+
:::::{step} Run the EDOT Collector
323+
324+
Run the following command to run the EDOT Collector:
325+
326+
::::{tab-set}
327+
328+
:::{tab-item} Linux and macOS
329+
330+
```bash
331+
sudo ./otelcol --config otel.yml
332+
```
333+
:::
334+
335+
:::{tab-item} Windows
336+
337+
```powershell
338+
.\elastic-agent.exe otel --config otel.yml
339+
```
340+
:::
341+
342+
::::
343+
344+
:::{note}
345+
The Collector opens ports `4317` and `4318` to receive application data from locally running OTel SDKs without authentication. This allows the SDKs to send data without any further configuration needed as they use this endpoint by default.
346+
:::
347+
:::::
348+
::::::
349+
350+
## Troubleshoot your EDOT Collector configuration [logs-stream-edot-troubleshooting]
351+
352+
If you're not seeing your log files in the UI, verify the following:
353+
354+
* The path to your logs file under `include` is correct.
355+
* Your API key is properly set in the environment variables.
356+
* The OTLP endpoint URL is correct and accessible.
357+
* The Collector is running without errors (check the console output).
358+
359+
If you're still running into issues, see [EDOT Collector troubleshooting](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Configure EDOT Collector](opentelemetry://reference/edot-collector/config/index.md).
360+
361+
## Next steps [logs-stream-edot-next-steps]
362+
363+
After you have your EDOT Collector configured and are streaming log data to {{es}}:
364+
365+
* Refer to the [Parse and organize logs](/solutions/observability/logs/parse-route-logs.md) documentation for information on extracting structured fields from your log data, rerouting your logs to different data streams, and filtering and aggregating your log data.
366+
* Refer to the [Filter and aggregate logs](/solutions/observability/logs/filter-aggregate-logs.md) documentation for information on filtering and aggregating your log data to find specific information, gain insight, and monitor your systems more efficiently.
367+
* To collect telemetry from applications and use the EDOT Collector as a gateway, instrument your target applications following the setup instructions:
368+
- [Android](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/android/)
369+
- [.NET](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/dotnet/setup/)
370+
- [iOS](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/ios/)
371+
- [Java](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/java/setup/)
372+
- [Node.js](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/nodejs/setup/)
373+
- [PHP](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/php/setup/)
374+
- [Python](https://www.elastic.co/docs/reference/opentelemetry/edot-sdks/python/setup/)

0 commit comments

Comments
 (0)