Skip to content

Commit 073bdde

Browse files
mdbirnstiehlmdbirnstiehl
committed
fix error
1 parent 3bda0fa commit 073bdde

File tree

1 file changed

+369
-0
lines changed

1 file changed

+369
-0
lines changed

‎raw-migrated-files/observability-docs/observability/logs-stream.md‎

Lines changed: 369 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,369 @@
1+
# Stream any log file [logs-stream]
2+
3+
This guide shows you how to manually configure a standalone {{agent}} to send your log data to {{es}} using the `elastic-agent.yml` file.
4+
5+
If you don’t want to manually configure the {{agent}}, you can use the **Monitor hosts with {{agent}}** quickstart. Refer to the [quickstart documentation](../../../solutions/observability/get-started/quickstart-monitor-hosts-with-elastic-agent.md) for more information.
6+
7+
Continue with this guide for instructions on manual configuration.
8+
9+
10+
## Prerequisites [logs-stream-prereq]
11+
12+
To follow the steps in this guide, you need an {{stack}} deployment that includes:
13+
14+
* {{es}} for storing and searching data
15+
* {{kib}} for visualizing and managing data
16+
* Kibana user with `All` privileges on {{fleet}} and Integrations. Since many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.
17+
* Integrations Server (included by default in every {{ess}} deployment)
18+
19+
To get started quickly, spin up a deployment of our hosted {{ess}}. The {{ess}} is available on AWS, GCP, and Azure. [Try it out for free](https://cloud.elastic.co/registration?page=docs&placement=docs-body).
20+
21+
22+
## Install and configure the standalone {{agent}} [logs-stream-install-config-agent]
23+
24+
Complete these steps to install and configure the standalone {{agent}} and send your log data to {{es}}:
25+
26+
1. [Download and extract the {{agent}} installation package.](../../../solutions/observability/logs/stream-any-log-file.md#logs-stream-extract-agent)
27+
2. [Install and start the {{agent}}.](../../../solutions/observability/logs/stream-any-log-file.md#logs-stream-install-agent)
28+
3. [Configure the {{agent}}.](../../../solutions/observability/logs/stream-any-log-file.md#logs-stream-agent-config)
29+
30+
31+
### Step 1: Download and extract the {{agent}} installation package [logs-stream-extract-agent]
32+
33+
On your host, download and extract the installation package that corresponds with your system:
34+
35+
:::::::{tab-set}
36+
37+
::::::{tab-item} macOS
38+
Version 9.0.0-beta1 of {{agent}} has not yet been released.
39+
::::::
40+
41+
::::::{tab-item} Linux
42+
Version 9.0.0-beta1 of {{agent}} has not yet been released.
43+
::::::
44+
45+
::::::{tab-item} Windows
46+
Version 9.0.0-beta1 of {{agent}} has not yet been released.
47+
::::::
48+
49+
::::::{tab-item} DEB
50+
Version 9.0.0-beta1 of {{agent}} has not yet been released.
51+
::::::
52+
53+
::::::{tab-item} RPM
54+
Version 9.0.0-beta1 of {{agent}} has not yet been released.
55+
::::::
56+
57+
:::::::
58+
59+
### Step 2: Install and start the {{agent}} [logs-stream-install-agent]
60+
61+
After downloading and extracting the installation package, you’re ready to install the {{agent}}. From the agent directory, run the install command that corresponds with your system:
62+
63+
::::{note}
64+
On macOS, Linux (tar package), and Windows, run the `install` command to install and start {{agent}} as a managed service and start the service. The DEB and RPM packages include a service unit for Linux systems with systemd, For these systems, you must enable and start the service.
65+
::::
66+
67+
68+
:::::::{tab-set}
69+
70+
::::::{tab-item} macOS
71+
::::{tip}
72+
You must run this command as the root user because some integrations require root privileges to collect sensitive data.
73+
::::
74+
75+
76+
```shell
77+
sudo ./elastic-agent install
78+
```
79+
::::::
80+
81+
::::::{tab-item} Linux
82+
::::{tip}
83+
You must run this command as the root user because some integrations require root privileges to collect sensitive data.
84+
::::
85+
86+
87+
```shell
88+
sudo ./elastic-agent install
89+
```
90+
::::::
91+
92+
::::::{tab-item} Windows
93+
Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select **Run As Administrator**).
94+
95+
From the PowerShell prompt, change to the directory where you installed {{agent}}, and run:
96+
97+
```shell
98+
.\elastic-agent.exe install
99+
```
100+
::::::
101+
102+
::::::{tab-item} DEB
103+
::::{tip}
104+
You must run this command as the root user because some integrations require root privileges to collect sensitive data.
105+
::::
106+
107+
108+
```shell
109+
sudo systemctl enable elastic-agent <1>
110+
sudo systemctl start elastic-agent
111+
```
112+
113+
1. The DEB package includes a service unit for Linux systems with systemd. On these systems, you can manage {{agent}} by using the usual systemd commands. If you don’t have systemd, run `sudo service elastic-agent start`.
114+
::::::
115+
116+
::::::{tab-item} RPM
117+
::::{tip}
118+
You must run this command as the root user because some integrations require root privileges to collect sensitive data.
119+
::::
120+
121+
122+
```shell
123+
sudo systemctl enable elastic-agent <1>
124+
sudo systemctl start elastic-agent
125+
```
126+
127+
1. The RPM package includes a service unit for Linux systems with systemd. On these systems, you can manage {{agent}} by using the usual systemd commands. If you don’t have systemd, run `sudo service elastic-agent start`.
128+
::::::
129+
130+
:::::::
131+
During installation, you’re prompted with some questions:
132+
133+
1. When asked if you want to install the agent as a service, enter `Y`.
134+
2. When asked if you want to enroll the agent in Fleet, enter `n`.
135+
136+
137+
### Step 3: Configure the {{agent}} [logs-stream-agent-config]
138+
139+
With your agent installed, configure it by updating the `elastic-agent.yml` file.
140+
141+
142+
#### Locate your configuration file [logs-stream-yml-location]
143+
144+
After installing the agent, you’ll find the `elastic-agent.yml` in one of the following locations according to your system:
145+
146+
:::::::{tab-set}
147+
148+
::::::{tab-item} macOS
149+
Main {{agent}} configuration file location:
150+
151+
`/Library/Elastic/Agent/elastic-agent.yml`
152+
::::::
153+
154+
::::::{tab-item} Linux
155+
Main {{agent}} configuration file location:
156+
157+
`/opt/Elastic/Agent/elastic-agent.yml`
158+
::::::
159+
160+
::::::{tab-item} Windows
161+
Main {{agent}} configuration file location:
162+
163+
`C:\Program Files\Elastic\Agent\elastic-agent.yml`
164+
::::::
165+
166+
::::::{tab-item} DEB
167+
Main {{agent}} configuration file location:
168+
169+
`/etc/elastic-agent/elastic-agent.yml`
170+
::::::
171+
172+
::::::{tab-item} RPM
173+
Main {{agent}} configuration file location:
174+
175+
`/etc/elastic-agent/elastic-agent.yml`
176+
::::::
177+
178+
:::::::
179+
180+
#### Update your configuration file [logs-stream-example-config]
181+
182+
The following is an example of a standalone {{agent}} configuration. To configure your {{agent}}, replace the contents of the `elastic-agent.yml` file with this configuration:
183+
184+
```yaml
185+
outputs:
186+
default:
187+
type: elasticsearch
188+
hosts: '<your-elasticsearch-endpoint>:<port>'
189+
api_key: 'your-api-key'
190+
inputs:
191+
- id: your-log-id
192+
type: filestream
193+
streams:
194+
- id: your-log-stream-id
195+
data_stream:
196+
dataset: example
197+
paths:
198+
- /var/log/your-logs.log
199+
```
200+
201+
Next, set the values for these fields:
202+
203+
* `hosts` – Copy the {{es}} endpoint from **Help menu (![help icon](../../../images/observability-help-icon.png "")) → Connection details**. For example, `https://my-deployment.es.us-central1.gcp.cloud.es.io:443`.
204+
* `api-key` – Use an API key to grant the agent access to {{es}}. To create an API key for your agent, refer to the [Create API keys for standalone agents](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/grant-access-to-elasticsearch.md#create-api-key-standalone-agent) documentation.
205+
206+
::::{note}
207+
The API key format should be `<id>:<key>`. Make sure you selected **Beats** when you created your API key. Base64 encoded API keys are not currently supported in this configuration.
208+
::::
209+
210+
* `inputs.id` – A unique identifier for your input.
211+
* `type` – The type of input. For collecting logs, set this to `filestream`.
212+
* `streams.id` – A unique identifier for your stream of log data.
213+
* `data_stream.dataset` – The name for your dataset data stream. Name this data stream anything that signifies the source of the data. In this configuration, the dataset is set to `example`. The default value is `generic`.
214+
* `paths` – The path to your log files. You can also use a pattern like `/var/log/your-logs.log*`.
215+
216+
217+
#### Restart the {{agent}} [logs-stream-restart-agent]
218+
219+
After updating your configuration file, you need to restart the {{agent}}:
220+
221+
First, stop the {{agent}} and its related executables using the command that works with your system:
222+
223+
:::::::{tab-set}
224+
225+
::::::{tab-item} macOS
226+
```shell
227+
sudo launchctl unload /Library/LaunchDaemons/co.elastic.elastic-agent.plist
228+
```
229+
230+
::::{note}
231+
{{agent}} will restart automatically if the system is rebooted.
232+
::::
233+
::::::
234+
235+
::::::{tab-item} Linux
236+
```shell
237+
sudo service elastic-agent stop
238+
```
239+
240+
::::{note}
241+
{{agent}} will restart automatically if the system is rebooted.
242+
::::
243+
::::::
244+
245+
::::::{tab-item} Windows
246+
```shell
247+
Stop-Service Elastic Agent
248+
```
249+
250+
If necessary, use Task Manager on Windows to stop {{agent}}. This will kill the `elastic-agent` process and any sub-processes it created (such as {{beats}}).
251+
252+
::::{note}
253+
{{agent}} will restart automatically if the system is rebooted.
254+
::::
255+
::::::
256+
257+
::::::{tab-item} DEB
258+
The DEB package includes a service unit for Linux systems with systemd. On these systems, you can manage {{agent}} by using the usual systemd commands.
259+
260+
Use `systemctl` to stop the agent:
261+
262+
```shell
263+
sudo systemctl stop elastic-agent
264+
```
265+
266+
Otherwise, use:
267+
268+
```shell
269+
sudo service elastic-agent stop
270+
```
271+
272+
::::{note}
273+
{{agent}} will restart automatically if the system is rebooted.
274+
::::
275+
::::::
276+
277+
::::::{tab-item} RPM
278+
The RPM package includes a service unit for Linux systems with systemd. On these systems, you can manage {{agent}} by using the usual systemd commands.
279+
280+
Use `systemctl` to stop the agent:
281+
282+
```shell
283+
sudo systemctl stop elastic-agent
284+
```
285+
286+
Otherwise, use:
287+
288+
```shell
289+
sudo service elastic-agent stop
290+
```
291+
292+
::::{note}
293+
{{agent}} will restart automatically if the system is rebooted.
294+
::::
295+
::::::
296+
297+
:::::::
298+
Next, restart the {{agent}} using the command that works with your system:
299+
300+
:::::::{tab-set}
301+
302+
::::::{tab-item} macOS
303+
```shell
304+
sudo launchctl load /Library/LaunchDaemons/co.elastic.elastic-agent.plist
305+
```
306+
::::::
307+
308+
::::::{tab-item} Linux
309+
```shell
310+
sudo service elastic-agent start
311+
```
312+
::::::
313+
314+
::::::{tab-item} Windows
315+
```shell
316+
Start-Service Elastic Agent
317+
```
318+
::::::
319+
320+
::::::{tab-item} DEB
321+
The DEB package includes a service unit for Linux systems with systemd. On these systems, you can manage {{agent}} by using the usual systemd commands.
322+
323+
Use `systemctl` to start the agent:
324+
325+
```shell
326+
sudo systemctl start elastic-agent
327+
```
328+
329+
Otherwise, use:
330+
331+
```shell
332+
sudo service elastic-agent start
333+
```
334+
::::::
335+
336+
::::::{tab-item} RPM
337+
The RPM package includes a service unit for Linux systems with systemd. On these systems, you can manage {{agent}} by using the usual systemd commands.
338+
339+
Use `systemctl` to start the agent:
340+
341+
```shell
342+
sudo systemctl start elastic-agent
343+
```
344+
345+
Otherwise, use:
346+
347+
```shell
348+
sudo service elastic-agent start
349+
```
350+
::::::
351+
352+
:::::::
353+
354+
## Troubleshoot your {{agent}} configuration [logs-stream-troubleshooting]
355+
356+
If you’re not seeing your log files in {{kib}}, verify the following in the `elastic-agent.yml` file:
357+
358+
* The path to your logs file under `paths` is correct.
359+
* Your API key is in `<id>:<key>` format. If not, your API key may be in an unsupported format, and you’ll need to create an API key in **Beats** format.
360+
361+
If you’re still running into issues, see [{{agent}} troubleshooting](../../../troubleshoot/ingest/fleet/common-problems.md) and [Configure standalone Elastic Agents](asciidocalypse://docs/docs-content/docs/reference/ingestion-tools/fleet/configure-standalone-elastic-agents.md).
362+
363+
364+
## Next steps [logs-stream-next-steps]
365+
366+
After you have your agent configured and are streaming log data to {{es}}:
367+
368+
* Refer to the [Parse and organize logs](../../../solutions/observability/logs/parse-route-logs.md) documentation for information on extracting structured fields from your log data, rerouting your logs to different data streams, and filtering and aggregating your log data.
369+
* Refer to the [Filter and aggregate logs](../../../solutions/observability/logs/filter-aggregate-logs.md) documentation for information on filtering and aggregating your log data to find specific information, gain insight, and monitor your systems more efficiently.

0 commit comments

Comments
 (0)