Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion manage-data/ingest/tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Depending on the type of data you want to ingest, you have a number of methods a

| Tools | Usage | Links to more information |
| ------- | --------------- | ------------------------- |
| Integrations | Ingest data using a variety of Elastic integrations. | [Elastic Integrations](asciidocalypse://docs/integration-docs/docs/reference/index.md) |
| Integrations | Ingest data using a variety of Elastic integrations. | [Elastic Integrations](integration-docs://reference/index.md) |
| File upload | Upload data from a file and inspect it before importing it into {{es}}. | [Upload data files](/manage-data/ingest/upload-data-files.md) |
| APIs | Ingest data through code by using the APIs of one of the language clients or the {{es}} HTTP APIs. | [Document APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-document) |
| OpenTelemetry | Collect and send your telemetry data to Elastic Observability | [Elastic Distributions of OpenTelemetry](https://github.com/elastic/opentelemetry?tab=readme-ov-file#elastic-distributions-of-opentelemetry) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Access to individual features is governed by Elastic user roles. Consult your ad

| Feature | Description | Available in |
| --- | --- | --- |
| [Integrations](asciidocalypse://docs/integration-docs/docs/reference/index.md) | Connect your data to your project. | [![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) |
| [Integrations](integration-docs://reference/index.md) | Connect your data to your project. | [![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) |
| [Fleet and Elastic Agent](/reference/ingestion-tools/fleet/index.md) | Add monitoring for logs, metrics, and other types of data to a host. | [![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) |
| [{{data-sources-cap}}](../../../explore-analyze/find-and-organize/data-views.md) | Manage the fields in the data views that retrieve your data from {{es-serverless}}. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) |
| [Index management](../../../manage-data/data-store/index-basics.md) | View index settings, mappings, and statistics and perform operations on indices. | [![Elasticsearch](../../../images/serverless-es-badge.svg "")](../../../solutions/search.md)[![Observability](../../../images/serverless-obs-badge.svg "")](../../../solutions/observability.md)[![Security](../../../images/serverless-sec-badge.svg "")](../../../solutions/security/elastic-security-serverless.md) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ inputs: <7>
9. For available input types, refer to [{{agent}} inputs](/reference/ingestion-tools/fleet/elastic-agent-inputs-list.md).
10. Learn about [Data streams](/reference/ingestion-tools/fleet/data-streams.md) for time series data.
11. Specify a unique ID for each individual input stream. Naming the ID by appending the associated `data_stream` dataset (for example `{{user-defined-unique-id}}-apache.access` or `{{user-defined-unique-id}}-apache.error`) is a recommended practice, but any unique ID will work.
12. Refer to [Logs](asciidocalypse://docs/integration-docs/docs/reference/apache.md#apache-logs) in the Apache HTTP Server integration documentation for the logs available to ingest and exported fields.
12. Refer to [Logs](integration-docs://reference/apache/index.md) in the Apache HTTP Server integration documentation for the logs available to ingest and exported fields.
13. Path to the log files to be monitored.


Expand Down Expand Up @@ -128,7 +128,7 @@ inputs: <7>
9. Learn about [Data streams](/reference/ingestion-tools/fleet/data-streams.md) for time series data.
10. Specify a unique ID for each individual input stream. Naming the ID by appending the associated `data_stream` dataset (for example `{{user-defined-unique-id}}-apache.status`) is a recommended practice, but any unique ID will work.
11. A user-defined dataset. You can specify anything that makes sense to signify the source of the data.
12. Refer to [Metrics](asciidocalypse://docs/integration-docs/docs/reference/apache.md#apache-metrics) in the Apache HTTP Server integration documentation for the type of metrics collected and exported fields.
12. Refer to [Metrics](integration-docs://reference/apache/index.md) in the Apache HTTP Server integration documentation for the type of metrics collected and exported fields.



Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ inputs: <7>
9. For available input types, refer to [{{agent}} inputs](/reference/ingestion-tools/fleet/elastic-agent-inputs-list.md).
10. Learn about [Data streams](/reference/ingestion-tools/fleet/data-streams.md) for time series data.
11. Specify a unique ID for each individual input stream. Naming the ID by appending the associated `data_stream` dataset (for example `{{user-defined-unique-id}}-nginx.access` or `{{user-defined-unique-id}}-nginx.error`) is a recommended practice, but any unique ID will work.
12. Refer to [Logs reference](asciidocalypse://docs/integration-docs/docs/reference/nginx.md#nginx-logs-reference) in the Nginx HTTP integration documentation for the logs available to ingest and exported fields.
12. Refer to [Logs reference](integration-docs://reference/nginx/index.md#logs-reference) in the Nginx HTTP integration documentation for the logs available to ingest and exported fields.
13. Path to the log files to be monitored.


Expand Down Expand Up @@ -135,7 +135,7 @@ inputs: <7>
10. Learn about [Data streams](/reference/ingestion-tools/fleet/data-streams.md) for time series data.
11. Specify a unique ID for each individual input stream. Naming the ID by appending the associated `data_stream` dataset (for example `{{user-defined-unique-id}}-nginx.stubstatus`) is a recommended practice, but any unique ID will work.
12. A user-defined dataset. You can specify anything that makes sense to signify the source of the data.
13. Refer to [Metrics reference](asciidocalypse://docs/integration-docs/docs/reference/nginx.md#nginx-metrics-reference) in the Nginx integration documentation for the type of metrics collected and exported fields.
13. Refer to [Metrics reference](integration-docs://reference/nginx/index.md#metrics-reference) in the Nginx integration documentation for the type of metrics collected and exported fields.



Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ mapped_pages:

Kubernetes [metadata](/solutions/observability/infra-and-hosts/tutorial-observe-kubernetes-deployments.md#beats-metadata) refer to contextual information extracted from Kubernetes resources. Metadata information enrich metrics and logs collected from a Kubernetes cluster, enabling deeper insights into Kubernetes environments.

When the {{agent}}'s policy includes the [{{k8s}} Integration](asciidocalypse://docs/integration-docs/docs/reference/kubernetes.md) which configures the collection of Kubernetes related metrics and container logs, the mechanisms used for the metadata enrichment are:
When the {{agent}}'s policy includes the [{{k8s}} Integration](integration-docs://reference/kubernetes.md) which configures the collection of Kubernetes related metrics and container logs, the mechanisms used for the metadata enrichment are:

* [Kubernetes Provider](/reference/ingestion-tools/fleet/kubernetes-provider.md) for log collection
* Kubernetes metadata enrichers for metrics
Expand Down
2 changes: 1 addition & 1 deletion reference/ingestion-tools/fleet/dissect-processor.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,5 +83,5 @@ This configuration produces fields like:

`service.name` is an ECS [keyword field](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md), which means that you can use it in {{es}} for filtering, sorting, and aggregations.

When possible, use ECS-compatible field names. For more information, see the [Elastic Common Schema](asciidocalypse://docs/integration-docs/docs/reference/index.md) documentation.
When possible, use ECS-compatible field names. For more information, see the [Elastic Common Schema](integration-docs://reference/index.md) documentation.

12 changes: 6 additions & 6 deletions reference/ingestion-tools/fleet/elastic-agent-unprivileged.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,12 @@ As well, a warning is displayed in {{kib}} if you try to add an integration that

Examples of integrations that require {{agent}} to have administrative privileges are:

* [{{elastic-defend}}](asciidocalypse://docs/integration-docs/docs/reference/endpoint.md)
* [Auditd Manager](asciidocalypse://docs/integration-docs/docs/reference/auditd_manager.md)
* [File Integrity Monitoring](asciidocalypse://docs/integration-docs/docs/reference/fim.md)
* [Network Packet Capture](asciidocalypse://docs/integration-docs/docs/reference/network_traffic.md)
* [System Audit](asciidocalypse://docs/integration-docs/docs/reference/system_audit.md)
* [Universal Profiling Agent](asciidocalypse://docs/integration-docs/docs/reference/profiler_agent.md)
* [{{elastic-defend}}](integration-docs://reference/endpoint/index.md)
* [Auditd Manager](integration-docs://reference/auditd_manager/index.md)
* [File Integrity Monitoring](integration-docs://reference/fim/index.md)
* [Network Packet Capture](integration-docs://reference/network_traffic/index.md)
* [System Audit](integration-docs://reference/system_audit/index.md)
* [Universal Profiling Agent](integration-docs://reference/profiler_agent/index.md)


## Viewing an {{agent}} privilege mode [unprivileged-view-mode]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -311,4 +311,4 @@ Congratulations! You have successfully set up monitoring for nginx using standal
## What’s next? [_whats_next]

* Learn more about [{{fleet}} and {{agent}}](/reference/ingestion-tools/fleet/index.md).
* Learn more about [{{integrations}}](asciidocalypse://docs/integration-docs/docs/reference/index.md).
* Learn more about [{{integrations}}](integration-docs://reference/index.md).
Original file line number Diff line number Diff line change
Expand Up @@ -310,4 +310,4 @@ Congratulations! You have successfully set up monitoring for nginx using standal
## What’s next? [_whats_next_2]

* Learn more about [{{fleet}} and {{agent}}](/reference/ingestion-tools/fleet/index.md).
* Learn more about [{{integrations}}](asciidocalypse://docs/integration-docs/docs/reference/index.md).
* Learn more about [{{integrations}}](integration-docs://reference/index.md).
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ The host to use for metrics retrieval. If not defined, the host will be set as t

### `co.elastic.hints/data_stream` [_co_elastic_hintsdata_stream]

The list of data streams to enable. If not specified, the integration’s default data streams are used. To find the defaults, refer to the [Elastic integrations documentation](asciidocalypse://docs/integration-docs/docs/reference/index.md).
The list of data streams to enable. If not specified, the integration’s default data streams are used. To find the defaults, refer to the [Elastic integrations documentation](integration-docs://reference/index.md).

If data streams are specified, additional hints can be defined per data stream. For example, `co.elastic.hints/info.period: 5m` if the data stream specified is `info` for the [Redis module](beats://reference/metricbeat/metricbeat-module-redis.md).

Expand Down
2 changes: 1 addition & 1 deletion reference/ingestion-tools/fleet/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Looking for a general guide that explores all of your options for ingesting data

## {{integrations}}

[{{integrations}}](asciidocalypse://docs/integration-docs/docs/reference/index.md) provide an easy way to connect Elastic to external services and systems, and quickly get insights or take action. They can collect new sources of data, and they often ship with out-of-the-box assets like dashboards, visualizations, and pipelines to extract structured fields out of logs and events. This makes it easier to get insights within seconds. Integrations are available for popular services and platforms like Nginx or AWS, as well as many generic input types like log files.
[{{integrations}}](integration-docs://reference/index.md) provide an easy way to connect Elastic to external services and systems, and quickly get insights or take action. They can collect new sources of data, and they often ship with out-of-the-box assets like dashboards, visualizations, and pipelines to extract structured fields out of logs and events. This makes it easier to get insights within seconds. Integrations are available for popular services and platforms like Nginx or AWS, as well as many generic input types like log files.

{{kib}} provides a web-based UI to add and manage integrations. You can browse a unified view of available integrations that shows both {{agent}} and {{beats}} integrations.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Custom pipelines can be used to add custom data processing, like adding fields,

## Metadata enrichment for Kubernetes [_metadata_enrichment_for_kubernetes]

The [{{k8s}} Integration](asciidocalypse://docs/integration-docs/docs/reference/kubernetes.md) is used to collect logs and metrics from Kubernetes clusters with {{agent}}. During the collection, the integration enhances the collected information with extra useful information that users can correlate with different Kubernetes assets. This additional information added on top of collected data, such as labels, annotations, ancestor names of Kubernetes assets, and others, are called metadata.
The [{{k8s}} Integration](integration-docs://reference/kubernetes.md) is used to collect logs and metrics from Kubernetes clusters with {{agent}}. During the collection, the integration enhances the collected information with extra useful information that users can correlate with different Kubernetes assets. This additional information added on top of collected data, such as labels, annotations, ancestor names of Kubernetes assets, and others, are called metadata.

The [{{k8s}} Provider](/reference/ingestion-tools/fleet/kubernetes-provider.md) offers the `add_resource_metadata` option to configure the metadata enrichment options.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Using our lab environment as an example, we can observe the following resource c

### CPU and RSS memory size [_cpu_and_rss_memory_size]

We tested using an AWS `m7i.large` instance type with 2 vCPUs, 8.0 GB of memory, and up to 12.5 Gbps of bandwidth. The tests ingested a single log file using both the [throughput and scale preset](/reference/ingestion-tools/fleet/elasticsearch-output.md#output-elasticsearch-performance-tuning-settings) with self monitoring enabled. These tests are representative of use cases that attempt to ingest data as fast as possible. This does not represent the resource overhead when using [{{elastic-defend}}](asciidocalypse://docs/integration-docs/docs/reference/endpoint.md).
We tested using an AWS `m7i.large` instance type with 2 vCPUs, 8.0 GB of memory, and up to 12.5 Gbps of bandwidth. The tests ingested a single log file using both the [throughput and scale preset](/reference/ingestion-tools/fleet/elasticsearch-output.md#output-elasticsearch-performance-tuning-settings) with self monitoring enabled. These tests are representative of use cases that attempt to ingest data as fast as possible. This does not represent the resource overhead when using [{{elastic-defend}}](integration-docs://reference/endpoint/index.md).

| | | |
| --- | --- | --- |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ The {{fleet}} integration assets are not supposed to work when sending arbitrary

While it’s possible to include {{fleet}} and {{agent}} integration assets in a custom integration, this is not recommended nor supported. Assets from another integration should not be referenced directly from a custom integration.

As an example scenario, one may want to ingest Redis logs from Kafka. This can be done using the [Redis integration](asciidocalypse://docs/integration-docs/docs/reference/redis-intro.md), but only certain files and paths are allowed. It’s technically possible to use the [Custom Kafka Logs integration](asciidocalypse://docs/integration-docs/docs/reference/kafka_log.md) with a custom ingest pipeline, referencing the ingest pipeline of the Redis integration to ingest logs into the index templates of the Custom Kafka Logs integration data streams.
As an example scenario, one may want to ingest Redis logs from Kafka. This can be done using the [Redis integration](integration-docs://reference/redis-intro.md), but only certain files and paths are allowed. It’s technically possible to use the [Custom Kafka Logs integration](integration-docs://reference/kafka_log.md) with a custom ingest pipeline, referencing the ingest pipeline of the Redis integration to ingest logs into the index templates of the Custom Kafka Logs integration data streams.

However, referencing assets of an integration from another custom integration is not recommended nor supported. A configuration as described above can break when the integration is upgraded, as can happen automatically.

Expand Down
2 changes: 1 addition & 1 deletion reference/ingestion-tools/fleet/manage-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ mapped_pages:


::::{admonition}
Integrations are available for a wide array of popular services and platforms. To see the full list of available integrations, go to the **Integrations** page in {{kib}}, or visit [Elastic Integrations](asciidocalypse://docs/integration-docs/docs/reference/index.md).
Integrations are available for a wide array of popular services and platforms. To see the full list of available integrations, go to the **Integrations** page in {{kib}}, or visit [Elastic Integrations](integration-docs://reference/index.md).

{{agent}} integrations provide a simple, unified way to collect data from popular apps and services, and protect systems from security threats.

Expand Down
Loading
Loading