Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions manage-data/ingest/transform-enrich/error-handling.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
mapped_pages:
- https://www.elastic.co/docs/manage-data/ingest/transform-enrich/error-handling.html
applies_to:
stack: ga
serverless: ga
Expand Down
25 changes: 11 additions & 14 deletions manage-data/ingest/transform-enrich/ingest-lag.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
mapped_pages:
- https://www.elastic.co/docs/manage-data/ingest/transform-enrich/calculate-ingest-lag.html
applies_to:
stack: ga
serverless: ga
Expand Down Expand Up @@ -99,16 +97,6 @@ As discussed above `@timestamp` is set to the timestamp from within the collecte

Regardless of the chosen architecture, add a `remove` processor at the end of the pipeline to drop the `_tmp` field. The raw timestamps from the various processing steps are not needed, as the latency in seconds should be sufficient. For additional pipeline architectures, refer to [Ingest architectures](../ingest-reference-architectures.md).

## Logstash

When Logstash is added to the architecture we must add a timestamp, this can only be done by using Ruby and the simplest form is this:

```
ruby {
code => "event.set('[_tmp][logstash_seen]', Time.now());"
}
```

### Elastic Agent to Elasticsearch

Use `@timestamp` and `event.ingested` to calculate the difference. This will give you the following document. The `event.ingestion.latency` is in seconds.
Expand Down Expand Up @@ -156,8 +144,17 @@ POST _ingest/pipeline/_simulate
}
}
```
### Logstash

When Logstash is added to the architecture we must add a timestamp, this can only be done by using Ruby and the simplest form is this:

```
ruby {
code => "event.set('[_tmp][logstash_seen]', Time.now());"
}
```

### Elastic Agent => Logstash => Elasticsearch
#### Elastic Agent => Logstash => Elasticsearch

Elastic Agent populates the `@timestamp` field, but Logstash doesn't add any timestamp by default. Add a temporary timestamp, for example by setting `_tmp.logstash_seen`. With this, you can calculate the following latency values:

Expand Down Expand Up @@ -236,7 +233,7 @@ POST _ingest/pipeline/_simulate
}
```

### Elastic Agent => Logstash => Kafka => Logstash => Elasticsearch
#### Elastic Agent => Logstash => Kafka => Logstash => Elasticsearch

As with the previous scenario, adding an additional hop introduces another point where latency can occur. The recommendation is to add another temporary timestamp field. For more details, refer to the previous section.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
mapped_pages:
- https://www.elastic.co/docs/manage-data/ingest/transform-enrich/common-mistakes.html
applies_to:
stack: ga
serverless: ga
Expand Down