Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions manage-data/ingest/transform-enrich/error-handling.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
mapped_pages:
- https://www.elastic.co/docs/manage-data/ingest/transform-enrich/error-handling.html
applies_to:
stack: ga
serverless: ga
Expand Down
21 changes: 9 additions & 12 deletions manage-data/ingest/transform-enrich/ingest-lag.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
mapped_pages:
- https://www.elastic.co/docs/manage-data/ingest/transform-enrich/calculate-ingest-lag.html
applies_to:
stack: ga
serverless: ga
Expand Down Expand Up @@ -99,16 +97,6 @@ As discussed above `@timestamp` is set to the timestamp from within the collecte

Regardless of the chosen architecture, add a `remove` processor at the end of the pipeline to drop the `_tmp` field. The raw timestamps from the various processing steps are not needed, as the latency in seconds should be sufficient. For additional pipeline architectures, refer to [Ingest architectures](../ingest-reference-architectures.md).

## Logstash

When Logstash is added to the architecture we must add a timestamp, this can only be done by using Ruby and the simplest form is this:

```
ruby {
code => "event.set('[_tmp][logstash_seen]', Time.now());"
}
```

### Elastic Agent to Elasticsearch

Use `@timestamp` and `event.ingested` to calculate the difference. This will give you the following document. The `event.ingestion.latency` is in seconds.
Expand Down Expand Up @@ -156,6 +144,15 @@ POST _ingest/pipeline/_simulate
}
}
```
### Logstash

When Logstash is added to the architecture we must add a timestamp, this can only be done by using Ruby and the simplest form is this:

```
ruby {
code => "event.set('[_tmp][logstash_seen]', Time.now());"
}
```

### Elastic Agent => Logstash => Elasticsearch

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
mapped_pages:
- https://www.elastic.co/docs/manage-data/ingest/transform-enrich/common-mistakes.html
applies_to:
stack: ga
serverless: ga
Expand Down