You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/data-streams/failure-store-recipes.asciidoc
+12Lines changed: 12 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,7 @@ When something goes wrong during ingestion it is often not an isolated event. In
8
8
When a document fails in an ingest pipeline it can be difficult to figure out exactly what went wrong and where. When these failures are captured by the failure store during this part of the ingestion process, they will contain additional debugging information. Failed documents will note the type of processor and which pipeline was executing when the failure occurred. Failed documents will also contain a pipeline trace which keeps track of any nested pipeline calls that the document was in at time of failure.
9
9
10
10
To demonstrate this, we will follow a failed document through an unfamiliar data stream and ingest pipeline:
11
+
11
12
[source,console]
12
13
----
13
14
POST my-datastream-ingest/_doc
@@ -18,6 +19,7 @@ POST my-datastream-ingest/_doc
18
19
}
19
20
}
20
21
----
22
+
21
23
[source,console-result]
22
24
----
23
25
{
@@ -38,10 +40,12 @@ POST my-datastream-ingest/_doc
38
40
<1> The document was sent to the failure store.
39
41
40
42
Now we search the failure store to check the failure document to see what went wrong.
43
+
41
44
[source,console]
42
45
----
43
46
GET my-datastream-ingest::failures/_search
44
47
----
48
+
45
49
[source,console-result]
46
50
----
47
51
{
@@ -129,10 +133,12 @@ GET _ingest/pipeline/ingest-step-2
129
133
<2> This field was missing from the document at this point.
130
134
131
135
There is only a set processor in the `ingest-step-2` pipeline so this is likely not where the root problem is. Remembering the `pipeline_trace` field on the failure we find that `ingest-step-1` was the original pipeline called for this document. It is likely the data stream's default pipeline. Pulling its definition we find the following:
136
+
132
137
[source,console]
133
138
----
134
139
GET _ingest/pipeline/ingest-step-1
135
140
----
141
+
136
142
[source,console-result]
137
143
----
138
144
{
@@ -214,6 +220,7 @@ PUT _ingest/pipeline/complicated-processor
214
220
----
215
221
216
222
We ingest some data and find that it was sent to the failure store.
223
+
217
224
[source,console]
218
225
----
219
226
POST my-datastream-ingest/_doc?pipeline=complicated-processor
@@ -222,6 +229,7 @@ POST my-datastream-ingest/_doc?pipeline=complicated-processor
222
229
"counter_name": "test"
223
230
}
224
231
----
232
+
225
233
[source,console-result]
226
234
----
227
235
{
@@ -240,10 +248,12 @@ POST my-datastream-ingest/_doc?pipeline=complicated-processor
240
248
}
241
249
----
242
250
On checking the failure, we can quickly identify the tagged processor that caused the problem.
251
+
243
252
[source,console]
244
253
----
245
254
GET my-datastream-ingest::failures/_search
246
255
----
256
+
247
257
[source,console-result]
248
258
----
249
259
{
@@ -357,6 +367,7 @@ Failures that occurred during ingest processing will be stored as they were befo
357
367
358
368
====== Step 1: Separate out which failures to replay
359
369
Start off by constructing a query that can be used to consistently identify which failures will be remediated.
370
+
360
371
[source,console]
361
372
----
362
373
POST my-datastream-ingest-example::failures/_search
@@ -459,6 +470,7 @@ Take note of the documents that are returned. We can use these to simulate that
459
470
460
471
====== Step 2: Fix the original problem
461
472
Because ingest pipeline failures need to be reprocessed by their original pipelines, any problems with those pipelines should be fixed before remediating failures. Investigating the pipeline mentioned in the example above shows that there is a processor that expects a field to be present that is not always present.
0 commit comments