You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: manage-data/ingest/transform-enrich/readable-maintainable-ingest-pipelines.md
+21-19Lines changed: 21 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,31 +35,31 @@ Below are some general guidelines for choosing the right option in a situation.
35
35
stack: ga 9.2.0
36
36
```
37
37
38
-
Starting with version [9.2](https://github.com/elastic/elasticsearch/pull/131581) we have access to the field API that enables the usage of this API in conditionals (the `if` statement of your processor). Otherwise you can always use the field API in the script processor itself.
38
+
The field API can be used in conditionals (the `if` statement of your processor) in addition to being used in the script processor itself.
39
39
40
40
:::{note}
41
41
This is the preferred way to access fields.
42
42
:::
43
43
44
44
**Benefits**
45
45
46
-
- Clean and easy to read
47
-
- Handles null values automatically
48
-
- Adds support for additional functions like `isEmpty()` to ease comparisions.
49
-
- Handles dots as part of field name
50
-
- Handles dots as dot walking for object notation
46
+
- Clean and easy to read.
47
+
- Handles null values automatically.
48
+
- Adds support for additional functions like `isEmpty()` to ease comparisons.
49
+
- Handles dots as part of field name.
50
+
- Handles dots as dot walking for object notation.
51
51
- Handles special characters.
52
52
53
53
**Limitations**
54
54
55
-
-Only available starting in 9.2 for conditionals.
55
+
-Not available in all versions for conditionals.
56
56
57
57
### Dot notation [dot-notation]
58
58
59
59
**Benefits**
60
60
61
61
- Clean and easy to read.
62
-
- Supports null safety operations `?`. Read more in [Use null safe operators (`?.`)](#null-safe-operators).
62
+
- Supports null safe operations `?`. Read more in [Use null safe operators (`?.`)](#null-safe-operators).
63
63
64
64
**Limitations**
65
65
@@ -78,7 +78,7 @@ This is the preferred way to access fields.
78
78
**Limitations**
79
79
80
80
- Slightly more verbose than dot notation.
81
-
- No support for null safety operations `?`.
81
+
- No support for null safe operations `?`.
82
82
Use [Dot notation](#dot-notation) instead.
83
83
84
84
### Mixed dot and bracket notation
@@ -391,8 +391,10 @@ All of the above discussed ways to [access fields](#access-fields) and retrieve
391
391
The fields API is the recommended way to add new fields.
392
392
:::
393
393
394
-
**Fields API**
395
-
We get the following field `cpu.usage` and we want to rename it to `system.cpu.total.norm.pct` which represents a scale from 0-1.0, where 1 is the equivalent of 100%.
394
+
For example, add a new `system.cpu.total.norm.pct` field based on the value of the `cpu.usage` field. The value of the existing `cpu.usage` field is a number on a scale of 0-100. The value of the new `system.cpu.total.norm.pct` field will be on a scale from 0-1.0 where 1 is the equivalent of 100 in the `cpu.usage` field.
395
+
396
+
**Option 1: Fields API (preferred)**
397
+
Create a new `system.cpu.total.norm.pct` field and set the value to the value of the `cpu.usage` field divided by `100.0`.
396
398
397
399
```json
398
400
POST _ingest/pipeline/_simulate
@@ -419,11 +421,11 @@ POST _ingest/pipeline/_simulate
419
421
}
420
422
}
421
423
```
422
-
1.Our field expects 0-1 and not 0-100, we will have to divide by 100 to get the right representation.
424
+
1.This field expects 0-1 and not 0-100. When renaming the field, divide this value by 100 to get the correct value.
423
425
2. The `field` API is exposed as `field(<field name>)`. The `set(<value>)` is responsible for setting the value. Inside we use the `$(<field name>, fallback)` to read the value out of the existing field. Lastly we divide by `100.0`. The `.0` is important, otherwise it will perform an integer only division and return just 0 instead of 0.9.
424
426
425
-
**No fields API**
426
-
Without the field API this can also be achieved. However there is much more code involved, as we have to ensure that we can walk the full path of `system.cpu.total.norm.pct`.
427
+
**Option 2: Without the fields API**
428
+
Without the field API, there is much more code involvedto ensure that you can walk the full path of `system.cpu.total.norm.pct`.
427
429
428
430
```json
429
431
{
@@ -446,10 +448,10 @@ Without the field API this can also be achieved. However there is much more code
446
448
}
447
449
}
448
450
```
449
-
1.We need to check whether the objects are null or not and then create them.
450
-
2.We create a new HashMap to store all the objects in it.
451
-
3. Instead of writing `new HashMap()` we can use the shortcut `[:]`.
452
-
4.We perform the same calculation as above and set the value.
451
+
1.Check whether the objects are null or not and then create them.
452
+
2.Create a new `HashMap` to store all the objects in it.
453
+
3. Instead of writing `new HashMap()`, use the shortcut `[:]`.
454
+
4.Perform the same calculation as above and set the value.
453
455
454
456
### Calculate `event.duration` in a complex manner
455
457
@@ -654,7 +656,7 @@ In this example, `{{tags.0}}` retrieves the first element of the `tags` array (`
654
656
655
657
### Transform into a JSON string
656
658
657
-
Whenever you need to store the original `_source` within a field `event.original`, we can use mustache function `{{#toJson}}<field>{{/toJson}}`.
659
+
Whenever you need to store the original `_source` within a field `event.original`, use mustache function `{{#toJson}}<field>{{/toJson}}`.
0 commit comments