Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion _data-prepper/pipelines/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,5 @@ OpenSearch Data Prepper offers a range of built-in functions that can be used wi
- [`getMetadata()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/get-metadata/)
- [`hasTags()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/has-tags/)
- [`join()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/join/)
- [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/)
- [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/)
- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/)
79 changes: 79 additions & 0 deletions _data-prepper/pipelines/sublist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
layout: default
title: subList()
parent: Functions
grand_parent: Pipelines
nav_order: 35
---

# subList(<key>, <start_index, inclusive>, <end_index, exclusive>)

Check failure on line 9 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'subList(, <, inclusive>, <, exclusive>)' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'subList(, \u003c, inclusive\u003e, \u003c, exclusive\u003e)' is a heading and should be in sentence case.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 9, "column": 1}}}, "severity": "ERROR"}

The `subList()` function extracts a sublist from a list field in an event. It takes the following arguments:

Check failure on line 11 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 11, "column": 37}}}, "severity": "ERROR"}

- A JSON pointer to a list field in the event
- A start index (inclusive)
- An end index (exclusive)

The function returns the portion of the list between the specified start and end indexes. If the end index is `-1`, the function extracts elements from the start index to the end of the list.


## Examples

The following examples show how the `sublist()` function works.

### add_entries processor

Check failure on line 24 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.HeadingCapitalization] 'processor' is a heading and should be in sentence case. Raw Output: {"message": "[OpenSearch.HeadingCapitalization] 'processor' is a heading and should be in sentence case.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 24, "column": 17}}}, "severity": "ERROR"}

You can use `subList()` in the `add_entries` processor, as shown in the following example.

For example, the function will use the following input to extact a sublist:

Check failure on line 28 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 28, "column": 68}}}, "severity": "ERROR"}

Check failure on line 28 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: extact. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: extact. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 28, "column": 59}}}, "severity": "ERROR"}

```
input: {"my_list": [ 0, 1, 2, 3, 4, 5, 6]}
```

Then, the following configuration uses the `add_entries` processor to extract a sublist from `my_list`, starting at index 1 and ending before index 4:

Check failure on line 34 in _data-prepper/pipelines/sublist.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: sublist. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/sublist.md", "range": {"start": {"line": 34, "column": 81}}}, "severity": "ERROR"}

```yaml
add_entries:
entries:
- key: "my_list"
value_expression: '/subList(/my_list, 1, 4)'
overwrite_if_key_exists: true
```
{% include copy-curl.html %}

The following output shows the resulting list after extracting elements from index 1 to 3 and overwriting the original list:

```
output: my_list: [1, 2, 3]
```

### Specific ranges

Each of the following examples demonstrates how the `subList()` function extracts a specific range of elements from a list.

The following example extracts elements from index `0` to `2` (excluding index `3`), resulting in the first three elements of the list:

```json
{
"event": {
"/my_list": [0, 1, 2, 3, 4, 5, 6]
},
"expression": "subList(/my_list, 4, -1)",
"expected_output": [4, 5, 6]
}
```
{% include copy-curl.html %}

The following example uses `-1` as the end index, which tells the function to include all elements from index `4` to the end of the list:

```json
{
"event": {
"/my_list": [0, 1, 2, 3, 4, 5, 6]
},
"expression": "subList(/my_list, 4, -1)",
"expected_output": [4, 5, 6]
}
```
{% include copy-curl.html %}
Loading