|
| 1 | +# Splitter Component |
1 | 2 |
|
2 | | - |
3 | | -[](https://circleci.com/gh/elasticio/splitter-component/tree/master) |
4 | | -# splitter-component |
5 | | -Splitter is the basic component for the [elastic.io platform](http://www.elastic.io). |
| 3 | +## Table of Contents |
| 4 | +* [Description](#description) |
| 5 | +* [Actions](#actions) |
| 6 | + * [Split on JSONata Expression](#Split-on-JSONata-Expression) |
| 7 | + * [Re-assemble Messages](#Re-assemble-Messages) |
6 | 8 |
|
7 | 9 | ## Description |
8 | | -The Splitter processes income messages containing multiple elements that might have to be processed in different ways. The Splitter emits out the composite message into individual messages, each containing data related to one item. |
9 | 10 |
|
10 | | -## Actions |
11 | | -### Split Message By Array |
12 | | -**This action is deprecated, please use Split on JSONata Expression instead.** |
13 | | - |
14 | | -Splits a message into multiple messages using a given separator. The separator is treated as a path to a property inside the message. A message is split when a property is an array and emitted are multiple messages. Otherwise the original message is emitted. |
15 | | - |
16 | | -For example, we have a message: |
17 | | - |
18 | | -``` |
19 | | -{ |
20 | | - "users": [ |
21 | | - { |
22 | | - "name": "John" |
23 | | - }, |
24 | | - { |
25 | | - "name": "Mike" |
26 | | - } |
27 | | - ] |
28 | | -} |
29 | | -``` |
30 | | - |
31 | | -The splitting expression is "users", action will return output: |
32 | | -``` |
33 | | -{ |
34 | | - "name": "John" |
35 | | -} |
36 | | -
|
37 | | -{ |
38 | | - "name": "Mike" |
39 | | -} |
40 | | -``` |
41 | | -*Notes:* |
42 | | - |
43 | | -- *When splitting expression refers to an object splitter returns this object;* |
44 | | -- *When splitting expression contains primitive value like ```users:"John"``` or array of primitives like ```users:["John", "Mike", "Anna"]``` splitter emits error.* |
| 11 | +The Splitter Component is a fundamental part of our platform. Its primary purpose is to split arrays into separate messages or to combine separate messages into one. |
45 | 12 |
|
| 13 | +## Actions |
| 14 | + |
46 | 15 | ### Split on JSONata Expression |
47 | 16 |
|
48 | | -This component takes the incoming message body and applies the configured JSONata transformation on it. The evaluated transformation must be an array. This array is split and emitted into multiple messages. |
49 | | - |
50 | | -For example, given the following message: |
51 | | - |
52 | | -``` |
53 | | -{ |
54 | | - "FirstName": "Fred", |
55 | | - "Surname": "Smith", |
56 | | - "Phone": [ |
57 | | - { |
58 | | - "type": "home", |
59 | | - "number": "0203 544 1234" |
| 17 | +Splits a single message into several separate messages. |
| 18 | + |
| 19 | +#### Configuration Fields |
| 20 | + |
| 21 | +* **JSONata Expression** - (string, required): Enter the expression that will be evaluated as an array. |
| 22 | + |
| 23 | + <details><summary>Example</summary> |
| 24 | + |
| 25 | + You have the following incoming message: |
| 26 | + |
| 27 | + ```json |
| 28 | + { |
| 29 | + "FirstName": "Fred", |
| 30 | + "Surname": "Smith", |
| 31 | + "Phone": [ |
| 32 | + { |
| 33 | + "type": "home", |
| 34 | + "number": "0203 544 1234" |
| 35 | + }, |
| 36 | + { |
| 37 | + "type": "office", |
| 38 | + "number": "01962 001234" |
| 39 | + }, |
| 40 | + { |
| 41 | + "type": "mobile", |
| 42 | + "number": "077 7700 1234" |
| 43 | + } |
| 44 | + ] |
| 45 | + } |
| 46 | + ``` |
| 47 | + |
| 48 | + If the JSONata expression is set to `Phone.{type: number}`, you will get three messages: |
| 49 | + |
| 50 | + ```json |
| 51 | + { |
| 52 | + "home": "0203 544 1234" |
| 53 | + } |
| 54 | + ``` |
| 55 | + ```json |
| 56 | + { |
| 57 | + "office": "01962 001234" |
| 58 | + } |
| 59 | + ``` |
| 60 | + ```json |
| 61 | + { |
| 62 | + "mobile": "077 7700 1234" |
| 63 | + } |
| 64 | + ``` |
| 65 | + </details> <br> |
| 66 | + |
| 67 | +#### Input Metadata |
| 68 | + |
| 69 | +None |
| 70 | + |
| 71 | +#### Output Metadata |
| 72 | + |
| 73 | +Each item of the array will be emitted as a separate message. |
| 74 | + |
| 75 | +### Re-assemble Messages |
| 76 | + |
| 77 | +Combines separate messages into one. |
| 78 | + |
| 79 | +#### Configuration Fields |
| 80 | + |
| 81 | +* **Behavior** - (dropdown, required): Select one of the following options: |
| 82 | + * `Group on fixed amount of messages` - Messages keeps collecting continuously. Once the group size is reached, the group is emitted and the new group starts collecting immediately. If the number of incoming messages for a particular group is less than the defined group size, the group will be stored in the internal storage (Maester) and proceed collecting messages into the open group. |
| 83 | + * `Group on timeout` - All incoming messages will be gathered until there are no more incoming messages within the specified timeframe (delay timer), at which point messages will be emitted for each group. |
| 84 | + * `Group on amount of messages or timeout` - Specify both group size and delay timer. Once a group is complete, that group will be emitted. If there are no more incoming messages within the specified timeframe, partially completed groups will also be emitted. |
| 85 | +* **Emit result as array** - (checkbox, optional): If selected, `messageData` in the response object will be an array of messages without message IDs. |
| 86 | + |
| 87 | + <details><summary>Example with unchecked</summary> |
| 88 | + |
| 89 | + ```json |
| 90 | + { |
| 91 | + "groupSize": 2, |
| 92 | + "groupId": "test22", |
| 93 | + "messageData": { |
| 94 | + "d899b000-5455-4c7a-9781-f16203426b93": { |
| 95 | + "dataFromMessage": "Message1" |
60 | 96 | }, |
| 97 | + "bdfca2b1-7aa7-444c-916d-3a2c17fc5dd6": { |
| 98 | + "dataFromMessage": "Message2" |
| 99 | + } |
| 100 | + } |
| 101 | + } |
| 102 | + ``` |
| 103 | + </details> <br> |
| 104 | + <details><summary>Example with checked</summary> |
| 105 | + |
| 106 | + ```json |
| 107 | + { |
| 108 | + "groupSize": 2, |
| 109 | + "groupId": "test22", |
| 110 | + "messageData": [ |
61 | 111 | { |
62 | | - "type": "office", |
63 | | - "number": "01962 001234" |
| 112 | + "dataFromMessage": "Message1" |
64 | 113 | }, |
65 | 114 | { |
66 | | - "type": "mobile", |
67 | | - "number": "077 7700 1234" |
| 115 | + "dataFromMessage": "Message2" |
68 | 116 | } |
69 | 117 | ] |
70 | | -} |
71 | | -``` |
72 | | - |
73 | | -and the JSONata expression `Phone.{type: number}`, an object constructor, the action will return output: |
74 | | -``` |
75 | | -{ |
76 | | - "home": "0203 544 1234" |
77 | | -} |
78 | | -
|
79 | | -{ |
80 | | - "office": "01962 001234" |
81 | | -} |
82 | | -
|
83 | | -{ |
84 | | - "mobile": "077 7700 1234" |
85 | | -} |
86 | | -``` |
87 | | -*Notes:* |
88 | | - |
89 | | -- *If the evaluated array contains primitive values like ```users:["John", "Mike", "Anna"]```, the splitter emits error.* |
90 | | - |
91 | | -#### List of Expected Config fields |
92 | | -```Split Property``` - use this field to choose a separator. |
93 | | - |
94 | | -### Re-assemble Messages |
95 | | - |
96 | | -Inverse of the split action: Given a stream of incoming messages a sum message is generated. |
97 | | - |
98 | | -#### List of Expected Config fields |
99 | | -```Behavior``` - Has 3 different behaviour variants(options): |
100 | | -* Produce Groups of Fixed Size (Don't Emit Partial Groups): A message is emitted once the group size is reached for the given group. If arriving messages for a particular group are less than the defined group size then the group will not be emitted. |
101 | | -* Group All Incoming Messages: All incoming messages will be gathered until there are no more incoming messages in the specifeid timeframe (delay timer) at which point messages will be emitted for each group. |
102 | | -* Produce Groups of Fixed Size (Emit Partial Groups): Specify both group size and delay timer. Once a group is complete, that group will be emitted. Once there are no more incoming messages, then partially completed groups will also be emitted. |
103 | | - |
104 | | -Supported: |
105 | | -* Messages can be re-ordered in the flow |
106 | | -* If messages are re-delivered as a result of the platform's at once delivery guarantee, does not trigger false positives |
107 | | -* Messages from one original message can be interleaved with messages from another original message |
108 | | -(e.g. Two overlapping webhook calls arrive and the flow has components where parallel processing > 1.) |
109 | | - |
110 | | -Limitations: |
111 | | -* All groups must have one or more messages. (i.e. No groups of size 0). |
112 | | -Can't do re-grouping when a split is done on an empty array. (i.e. No empty for each pattern supported). |
113 | | -If all the messages in the group do not arrive, then the group will not be emitted. |
114 | | -* The group is dropped if there are any unexpected restarts to the container. |
115 | | -* In case only a groupSize is given and no delay timer is specified. The size of the group must be known by all group members. |
116 | | -* In case of using the delay timer. Messages are only emitted when all parts arrive. Emitting a message only when the first part arrives isn't supported. |
117 | | -* The delay timer can not exceed 20,000 milliseconds. If more than this maximum is given, then this maximum will be used instead. |
| 118 | + } |
| 119 | + ``` |
| 120 | + </details> <br> |
118 | 121 |
|
119 | | -#### List of Expected Config fields |
120 | | -```groupId``` - Globally unique id for the group to distinguish it from other groups. This value needs to be the same for all messages in a group. |
| 122 | +#### Input Metadata |
121 | 123 |
|
122 | | -```messageId``` - Id for a message to distinguish it from other messages in the group. |
123 | | -Must be unique per group but does not have to be globally unique. This value needs to be different for all messages in a group. |
124 | | -In case a messageId occures multiple times, only the messageData of the latest message survives. |
125 | | -If the messageId is not defined, a random guid will be generated and used as messageID. |
| 124 | +* **Unique ID to describe the group** - (string, required): A unique ID for the group to distinguish it from other groups. |
| 125 | +* **Unique ID to describe this message** - (string, optional): An ID for a message to distinguish it from other messages in the group. Must be unique per group but does not have to be globally unique. This value needs to be different for all messages in a group. If a messageId occurs multiple times, only the messageData of the latest message will be retained. If the messageId is not defined, a random GUID will be generated and used as the messageID. |
| 126 | +* **Message Data** - (object, optional): Data from individual messages. |
| 127 | + |
| 128 | +If `Group on fixed amount of messages` or `Group on amount of messages or timeout` is selected: |
| 129 | +* **Number of messages expected to be reassembled into the group** - (number, optional): The number of messages when the group is considered full. |
126 | 130 |
|
127 | | -```messageData``` - Data from individual messages can be inserted here in form of an object. This object is then inserted into an array which is available in the message emitted for this group. |
| 131 | +If `Group on timeout` or `Group on amount of messages or timeout` is selected: |
| 132 | +* **Delay timer (in ms)** - (number, optional): The time the process waits when no incoming messages before emitting. Maximum is 20000 ms (20 sec). If you try to put here more than allowed, than default value will be used |
128 | 133 |
|
129 | | -```groupSize``` - Number of messages in the group. |
| 134 | +#### Output Metadata |
130 | 135 |
|
131 | | -```Delay timer (in ms)``` - Time the process waits when no incoming messages before emiting (Max 40,000 milliseconds) |
| 136 | +* **groupSize** - (number, required): The number of messages in this group. |
| 137 | +* **groupId** - (number, required): The ID of this group. |
| 138 | +* **messageData** - (number, required): If `Emit result as array` is selected, this will be an array of messages from previous steps; otherwise, it will be an object with keys as `Unique ID to describe this message` and values as messages from previous steps. |
132 | 139 |
|
133 | | -## Known limitations (common for the component) |
134 | | -None. |
| 140 | +#### Known Limitations |
135 | 141 |
|
136 | | -## Documentation links |
137 | | -More information and some examples can be found here: [Splitter documentation](https://www.elastic.io/connectors/splitter-integration/). |
| 142 | +* The total size of stored messages in groups should be less than 5MB; otherwise, the component will emit the group regardless of the selected behavior. |
| 143 | +* Messages are stored in component memory during execution - "Suspending" the flow will erase them. |
| 144 | +* With option `Produce Groups of Fixed Size (Don't Emit Partial Groups)` if group is not ready, messages will be stored inside internal storage (Maester) for up to two days |
0 commit comments