Skip to content

Commit 8d0e591

Browse files
acarbonettoacarbonetto
authored
PPL: Add json_set and json_extend command to spark-ppl (opensearch-project#1038)
* PPL: Add the json_extend command Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * PPL: add json_set function Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * scalafmt Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Remove comment Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Update JSON_APPEND, JSON_EXTEND, JSON_SET Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Consolidate nested function calls Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Add IT tests for json_set, json_expand Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Fix examples Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Fix IT tests Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Fix unit tests Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Fix syntax errors in IT tests Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Add IT tests for flatten calls Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Refactor nested traverse Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Update ppl-json.md examples Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> * Re-add missing import Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> --------- Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
1 parent 05dbf33 commit 8d0e591

File tree

9 files changed

+792
-100
lines changed

9 files changed

+792
-100
lines changed

‎docs/ppl-lang/functions/ppl-json.md‎

Lines changed: 86 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -202,14 +202,39 @@ Example:
202202
| null |
203203
+----------------+
204204

205+
### `JSON_SET`
206+
207+
**Description**
208+
209+
`json_set(json_string, array(path1, value1, path2, value2, ...))` Inserts or updates one or more values at the corresponding paths in the specified JSON object.
210+
211+
**Argument type:**
212+
- \<json_string\> must be a JSON_STRING.
213+
- \<path\> must be a STRING.
214+
- \<value\> must be a JSON_STRING.
215+
216+
**Return type:** JSON_STRING
217+
218+
An updated JSON object format.
219+
220+
Example:
221+
222+
os> source=people | eval updated = json_set('{"a":[{"b":1},{"b":2}]}', array('$.a[*].b', '3', '$.a', '{"c":4}')) | head 1 | fields updated
223+
fetched rows / total rows = 1/1
224+
+---------------------------------+
225+
| updated |
226+
+---------------------------------+
227+
| {"a":[{"b":3},{"b":3},{"c":4}]} |
228+
+---------------------------------+
229+
205230

206231
### `JSON_DELETE`
207232

208233
**Description**
209234

210-
`json_delete(json_string, [keys list])` Deletes json elements from a json object based on json specific keys. Return the updated object after keys deletion .
235+
`json_delete(json_string, array(key1, key2, ...))` Deletes json elements from a json object based on json specific keys. Returns the updated object after keys deletion.
211236

212-
**Arguments type:** JSON_STRING, List<STRING>
237+
**Arguments type:** JSON_STRING, List<JSON_STRING>
213238

214239
**Return type:** JSON_STRING
215240

@@ -245,9 +270,13 @@ Example:
245270

246271
**Description**
247272

248-
`json_append(json_string, [path_key, list of values to add ])` appends values to end of an array within the json elements. Return the updated json object after appending .
273+
`json_append(json_string, array(key1, value1, key2, value2, ...))` appends values to end of an array at key within the json elements. Returns the updated json object after appending.
274+
`json_append` is identical to `json_extend` function except that it does not flatten given arrays before appending them.
249275

250-
**Argument type:** JSON_STRING, List<STRING>
276+
**Argument type:**
277+
- \<json_string\> must be a JSON_STRING.
278+
- \<path\> must be a STRING.
279+
- \<value\> can be a JSON_STRING.
251280

252281
**Return type:** JSON_STRING
253282

@@ -262,28 +291,76 @@ Append adds the value to the end of the existing array with the following cases:
262291

263292
Example:
264293

265-
os> source=people | eval append = json_append(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`,array('student', '{"name":"Tomy","rank":5}')) | head 1 | fields append
294+
os> source=people | eval append = json_append(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`, array('student', '{"name":"Tomy","rank":5}')) | head 1 | fields append
266295
fetched rows / total rows = 1/1
267296
+-----------------------------------------------------------------------------------------------------------------------------------+
268297
| append |
269298
+-----------------------------------------------------------------------------------------------------------------------------------+
270299
|{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2},{"name":"Tomy","rank":5}]} |
271300
+-----------------------------------------------------------------------------------------------------------------------------------+
272301

273-
os> source=people | eval append = json_append(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`,array('teacher', 'Tom', 'Walt')) | head 1 | fields append
302+
os> source=people | eval append = json_append(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`, array('teacher', '"Tom"', 'teacher', '"Walt"')) | head 1 | fields append
274303
fetched rows / total rows = 1/1
275304
+-----------------------------------------------------------------------------------------------------------------------------------+
276305
| append |
277306
+-----------------------------------------------------------------------------------------------------------------------------------+
278307
|{"teacher":["Alice","Tom","Walt"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]} |
279308
+-----------------------------------------------------------------------------------------------------------------------------------+
280309

281-
282-
os> source=people | eval append = json_append(`{"school":{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}}`,array('school.teacher', 'Tom', 'Walt')) | head 1 | fields append
310+
os> source=people | eval append = json_append(`{"school":{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}}`, array('school.teacher', '["Tom", "Walt"]')) | head 1 | fields append
283311
fetched rows / total rows = 1/1
284312
+-------------------------------------------------------------------------------------------------------------------------+
285313
| append |
286314
+-------------------------------------------------------------------------------------------------------------------------+
315+
|{"school":{"teacher":["Alice",["Tom","Walt"]],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}} |
316+
+-------------------------------------------------------------------------------------------------------------------------+
317+
318+
### `JSON_EXTEND`
319+
320+
**Description**
321+
322+
`json_extend(json_string, array(key1, value1, key2, value2, ...))` extends values to end of an array at path_key within the json elements. Returns the updated json object after extending.
323+
`json_extend` is identical to `json_append` function except that it flattens given arrays before appending.
324+
325+
**Argument type:**
326+
- \<json_string\> must be a JSON_STRING.
327+
- \<path\> must be a STRING.
328+
- \<value\> can be a JSON_STRING.
329+
330+
**Return type:** JSON_STRING
331+
332+
A string JSON object format.
333+
334+
**Note**
335+
Extends adds the value to the end of the existing array with the following cases:
336+
- path is an object value - append is ignored and the value is returned
337+
- path is an existing array not empty - the value are added to the array's tail
338+
- path not found - the value are added to the root of the json tree
339+
- path is an existing array is empty - create a new array with the given value
340+
341+
Example:
342+
343+
os> source=people | eval extend = json_extend(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`, array('student', '{"name":"Tommy","rank":5}')) | head 1 | fields extend
344+
fetched rows / total rows = 1/1
345+
+-----------------------------------------------------------------------------------------------------------------------------------+
346+
| extend |
347+
+-----------------------------------------------------------------------------------------------------------------------------------+
348+
|{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2},{"name":"Tommy","rank":5}]} |
349+
+-----------------------------------------------------------------------------------------------------------------------------------+
350+
351+
os> source=people | eval extend = json_extend(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`, array('teacher', '"Tom"', 'teacher', '"Walt"')) | head 1 | fields extend
352+
fetched rows / total rows = 1/1
353+
+-----------------------------------------------------------------------------------------------------------------------------------+
354+
| extend |
355+
+-----------------------------------------------------------------------------------------------------------------------------------+
356+
|{"teacher":["Alice","Tom","Walt"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]} |
357+
+-----------------------------------------------------------------------------------------------------------------------------------+
358+
359+
os> source=people | eval extend = json_extend(`{"school":{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}}`, array('school.teacher', '["Tom", "Walt"]')) | head 1 | fields extend
360+
fetched rows / total rows = 1/1
361+
+-------------------------------------------------------------------------------------------------------------------------+
362+
| extend |
363+
+-------------------------------------------------------------------------------------------------------------------------+
287364
|{"school":{"teacher":["Alice","Tom","Walt"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}} |
288365
+-------------------------------------------------------------------------------------------------------------------------+
289366

@@ -293,7 +370,7 @@ Example:
293370

294371
`json_keys(jsonStr)` Returns all the keys of the outermost JSON object as an array.
295372

296-
**Argument type:** STRING
373+
**Argument type:** JSON_STRING
297374

298375
A STRING expression of a valid JSON object format.
299376

0 commit comments

Comments
 (0)