JSON parsing error in file-exports module when --batch option is used

[kharthigeyan]

The file export when done in batches then during the merge of the output JSON data, a comma is missed which invalidating the whole JSON output data.

JSON output data issue

{"key":"","...}
<<comma is missing here>>
{"key":"","...}

Problem command ⚠️

Example RIOT file-export

riot --version
#------------------------------------------------------------
#riot 4.3.0
#------------------------------------------------------------
#Build time:   2025-04-01 00:17:45Z
#Revision:     e958616d3942f2ff6c001e379ddebadc4d9bf8b2
#JVM:          21.0.7 (Ubuntu 21.0.7+6-Ubuntu-0ubuntu122.04)
#------------------------------------------------------------


riot file-export \
--uri redis://my--redis.prod.prod.us-west-1.aws.groupondev.com \
myredis-data-dump.json \
--content-type=STRUCT \
--batch 10000 \
--scan-count 10000 \
--threads 1 \
--type=json

Validating JSON ⚠️

So the start of the each batch first record got the missing comma.

jq . ./myredis-data-dump.json
# jq: parse error: Expected separator between values at line 10001, column 149

# after fixing the problem in the line no: 10001
jq . ./myredis-data-dump.json
#jq: parse error: Expected separator between values at line 20001, column 150

Workaround ✅

redis-cli -h my--redis.prod.prod.us-west-1.aws.groupondev.com info keyspace
# Keyspace
db0:keys=404120,expires=404120,avg_ttl=61536672

# so choose batch size larger than keyspace, so all export done in one file avoiding the data merge.
riot file-export \
--uri redis://my--redis.prod.prod.us-west-1.aws.groupondev.com \
myredis-data-dump.json \
--content-type=STRUCT \
--batch 500000 \
--scan-count 500000 \
--threads 1 \
--type=json

jq . ./myredis-data-dump.json  >/dev/null
# no parse error message which means, there is no problem in the exported JSON data.