json parser's stop_on_invalid_record option is not functioning

[junya-okumura]

Hello,
I would like to ask a question because the stop_on_invalid_record option of the JSON parser did not work as expected.

My environment
OS
NAME="CentOS Linux"
VERSION="7 (Core)"
embulk --version
embulk 0.9.19

$ cat /tmp/sample.json
There is an invalid record in the third line.
{invalid_json}

{"hoge":"aaa"}
{"hoge":"bbb"}
{invalid_json}
{"hoge":"ccc"}

embulk conf
json_test.yml

in:
  type: file
  path_prefix: /tmp/sample.json 
  parser:
    type: json
    stop_on_invalid_record: false
    charset: UTF-8
    newline: LF
    columns:
    - {name: hoge, type: string}
out:
  type: file
  path_prefix: /tmp/tmp01_
  file_ext: csv
  formatter:
    type: csv
    encoding: UTF-8
exec:
  min_output_tasks: 1

$ embulk run json_test.yml 

2023-10-04 16:29:38.753 +0900: Embulk v0.9.19
2023-10-04 16:29:39.333 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2023-10-04 16:29:41.756 +0900 [INFO] (main): Gem's home and path are set by default: "/home/hpadmin/.embulk/lib/gems"
2023-10-04 16:29:42.818 +0900 [INFO] (main): Started Embulk v0.9.19
2023-10-04 16:29:42.880 +0900 [INFO] (0001:transaction): Listing local files at directory '/tmp' filtering filename by prefix 'sample.json'
2023-10-04 16:29:42.881 +0900 [INFO] (0001:transaction): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2023-10-04 16:29:42.883 +0900 [INFO] (0001:transaction): Loading files [/tmp/sample.json]
2023-10-04 16:29:42.902 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=4 / tasks=1
2023-10-04 16:29:42.925 +0900 [INFO] (0001:transaction): {done:  0 / 1, running: 0}
2023-10-04 16:29:43.089 +0900 [INFO] (0013:task-0000): Writing local file '/tmp/tmp01_000.00.csv'
2023-10-04 16:29:43.117 +0900 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
org.embulk.exec.PartialExecutionException: org.embulk.spi.DataException: Failed to parse JSON: /tmp/sample.json
	at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
	at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
	at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
	at org.embulk.spi.Exec.doWith(Exec.java:22)
	at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
	at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:205)
	at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292)
	at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156)
	at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:433)
	at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
	at org.embulk.cli.Main.main(Main.java:64)
Caused by: org.embulk.spi.DataException: Failed to parse JSON: /tmp/sample.json
	at org.embulk.standards.JsonParserPlugin.run(JsonParserPlugin.java:177)
	at org.embulk.spi.FileInputRunner.run(FileInputRunner.java:140)
	at org.embulk.spi.util.Executors.process(Executors.java:62)
	at org.embulk.spi.util.Executors.process(Executors.java:38)
	at org.embulk.exec.LocalExecutorPlugin$DirectExecutor$1.call(LocalExecutorPlugin.java:170)
	at org.embulk.exec.LocalExecutorPlugin$DirectExecutor$1.call(LocalExecutorPlugin.java:167)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.embulk.spi.json.JsonParseException: Failed to parse JSON: in
	at org.embulk.spi.json.JsonParser$AbstractParseContext.next(JsonParser.java:144)
	at org.embulk.standards.JsonParserPlugin.run(JsonParserPlugin.java:135)
	... 9 more
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('i' (code 105)): was expecting double-quote to start field name
 at [Source: org.embulk.spi.util.FileInputInputStream@38089dae; line: 3, column: 3]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:462)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleOddName(UTF8StreamJsonParser.java:2003)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1650)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:740)
	at org.embulk.spi.json.JsonParser$AbstractParseContext.jsonTokenToValue(JsonParser.java:193)
	at org.embulk.spi.json.JsonParser$AbstractParseContext.next(JsonParser.java:142)
	... 10 more

Error: org.embulk.spi.DataException: Failed to parse JSON: /tmp/sample.json

output file
cat /tmp/tmp01_000.00.csv

hoge

umm

I was expecting that the invalid records would be skipped, and the following file would be generated.

{"hoge":"aaa"}
{"hoge":"bbb"}
{"hoge":"ccc"}

In an environment where invalid records are being input, how can I ensure that invalid records are skipped, and processing continues?

[dmikurube]

AFAIR, stop_on_invalid_record: false works when the input is valid as JSON, but the specified field does not exist. (Ex. for config root: foo, it returns "bar" for input {"foo": "bar"}. But it would fail for input {"non-foo": "bar"}.)

It cannot parse inputs that is broken as JSON.

[junya-okumura]

I see, so it's not expected to handle formats that are broken as JSON. Thank you.