Consecutive comma handling in JSONArray breaks lenient parsing in 20251224

[DavidJ-Stewart]

Bug Report: Consecutive comma handling in JSONArray breaks lenient parsing in 20251224

Summary

In org.json version 20251224, parsing arrays with consecutive commas (e.g., [1,,3]) no longer inserts JSONObject.NULL for elided elements as documented. Instead, it silently stops parsing and returns a truncated array.

Versions Affected

• Working: 20240303 and earlier
• Broken: 20251224 (and likely 20250107, 20250517)

Expected Behavior (per documentation)

From JSONArray.java javadoc (lines 46-48):

The constructors are more forgiving in the texts they will accept:

• The null value will be inserted when there is , (comma) elision.

When parsing [1,,3]:

• Expected: 3 elements: [1, null, 3]

Actual Behavior (20251224)

When parsing [1,,3]:

• Actual: 1 element: [1]

The parser stops at the consecutive comma and returns early, silently dropping all subsequent elements.

Reproduction

import org.json.JSONArray;

public class ConsecutiveCommaTest {
    public static void main(String[] args) {
        JSONArray arr = new JSONArray("[1,,3]");
        System.out.println("Length: " + arr.length());  // Expected: 3, Actual: 1
        System.out.println("Array: " + arr.toString()); // Expected: [1,null,3], Actual: [1]
    }
}

Root Cause

The issue was introduced in PR #937 (merged 2025-01-15).

In JSONArray.java, the checkForSyntaxError() method (around lines 155-160):

if (nextChar == ',') {
    // consecutive commas are not allowed in strict mode
    if (jsonParserConfiguration.isStrictMode()) {
        throw x.syntaxError("Strict mode error: Expected a valid array element");
    }
    return true;  // <-- BUG: Returns early even in lenient mode!
}

The return true; causes the parsing loop to exit immediately, even when strictMode = false.

Correct Behavior

In 20240303, the parsing loop (lines 96-98) correctly handled comma elision:

if (x.nextClean() == ',') {
    x.back();
    this.myArrayList.add(JSONObject.NULL);  // Insert null for elided element
}

The new code should either:

1. Not return early in lenient mode - continue parsing after the consecutive comma
2. Insert JSONObject.NULL before continuing, preserving the documented behavior

Suggested Fix

if (nextChar == ',') {
    // consecutive commas are not allowed in strict mode
    if (jsonParserConfiguration.isStrictMode()) {
        throw x.syntaxError("Strict mode error: Expected a valid array element");
    }
    // In lenient mode, back up so the comma elision logic can insert NULL
    x.back();
    break;  // Continue the loop instead of returning
}

Or restore the original comma elision logic that inserts JSONObject.NULL.

Impact

• Silent data loss when parsing arrays with consecutive commas
• Breaks backward compatibility with documented lenient parsing behavior
• Regression from working behavior in 20240303

Environment

• Java version: 17+
• org.json version: 20251224
• OS: Any

Related PRs/Issues

• PR Update JSONParserConfiguration usage in JSONTokener, JSONArray, and JSONObject #937 - "Update JSONParserConfiguration usage in JSONTokener, JSONArray, and JSONObject" (introduced the regression)
• PR Fixed JSONArray strict mode check for leading comma #1008 - "Fixed JSONArray strict mode check for leading comma" (related but does not fix this issue)

[javadev]

This JSON is not valid and may result in a parsing error

When parsing [1,,3]

[DavidJ-Stewart]

Totally can see that logic in strict mode. However..

• Previous lenient versions handled this - In 20240303, JSONArray explicitly added JSONObject.NULL for comma elision via this.myArrayList.add(JSONObject.NULL) in the constructor's parsing loop
• The documentation indicates handling it is intentional - The JSONArray javadoc still states:
  "The constructors are more forgiving in the texts they will accept: The null value will be inserted when there is , (comma) elision."
• Lenient mode vs strict mode - I'd expect strict mode to reject [1,,3], but lenient mode should preserve the documented behavior of inserting nulls.

The current behavior (return true on line 160) exits parsing early in lenient mode, losing data rather than inserting nulls as documented. Could this be reconsidered as a regression in lenient parsing behavior?