fix: unified lookahead for unescaped quotes (fixes #129, #144)

[majiayu000]

Summary

Unified fix for unescaped double quotes inside strings by using a single lookahead approach that checks whether a quote is followed by valid JSON tokens.

This PR handles:

• Issue repair unescaped double quotes followed by parentheses #144: Quotes followed by parentheses or another quote (e.g., "53"", "(5'3")")
• Issue Failing when parsing double quotes inside a string which are followed by a Comma #129: Quotes followed by comma (e.g., "become an "Airbnb-free zone", which...")

Examples of now correctly repaired JSON

// Issue #144
jsonrepair('{ "height": "53"" }')      // -> '{ "height": "53\"" }'
jsonrepair('{ "height": "(5\'3")" }')  // -> '{ "height": "(5\'3\")" }'

// Issue #129  
jsonrepair('{"key": "become an "Airbnb-free zone", which is a political decision."}')
// -> '{"key": "become an \"Airbnb-free zone\", which is a political decision."}'

Test plan

• Added unified test cases for issues Failing when parsing double quotes inside a string which are followed by a Comma #129 and repair unescaped double quotes followed by parentheses #144
• All 156 tests pass
• Applied fix to both regular and streaming implementations

Closes #144
Closes #129

🤖 Generated with Claude Code

[josdejong]

Thanks @majiayu000, I'll review your PR soon.

[josdejong]

I think the issue addressed here is more generic and probably the two PR's that you have provided (#148, #149) should become a single PR which extends the condition to check whether a double-quote " is actually an end quote by looking ahead two steps to see whether it is followed by valid JSON tokens. What do you think?

[josdejong]

Can you also add a test case for #114, for strings like a 40" televison?

[majiayu000]

Thanks for the feedback! You're right - I've unified the two PRs into this single one.

The new implementation uses a unified lookahead approach that checks whether a quote is followed by valid JSON tokens (looking ahead 2+ steps). This now handles:

• Quotes followed by comma (issue Failing when parsing double quotes inside a string which are followed by a Comma #129)
• Quotes followed by parentheses or another quote (issue repair unescaped double quotes followed by parentheses #144)

I'll close PR #148 as it's now covered here.

[josdejong]

Thanks for the update. The feedback I posted in #148 (review) still holds, can you address those concerns please? Can you also address #151 and #114 in this PR? I think #144, #129, #114, and #151 all have a similar cause and the same solution.

[majiayu000]

Thanks for the feedback! I've made the following updates:

• Refactored lookahead logic into isUnescapedQuoteSuspicious() and findNextValidEndQuote() helper functions
• Added test cases for issues Adding missing escape for double quote not working #114 and Escaping double quote doesn't work when it is followed by slash('/') character #151
• All 156 tests passing

Let me know if anything else needs adjustment.

[straxico]

@josdejong please merge and release new version

[josdejong]

I'm a bit in doubt about the solution. I think that it duplicates quite some parsing logic. What we basically try to do is parse two steps ahead to see what we encounter. Maybe we need to separate getting the next token from the input from the parsing step? Then we could quite easily look up the next two tokens without duplicating parsing logic. Or do you see other ways to better reuse the existing logic and keep the code better maintainable? What do you think @majiayu000?