Add Streaming Support for JSON Parsing/Loading

[orgads]

Summary

Following the successful implementation of streaming support for JSON generation/dumping (#524, #686), I propose adding similar streaming capabilities for JSON parsing/loading operations.

Background

The JSON library recently added support for streaming JSON generation, allowing direct writing to IO objects without intermediate buffering (#686). This significantly improved memory efficiency for large JSON dumps by eliminating the need to buffer the entire JSON structure in memory.

However, the parsing side still requires loading the entire JSON content into memory before parsing, which can be problematic for:

• Large JSON files that exceed available memory
• Stream processing scenarios where data arrives incrementally
• API responses that could be processed as they arrive
• Log file processing where JSON entries could be parsed one at a time

Proposed Enhancement

Add streaming support for JSON parsing through one or more of these approaches:

Option 1: Enhance existing JSON.parse/JSON.load

Allow these methods to accept IO objects and parse incrementally:

File.open('large.json', 'r') do |io|
  JSON.parse(io, stream: true) do |object|
    # Process each top-level object as it's parsed
    process_object(object)
  end
end

Option 2: New streaming-specific methods

Introduce dedicated methods like JSON.parse_stream or JSON.each:

JSON.parse_stream(io) do |object|
  # Process each object
end

# Or for line-delimited JSON (JSONL/NDJSON)
JSON.each_line(io) do |object|
  # Process each line as a separate JSON object
end

Option 3: SAX-style parsing API

Provide an event-driven parsing interface for maximum flexibility:

parser = JSON::StreamParser.new
parser.on_object do |obj|
  # Handle complete objects
end
parser.on_array do |arr|
  # Handle arrays
end
parser.parse(io)

Use Cases

1. Large file processing: Parse multi-gigabyte JSON files without loading them entirely into memory
2. Real-time data streams: Process JSON data from network streams as it arrives
3. Line-delimited JSON (JSONL/NDJSON): Efficiently process log files and data exports that use newline-delimited JSON format
4. Memory-constrained environments: Enable JSON processing in environments with limited memory
5. Progressive web APIs: Parse streaming JSON responses from APIs that support chunked transfer encoding

Benefits

• Memory efficiency: Constant memory usage regardless of input size
• Improved performance: Start processing data before the entire document is received
• Better scalability: Handle arbitrarily large JSON documents
• Consistency: Mirrors the streaming generation capability already implemented

Technical Considerations

• Would need implementation across all three backends (pure Ruby, C extension, Java)
• Should handle both single large JSON objects and streams of multiple JSON objects
• Error handling for partial/malformed JSON in streams
• Backward compatibility with existing parsing APIs
• Support for common streaming JSON formats (JSONL, JSON Text Sequences RFC 7464)

Related Work

• Streaming enhancements for dumping #524: Original issue for streaming JSON generation
• JSON.dump: write directly into the provided IO #686: Implementation of streaming JSON generation
• Similar functionality exists in other JSON libraries:
  • yajl-ruby: Streaming JSON parsing
  • oj: ScHandler for SAX-style parsing
  • Node.js stream-json
  • Python ijson

Questions for Discussion

1. Which API approach would be most consistent with Ruby conventions and the existing JSON library?
2. Should we support both SAX-style and object-stream patterns?
3. How should we handle different streaming JSON formats (single large object vs. line-delimited)?
4. What would be the performance targets compared to the current implementation?

Would love to hear thoughts from maintainers and the community on this enhancement!

[byroot]

Sorry, I don't read AI generated issues.

If you have a use case, express it like a human in a reasonably short issue.

[headius]

@byroot I read through the entire issue and even if it's AI-generated (I am unsure), it still covers all the points I'd want to see covered in a hand-written proposal. Do you agree that streaming parsing would be a useful feature to add?

I am, for the record, very interested in adding streaming parsing to the json library, since that would solve the other half of the "2GB array problem" we have in JRuby. I haven't attempted to do so up to this point because I'm not sure how to implement it, given the limitations of Ragel and other parts of the current design.

@orgads Whether or not this proposal was AI-generated, are you willing to help carry it forward? This is no trivial enhancement, but I think it could be a big improvement when dealing with large JSON datasets. I'd like to see it happen.

[orgads]

It is AI generated, but I reviewed it before posting.

I'm trying to get rid of yajl dependency for fluentd, and I think streaming support for json will enable that. The most relevant use-case there is Real-time data streams (number 2 in the proposal).

I don't have any experience with native Ruby code, so I'm not sure I'd be able to contribute on that front.

[byroot]

It is AI generated, but I reviewed it before posting.

Just a quick advice, the problem isn't so much AI vs humain, it's really about going to the point quickly and being clear, which AI almost always fail at.

I'm trying to get rid of yajl dependency for fluentd, and I think streaming support for json will enable that.

I figured because I'm on the issue in question in fluentd.

If you wish to get this going forward, what you can do is:

• Isolate the fluentd use case in a small script to demonstrate how they use YAJL and what their actual need is.
• (optional) Propose a json API that would solve that same need.

Keep it short and to the point, and I'll see what I can do.

[headius]

@orgads For what it's worth, I'm also interested in the use case of eliminating yajl (and oj) because neither of them have support for JRuby. 🙂

The points that @byroot makes are valid... we ideally we need a more concrete starting point than a "boil the ocean" AI proposal. If you don't think you'd be able to help implement this on the native side (and that's fair, there's quite a lot of native code in ruby/json), then perhaps we should start from an API proposal and go forward from there?

There's also the pure-Ruby version of this library still out there... perhaps we can prototype something in Ruby and move it into the native pieces later on?

We don't want to discourage you, but we need a measurable goal (specific API requirement) and manageable steps to move forward.

[samuel-williams-shopify]

Can you use newline delimited JSON instead?