Skip to content

Conversation

@hugovk
Copy link
Member

@hugovk hugovk commented Apr 17, 2025

cat input.jsonl
{"ingredients":["frog", "water", "chocolate", "glucose"]}
{"ingredients":["chocolate","steel bolts"]}

Before

./python.exe -m json --json-lines input.jsonl
I/O operation on closed file.

After

./python.exe -m json --json-lines input.jsonl
{
    "ingredients": [
        "frog",
        "water",
        "chocolate",
        "glucose"
    ]
}
{
    "ingredients": [
        "chocolate",
        "steel bolts"
    ]
}

@hugovk hugovk added the needs backport to 3.13 bugs and security fixes label Apr 17, 2025
@hugovk hugovk changed the title Fix "I/O operation on closed file" when parsing JSON Lines file gh-132631: Fix "I/O operation on closed file" when parsing JSON Lines file Apr 17, 2025
try:
if options.json_lines:
objs = (json.loads(line) for line in infile)
objs = tuple(json.loads(line) for line in infile)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has significant memory implications for large .jsonl files. It might be better to instead move the finally clause either to the outer try or just after the with outfile block, but that does have implications for the infile == outfile case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, yeah, moving it gives a blank file when infile == outfile...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we could load all the lines into a list. This uses more memory than before, but less than the tuple as we're only deserialising each line when consumed from the generator:

Suggested change
objs = tuple(json.loads(line) for line in infile)
lines = infile.readlines()
objs = (json.loads(line) for line in lines)

I don't think it's worth adding much extra complexity for this CLI, as if someone needs to process extremely large .jsonl files they can write their own code to do exactly what they need.

@python-cla-bot
Copy link

python-cla-bot bot commented Apr 18, 2025

All commit authors signed the Contributor License Agreement.

CLA signed

Co-authored-by: Brian Schubert <[email protected]>
@serhiy-storchaka serhiy-storchaka added the needs backport to 3.14 bugs and security fixes label May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants