The object returned by `json.loads(s)` cannot release memory after calling the `del` method

[xinsheng3]

Bug report

Bug description:

import base64
import json
import random
import string
import sys
import uuid
import time
import zlib

import psutil


def generate_large_file(filename='large_file', num_objects=1):
    json_obj = dict()
    for i in range(num_objects):
        name = ''.join(random.choice(string.ascii_lowercase) for _ in range(10))
        age = random.randint(18, 100)
        uuid_key = uuid.uuid4()
        uu1 = uuid.uuid4()
        uu2 = uuid.uuid4()
        uu3 = uuid.uuid4()
        uu4 = uuid.uuid4()
        uu5 = uuid.uuid4()
        uu6 = uuid.uuid4()

        json_obj[str(uuid_key)] = {"name": name,
                                   "age": age,
                                   str(uu1): {"name": name, "age": age,
                                              str(uu2): {"name": name, "age": age,
                                                         str(uu3): {"name": name, "age": age,
                                                                    str(uu4): {"name": name, "age": age,
                                                                               str(uu5): {"name": name, "age": age,
                                                                                          str(uu6): {"name": name, "age": age}}}}}},

                                   }

    json_obj = json.dumps(json_obj)
    zlib_obj = zlib.compress(json_obj.encode('utf-8'))
    b64_obj = base64.b64encode(zlib_obj)
    with open(filename, "wb") as f:
        f.write(b64_obj)


def get_process_memory(num):
    mem_info = psutil.Process().memory_info()
    res = mem_info.rss / (1024 ** 2)
    print("xxx %s process memory usage is %s M" % (num, res))


generate_large_file(num_objects=int(sys.argv[1]))

get_process_memory(1)

with open("./large_file", "rb") as f:
    content = f.read()
b64 = base64.b64decode(content)
z_content = zlib.decompress(b64)
data = json.loads(z_content)
get_process_memory(2)

del content
del b64
del z_content
del data

time.sleep(3)
get_process_memory(3)

test_github_issue.py test:

[root@localhost opt]# python3 test_github_issue.py 61000
xxx 1 process memory usage is 40.01171875 M
xxx 2 process memory usage is 281.01171875 M
xxx 3 process memory usage is 141.24609375 M

[root@localhost opt]# python3 test_github_issue.py 111000
xxx 1 process memory usage is 29.1640625 M
xxx 2 process memory usage is 430.328125 M
xxx 3 process memory usage is 25.44140625 M

why "xxx 3" - "xxx 1" != 0 ?

Python3.9.0 or later. The above verification data is in Python 3.11.6.

CPython versions tested on:

3.9

Operating systems tested on:

Linux

[sergey-miryanov]

There are no guaranties that freed memory will be returned immediately to the OS.

[xinsheng3]

There are no guaranties that freed memory will be returned immediately to the OS.

Are there any parameters or environment variables that can control immediate garbage collection? Directly calling gc.collect() doesn't work.

[xinsheng3]

Memory can be fully released under certain loop conditions.

[sergey-miryanov]

It is not a GC responsibility - there is an allocator "under" it that handles allocation, deallocation and returns memory to OS. More than that I unfortunately can't help.

[picnixz]

Calling gc.collect() while the GC is already doing a collection is an undefined behavior (this is documented). Since the GC can trigger collection every so often, it may be surprsing. Also, it may happen that some memory is freed and reused, that's something left at the discretion of the OS. And it may also happen that it takes time to make the collection.

What's important is that the memory doesn't increase constantly (that is, after some time, it's properly freed).

[picnixz]

Also, your experiments are run on Python 3.9 which is almost EOL. In Python 3.13 (or was it 14?), the GC algorithm changed and in 3.10, we changed the memory allocation of some types (some of them were static instead of heap types). I'm going to close this one as wont fix because there's not much we can on our side. If there is a leak, however, on Python 3.13+ (which can be verified by checking if ./python -X showrefcount yourscript.py prints something that is different than [0 refs, 0 blocks]), then I'll reopen it.

[xinsheng3]

Also, your experiments are run on Python 3.9 which is almost EOL. In Python 3.13 (or was it 14?), the GC algorithm changed and in 3.10, we changed the memory allocation of some types (some of them were static instead of heap types). I'm going to close this one as wont fix because there's not much we can on our side. If there is a leak, however, on Python 3.13+ (which can be verified by checking if ./python -X showrefcount yourscript.py prints something that is different than [0 refs, 0 blocks]), then I'll reopen it.

This issue is tested on python3.11.6 。
Also , tested 3.7.0 3.8.0 3.9.0 3.10.0 3.11.0 3.12.0 3.13.0 3.14.0 3.15.0a0, This issue has occurred in all cases.