Skip to content

Memory leak in CIMultiDict.pop() method? #1232

@F1int0m

Description

@F1int0m

I noticed a big memory leak in my application.

With tracemalloc I found that memory is leaking little by little when calling the .pop() method.
I used multidict version: 6.6.3

From #1134 I took an example for the script and got this:

import os

import psutil
from multidict import MultiDict, __version__ as multidict_version

print(f"multidict version: {multidict_version}")


def get_memory_usage():
    process = psutil.Process(os.getpid())
    memory_info = process.memory_info()
    return memory_info.rss / (1024 * 1024)


keys = [f"X-Any-{i}" for i in range(1000)]
headers = {key: key * 2 for key in keys}


def main():
    for _ in range(1000):
        for _ in range(1000):
            result = MultiDict(headers)

            for key in keys:
                result.pop(key, None) # leak
                # result.popall(key) # leak
                # del result[key] # leak

                # result.pop('123321', None) # very slow leak

        del result
        usage = get_memory_usage()
        print(f"Memory usage: {usage:.2f} MB")
        if usage > 200:
            raise MemoryError("Memory usage exceeded 200 MB")


if __name__ == "__main__":
    main()

From the notable:

  1. If you increase if usage > 200 to a super large value, the script runs without errors (but should this script really consume so much memory? The maximum I saw was about 6000 MB, and at the end of the script the memory stabilizes and remains at some still large value)
  2. Just in case, I tried different functions, most of them behave the same. But if you do .pop from a non-existent key, then the memory leaks very slowly, about 0.5MB in 1-2 minute.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions