Skip to content

load 5.5mb dictionary consume 1GB memory? #152

@Benedixx

Description

@Benedixx

Hello there, I have corpus dictionary with 5.5 MB filesize as .txt file, when i monitor my code memory usage using debug mode, the memory usage spiked up to 1+GB memory. You can see the graph here :

image

Is there any way to optimize this? i already tried to load dictionary as compressed pickle or pickle but i think the dictionary is not loaded because it's not detecting any typo on my text.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions