Skip to content

Memory leak due to packet parsers #722

@aikar

Description

@aikar

I've noticed a memory leak in our processes, and after days of debugging, I've finally narrowed it down to the packet parsers.

The issue stems with the nature that the parser cache is stored on the connection object.

Most people are ideally using connection pools, and due to the nature that connections aren't closed (until #651 at least) usually, this problem doesn't really make itself apparent.

But once we start cycling connections, the leak starts growing... and slowly.

Here's the fun part, it doesn't show up in the heap dump fully!
A process can show a much larger memory footprint than the Heap Dump says its around.

The V8 engine is creating Compiled Code for these functions, and I assume that V8 does not clear this compiled code cache.
Some of the memory is represented in the heap dump, but a large chunk isn't.

I orchestrated my environment with my #651 PR to auto time out connections every minute, opening 20 connections and closing them rapidly.

My process leaked upwards to 1GB of memory in a few hours. Comparing Heap Snapshots, all of the memory went mostly to strings, copies of the same BinaryRow parsers over and over, and compiled code.
But the size reported in Strings being 200MB~, the real RSS size was 1GB.

I am about to submit a PR to resolve this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions