Skip to content

Mattermost suddenly goes out of memory (OOM) and reboots #20625

@DummyThatMatters

Description

@DummyThatMatters

Summary

Mattermost goes for an unexpected reboot periodically (every 1-2 working days appox.) due to sudden increase in memory consumption.

Steps to reproduce

Mattermost 7.0.1 Team Edition, deployed on a pod in openshift (tryed to allocate from 2.6GB to 5.5GB RAM with the same result). Postgres 14.2 as a DB.
Around ~4000 users, ~1200 of them are active
~ 26000 messages per day load.

Expected behavior

Mattermost workes stably without reboots.

Observed behavior (that appears unintentional)

Mattermost goes for a reboot every 1-2 working days. The cause of reboot is OOM. Here is the example of log of the memory consumption:
image
image

The same increase of load can be observed on CPU part as well:
image

As you can see there is sudden growth of resource utilisation out of nowhere. The logs are relatevely clean and logs raito didnt show any increase of operations number or increased user activity.

We have done our small investigation and we think that it can be caused by unproper functioning of getPostsForChannel method. That assumption being done by inspecting mattermost go profile.
Here is example of heap tree made via pprof tool:
profile003

Please help with investigating, we can provide additional info if it needed (if it can be collected via our tools and does not contain corporate data)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions