-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Description
Summary
Mattermost goes for an unexpected reboot periodically (every 1-2 working days appox.) due to sudden increase in memory consumption.
Steps to reproduce
Mattermost 7.0.1 Team Edition, deployed on a pod in openshift (tryed to allocate from 2.6GB to 5.5GB RAM with the same result). Postgres 14.2 as a DB.
Around ~4000 users, ~1200 of them are active
~ 26000 messages per day load.
Expected behavior
Mattermost workes stably without reboots.
Observed behavior (that appears unintentional)
Mattermost goes for a reboot every 1-2 working days. The cause of reboot is OOM. Here is the example of log of the memory consumption:


The same increase of load can be observed on CPU part as well:

As you can see there is sudden growth of resource utilisation out of nowhere. The logs are relatevely clean and logs raito didnt show any increase of operations number or increased user activity.
We have done our small investigation and we think that it can be caused by unproper functioning of getPostsForChannel method. That assumption being done by inspecting mattermost go profile.
Here is example of heap tree made via pprof tool:
Please help with investigating, we can provide additional info if it needed (if it can be collected via our tools and does not contain corporate data)