Memory Utilization Spike

[Hmonat]

Hello,

I am currently facing issue related to memory spike on our flask application. I am attaching WSGI file which are using.
NOTE: I have masked some of the Daemon process names so please ignore them.
Can you suggest anything which can lead to memory spike in this file and also what can be the possible fix for the same.

wsgi.txt

Thanks & Regards
Himanshu Monat

[GrahamDumpleton]

Have you definitively determined that the memory spike is in your WSGI application processes (the mod_wsgi managed daemon mode processes)?

You need to rule out whether it isn't Apache child worker processes instead, which can grow in memory size for various reasons, especially if using event/worker MPM with Apache.

A couple of things I would first suggest.

Ensure you are setting:

WSGIRestrictEmbedded On

outside of any VirtualHost if you are exclusively using mod_wsgi daemon mode.

This stops wasted memory in Apache child worker processes from the Python interpreter being initialised in them when not required.

It will also cause an error if you misconfigure things and don't properly delegate WSGI applications to the intended mod_wsgi daemon processes. If such a misconfiguration isn't caught, you could have the WSGI application being run in the Apache child worker processes which would cause overall system memory to grow as now have more copies of your WSGI application in different processes.

Next thing, especially if only using this Apache to host WSGI applications with mod_wsgi and using only mod_wsgi daemon mode, is to set:

ThreadStackSize 262144

Depends on Apache version and also what Linux OS may override it to if it does override it, but the default stack memory size per thread in Apache child worker processes is 8MB. This means that Apache child worker processes when using event/worker MPM can actually grow quite large in size over time since the default stack size is larger than it needs to be.

These two things will ensure trimming down Apache child worker processes as much as possible.

Now when you say memory spike, do you mean the memory usage grows all of a sudden and then holds at the new memory usage level, or do you really mean the memory spikes up quite a lot, but then goes back to its prior level after a period without the processes being restarted.

If the latter, you would be encountering a special case where when large blocks of memory are allocated by an application and it has to allocate more memory from the operating system, this occurs at the top of memory. If it so happens that the block of memory is released before any new memory has to be allocated from the operating system at the top of memory, then the process can actually release it back to the operating system and process memory usage can drop.

This would likely only occur when application for some reason needs a very large block of memory which cannot be satisfied from the current in process free memory list.

Whether it is this case or process memory just grows and then holds at the new level, this is going to be something about your WSGI application and Python code. Perhaps someone did a very large file upload and the code naively tries to read it all into memory at the same time, as opposed to streaming the data from the request and writing it to disk in chunks as they arrive.

These latter sorts of issues it is hard to give too much guidance on since I don't know anything about your application and what it does.

[GrahamDumpleton]

Forgot one thing. If it is your Python code and how it works that just causes these spikes, and you can't fix the code immediately, you can add the restart-interval option to WSGIDaemonProcess to have it restart the mod_wsgi daemon processes periodically so that base memory usage drops back. Eg., restart-interval=600 would restart the mod_wsgi daemon processes 10 minutes after first request handled by that process. Note that it will delay restart if there are pending requests and wait for the process to be inactive. I can't remember if restart will be forced after the graceful timeout period in this case if the process does not become inactive.