OutOfMemory event for garbage collector

[msedi]

We are having an environment with a total amount of used memory of around 50TB during a single run. Since we do not have systems with 50TB RAM, we need to manage our memory in a file-backed storage. Although memory-mapped files would be a solution here, they have certain drawbacks so that they cannot be used currently.

For better performance it is of high interest to keep the data in memory as long as possible so that we needed to introduce usage counters and last access times to better page-out data that hasn't been used for a long time.

We are doing something similar as windows does to identify memory consitions (MmMemoryInspector thread) modified pages and write pages to disk and free the memory (MmModifiedPageWriter thread). The inspection is done by constantly inspecting the system memory load and initiate writing dirty pages out to the disk.

While this all works fine, the problem is that this manager doesn't see any allocations done by a regular user-called new[], which is mostly no problem because the system is balancing itself and the MmMemoryInspector is paging out memory if needed. The problem is if the user allocates a large amount of memory and there is not enough free memory, where are getting an OutOfMemoryException since the MmMemoryInspector didn't see allocation early enough, and the MmModifiedPageWriter couldn't be triggered to page data out to disk.

Event hooking in with GC.RegisterForFullGCNotification is doesn't help because this is also only another thread that cannot intercept the OutOfMemoryException.

A possible solution would be to create our own AllocateArray<T> method that try..catches the new instruction in order to trigger the MmModifiedPageWriter but this makes the system dependent on the knowledge of our developers to know about the special AllocateArray<T> method.

I would prefer some sort of an event that fires before an out of memory exception would happen. A long time ago we where using MemoryFailPoints, but for the amount of data we had the performance dropped a lot and even then I needed our own AllocateArray<T> method.

C++ once had a set_new_handler which would be very nice to have so that I can intercept the memory allocation, but I also agree that this is maybe not the best way to solve this problem.

Is there any idea how to solve it?