Win 32 Code 8 "Not enough memory resources are available to process this command" when accessing files over tsclient

[ahsanaman92]

Hello,

I am running into an issue when writing to a FileStream over tsclient on a windows 10 VM in HyperV.

Technical Details of the Issue

Setup:

• On a windows 10 HyperV VM, share C drive of the host machine by following this procedure: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/learn-more/use-local-resources-on-hyper-v-virtual-machine-with-vmconnect

• Create a MS Word document on the host machine, for example TestMemoryIssue.docx under C:\Test\ (contents don’t matter, I was testing with a 17 KB file with a few sentences in it)

• Representative code snippet that reproduces the issue:

• Compile and run the following C# (.Net Framework 4.8) program (I did it in VS 2022 Professional)

static void Main(string[] args)
{
// Throws exception when SetLength is called below
string filePath = “\\tsclient\C\Test\TestMemoryIssue.docx”;

using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite))
{
long length = fs. Length;
long newLength = length + 5;
fs. SetLength(newLength);
}
}

Exception thrown:

• SetLength calls SetLengthCore which subsequently calls _SetEndOfFile(handle) in filestream.cs (MS reference: https://referencesource.microsoft.com/#mscorlib/system/io/filestream.cs,d6c30590c2fd88be). Here Win32 error code 8 (ERROR_NOT_ENOUGH_MEMORY) is thrown as seen in the screenshot below.

The workarounds tried:

• Increase the limits (second and third in SharedSection) in the registry mentioned in this article: https://learn.microsoft.com/en-us/troubleshoot/windows-server/performance/desktop-heap-limitation-out-of-memory

• Debug the kernel through windbg using the !dskheap command → dskheap utility errors out as per this open issue: !dskheap extension fails on recent Windows releases microsoft/WinDbg-Feedback#54

• Opened FileStream like this: FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite) and like this: FileStream fs =File.Open(filePath, FileMode.Open, FileAccess.ReadWrite)

• Changed the IRPStackSize to 20 in the registry at \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

• VM RAM doesn’t seem to make a difference. RAM was varied between 2GB and 6GB (CPU utilization was < 10%, RAM at < 50% and disk active time < 50% in the task manager)

Questions:

• Is this a known issue with tsclient?
• How would I go about debugging it? Knowing that !dskheap command in windbg does not work and dheapmon doesn’t seem to be available anymore?
• Are there any workarounds other than the ones I have already tried?
• My current working theory is that Desktop heap memory limit is being hit, I am not sure if anyone here can you validate that?