Yarp app pool intermittently crash

[JasonXuMSCSS]

Describe the bug

I have a customer opened support ticket for this issue. Latest Yarp hosted on IIS with high volume, the w3wp.exe process for Yarp intermittently crash. crash can happen 30 times a day, and can be also once in 2 months. unpredictable.

To Reproduce

VM Azure Hosting latest Yarp and a bunch of ASP.Net Core app. Issue is the w3wp process for YARP intermittently crash. crash can happen 30 times a day, and can be also once in 2 months. unpredictable. Recently more frequently.

crash mostly happened on server 04 and 05. Very occasionally, crash can happen on other servers too, there is one server never had crash.

The Yarp takes incoming requests, and routes to web apps (same ASP.Net core version apps) hosted on the same IIS server.
The request count is very high about 1000 concurrent requests per second at peak. But crash can happen any time.

And although there are over 1 dozen ASP.Net core apps hosted on every server, only the YARP app pool crashes. checked YARP app pool and it is set to “No Managed Code”, “Always Running” “Idle Timeout” 0, no any recycle time or condition is set.
When YARP app pool crashes, nothing else crashes.

Further technical details

Crash dumps were collected and call stack at crash varies a little in each dump, see crash call stacks below:

00 (Inline Function) --------`--------     iiscore!W3_CONTEXT_BASE::ContinueNotificationLoop(unsigned long cbCompletion = 0xb60, HRESULT hrCompletionStatus = 0x00000000)+0x4 [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.cxx @ 1149] 
01 000000b3`a51ffbc0 00007ffb`04792e0a     iiscore!W3_MAIN_CONTEXT::OnIoCompletion(unsigned long dwIoOperation = 1, void * pvContext = 0x00000368`36da6130, unsigned long cbCompletion = 0xb60, unsigned long dwCompletionStatus = 0, struct _OVERLAPPED * __formal = 0x00000368`36da6100)+0xd8 [servercommon\inetsrv\iis\iisrearc\iis70\core\maincontext.cxx @ 200] 
02 (Inline Function) --------`--------     w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x1a3e [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 741] 
03 (Inline Function) --------`--------     w3dt!UL_NATIVE_REQUEST::DoWork+0x1a80 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 369] 
04 000000b3`a51ffc20 00007ffb`04791b45     w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1a8a [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 2368] 
05 000000b3`a51ffc60 00007ffb`09eb1133     w3dt!WP_CONTEXT::OnCompletion(unsigned long dwErrorCode = 0x36da6130, unsigned long dwNumberOfBytesTransfered = 0xb60, struct _OVERLAPPED * lpOverlapped = 0x00000368`36da6100)+0x45 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\wpcontext.cxx @ 348] 
06 000000b3`a51ffca0 00007ffb`09eb1048     w3tp!THREAD_POOL_DATA::ThreadPoolThread(void)+0x83 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 1108] 
07 000000b3`a51ffcf0 00007ffb`09eb1b59     w3tp!THREAD_POOL_DATA::ThreadPoolThread(void * pvThis = 0x000001ea`c7868b10)+0x38 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 965] 
08 000000b3`a51ffd30 00007ffb`13064cb0     w3tp!THREAD_MANAGER::ThreadManagerThread(void * ThreadParam = 0x000001ea`c7834c50)+0x59 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_manager.cxx @ 877] 
09 000000b3`a51ffd70 00007ffb`1437edcb     kernel32!BaseThreadInitThunk(unsigned long RunProcessInit = <Value unavailable error>, <function> * StartAddress = <Value unavailable error>, void * Argument = <Value unavailable error>)+0x10 [clientcore\base\win32\client\thread.c @ 75] 
0a 000000b3`a51ffda0 00000000`00000000     ntdll!RtlUserThreadStart(<function> * StartAddress = 0x00000000`00000000, void * Argument = 0x00000000`00000000)+0x2b [minkernel\ntdll\rtlstrt.c @ 1152] 

00 00000043`a7e4fab0 00007ffa`c6be1edd     KERNELBASE!RaiseException(unsigned long dwExceptionCode = <Value unavailable error>, unsigned long dwExceptionFlags = 0xcb9377c8, unsigned long nNumberOfArguments = 0xcb9377c8, unsigned int64 * lpArguments = 0x00000000`00000000)+0x6c [minkernel\kernelbase\xcpt.c @ 955] 
01 00000043`a7e4fb90 00007ffa`c6b81599     iiscore!W3_MAIN_CONTEXT::OnIoCompletion(unsigned long dwIoOperation = 2, void * pvContext = 0x000002bb`c7fe78e0, unsigned long cbCompletion = 0x20000, unsigned long dwCompletionStatus = <Value unavailable error>, struct _OVERLAPPED * __formal = 0x000002bb`c7fe7888)+0x9d [servercommon\inetsrv\iis\iisrearc\iis70\core\maincontext.cxx @ 184] 
02 (Inline Function) --------`--------     w3dt!UL_NATIVE_REQUEST::DoStateProcess(void)+0x3a [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 741] 
03 (Inline Function) --------`--------     w3dt!UL_NATIVE_REQUEST::DoWork(void)+0x6f [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 369] 
04 00000043`a7e4fbf0 00007ffa`c6b81b45     w3dt!UL_SEND_CONTEXT::DoWork(unsigned long cbData = <Value unavailable error>, unsigned long dwError = <Value unavailable error>, struct _OVERLAPPED * lpo = 0x000002bb`c7fe7888)+0x79 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 2350] 
05 00000043`a7e4fc30 00007ffa`cb3b1133     w3dt!WP_CONTEXT::OnCompletion(unsigned long dwErrorCode = 0xcb9377c8, unsigned long dwNumberOfBytesTransfered = 0x20000, struct _OVERLAPPED * lpOverlapped = 0x000002bb`c7fe7888)+0x45 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\wpcontext.cxx @ 348] 
06 00000043`a7e4fc70 00007ffa`cb3b1048     w3tp!THREAD_POOL_DATA::ThreadPoolThread(void)+0x83 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 1108] 
07 00000043`a7e4fcc0 00007ffa`cb3b1b59     w3tp!THREAD_POOL_DATA::ThreadPoolThread(void * pvThis = 0x000001bd`83613b30)+0x38 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 965] 
08 00000043`a7e4fd00 00007ffa`d4334cb0     w3tp!THREAD_MANAGER::ThreadManagerThread(void * ThreadParam = 0x000001bd`83be7600)+0x59 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_manager.cxx @ 877] 
09 00000043`a7e4fd40 00007ffa`d5ddedcb     kernel32!BaseThreadInitThunk(unsigned long RunProcessInit = <Value unavailable error>, <function> * StartAddress = <Value unavailable error>, void * Argument = <Value unavailable error>)+0x10 [clientcore\base\win32\client\thread.c @ 75] 
0a 00000043`a7e4fd70 00000000`00000000     ntdll!RtlUserThreadStart(<function> * StartAddress = 0x00000000`00000000, void * Argument = 0x00000000`00000000)+0x2b [minkernel\ntdll\rtlstrt.c @ 1152] 

00 (Inline Function) --------`--------     iiscore!W3_CONTEXT_BASE::ReferenceW3Context(void) [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.h @ 562] 
01 (Inline Function) --------`--------     iiscore!IISCORE_ASYNC_CONTEXT::SetIoCompletionContext(void)+0x22 [servercommon\inetsrv\iis\iisrearc\iis70\core\w3request.cxx @ 3638] 
02 0000007b`5b97f2a0 00007ffe`8cfb594a     iiscore!NOTIFICATION_SEND_RESPONSE::DoWork(unsigned long cbCompletion = 0, HRESULT hrCompletionStatus = <Value unavailable error>, unsigned long dwCompletedOperation = <Value unavailable error>, int fCompletion = 0n0)+0xbf [servercommon\inetsrv\iis\iisrearc\iis70\core\w3response.cxx @ 3457] 
03 (Inline Function) --------`--------     iiscore!W3_CONTEXT_BASE::StartNotificationLoop(void)+0x6a [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.cxx @ 1083] 
04 0000007b`5b97f320 00007ffe`8cfb5cda     iiscore!W3_RESPONSE::Flush(int fAsync = 0n1, int fMoreData = <Value unavailable error>, class W3_CONTEXT_BASE * pW3BaseContext = 0x000002bb`babc0790, unsigned long * pcbSent = 0x0000007b`5b97f434, int * pfCompletionExpected = 0x0000007b`5b97f430)+0x39a [servercommon\inetsrv\iis\iisrearc\iis70\core\w3response.cxx @ 2016] 
05 0000007b`5b97f3d0 00007ffe`8cfb1f43     iiscore!NOTIFICATION_MAIN::DoWork(unsigned long cbCompletion = 0, HRESULT hrCompletionStatus = 0x00000000, unsigned long dwCompletedOperation = 1, int fCompletion = 0n1)+0x25a [servercommon\inetsrv\iis\iisrearc\iis70\core\mainnotification.cxx @ 255] 
06 (Inline Function) --------`--------     iiscore!W3_CONTEXT_BASE::ContinueNotificationLoop(unsigned long cbCompletion = <Value unavailable error>, HRESULT hrCompletionStatus = 0x00000000)+0x2f [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.cxx @ 1149] 
07 0000007b`5b97f6d0 00007ffe`90422e0a     iiscore!W3_MAIN_CONTEXT::OnIoCompletion(unsigned long dwIoOperation = 1, void * pvContext = 0x000002bb`babc0790, unsigned long cbCompletion = 0x7b0, unsigned long dwCompletionStatus = <Value unavailable error>, struct _OVERLAPPED * __formal = 0x000002bb`babc0760)+0x103 [servercommon\inetsrv\iis\iisrearc\iis70\core\maincontext.cxx @ 200] 
08 (Inline Function) --------`--------     w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x1a3e [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 741] 
09 (Inline Function) --------`--------     w3dt!UL_NATIVE_REQUEST::DoWork+0x1a80 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 369] 
0a 0000007b`5b97f730 00007ffe`90421b45     w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1a8a [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 2368] 
0b 0000007b`5b97f770 00007ffe`939a1133     w3dt!WP_CONTEXT::OnCompletion(unsigned long dwErrorCode = 0xffffffff, unsigned long dwNumberOfBytesTransfered = 0x7b0, struct _OVERLAPPED * lpOverlapped = 0x000002bb`babc0760)+0x45 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\wpcontext.cxx @ 348] 
0c 0000007b`5b97f7b0 00007ffe`939a1048     w3tp!THREAD_POOL_DATA::ThreadPoolThread(void)+0x83 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 1108] 
0d 0000007b`5b97f800 00007ffe`939a1b59     w3tp!THREAD_POOL_DATA::ThreadPoolThread(void * pvThis = 0x000001bd`71a23310)+0x38 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 965] 
0e 0000007b`5b97f840 00007ffe`9d904cb0     w3tp!THREAD_MANAGER::ThreadManagerThread(void * ThreadParam = 0x000001bd`71ac0010)+0x59 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_manager.cxx @ 877] 
0f 0000007b`5b97f880 00007ffe`9e41edcb     kernel32!BaseThreadInitThunk(unsigned long RunProcessInit = <Value unavailable error>, <function> * StartAddress = <Value unavailable error>, void * Argument = <Value unavailable error>)+0x10 [clientcore\base\win32\client\thread.c @ 75] 
10 0000007b`5b97f8b0 00000000`00000000     ntdll!RtlUserThreadStart(<function> * StartAddress = 0x00000000`00000000, void * Argument = 0x00000000`00000000)+0x2b [minkernel\ntdll\rtlstrt.c @ 1152] 

• Include the version of the packages you are using: upgraded to latest version of Yarp and issue persists.
• The platform (Linux/macOS/Windows) Windows Server 2022, Azure VM.

[MihaZupan]

What kind of crash is it (OOM, access violation, stack overflow, ...)?

YARP doesn't directly interact with IIS in any way, that all goes through ASP.NET Core APIs.
Use of unsafe code in YARP itself is also very minimal, and I don't see previous mentions of similar stack traces in any Yarp/ASP.NET issue.

Other than the fact this application uses YARP, is there any indication from stack traces / the dump that the issue is with YARP? Are you able to share the crash dump?
It's more likely to be an issue with IIS itself / the ASP.NET wrapper / user logic in that app if they're doing anything unsafe.

[JasonXuMSCSS]

@MihaZupan Most of them are C0000005, one is C0000002. Customer told me they are using out of box Yarp latest version. I had never seen IIS crash like this before. also there are over a dozen of ASP.Net Core apps on the same IIS server the Yarp routing requests to, none of them crash. I will ping you in Teams

[Tratcher]

This is almost certainly an IIS issue, it owns those callstacks. YARP's usage is somewhat unusual and may uncover uncommon IIS bugs, but it doesn't have access to anything to directly cause this. Recommend transferring to AspNetCore.