Fix edge case in stack overflow handling (#85272)

There is a problematic case when the stack overflow happens in native code
and there is an explicit frame between the managed and native code. Some
time ago, I have added a fix for a problem when the stack overflow
happened in native code, but I haven't found that it actually works only
in case there is no explicit frame between the managed code and the
failing native code frame. That fix makes the `FaultingExceptionFrame`
that is created for the stack overflow to contain context of the managed
code frame, so the stack walker uses that to move to the next frame. But
while doing so, it hits the other explicit frame that it doesn't expect
there and fires an assert.

This fix handles the problematic case correctly.

diff --git a/src/coreclr/vm/eepolicy.cpp b/src/coreclr/vm/eepolicy.cpp
index 35671c4f2ef03f60b1fc47deae7c3736f9317be3..af27338544e86125b9e6b2ef4d13029974caa796 100644 (file)
--- a/src/coreclr/vm/eepolicy.cpp
+++ b/src/coreclr/vm/eepolicy.cpp
@@ -612,14 +612,39 @@ void DECLSPEC_NORETURN EEPolicy::HandleFatalStackOverflow(EXCEPTION_POINTERS *pE
     if (pExceptionInfo && pExceptionInfo->ContextRecord)
     {
         GCX_COOP();
+        CONTEXT *pExceptionContext = pExceptionInfo->ContextRecord;
+
 #if defined(TARGET_X86) && defined(TARGET_WINDOWS)
         // For Windows x86, we don't have a reliable method to unwind to the first managed call frame,
         // so we handle at least the cases when the stack overflow happens in JIT helpers
         AdjustContextForJITHelpers(pExceptionInfo->ExceptionRecord, pExceptionInfo->ContextRecord);
 #else
-        Thread::VirtualUnwindToFirstManagedCallFrame(pExceptionInfo->ContextRecord);
+        // There are three possible kinds of locations where the stack overflow can happen:
+        // 1. In managed code
+        // 2. In native code with no explicit frame above the topmost managed frame
+        // 3. In native code with a explicit frame(s) above the topmost managed frame
+        // The FaultingExceptionFrame's context needs to point to the topmost managed code frame except for the case 3.
+        // In that case, it needs to point to the actual frame where the stack overflow happened, otherwise the stack 
+        // walker would skip the explicit frame(s) and misbehave.
+        Thread *pThread = GetThreadNULLOk();
+        if (pThread)
+        {
+            // Use the context in the FaultingExceptionFrame as a temporary store for unwinding to the first managed frame
+            CopyOSContext((&fef)->GetExceptionContext(), pExceptionInfo->ContextRecord);
+            Thread::VirtualUnwindToFirstManagedCallFrame((&fef)->GetExceptionContext());
+            if (GetSP((&fef)->GetExceptionContext()) > (TADDR)pThread->GetFrame())
+            {
+                // If the unwind has crossed any explicit frame, use the original exception context.
+                pExceptionContext = pExceptionInfo->ContextRecord;
+            }
+            else
+            {
+                // Otherwise use the first managed frame context.
+                pExceptionContext = (&fef)->GetExceptionContext();
+            }
+        }
 #endif
-        fef.InitAndLink(pExceptionInfo->ContextRecord);
+        fef.InitAndLink(pExceptionContext);
     }
 
     static volatile LONG g_stackOverflowCallStackLogged = 0;