[release/6.0] Change macOS activatin injection failure handling (#63393)

Backport of #59045 to release/6.0

The pthread_kill can fail with ENOTSUP on macOS when the target thread
is a dispatch queue thread. Instead of aborting the process, it is
better to fail to inject the activation and rely on return address
hijacking and other means of syncing with GC.

diff --git a/src/coreclr/pal/src/exception/signal.cpp b/src/coreclr/pal/src/exception/signal.cpp
index cbcd1ca..59b513b 100644 (file)
--- a/src/coreclr/pal/src/exception/signal.cpp
+++ b/src/coreclr/pal/src/exception/signal.cpp
@@ -845,6 +845,15 @@ PAL_ERROR InjectActivationInternal(CorUnix::CPalThread* pThread)
     // We can get EAGAIN when printing stack overflow stack trace and when other threads hit
     // stack overflow too. Those are held in the sigsegv_handler with blocked signals until
     // the process exits.
+
+#ifdef __APPLE__
+    // On Apple, pthread_kill is not allowed to be sent to dispatch queue threads
+    if (status == ENOTSUP)
+    {
+        return ERROR_NOT_SUPPORTED;
+    }
+#endif
+
     if ((status != 0) && (status != EAGAIN))
     {
         // Failure to send the signal is fatal. There are only two cases when sending