Avoid AwaitTaskContinuation allocation in some awaits (dotnet/coreclr#20159)
In .NET Core 2.1, I added a bunch of optimizations to async methods that are based on reusing the async state machine object itself for other purposes in order to avoid related allocations. One of those optimizations was using the boxed state machine itself as the continuation object that could be queued onto a Task, and in the common case where the continuation could be executed synchronously, there would then not be any further allocations. However, if the continuation needed to be run asynchronously (e.g. because the Task required it via RunContinuationsAsynchronously), the code would allocate a new work item object and queue that to the thread pool to execute. This then also forced the state machine object to lazily allocate the Action delegate for its MoveNext method. This PR extends the system slightly to also cover that asynchronous execution case, by making the state machine box itself being queueable to the thread pool. In doing so, it avoids that AwaitTaskContinuation allocation and also avoids forcing the delegate into existence. (As is the case for other optimizations, this one is only employed when ETW logging isn't enabled; if it is enabled, we need to flow more information, and enabling that would penalize the non-logging case.)
Commit migrated from https://github.com/dotnet/coreclr/commit/
03ead514a3aa414b26246a4160d0593811fd1044