1 // Licensed to the .NET Foundation under one or more agreements.
2 // The .NET Foundation licenses this file to you under the MIT license.
3 // See the LICENSE file in the project root for more information.
9 // This module implements the tracking and execution of rejit requests. In order to avoid
10 // any overhead on the non-profiled case we don't intrude on any 'normal' data structures
11 // except one member on the AppDomain to hold our main hashtable and crst (the
12 // ReJitManager). See comments in rejit.h to understand relationships between ReJitInfo,
13 // SharedReJitInfo, and ReJitManager, particularly SharedReJitInfo::InternalFlags which
14 // capture the state of a rejit request, and ReJitInfo::InternalFlags which captures the
15 // state of a particular MethodDesc from a rejit request.
17 // A ReJIT request (tracked via SharedReJitInfo) is made at the level of a (Module *,
18 // methodDef) pair, and thus affects all instantiations of a generic. Each MethodDesc
19 // affected by a ReJIT request has its state tracked via a ReJitInfo instance. A
20 // ReJitInfo can represent a rejit request against an already-jitted MethodDesc, or a
21 // rejit request against a not-yet-jitted MethodDesc (called a "pre-rejit" request). A
22 // Pre-ReJIT request happens when a profiler specifies a (Module *, methodDef) pair that
23 // has not yet been JITted, or that represents a generic function which always has the
24 // potential to JIT new instantiations in the future.
26 // Top-level functions in this file of most interest are:
28 // * (static) code:ReJitManager::RequestReJIT:
29 // Profiling API just delegates all rejit requests directly to this function. It is
30 // responsible for recording the request into the appropriate ReJITManagers and for
31 // jump-stamping any already-JITted functions affected by the request (so that future
32 // calls hit the prestub)
34 // * code:ReJitManager::DoReJitIfNecessary:
35 // MethodDesc::DoPrestub calls this to determine whether it's been invoked to do a rejit.
36 // If so, ReJitManager::DoReJitIfNecessary is responsible for (indirectly) gathering the
37 // appropriate IL and codegen flags, calling UnsafeJitFunction(), and redirecting the
38 // jump-stamp from the prestub to the newly-rejitted code.
40 // * code:PublishMethodHolder::PublishMethodHolder
41 // MethodDesc::MakeJitWorker() calls this to determine if there's an outstanding
42 // "pre-rejit" request for a MethodDesc that has just been jitted for the first time. We
43 // also call this from MethodDesc::CheckRestore when restoring generic methods.
44 // The holder applies the jump-stamp to the
45 // top of the originally JITted code, with the jump target being the prestub.
46 // When ReJIT is enabled this holder enters the ReJIT
47 // lock to enforce atomicity of doing the pre-rejit-jmp-stamp & publishing/restoring
48 // the PCODE, which is required to avoid races with a profiler that calls RequestReJIT
49 // just as the method finishes compiling/restoring.
51 // * code:PublishMethodTableHolder::PublishMethodTableHolder
52 // Does the same thing as PublishMethodHolder except iterating over every
53 // method in the MethodTable. This is called from MethodTable::SetIsRestored.
55 // * code:ReJitManager::GetCurrentReJitFlags:
56 // CEEInfo::canInline() calls this as part of its calculation of whether it may inline a
57 // given method. (Profilers may specify on a per-rejit-request basis whether the rejit of
58 // a method may inline callees.)
63 // For a given Module/MethodDef there is at most 1 SharedReJitInfo that is not Reverted,
64 // though there may be many that are in the Reverted state. If a method is rejitted
65 // multiple times, with multiple versions actively in use on the stacks, then all but the
66 // most recent are put into the Reverted state even though they may not yet be physically
67 // reverted and pitched yet.
69 // For a given MethodDesc there is at most 1 ReJitInfo in the kJumpToPrestub or kJumpToRejittedCode
72 // The ReJitManager::m_crstTable lock is held whenever reading or writing to that
73 // ReJitManager instance's table (including state transitions applied to the ReJitInfo &
74 // SharedReJitInfo instances stored in that table).
76 // The ReJitManager::m_crstTable lock is never held during callbacks to the profiler
77 // such as GetReJITParameters, ReJITStarted, JITComplete, ReportReJITError
79 // Any thread holding the ReJitManager::m_crstTable lock can't block during runtime suspension
80 // therefore it can't call any GC_TRIGGERS functions
82 // Transitions between SharedRejitInfo states happen only in the following cicumstances:
83 // 1) New SharedRejitInfo added to table (Requested State)
84 // Inside RequestRejit
85 // Global Crst held, table Crst held
87 // 2) Requested -> GettingReJITParameters
88 // Inside DoRejitIfNecessary
89 // Global Crst NOT held, table Crst held
91 // 3) GettingReJITParameters -> Active
92 // Inside DoRejitIfNecessary
93 // Global Crst NOT held, table Crst held
96 // Inside RequestRejit or RequestRevert
97 // Global Crst held, table Crst held
100 // Transitions between RejitInfo states happen only in the following circumstances:
101 // 1) New RejitInfo added to table (kJumpNone state)
102 // Inside RequestRejit, DoJumpStampIfNecessary
103 // Global Crst MAY/MAY NOT be held, table Crst held
104 // Allowed SharedReJit states: Requested, GettingReJITParameters, Active
106 // 2) kJumpNone -> kJumpToPrestub
107 // Inside RequestRejit, DoJumpStampIfNecessary
108 // Global Crst MAY/MAY NOT be held, table Crst held
109 // Allowed SharedReJit states: Requested, GettingReJITParameters, Active
111 // 3) kJumpToPreStub -> kJumpToRejittedCode
112 // Inside DoReJitIfNecessary
113 // Global Crst NOT held, table Crst held
114 // Allowed SharedReJit states: Active
117 // Inside RequestRevert, RequestRejit
118 // Global Crst held, table crst held
119 // Allowed SharedReJit states: Reverted
122 // #Beware Invariant misconceptions - don't make bad assumptions!
123 // Even if a SharedReJitInfo is in the Reverted state:
124 // a) RejitInfos may still be in the kJumpToPreStub or kJumpToRejittedCode state
125 // Reverted really just means the runtime has started reverting, but it may not
126 // be complete yet on the thread executing Revert or RequestRejit.
127 // b) The code for this version of the method may be executing on any number of
128 // threads. Even after transitioning all rejit infos to kJumpNone state we
129 // have no power to abort or hijack threads already running the rejitted code.
131 // Even if a SharedReJitInfo is in the Active state:
132 // a) The corresponding ReJitInfos may not be jump-stamped yet.
133 // Some thread is still in the progress of getting this thread jump-stamped
134 // OR it is a place-holder ReJitInfo.
135 // b) An older ReJitInfo linked to a reverted SharedReJitInfo could still be
136 // in kJumpToPreStub or kJumpToReJittedCode state. RequestRejit is still in
137 // progress on some thread.
140 // #Known issues with REJIT at this time:
141 // NGEN inlined methods will not be properly rejitted
142 // Exception callstacks through rejitted code do not produce correct StackTraces
143 // Live debugging is not supported when rejit is enabled
144 // Rejit leaks rejitted methods, RejitInfos, and SharedRejitInfos until AppDomain unload
145 // Dump debugging doesn't correctly locate RejitInfos that are keyed by MethodDesc
146 // Metadata update creates large memory increase switching to RW (not specifically a rejit issue)
148 // ======================================================================================
152 #include "method.hpp"
153 #include "eeconfig.h"
154 #include "methoditer.h"
155 #include "dbginterface.h"
156 #include "threadsuspend.h"
159 #ifdef FEATURE_CODE_VERSIONING
161 #include "../debug/ee/debugger.h"
162 #include "../debug/ee/walker.h"
163 #include "../debug/ee/controller.h"
164 #include "codeversion.h"
166 // This HRESULT is only used as a private implementation detail. Corerror.xml has a comment in it
167 // reserving this value for our use but it doesn't appear in the public headers.
168 #define CORPROF_E_RUNTIME_SUSPEND_REQUIRED _HRESULT_TYPEDEF_(0x80131381L)
170 // This is just used as a unique id. Overflow is OK. If we happen to have more than 4+Billion rejits
171 // and somehow manage to not run out of memory, we'll just have to redefine ReJITID as size_t.
173 static ReJITID s_GlobalReJitId = 1;
176 CrstStatic ReJitManager::s_csGlobalRequest;
179 //---------------------------------------------------------------------------------------
183 CORJIT_FLAGS ReJitManager::JitFlagsFromProfCodegenFlags(DWORD dwCodegenFlags)
185 LIMITED_METHOD_DAC_CONTRACT;
187 CORJIT_FLAGS jitFlags;
188 if ((dwCodegenFlags & COR_PRF_CODEGEN_DISABLE_ALL_OPTIMIZATIONS) != 0)
190 jitFlags.Set(CORJIT_FLAGS::CORJIT_FLAG_DEBUG_CODE);
192 if ((dwCodegenFlags & COR_PRF_CODEGEN_DISABLE_INLINING) != 0)
194 jitFlags.Set(CORJIT_FLAGS::CORJIT_FLAG_NO_INLINING);
197 // In the future more flags may be added that need to be converted here (e.g.,
198 // COR_PRF_CODEGEN_ENTERLEAVE / CORJIT_FLAG_PROF_ENTERLEAVE)
203 //---------------------------------------------------------------------------------------
204 // ProfilerFunctionControl implementation
206 ProfilerFunctionControl::ProfilerFunctionControl(LoaderHeap * pHeap) :
212 m_cInstrumentedMapEntries(0),
213 m_rgInstrumentedMapEntries(NULL)
215 LIMITED_METHOD_CONTRACT;
218 ProfilerFunctionControl::~ProfilerFunctionControl()
220 LIMITED_METHOD_CONTRACT;
222 // Intentionally not deleting m_pbIL or m_rgInstrumentedMapEntries, as its ownership gets transferred to the
223 // SharedReJitInfo that manages that rejit request.
227 HRESULT ProfilerFunctionControl::QueryInterface(REFIID id, void** pInterface)
229 LIMITED_METHOD_CONTRACT;
231 if ((id != IID_IUnknown) &&
232 (id != IID_ICorProfilerFunctionControl))
235 return E_NOINTERFACE;
243 ULONG ProfilerFunctionControl::AddRef()
245 LIMITED_METHOD_CONTRACT;
247 return InterlockedIncrement(&m_refCount);
250 ULONG ProfilerFunctionControl::Release()
252 LIMITED_METHOD_CONTRACT;
254 ULONG refCount = InterlockedDecrement(&m_refCount);
264 //---------------------------------------------------------------------------------------
266 // Profiler calls this to specify a set of flags from COR_PRF_CODEGEN_FLAGS
267 // to control rejitting a particular methodDef.
270 // * flags - set of flags from COR_PRF_CODEGEN_FLAGS
276 HRESULT ProfilerFunctionControl::SetCodegenFlags(DWORD flags)
278 LIMITED_METHOD_CONTRACT;
280 m_dwCodegenFlags = flags;
284 //---------------------------------------------------------------------------------------
286 // Profiler calls this to specify the IL to use when rejitting a particular methodDef.
289 // * cbNewILMethodHeader - Size in bytes of pbNewILMethodHeader
290 // * pbNewILMethodHeader - Pointer to beginning of IL header + IL bytes.
293 // HRESULT indicating success or failure.
296 // Caller owns allocating and freeing pbNewILMethodHeader as expected.
297 // SetILFunctionBody copies pbNewILMethodHeader into a separate buffer.
300 HRESULT ProfilerFunctionControl::SetILFunctionBody(ULONG cbNewILMethodHeader, LPCBYTE pbNewILMethodHeader)
310 if (cbNewILMethodHeader == 0)
315 if (pbNewILMethodHeader == NULL)
320 _ASSERTE(m_cbIL == 0);
321 _ASSERTE(m_pbIL == NULL);
323 #ifdef DACCESS_COMPILE
324 m_pbIL = new (nothrow) BYTE[cbNewILMethodHeader];
326 // IL is stored on the appropriate loader heap, and its memory will be owned by the
327 // SharedReJitInfo we copy the pointer to.
328 m_pbIL = (LPBYTE) (void *) m_pHeap->AllocMem_NoThrow(S_SIZE_T(cbNewILMethodHeader));
332 return E_OUTOFMEMORY;
335 m_cbIL = cbNewILMethodHeader;
336 memcpy(m_pbIL, pbNewILMethodHeader, cbNewILMethodHeader);
341 HRESULT ProfilerFunctionControl::SetILInstrumentedCodeMap(ULONG cILMapEntries, COR_IL_MAP * rgILMapEntries)
343 #ifdef DACCESS_COMPILE
344 // I'm not sure why any of these methods would need to be compiled in DAC? Could we remove the
345 // entire class from the DAC'ized code build?
346 _ASSERTE(!"This shouldn't be called in DAC");
358 if (cILMapEntries >= (MAXULONG / sizeof(COR_IL_MAP)))
360 // Too big! The allocation below would overflow when calculating the size.
364 if (g_pDebugInterface == NULL)
366 return CORPROF_E_DEBUGGING_DISABLED;
370 // copy the il map and il map entries into the corresponding fields.
371 m_cInstrumentedMapEntries = cILMapEntries;
373 // IL is stored on the appropriate loader heap, and its memory will be owned by the
374 // SharedReJitInfo we copy the pointer to.
375 m_rgInstrumentedMapEntries = (COR_IL_MAP*) (void *) m_pHeap->AllocMem_NoThrow(S_SIZE_T(cILMapEntries * sizeof(COR_IL_MAP)));
377 if (m_rgInstrumentedMapEntries == NULL)
378 return E_OUTOFMEMORY;
381 memcpy_s(m_rgInstrumentedMapEntries, sizeof(COR_IL_MAP) * cILMapEntries, rgILMapEntries, sizeof(COR_IL_MAP) * cILMapEntries);
384 #endif // DACCESS_COMPILE
387 //---------------------------------------------------------------------------------------
389 // ReJitManager may use this to access the codegen flags the profiler had set on this
390 // ICorProfilerFunctionControl.
393 // * codegen flags previously set via SetCodegenFlags; 0 if none were set.
395 DWORD ProfilerFunctionControl::GetCodegenFlags()
397 return m_dwCodegenFlags;
400 //---------------------------------------------------------------------------------------
402 // ReJitManager may use this to access the IL header + instructions the
403 // profiler had set on this ICorProfilerFunctionControl via SetIL
406 // * Pointer to ProfilerFunctionControl-allocated buffer containing the
407 // IL header and instructions the profiler had provided.
409 LPBYTE ProfilerFunctionControl::GetIL()
414 //---------------------------------------------------------------------------------------
416 // ReJitManager may use this to access the count of instrumented map entry flags the
417 // profiler had set on this ICorProfilerFunctionControl.
420 // * size of the instrumented map entry array
422 ULONG ProfilerFunctionControl::GetInstrumentedMapEntryCount()
424 return m_cInstrumentedMapEntries;
427 //---------------------------------------------------------------------------------------
429 // ReJitManager may use this to access the instrumented map entries the
430 // profiler had set on this ICorProfilerFunctionControl.
433 // * the array of instrumented map entries
435 COR_IL_MAP* ProfilerFunctionControl::GetInstrumentedMapEntries()
437 return m_rgInstrumentedMapEntries;
440 //---------------------------------------------------------------------------------------
441 // ReJitManager implementation
443 // All the state-changey stuff is kept up here in the !DACCESS_COMPILE block.
444 // The more read-only inspection-y stuff follows the block.
446 #ifndef DACCESS_COMPILE
447 NativeImageInliningIterator::NativeImageInliningIterator() :
450 m_dynamicBuffer(NULL),
451 m_dynamicBufferSize(0),
457 HRESULT NativeImageInliningIterator::Reset(Module *pModule, MethodDesc *pInlinee)
459 _ASSERTE(pModule != NULL);
460 _ASSERTE(pInlinee != NULL);
463 m_pInlinee = pInlinee;
468 // Trying to use the existing buffer
470 Module *inlineeModule = m_pInlinee->GetModule();
471 mdMethodDef mdInlinee = m_pInlinee->GetMemberDef();
472 COUNT_T methodsAvailable = m_pModule->GetNativeOrReadyToRunInliners(inlineeModule, mdInlinee, m_dynamicBufferSize, m_dynamicBuffer, &incompleteData);
474 // If the existing buffer is not large enough, reallocate.
475 if (methodsAvailable > m_dynamicBufferSize)
477 COUNT_T newSize = max(methodsAvailable, s_bufferSize);
478 m_dynamicBuffer = new MethodInModule[newSize];
479 m_dynamicBufferSize = newSize;
481 methodsAvailable = m_pModule->GetNativeOrReadyToRunInliners(inlineeModule, mdInlinee, m_dynamicBufferSize, m_dynamicBuffer, &incompleteData);
482 _ASSERTE(methodsAvailable <= m_dynamicBufferSize);
485 EX_CATCH_HRESULT(hr);
499 BOOL NativeImageInliningIterator::Next()
501 if (m_currentPos < 0)
507 return m_currentPos < m_dynamicBufferSize;
510 MethodDesc *NativeImageInliningIterator::GetMethodDesc()
512 if (m_currentPos == (COUNT_T)-1 || m_currentPos >= m_dynamicBufferSize)
517 MethodInModule mm = m_dynamicBuffer[m_currentPos];
518 Module *pModule = mm.m_module;
519 mdMethodDef mdInliner = mm.m_methodDef;
520 return pModule->LookupMethodDef(mdInliner);
523 //---------------------------------------------------------------------------------------
525 // ICorProfilerInfo4::RequestReJIT calls into this guy to do most of the
526 // work. Takes care of finding the appropriate ReJitManager instances to
527 // record the rejit requests and perform jmp-stamping.
530 // * cFunctions - Element count of rgModuleIDs & rgMethodDefs
531 // * rgModuleIDs - Parallel array of ModuleIDs to rejit
532 // * rgMethodDefs - Parallel array of methodDefs to rejit
535 // HRESULT indicating success or failure of the overall operation. Each
536 // individual methodDef (or MethodDesc associated with the methodDef)
537 // may encounter its own failure, which is reported by the ReJITError()
538 // callback, which is called into the profiler directly.
542 HRESULT ReJitManager::RequestReJIT(
544 ModuleID rgModuleIDs[],
545 mdMethodDef rgMethodDefs[],
546 COR_PRF_REJIT_FLAGS flags)
548 return ReJitManager::UpdateActiveILVersions(cFunctions, rgModuleIDs, rgMethodDefs, NULL, FALSE, flags);
552 HRESULT ReJitManager::UpdateActiveILVersions(
554 ModuleID rgModuleIDs[],
555 mdMethodDef rgMethodDefs[],
556 HRESULT rgHrStatuses[],
558 COR_PRF_REJIT_FLAGS flags)
569 // Serialize all RequestReJIT() and Revert() calls against each other (even across AppDomains)
570 CrstHolder ch(&(s_csGlobalRequest));
574 // Request at least 1 method to reJIT!
575 _ASSERTE ((cFunctions != 0) && (rgModuleIDs != NULL) && (rgMethodDefs != NULL));
577 // Temporary storage to batch up all the ReJitInfos that will get jump stamped
578 // later when the runtime is suspended.
580 //DESKTOP WARNING: On CoreCLR we are safe but if this code ever gets ported back
581 //there aren't any protections against domain unload. Any of these moduleIDs
582 //code version managers, or code versions would become invalid if the domain which
583 //contains them was unloaded.
584 SHash<CodeActivationBatchTraits> mgrToCodeActivationBatch;
585 CDynArray<CodeVersionManager::CodePublishError> errorRecords;
586 for (ULONG i = 0; i < cFunctions; i++)
588 Module * pModule = reinterpret_cast< Module * >(rgModuleIDs[i]);
589 if (pModule == NULL || TypeFromToken(rgMethodDefs[i]) != mdtMethodDef)
591 ReportReJITError(pModule, rgMethodDefs[i], NULL, E_INVALIDARG);
595 if (pModule->IsBeingUnloaded())
597 ReportReJITError(pModule, rgMethodDefs[i], NULL, CORPROF_E_DATAINCOMPLETE);
601 if (pModule->IsReflection())
603 ReportReJITError(pModule, rgMethodDefs[i], NULL, CORPROF_E_MODULE_IS_DYNAMIC);
607 if (!pModule->GetMDImport()->IsValidToken(rgMethodDefs[i]))
609 ReportReJITError(pModule, rgMethodDefs[i], NULL, E_INVALIDARG);
613 MethodDesc * pMD = pModule->LookupMethodDef(rgMethodDefs[i]);
617 _ASSERTE(!pMD->IsNoMetadata());
619 // Weird, non-user functions can't be rejitted
622 // Intentionally not reporting an error in this case, to be consistent
623 // with the pre-rejit case, as we have no opportunity to report an error
624 // in a pre-rejit request for a non-IL method, since the rejit manager
625 // never gets a call from the prestub worker for non-IL methods. Thus,
626 // since pre-rejit requests silently ignore rejit requests for non-IL
627 // methods, regular rejit requests will also silently ignore rejit requests for
628 // non-IL methods to be consistent.
633 hr = UpdateActiveILVersion(&mgrToCodeActivationBatch, pModule, rgMethodDefs[i], fIsRevert, static_cast<COR_PRF_REJIT_FLAGS>(flags | COR_PRF_REJIT_INLINING_CALLBACKS));
639 if ((flags & COR_PRF_REJIT_BLOCK_INLINING) == COR_PRF_REJIT_BLOCK_INLINING)
641 hr = UpdateNativeInlinerActiveILVersions(&mgrToCodeActivationBatch, pMD, fIsRevert, flags);
647 hr = UpdateJitInlinerActiveILVersions(&mgrToCodeActivationBatch, pMD, fIsRevert, flags);
653 } // for (ULONG i = 0; i < cFunctions; i++)
655 // For each code versioning mgr, if there's work to do, suspend EE if needed,
656 // enter the code versioning mgr's crst, and do the batched work.
657 BOOL fEESuspended = FALSE;
658 SHash<CodeActivationBatchTraits>::Iterator beginIter = mgrToCodeActivationBatch.Begin();
659 SHash<CodeActivationBatchTraits>::Iterator endIter = mgrToCodeActivationBatch.End();
662 MethodDescBackpatchInfoTracker::ConditionalLockHolder lockHolder;
664 for (SHash<CodeActivationBatchTraits>::Iterator iter = beginIter; iter != endIter; iter++)
666 CodeActivationBatch * pCodeActivationBatch = *iter;
667 CodeVersionManager * pCodeVersionManager = pCodeActivationBatch->m_pCodeVersionManager;
669 int cMethodsToActivate = pCodeActivationBatch->m_methodsToActivate.Count();
670 if (cMethodsToActivate == 0)
676 // SetActiveILCodeVersions takes the SystemDomain crst, which needs to be acquired before the
678 SystemDomain::LockHolder lh;
682 // As a potential future optimization we could speculatively try to update the jump stamps without
683 // suspending the runtime. That needs to be plumbed through BatchUpdateJumpStamps though.
684 ThreadSuspend::SuspendEE(ThreadSuspend::SUSPEND_FOR_REJIT);
688 _ASSERTE(ThreadStore::HoldingThreadStore());
689 hr = pCodeVersionManager->SetActiveILCodeVersions(pCodeActivationBatch->m_methodsToActivate.Ptr(), pCodeActivationBatch->m_methodsToActivate.Count(), fEESuspended, &errorRecords);
696 ThreadSuspend::RestartEE(FALSE, TRUE);
702 _ASSERTE(hr == E_OUTOFMEMORY);
706 // Report any errors that were batched up
707 for (int i = 0; i < errorRecords.Count(); i++)
709 if (rgHrStatuses != NULL)
711 for (DWORD j = 0; j < cFunctions; j++)
713 if (rgMethodDefs[j] == errorRecords[i].methodDef &&
714 reinterpret_cast<Module*>(rgModuleIDs[j]) == errorRecords[i].pModule)
716 rgHrStatuses[j] = errorRecords[i].hrStatus;
722 ReportReJITError(&(errorRecords[i]));
727 // We got through processing everything, but profiler will need to see the individual ReJITError
728 // callbacks to know what, if anything, failed.
733 HRESULT ReJitManager::UpdateActiveILVersion(
734 SHash<CodeActivationBatchTraits> *pMgrToCodeActivationBatch,
736 mdMethodDef methodDef,
738 COR_PRF_REJIT_FLAGS flags)
749 _ASSERTE(pMgrToCodeActivationBatch != NULL);
750 _ASSERTE(pModule != NULL);
751 _ASSERTE(methodDef != mdTokenNil);
755 CodeVersionManager * pCodeVersionManager = pModule->GetCodeVersionManager();
756 _ASSERTE(pCodeVersionManager != NULL);
757 CodeActivationBatch * pCodeActivationBatch = pMgrToCodeActivationBatch->Lookup(pCodeVersionManager);
758 if (pCodeActivationBatch == NULL)
760 pCodeActivationBatch = new (nothrow)CodeActivationBatch(pCodeVersionManager);
761 if (pCodeActivationBatch == NULL)
763 return E_OUTOFMEMORY;
769 // This guy throws when out of memory, but remains internally
770 // consistent (without adding the new element)
771 pMgrToCodeActivationBatch->Add(pCodeActivationBatch);
773 EX_CATCH_HRESULT(hr);
775 _ASSERT(hr == S_OK || hr == E_OUTOFMEMORY);
783 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
785 // Bind the il code version
786 ILCodeVersion* pILCodeVersion = pCodeActivationBatch->m_methodsToActivate.Append();
787 if (pILCodeVersion == NULL)
789 return E_OUTOFMEMORY;
793 // activate the original version
794 *pILCodeVersion = ILCodeVersion(pModule, methodDef);
798 // activate an unused or new IL version
799 hr = ReJitManager::BindILVersion(pCodeVersionManager, pModule, methodDef, pILCodeVersion, flags);
802 _ASSERTE(hr == E_OUTOFMEMORY);
812 HRESULT ReJitManager::UpdateNativeInlinerActiveILVersions(
813 SHash<CodeActivationBatchTraits> *pMgrToCodeActivationBatch,
814 MethodDesc *pInlinee,
816 COR_PRF_REJIT_FLAGS flags)
827 _ASSERTE(pMgrToCodeActivationBatch != NULL);
828 _ASSERTE(pInlinee != NULL);
832 // Iterate through all modules, for any that are NGEN or R2R need to check if there are inliners there and call
833 // RequestReJIT on them
834 // TODO: is the default domain enough for coreclr?
835 AppDomain::AssemblyIterator domainAssemblyIterator = SystemDomain::System()->DefaultDomain()->IterateAssembliesEx((AssemblyIterationFlags) (kIncludeLoaded));
836 CollectibleAssemblyHolder<DomainAssembly *> pDomainAssembly;
837 NativeImageInliningIterator inlinerIter;
838 while (domainAssemblyIterator.Next(pDomainAssembly.This()))
840 _ASSERTE(pDomainAssembly != NULL);
841 _ASSERTE(pDomainAssembly->GetAssembly() != NULL);
843 DomainModuleIterator domainModuleIterator = pDomainAssembly->IterateModules(kModIterIncludeLoaded);
844 while (domainModuleIterator.Next())
846 Module * pCurModule = domainModuleIterator.GetModule();
847 if (pCurModule->HasNativeOrReadyToRunInlineTrackingMap())
849 inlinerIter.Reset(pCurModule, pInlinee);
851 MethodDesc *pInliner = NULL;
852 while (inlinerIter.Next())
854 pInliner = inlinerIter.GetMethodDesc();
856 CodeVersionManager *pCodeVersionManager = pCurModule->GetCodeVersionManager();
857 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
858 ILCodeVersion ilVersion = pCodeVersionManager->GetActiveILCodeVersion(pInliner);
859 if (!ilVersion.HasDefaultIL())
861 // This method has already been ReJITted, no need to request another ReJIT at this point.
862 // The ReJITted method will be in the JIT inliner check below.
867 hr = UpdateActiveILVersion(pMgrToCodeActivationBatch, pInliner->GetModule(), pInliner->GetMemberDef(), fIsRevert, flags);
870 ReportReJITError(pInliner->GetModule(), pInliner->GetMemberDef(), NULL, hr);
881 HRESULT ReJitManager::UpdateJitInlinerActiveILVersions(
882 SHash<CodeActivationBatchTraits> *pMgrToCodeActivationBatch,
883 MethodDesc *pInlinee,
885 COR_PRF_REJIT_FLAGS flags)
896 _ASSERTE(pMgrToCodeActivationBatch != NULL);
897 _ASSERTE(pInlinee != NULL);
901 Module *pModule = pInlinee->GetModule();
902 if (pModule->HasJitInlineTrackingMap())
904 // JITInlineTrackingMap::VisitInliners wants to be in cooperative mode,
905 // but UpdateActiveILVersion wants to be in preemptive mode. Rather than do
906 // a bunch of mode switching just batch up the inliners.
907 InlineSArray<MethodDesc *, 10> inliners;
908 auto lambda = [&](MethodDesc *inliner, MethodDesc *inlinee)
910 _ASSERTE(!inliner->IsNoMetadata());
916 // InlineSArray can throw if we run out of memory,
917 // need to guard against it.
918 inliners.Append(inliner);
920 EX_CATCH_HRESULT(hr);
922 return SUCCEEDED(hr);
929 JITInlineTrackingMap *pMap = pModule->GetJitInlineTrackingMap();
930 pMap->VisitInliners(pInlinee, lambda);
938 // InlineSArray iterator can throw
939 for (auto it = inliners.Begin(); it != inliners.End(); ++it)
941 Module *inlinerModule = (*it)->GetModule();
942 mdMethodDef inlinerMethodDef = (*it)->GetMemberDef();
943 hr = UpdateActiveILVersion(pMgrToCodeActivationBatch, inlinerModule, inlinerMethodDef, fIsRevert, flags);
946 ReportReJITError(inlinerModule, inlinerMethodDef, NULL, hr);
950 EX_CATCH_HRESULT(hr);
957 HRESULT ReJitManager::BindILVersion(
958 CodeVersionManager *pCodeVersionManager,
960 mdMethodDef methodDef,
961 ILCodeVersion *pILCodeVersion,
962 COR_PRF_REJIT_FLAGS flags)
970 PRECONDITION(CheckPointer(pCodeVersionManager));
971 PRECONDITION(CheckPointer(pModule));
972 PRECONDITION(CheckPointer(pILCodeVersion));
976 _ASSERTE(pCodeVersionManager->LockOwnedByCurrentThread());
977 _ASSERTE((pModule != NULL) && (methodDef != mdTokenNil));
979 // Check if there was there a previous rejit request for this method that hasn't been exposed back
980 // to the profiler yet
981 ILCodeVersion ilCodeVersion = pCodeVersionManager->GetActiveILCodeVersion(pModule, methodDef);
982 BOOL fDoCallback = (flags & COR_PRF_REJIT_INLINING_CALLBACKS) == COR_PRF_REJIT_INLINING_CALLBACKS;
984 if (ilCodeVersion.GetRejitState() == ILCodeVersion::kStateRequested)
986 // We can 'reuse' this instance because the profiler doesn't know about
987 // it yet. (This likely happened because a profiler called RequestReJIT
988 // twice in a row, without us having a chance to jmp-stamp the code yet OR
989 // while iterating through instantiations of a generic, the iterator found
990 // duplicate entries for the same instantiation.)
991 // TODO: this assert likely needs to be removed. This code path should be
992 // hit for any duplicates, and that can happen regardless of whether this
993 // is the first ReJIT or not.
994 _ASSERTE(ilCodeVersion.HasDefaultIL());
996 *pILCodeVersion = ilCodeVersion;
1000 // There could be a case where the method that a profiler requested ReJIT on also ends up in the
1001 // inlining graph from a different method. In that case we should override the previous setting,
1002 // but we should never override a request to get the callback with a request to suppress it.
1003 pILCodeVersion->SetEnableReJITCallback(true);
1009 // Either there was no ILCodeVersion yet for this MethodDesc OR whatever we've found
1010 // couldn't be reused (and needed to be reverted). Create a new ILCodeVersion to return
1012 HRESULT hr = pCodeVersionManager->AddILCodeVersion(pModule, methodDef, InterlockedIncrement(reinterpret_cast<LONG*>(&s_GlobalReJitId)), pILCodeVersion);
1013 pILCodeVersion->SetEnableReJITCallback(fDoCallback);
1017 //---------------------------------------------------------------------------------------
1019 // ICorProfilerInfo4::RequestRevert calls into this guy to do most of the
1020 // work. Takes care of finding the appropriate ReJitManager instances to
1021 // perform the revert
1024 // * cFunctions - Element count of rgModuleIDs & rgMethodDefs
1025 // * rgModuleIDs - Parallel array of ModuleIDs to revert
1026 // * rgMethodDefs - Parallel array of methodDefs to revert
1027 // * rgHrStatuses - [out] Parallel array of HRESULTs indicating success/failure
1028 // of reverting each (ModuleID, methodDef).
1031 // HRESULT indicating success or failure of the overall operation. Each
1032 // individual methodDef (or MethodDesc associated with the methodDef)
1033 // may encounter its own failure, which is reported by the rgHrStatuses
1038 HRESULT ReJitManager::RequestRevert(
1040 ModuleID rgModuleIDs[],
1041 mdMethodDef rgMethodDefs[],
1042 HRESULT rgHrStatuses[])
1053 return UpdateActiveILVersions(cFunctions, rgModuleIDs, rgMethodDefs, rgHrStatuses, TRUE, static_cast<COR_PRF_REJIT_FLAGS>(0));
1057 HRESULT ReJitManager::ConfigureILCodeVersion(ILCodeVersion ilCodeVersion)
1059 STANDARD_VM_CONTRACT;
1061 CodeVersionManager* pCodeVersionManager = ilCodeVersion.GetModule()->GetCodeVersionManager();
1062 _ASSERTE(!pCodeVersionManager->LockOwnedByCurrentThread());
1066 Module* pModule = ilCodeVersion.GetModule();
1067 mdMethodDef methodDef = ilCodeVersion.GetMethodDef();
1068 BOOL fNeedsParameters = FALSE;
1069 BOOL fWaitForParameters = FALSE;
1072 // Serialize access to the rejit state
1073 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
1074 switch (ilCodeVersion.GetRejitState())
1076 case ILCodeVersion::kStateRequested:
1077 ilCodeVersion.SetRejitState(ILCodeVersion::kStateGettingReJITParameters);
1078 fNeedsParameters = TRUE;
1081 case ILCodeVersion::kStateGettingReJITParameters:
1082 fWaitForParameters = TRUE;
1090 if (fNeedsParameters)
1093 ReleaseHolder<ProfilerFunctionControl> pFuncControl = NULL;
1095 if (ilCodeVersion.GetEnableReJITCallback())
1097 // Here's where we give a chance for the rejit requestor to
1098 // examine and modify the IL & codegen flags before it gets to
1099 // the JIT. This allows one to add probe calls for things like
1100 // code coverage, performance, or whatever. These will be
1101 // stored in pShared.
1102 _ASSERTE(pModule != NULL);
1103 _ASSERTE(methodDef != mdTokenNil);
1105 new (nothrow)ProfilerFunctionControl(pModule->GetLoaderAllocator()->GetLowFrequencyHeap());
1106 if (pFuncControl == NULL)
1112 BEGIN_PIN_PROFILER(CORProfilerPresent());
1113 hr = g_profControlBlock.pProfInterface->GetReJITParameters(
1121 if (!ilCodeVersion.GetEnableReJITCallback() || FAILED(hr))
1124 // Historically on failure we would revert to the kRequested state and fall-back
1125 // to the initial code gen. The next time the method ran it would try again.
1127 // Preserving that behavior is possible, but a bit awkward now that we have
1128 // Precode swapping as well. Instead of doing that I am acting as if GetReJITParameters
1129 // had succeeded, using the original IL, no jit flags, and no modified IL mapping.
1130 // This is similar to a fallback except the profiler won't get any further attempts
1131 // to provide the parameters correctly. If the profiler wants another attempt it would
1132 // need to call RequestRejit again.
1134 // This code path also happens if the GetReJITParameters callback was suppressed due to
1135 // the method being ReJITted as an inliner by the runtime (instead of by the user).
1136 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
1137 if (ilCodeVersion.GetRejitState() == ILCodeVersion::kStateGettingReJITParameters)
1139 ilCodeVersion.SetRejitState(ILCodeVersion::kStateActive);
1140 ilCodeVersion.SetIL(ILCodeVersion(pModule, methodDef).GetIL());
1146 // Only call if the GetReJITParamters call failed
1147 ReportReJITError(pModule, methodDef, pModule->LookupMethodDef(methodDef), hr);
1153 _ASSERTE(pFuncControl != NULL);
1155 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
1156 if (ilCodeVersion.GetRejitState() == ILCodeVersion::kStateGettingReJITParameters)
1158 // Inside the above call to ICorProfilerCallback4::GetReJITParameters, the profiler
1159 // will have used the specified pFuncControl to provide its IL and codegen flags.
1160 // So now we transfer it out to the SharedReJitInfo.
1161 ilCodeVersion.SetJitFlags(pFuncControl->GetCodegenFlags());
1162 ilCodeVersion.SetIL((COR_ILMETHOD*)pFuncControl->GetIL());
1163 // ilCodeVersion is now the owner of the memory for the IL buffer
1164 ilCodeVersion.SetInstrumentedILMap(pFuncControl->GetInstrumentedMapEntryCount(),
1165 pFuncControl->GetInstrumentedMapEntries());
1166 ilCodeVersion.SetRejitState(ILCodeVersion::kStateActive);
1170 else if (fWaitForParameters)
1172 // This feels annoying, but it doesn't appear like we have the good threading primitves
1173 // for this. What I would like is an AutoResetEvent that atomically exits the table
1174 // Crst when I wait on it. From what I can tell our AutoResetEvent doesn't have
1175 // that atomic transition which means this ordering could occur:
1176 // [Thread 1] detect kStateGettingParameters and exit table lock
1177 // [Thread 2] enter table lock, transition kStateGettingParameters -> kStateActive
1178 // [Thread 2] signal AutoResetEvent
1179 // [Thread 2] exit table lock
1180 // [Thread 1] wait on AutoResetEvent (which may never be signaled again)
1182 // Another option would be ManualResetEvents, one for each SharedReJitInfo, but
1183 // that feels like a lot of memory overhead to handle a case which occurs rarely.
1184 // A third option would be dynamically creating ManualResetEvents in a side
1185 // dictionary on demand, but that feels like a lot of complexity for an event
1186 // that occurs rarely.
1188 // I just ended up with this simple polling loop. Assuming profiler
1189 // writers implement GetReJITParameters performantly we will only iterate
1190 // this loop once, and even then only in the rare case of threads racing
1191 // to JIT the same IL. If this really winds up causing performance issues
1192 // We can build something more sophisticated.
1196 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
1197 if (ilCodeVersion.GetRejitState() == ILCodeVersion::kStateActive)
1199 break; // the other thread got the parameters succesfully, go race to rejit
1202 ClrSleepEx(1, FALSE);
1209 #endif // DACCESS_COMPILE
1210 // The rest of the ReJitManager methods are safe to compile for DAC
1212 //---------------------------------------------------------------------------------------
1214 // Used by profiler to get the ReJITID corrseponding to a (MethodDesc *, PCODE) pair.
1215 // Can also be used to determine whether (MethodDesc *, PCODE) corresponds to a rejit
1216 // (vs. a regular JIT) for the purposes of deciding whether to notify the debugger about
1217 // the rejit (and building the debugger JIT info structure).
1220 // * pMD - MethodDesc * of interestg
1221 // * pCodeStart - PCODE of the particular interesting JITting of that MethodDesc *
1224 // 0 if no such ReJITID found (e.g., PCODE is from a JIT and not a rejit), else the
1225 // ReJITID requested.
1228 ReJITID ReJitManager::GetReJitId(PTR_MethodDesc pMD, PCODE pCodeStart)
1235 PRECONDITION(CheckPointer(pMD));
1236 PRECONDITION(pCodeStart != NULL);
1240 // Fast-path: If the rejit map is empty, no need to look up anything. Do this outside
1241 // of a lock to impact our caller (the prestub worker) as little as possible. If the
1242 // map is nonempty, we'll acquire the lock at that point and do the lookup for real.
1243 CodeVersionManager* pCodeVersionManager = pMD->GetCodeVersionManager();
1244 if (pCodeVersionManager->GetNonDefaultILVersionCount() == 0)
1249 CodeVersionManager::TableLockHolder ch(pCodeVersionManager);
1250 return ReJitManager::GetReJitIdNoLock(pMD, pCodeStart);
1253 //---------------------------------------------------------------------------------------
1255 // See comment above code:ReJitManager::GetReJitId for main details of what this does.
1257 // This function is basically the same as GetReJitId, except caller is expected to take
1258 // the ReJitManager lock directly (via ReJitManager::TableLockHolder). This exists so
1259 // that ETW can explicitly take the triggering ReJitManager lock up front, and in the
1260 // proper order, to avoid lock leveling issues, and triggering issues with other locks it
1261 // takes that are CRST_UNSAFE_ANYMODE
1264 ReJITID ReJitManager::GetReJitIdNoLock(PTR_MethodDesc pMD, PCODE pCodeStart)
1271 PRECONDITION(CheckPointer(pMD));
1272 PRECONDITION(pCodeStart != NULL);
1276 // Caller must ensure this lock is taken!
1277 CodeVersionManager* pCodeVersionManager = pMD->GetCodeVersionManager();
1278 _ASSERTE(pCodeVersionManager->LockOwnedByCurrentThread());
1280 NativeCodeVersion nativeCodeVersion = pCodeVersionManager->GetNativeCodeVersion(pMD, pCodeStart);
1281 if (nativeCodeVersion.IsNull())
1285 return nativeCodeVersion.GetILCodeVersion().GetVersionId();
1288 //---------------------------------------------------------------------------------------
1290 // Called by profiler to retrieve an array of ReJITIDs corresponding to a MethodDesc *
1293 // * pMD - MethodDesc * to look up
1294 // * cReJitIds - Element count capacity of reJitIds
1295 // * pcReJitIds - [out] Place total count of ReJITIDs found here; may be more than
1296 // cReJitIds if profiler passed an array that's too small to hold them all
1297 // * reJitIds - [out] Place ReJITIDs found here. Count of ReJITIDs returned here is
1298 // min(cReJitIds, *pcReJitIds)
1301 // * S_OK: ReJITIDs successfully returned, array is big enough
1302 // * S_FALSE: ReJITIDs successfully found, but array was not big enough. Only
1303 // cReJitIds were returned and cReJitIds < *pcReJitId (latter being the total
1304 // number of ReJITIDs available).
1307 HRESULT ReJitManager::GetReJITIDs(PTR_MethodDesc pMD, ULONG cReJitIds, ULONG * pcReJitIds, ReJITID reJitIds[])
1314 PRECONDITION(CheckPointer(pMD));
1315 PRECONDITION(pcReJitIds != NULL);
1316 PRECONDITION(reJitIds != NULL);
1320 CodeVersionManager* pCodeVersionManager = pMD->GetCodeVersionManager();
1321 CodeVersionManager::TableLockHolder lock(pCodeVersionManager);
1325 ILCodeVersionCollection ilCodeVersions = pCodeVersionManager->GetILCodeVersions(pMD);
1326 for (ILCodeVersionIterator iter = ilCodeVersions.Begin(), end = ilCodeVersions.End();
1330 ILCodeVersion curILVersion = *iter;
1332 if (curILVersion.GetRejitState() == ILCodeVersion::kStateActive)
1334 if (cnt < cReJitIds)
1336 reJitIds[cnt] = curILVersion.GetVersionId();
1346 return (cnt > cReJitIds) ? S_FALSE : S_OK;
1349 #endif // FEATURE_CODE_VERSIONING
1350 #else // FEATURE_REJIT
1352 // On architectures that don't support rejit, just keep around some do-nothing
1353 // stubs so the rest of the VM doesn't have to be littered with #ifdef FEATURE_REJIT
1356 HRESULT ReJitManager::RequestReJIT(
1358 ModuleID rgModuleIDs[],
1359 mdMethodDef rgMethodDefs[],
1360 COR_PRF_REJIT_FLAGS flags)
1366 HRESULT ReJitManager::RequestRevert(
1368 ModuleID rgModuleIDs[],
1369 mdMethodDef rgMethodDefs[],
1370 HRESULT rgHrStatuses[])
1375 ReJITID ReJitManager::GetReJitId(PTR_MethodDesc pMD, PCODE pCodeStart)
1380 ReJITID ReJitManager::GetReJitIdNoLock(PTR_MethodDesc pMD, PCODE pCodeStart)
1385 HRESULT ReJitManager::GetReJITIDs(PTR_MethodDesc pMD, ULONG cReJitIds, ULONG * pcReJitIds, ReJITID reJitIds[])
1390 #endif // FEATURE_REJIT