From: Gleb Balykov Date: Fri, 22 Apr 2022 10:28:32 +0000 (+0300) Subject: [Tizen] Add mcj-edit.py tool that can modify MultiCoreJit profiles. X-Git-Tag: accepted/tizen/unified/20221103.165808~22 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=4bab09349da598984d69269d05ba06a73371aff8;p=platform%2Fupstream%2Fdotnet%2Fruntime.git [Tizen] Add mcj-edit.py tool that can modify MultiCoreJit profiles. Available commands: - split (splits mcj profile in app-dependent and app-independent based on modules) - merge (merges mcj profiles in one profile) - verify (verifies correctness of mcj profile) - find (finds method or module in mcj profile) - compare (compares mcj profiles) - clean-stats (cleans mcj profile usage stats) - print (prints mcj profile) - help (shows short summary on some aspects of mcj profile) - self-test (performs some testing) See README for more details on usage, see examples/ for examples of mcj profiles. --- diff --git a/src/coreclr/tools/mcj-edit/README.md b/src/coreclr/tools/mcj-edit/README.md new file mode 100644 index 0000000..15b05b5 --- /dev/null +++ b/src/coreclr/tools/mcj-edit/README.md @@ -0,0 +1,95 @@ +# MCJ profile editor + +`mcj-edit.py` is MCJ profile editor, see `--help` for all available options, see `help` for mcj profile short summary and for binary signature short summary. + +See `examples/` for examples of mcj profiles. + +## Examples of usage + +### To show mcj profile format summary: + +```sh +python3 mcj-edit.py help --mcj-format +``` + +### To show binary signature format summary: + +```sh +python3 mcj-edit.py help --binary-signature-format +``` + +### To show help: + +```sh +python3 mcj-edit.py --help +``` + +### To verify profile correctness: + +```sh +python3 mcj-edit.py verify -i `pwd`/profile.dat +``` + +### To print profile: + +```sh +python3 mcj-edit.py print -i `pwd`/profile.dat +``` + +Add `--raw` option to print raw profile, i.e. the way it is saved in file. + +Add `--header`, `--modules`, `--module-deps-and-methods` options to print only parts of profile. + +### To find method or module in profile: + +```sh +python3 mcj-edit.py find -i `pwd`/profile.dat --module System.Private.CoreLib --method-token 100663502 +``` + +Use `--module` option to find module by name, `--method-token` to find generic/non-generic method by token. + +### To compare profiles: + +Using load and compare: +```sh +python3 mcj-edit.py compare -i `pwd`/profile.dat -i /tmp/profile2.dat +``` + +To use sha256sum on files add `--sha256` option. + +### To split profile in app-dependent and app-independent: + +```sh +python3 mcj-edit.py split -i `pwd`/profile.dat --system-modules-list `pwd`/system_modules.txt +``` + +After this command two new files will be created: `pwd`/profile.dat.app for app-dependent profile and `pwd`/profile.dat.sys for app-independent profile. + +### To merge profiles: + +```sh +python3 mcj-edit.py merge -i `pwd`/profile.dat -i /tmp/profile2.dat -o `pwd`/profile.merged.dat +``` + +After this command new file wil be created: `pwd`/profile.merged.dat. + +**Note: merge can't be performed on two arbitrary mcj profiles! Next profiles can't be merged:** +- if one profile contains module with name `AAA` and version `X` and another profile contains module with same name `AAA` and version `Y` +- if one profile contains module with name `AAA` and assembly name `BBB` and another profile contains module with same name `AAA` and assembly name `CCC` +- if one profile contains module with name `AAA` with flags `X` and another profile contains module with same name `AAA` with flags `Y` (this situation should not happen now, flags are always 0) +- if one profile contains method with token/signature `XXX` with flags `X` and another profile contains method with same token/signature `XXX` with flags `Y` (this situation can happen now only if one profile was rewritten during use, i.e. JIT_BY_APP_THREAD_TAG was not set for some methods, i.e. COMPlus_MultiCoreJitNoProfileGather=1 was not set) + +### To run some tests: + +```sh +python3 mcj-edit.py self-test --rw-sha256 -i `pwd`/profile.dat +python3 mcj-edit.py self-test --rw -i `pwd`/profile.dat +python3 mcj-edit.py self-test --sm -i `pwd`/profile.dat --system-modules-list `pwd`/system_modules.txt +``` + +## pylint + +To run pylint: +```sh +pylint -d superfluous-parens,missing-function-docstring,missing-class-docstring,missing-module-docstring,too-many-boolean-expressions,too-many-arguments,too-many-public-methods,too-many-lines --max-line-length=120 --method-naming-style=camelCase --argument-naming-style=camelCase --attr-naming-style=camelCase --variable-naming-style=camelCase mcj-edit.py +``` diff --git a/src/coreclr/tools/mcj-edit/examples/helloworld/README.md b/src/coreclr/tools/mcj-edit/examples/helloworld/README.md new file mode 100644 index 0000000..09c8513 --- /dev/null +++ b/src/coreclr/tools/mcj-edit/examples/helloworld/README.md @@ -0,0 +1,3 @@ +# HelloWorld + +`profile.dat` is mcj profile for console helloworld launched without ni.dll, `system_modules.txt` is related number of system modules. diff --git a/src/coreclr/tools/mcj-edit/examples/helloworld/profile.dat b/src/coreclr/tools/mcj-edit/examples/helloworld/profile.dat new file mode 100644 index 0000000..9c3c6d9 Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/helloworld/profile.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/helloworld/system_modules.txt b/src/coreclr/tools/mcj-edit/examples/helloworld/system_modules.txt new file mode 100644 index 0000000..ca9d15b --- /dev/null +++ b/src/coreclr/tools/mcj-edit/examples/helloworld/system_modules.txt @@ -0,0 +1,6 @@ +System.Private.CoreLib +System.Console +System.Threading +Microsoft.Win32.Primitives +System.Collections +System.Memory diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/README.md b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/README.md new file mode 100644 index 0000000..65e01a7 --- /dev/null +++ b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/README.md @@ -0,0 +1,9 @@ +# Tizen CSharp Samples + +Built apps from https://github.com/Samsung/Tizen-CSharp-Samples + +With this setup all dlls have corresponding ni.dll: +- netcoreapp is in bubble with itself +- framework is in bubble with netcoreapp and itself +- frameworkref is in bubble with netcoreapp, framework and itself +- app is in bubble with all above diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Alarm.dat b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Alarm.dat new file mode 100644 index 0000000..a318701 Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Alarm.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Calculator.dat b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Calculator.dat new file mode 100644 index 0000000..bd6ea87 Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Calculator.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.HeartRateMonitor.dat b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.HeartRateMonitor.dat new file mode 100644 index 0000000..54ced78 Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.HeartRateMonitor.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Net.VoiceMemo.Tizen.Wearable.dat b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Net.VoiceMemo.Tizen.Wearable.dat new file mode 100644 index 0000000..dba9408 Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Net.VoiceMemo.Tizen.Wearable.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Weather.dat b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Weather.dat new file mode 100644 index 0000000..81761cd Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.Weather.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.XStopWatch.dat b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.XStopWatch.dat new file mode 100644 index 0000000..23845b5 Binary files /dev/null and b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/org.tizen.example.XStopWatch.dat differ diff --git a/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/system_modules.txt b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/system_modules.txt new file mode 100644 index 0000000..80c5d85 --- /dev/null +++ b/src/coreclr/tools/mcj-edit/examples/tizen-csharp-samples/system_modules.txt @@ -0,0 +1,265 @@ +Microsoft.CSharp +Microsoft.VisualBasic.Core +Microsoft.VisualBasic +Microsoft.Win32.Primitives +Microsoft.Win32.Registry +System.AppContext +System.Buffers +System.Collections.Concurrent +System.Collections.Immutable +System.Collections.NonGeneric +System.Collections.Specialized +System.Collections +System.ComponentModel.Annotations +System.ComponentModel.DataAnnotations +System.ComponentModel.EventBasedAsync +System.ComponentModel.Primitives +System.ComponentModel.TypeConverter +System.ComponentModel +System.Configuration +System.Console +System.Core +System.Data.Common +System.Data.DataSetExtensions +System.Data +System.Diagnostics.Contracts +System.Diagnostics.Debug +System.Diagnostics.DiagnosticSource +System.Diagnostics.FileVersionInfo +System.Diagnostics.Process +System.Diagnostics.StackTrace +System.Diagnostics.TextWriterTraceListener +System.Diagnostics.Tools +System.Diagnostics.TraceSource +System.Diagnostics.Tracing +System.Drawing.Primitives +System.Drawing +System.Dynamic.Runtime +System.Formats.Asn1 +System.Globalization.Calendars +System.Globalization.Extensions +System.Globalization +System.IO.Compression.Brotli +System.IO.Compression.FileSystem +System.IO.Compression.ZipFile +System.IO.Compression +System.IO.FileSystem.AccessControl +System.IO.FileSystem.DriveInfo +System.IO.FileSystem.Primitives +System.IO.FileSystem.Watcher +System.IO.FileSystem +System.IO.IsolatedStorage +System.IO.MemoryMappedFiles +System.IO.Pipes.AccessControl +System.IO.Pipes +System.IO.UnmanagedMemoryStream +System.IO +System.Linq.Expressions +System.Linq.Parallel +System.Linq.Queryable +System.Linq +System.Memory +System.Net.Http.Json +System.Net.Http +System.Net.HttpListener +System.Net.Mail +System.Net.NameResolution +System.Net.NetworkInformation +System.Net.Ping +System.Net.Primitives +System.Net.Quic +System.Net.Requests +System.Net.Security +System.Net.ServicePoint +System.Net.Sockets +System.Net.WebClient +System.Net.WebHeaderCollection +System.Net.WebProxy +System.Net.WebSockets.Client +System.Net.WebSockets +System.Net +System.Numerics.Vectors +System.Numerics +System.ObjectModel +System.Private.CoreLib +System.Private.DataContractSerialization +System.Private.Uri +System.Private.Xml.Linq +System.Private.Xml +System.Reflection.DispatchProxy +System.Reflection.Emit.ILGeneration +System.Reflection.Emit.Lightweight +System.Reflection.Emit +System.Reflection.Extensions +System.Reflection.Metadata +System.Reflection.Primitives +System.Reflection.TypeExtensions +System.Reflection +System.Resources.Reader +System.Resources.ResourceManager +System.Resources.Writer +System.Runtime.CompilerServices.Unsafe +System.Runtime.CompilerServices.VisualC +System.Runtime.Extensions +System.Runtime.Handles +System.Runtime.InteropServices.RuntimeInformation +System.Runtime.InteropServices +System.Runtime.Intrinsics +System.Runtime.Loader +System.Runtime.Numerics +System.Runtime.Serialization.Formatters +System.Runtime.Serialization.Json +System.Runtime.Serialization.Primitives +System.Runtime.Serialization.Xml +System.Runtime.Serialization +System.Runtime +System.Security.AccessControl +System.Security.Claims +System.Security.Cryptography.Algorithms +System.Security.Cryptography.Cng +System.Security.Cryptography.Csp +System.Security.Cryptography.Encoding +System.Security.Cryptography.OpenSsl +System.Security.Cryptography.Primitives +System.Security.Cryptography.X509Certificates +System.Security.Principal.Windows +System.Security.Principal +System.Security.SecureString +System.Security +System.ServiceModel.Web +System.ServiceProcess +System.Text.Encoding.CodePages +System.Text.Encoding.Extensions +System.Text.Encoding +System.Text.Encodings.Web +System.Text.Json +System.Text.RegularExpressions +System.Threading.Channels +System.Threading.Overlapped +System.Threading.Tasks.Dataflow +System.Threading.Tasks.Extensions +System.Threading.Tasks.Parallel +System.Threading.Tasks +System.Threading.Thread +System.Threading.ThreadPool +System.Threading.Timer +System.Threading +System.Transactions.Local +System.Transactions +System.ValueTuple +System.Web.HttpUtility +System.Web +System.Windows +System.Xml.Linq +System.Xml.ReaderWriter +System.Xml.Serialization +System.Xml.XDocument +System.Xml.XPath.XDocument +System.Xml.XPath +System.Xml.XmlDocument +System.Xml.XmlSerializer +System.Xml +System +WindowsBase +mscorlib +netstandard +ElmSharp.Wearable +ElmSharp +OpenTK +Tizen.Account.AccountManager +Tizen.Account.FidoClient +Tizen.Account.OAuth2 +Tizen.Account.SyncManager +Tizen.Applications.Alarm +Tizen.Applications.Badge +Tizen.Applications.Cion +Tizen.Applications.Common +Tizen.Applications.ComponentBased.ComponentManager +Tizen.Applications.ComponentBased.Default +Tizen.Applications.ComponentBased.Port +Tizen.Applications.ComponentBased +Tizen.Applications.DataControl +Tizen.Applications.EventManager +Tizen.Applications.MessagePort +Tizen.Applications.Notification +Tizen.Applications.NotificationEventListener +Tizen.Applications.PackageManager +Tizen.Applications.Preference +Tizen.Applications.RemoteView +Tizen.Applications.Service +Tizen.Applications.ThemeManager +Tizen.Applications.ToastMessage +Tizen.Applications.UI +Tizen.Applications.WatchApplication +Tizen.Applications.WatchfaceComplication +Tizen.Applications.WidgetApplication +Tizen.Applications.WidgetControl +Tizen.Content.Download +Tizen.Content.MediaContent +Tizen.Content.MimeType +Tizen.Context +Tizen.Inspections +Tizen.Location +Tizen.Log +Tizen.MachineLearning.Inference +Tizen.Maps +Tizen.Messaging.Push +Tizen.Messaging +Tizen.Multimedia.AudioIO +Tizen.Multimedia.Camera +Tizen.Multimedia.MediaCodec +Tizen.Multimedia.MediaPlayer +Tizen.Multimedia.Metadata +Tizen.Multimedia.Radio +Tizen.Multimedia.Recorder +Tizen.Multimedia.Remoting +Tizen.Multimedia.StreamRecorder +Tizen.Multimedia.Util +Tizen.Multimedia.Vision +Tizen.Multimedia +Tizen.NUI.Components +Tizen.NUI.Extension +Tizen.NUI.Wearable +Tizen.NUI.WindowSystem +Tizen.NUI +Tizen.Network.Bluetooth +Tizen.Network.Connection +Tizen.Network.IoTConnectivity +Tizen.Network.Nfc +Tizen.Network.Nsd +Tizen.Network.Smartcard +Tizen.Network.Stc +Tizen.Network.WiFi +Tizen.Peripheral +Tizen.PhonenumberUtils +Tizen.Pims.Calendar +Tizen.Pims.Contacts +Tizen.Runtime +Tizen.Security.DevicePolicyManager +Tizen.Security.PrivacyPrivilegeManager +Tizen.Security.SecureRepository +Tizen.Security.TEEC +Tizen.Security +Tizen.Sensor +Tizen.System.Feedback +Tizen.System.Information +Tizen.System.MediaKey +Tizen.System.PlatformConfig +Tizen.System.PowerUsage +Tizen.System.Storage +Tizen.System.SystemSettings +Tizen.System +Tizen.Telephony +Tizen.Tracer +Tizen.Uix.InputMethod +Tizen.Uix.InputMethodManager +Tizen.Uix.Stt +Tizen.Uix.SttEngine +Tizen.Uix.Tts +Tizen.Uix.TtsEngine +Tizen.Uix.VoiceControl +Tizen.Uix.VoiceControlManager +Tizen.Uix.VoiceControlWidget +Tizen.WebView +Tizen +XSF diff --git a/src/coreclr/tools/mcj-edit/mcj-edit.py b/src/coreclr/tools/mcj-edit/mcj-edit.py new file mode 100644 index 0000000..2cc145f --- /dev/null +++ b/src/coreclr/tools/mcj-edit/mcj-edit.py @@ -0,0 +1,4467 @@ +# +# The MIT License (MIT) +# +# Copyright (c) 2022 Samsung Electronics Co., Ltd. +# +# All rights reserved. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. +# + +from abc import ABC, abstractmethod +import argparse +import subprocess + +# ---------------------------------------------------------------------------------------------------------------------- +# Base class for error exceptions +class Error(Exception): + pass + +# Error with message +class MessageError(Error): + def __init__(self, message): + super().__init__() + self.message = message + +# Error with message about mcj profile inconsistency +class ProfileInconsistencyError(MessageError): + def __init__(self): + super().__init__("Inconsistency in mcj format") + +# Error with message about mcj-edit internal error +class InternalError(MessageError): + def __init__(self): + super().__init__("Internal error in mcj-edit") + +# Error with message about unsupported case in mcj-edit +class UnsupportedError(MessageError): + def __init__(self): + super().__init__("Unsupported case in mcj-edit") + +# ---------------------------------------------------------------------------------------------------------------------- +# Class with helper functions +class Utility: + + # Align value to specified alignment + @staticmethod + def alignUp(value, alignment): + if (value is None or alignment is None): + raise InternalError() + if (not issubclass(type(value), int) or not issubclass(type(alignment), int)): + raise InternalError() + if (value < 0 or alignment < 0): + raise InternalError() + return (value + alignment - 1) // alignment * alignment + + # Merge bytes into a value + @staticmethod + def mergeNBytes(bytesArr, isLittleEndian=True): + if (bytesArr is None or isLittleEndian is None): + raise InternalError() + if (not issubclass(type(bytesArr), list) or not issubclass(type(isLittleEndian), bool)): + raise InternalError() + + value = 0 + + for index, byte in enumerate(bytesArr): + if (not issubclass(type(byte), int)): + raise InternalError() + if (byte < 0 or byte > Utility.getMaxForNBytes(1)): + raise InternalError() + + if (isLittleEndian): + value = value | (byte << index * 8) + else: + value = (value << 8) | byte + + return value + + # Split bytes of value + @staticmethod + def splitNBytes(value, numbytes, isLittleEndian=True): + if (value is None or numbytes is None or isLittleEndian is None): + raise InternalError() + if (not issubclass(type(value), int) or not issubclass(type(numbytes), int) + or not issubclass(type(isLittleEndian), bool)): + raise InternalError() + if (value < 0 or numbytes < 0): + raise InternalError() + + bytesArr = [] + + for index in range(numbytes): # pylint: disable=unused-variable + if (isLittleEndian): + bytesArr.append(value & 0xff) + else: + bytesArr.insert(0, value & 0xff) + value = value >> 8 + + if (value > 0): + raise InternalError() + + return bytesArr + + # Pack bits: 8 high bits for value1, 24 low bits for value2 + @staticmethod + def pack8_24(value1, value2): # pylint: disable=invalid-name + if (value1 is None or value2 is None): + raise InternalError() + if (not issubclass(type(value1), int) or not issubclass(type(value2), int)): + raise InternalError() + if (value1 < 0 or value1 > Utility.getMaxForNBytes(1) + or value2 < 0 or value2 > Utility.getMaxForNBytes(3)): + raise InternalError() + return (value1 << 24) | value2 + + # Get max allowed value for unsigned N bits + @staticmethod + def getMaxForNBits(numBits): + if (numBits is None): + raise InternalError() + if (not issubclass(type(numBits), int)): + raise InternalError() + if (numBits < 0): + raise InternalError() + return (1 << numBits) - 1 + + # Get max allowed value for unsigned N bytes + @staticmethod + def getMaxForNBytes(numBytes): + if (numBytes is None): + raise InternalError() + if (not issubclass(type(numBytes), int)): + raise InternalError() + if (numBytes < 0): + raise InternalError() + return Utility.getMaxForNBits(numBytes * 8) + +# ---------------------------------------------------------------------------------------------------------------------- +# These are some constants from runtime, which should be checked before usage! +# See: +# - src/coreclr/vm/multicorejitimpl.h +# - ... +# +# TODO: add function that checks/sets these by parsing file from runtime +class RuntimeConstants: # pylint: disable=too-few-public-methods + METHOD_FLAGS_MASK = 0xff0000 + JIT_BY_APP_THREAD_TAG = 0x10000 + RECORD_TYPE_OFFSET = 24 + MAX_MODULES = 0x1000 + MODULE_MASK = 0xffff + MODULE_LEVEL_OFFSET = 16 + MAX_MODULE_LEVELS = 0x100 + MAX_METHODS = 0x4000 + #SIGNATURE_LENGTH_MASK = 0xffff + HEADER_W_COUNTER = 14 + HEADER_D_COUNTER = 3 + #MULTICOREJITLIFE = 60 * 1000 + #MAX_WALKBACK = 128 + + MULTICOREJIT_PROFILE_VERSION = 102 + MULTICOREJIT_HEADER_RECORD_ID = 1 + MULTICOREJIT_MODULE_RECORD_ID = 2 + MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID = 3 + MULTICOREJIT_METHOD_RECORD_ID = 4 + MULTICOREJIT_GENERICMETHOD_RECORD_ID = 5 + + # Next constant are used directly and have no named variables in runtime + X_MODULE_RECORD_LEN_MASK = 0xffffff + +# ---------------------------------------------------------------------------------------------------------------------- +# These are sizes of types in bytes +class RuntimeTypeSizes: # pylint: disable=too-few-public-methods + int = 4 + short = 2 + char = 1 + +# ---------------------------------------------------------------------------------------------------------------------- +# Header of MCJ profile. +# +# Corresponds to "struct HeaderRecord" from multicorejitimpl.h +# +# To create from bytes: HeaderRecord.createFromBytes(bytes) +# To create from data structures using references: HeaderRecord.createByRef(...) +# To create from data structures using copy: HeaderRecord.createByCopy(...) +# +class HeaderRecord: # pylint: disable=too-many-instance-attributes + Alignment = RuntimeTypeSizes.int + + Size = 6 * RuntimeTypeSizes.int \ + + Utility.alignUp(RuntimeConstants.HEADER_W_COUNTER * RuntimeTypeSizes.short, Alignment) \ + + RuntimeConstants.HEADER_D_COUNTER * RuntimeTypeSizes.int + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._recordId = None + self._version = None + self._timeStamp = None + self._moduleCount = None + self._methodCount = None + self._moduleDepCount = None + self._shortCounters = None + self._longCounters = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), HeaderRecord)): + return False + return self._recordId == rhs._recordId and self._version == rhs._version \ + and self._timeStamp == rhs._timeStamp and self._moduleCount == rhs._moduleCount \ + and self._methodCount == rhs._methodCount and self._moduleDepCount == rhs._moduleDepCount \ + and self._shortCounters == rhs._shortCounters and self._longCounters == rhs._longCounters + + # Get moduleCount + def getModuleCount(self): + return self._moduleCount + + # Get methodCount + def getMethodCount(self): + return self._methodCount + + # Get moduleDepCount + def getModuleDepCount(self): + return self._moduleDepCount + + # Set moduleCount + def setModuleCount(self, moduleCount): + self._moduleCount = moduleCount + self.verify() + + # Set methodCount + def setMethodCount(self, methodCount): + self._methodCount = methodCount + self.verify() + + # Set moduleDepCount + def setModuleDepCount(self, moduleDepCount): + self._moduleDepCount = moduleDepCount + self.verify() + + # 0-init usage stats + def dropGlobalUsageStats(self): + self._shortCounters = [0] * RuntimeConstants.HEADER_W_COUNTER + self._longCounters = [0] * RuntimeConstants.HEADER_D_COUNTER + self.verify() + + # Verify consistency + def verify(self): + if (self._recordId is None or self._version is None + or self._timeStamp is None or self._moduleCount is None + or self._methodCount is None or self._moduleDepCount is None + or self._shortCounters is None or self._longCounters is None): + raise InternalError() + + if (not issubclass(type(self._recordId), int) or not issubclass(type(self._version), int) + or not issubclass(type(self._timeStamp), int) or not issubclass(type(self._moduleCount), int) + or not issubclass(type(self._methodCount), int) or not issubclass(type(self._moduleDepCount), int) + or not issubclass(type(self._shortCounters), list) or not issubclass(type(self._longCounters), list)): + raise InternalError() + + if (self._recordId < 0 or self._recordId > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._version < 0 or self._version > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._timeStamp < 0 or self._timeStamp > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._moduleCount < 0 or self._moduleCount > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._methodCount < 0 or self._methodCount > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._moduleDepCount < 0 or self._moduleDepCount > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or len(self._shortCounters) != RuntimeConstants.HEADER_W_COUNTER + or len(self._longCounters) != RuntimeConstants.HEADER_D_COUNTER): + raise InternalError() + + for i in self._shortCounters: + if (i < 0 or i > Utility.getMaxForNBytes(RuntimeTypeSizes.short)): + raise InternalError() + + for i in self._longCounters: + if (i < 0 or i > Utility.getMaxForNBytes(RuntimeTypeSizes.short)): + raise InternalError() + + if (self._version != RuntimeConstants.MULTICOREJIT_PROFILE_VERSION): + raise ProfileInconsistencyError() + + if (self._moduleCount > RuntimeConstants.MAX_MODULES): + raise ProfileInconsistencyError() + + if (self._methodCount > RuntimeConstants.MAX_METHODS): + raise ProfileInconsistencyError() + + if (self._recordId != Utility.pack8_24(RuntimeConstants.MULTICOREJIT_HEADER_RECORD_ID, HeaderRecord.Size)): + raise ProfileInconsistencyError() + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> Raw HeaderRecord:") + print(offsetStr + "recordId = " + str(self._recordId)) + print(offsetStr + "version = " + str(self._version)) + print(offsetStr + "timeStamp = " + str(self._timeStamp)) + print(offsetStr + "moduleCount = " + str(self._moduleCount)) + print(offsetStr + "methodCount = " + str(self._methodCount)) + print(offsetStr + "moduleDepCount = " + str(self._moduleDepCount)) + print(offsetStr + "shortCounters = " + str(self._shortCounters)) + print(offsetStr + "longCounters = " + str(self._longCounters)) + + # Print pretty + def print(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> MultiCoreJit profile (version: " + str(self._version) + + ", time stamp: " + str(self._timeStamp) + ")") + print("") + print(offsetStr + "Number of used modules: " + str(self._moduleCount)) + print(offsetStr + "Number of method dependencies: " + str(self._methodCount)) + print(offsetStr + "Number of module dependencies: " + str(self._moduleDepCount)) + print("") + print(offsetStr + ">>> Stats for played profile (these are 0 if profile was not played):") + print("") + print(offsetStr + "Total number of methods: " + + str(self._shortCounters[0])) + print(offsetStr + "Number of methods, which had native code (i.e. no jit in mcj thread for them): " + + str(self._shortCounters[1])) + print(offsetStr + "Number of methods, which were tried to be jitted in mcj thread: " + + str(self._shortCounters[2])) + print(offsetStr + "Number of methods, which were successfully jitted in mcj thread: " + + str(self._shortCounters[3])) + print(offsetStr + "Number of methods, jit code for which was used after jit in mcj thread: " + + str(self._shortCounters[4])) + print(offsetStr + "Number of methods, which were skipped for some reason: " + + str(self._shortCounters[5])) + print(offsetStr + "Number of methods, which were skipped because of missing module: " + + str(self._shortCounters[6])) + print(offsetStr + "Number of methods, which were traversed backwards: " + + str(self._shortCounters[9])) + print("") + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None or not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) != HeaderRecord.Size): + raise ProfileInconsistencyError() + + index = 0 + + recordId = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + version = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + timeStamp = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + moduleCount = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + methodCount = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + moduleDepCount = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + shortCounters = [] + for i in range(0, RuntimeConstants.HEADER_W_COUNTER): # pylint: disable=unused-variable + shortCounters.append(Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short])) + index += RuntimeTypeSizes.short + + index = Utility.alignUp(index, HeaderRecord.Alignment) + + longCounters = [] + for i in range(0, RuntimeConstants.HEADER_D_COUNTER): # pylint: disable=unused-variable + longCounters.append(Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int])) + index += RuntimeTypeSizes.int + + return HeaderRecord.createByRef(recordId, version, timeStamp, moduleCount, methodCount, + moduleDepCount, shortCounters, longCounters) + + # Create from objects taking them by reference + @staticmethod + def createByRef(recordId, version, timeStamp, moduleCount, methodCount, + moduleDepCount, shortCounters, longCounters): + header = HeaderRecord() + + header._recordId = recordId # pylint: disable=protected-access + header._version = version # pylint: disable=protected-access + header._timeStamp = timeStamp # pylint: disable=protected-access + header._moduleCount = moduleCount # pylint: disable=protected-access + header._methodCount = methodCount # pylint: disable=protected-access + header._moduleDepCount = moduleDepCount # pylint: disable=protected-access + header._shortCounters = shortCounters # pylint: disable=protected-access + header._longCounters = longCounters # pylint: disable=protected-access + + header.verify() + return header + + # Create from objects taking them by copy + @staticmethod + def createByCopy(recordId, version, timeStamp, moduleCount, methodCount, + moduleDepCount, shortCounters, longCounters): + return HeaderRecord.createByRef(recordId, version, timeStamp, moduleCount, methodCount, + moduleDepCount, shortCounters.copy(), longCounters.copy()) + + # Copy object + def copy(self): + return HeaderRecord.createByCopy(self._recordId, self._version, self._timeStamp, + self._moduleCount, self._methodCount, self._moduleDepCount, + self._shortCounters, self._longCounters) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + bytesArr += Utility.splitNBytes(self._recordId, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(self._version, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(self._timeStamp, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(self._moduleCount, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(self._methodCount, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(self._moduleDepCount, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + for i in self._shortCounters: + bytesArr += Utility.splitNBytes(i, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + padding = Utility.alignUp(index, HeaderRecord.Alignment) - index + bytesArr += [0] * padding + index += padding + + for i in self._longCounters: + bytesArr += Utility.splitNBytes(i, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + if (index != HeaderRecord.Size): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# GUID. +# +# Corresponds to "GUID" from pal_mstypes.h +# +# To create from bytes: GUID.createFromBytes(bytes) +# To create from data structures using references: GUID.createByRef(...) +# To create from data structures using copy: GUID.createByCopy(...) +# +class GUID: + Alignment = RuntimeTypeSizes.int + + Size = RuntimeTypeSizes.int + 2 * RuntimeTypeSizes.short + 8 * RuntimeTypeSizes.char + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._data1 = None + self._data2 = None + self._data3 = None + self._data4 = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), GUID)): + return False + return self._data1 == rhs._data1 and self._data2 == rhs._data2 \ + and self._data3 == rhs._data3 and self._data4 == rhs._data4 + + # Verify consistency + def verify(self): + if (self._data1 is None or self._data2 is None or self._data3 is None or self._data4 is None): + raise InternalError() + + if (not issubclass(type(self._data1), int) or not issubclass(type(self._data2), int) + or not issubclass(type(self._data3), int) or not issubclass(type(self._data4), list)): + raise InternalError() + + if (self._data1 < 0 or self._data1 > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._data2 < 0 or self._data2 > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._data3 < 0 or self._data3 > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or len(self._data4) != 8): + raise InternalError() + + for i in self._data4: + if (i < 0 or i > Utility.getMaxForNBytes(RuntimeTypeSizes.char)): + raise InternalError() + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> Raw GUID:") + print(offsetStr + "data1 = " + str(self._data1)) + print(offsetStr + "data2 = " + str(self._data2)) + print(offsetStr + "data3 = " + str(self._data3)) + print(offsetStr + "data4 = " + str(self._data4)) + + # Print pretty + def print(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + "GUID: " + str(self._data1) + + "," + str(self._data2) + + "," + str(self._data3) + + "," + str(self._data4)) + print(offsetStr + "") + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None or not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) != GUID.Size): + raise ProfileInconsistencyError() + + index = 0 + + data1 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + data2 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + data3 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + data4 = [0] * 8 + for i in range(0, 8): + data4[i] = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.char]) + index += RuntimeTypeSizes.char + + return GUID.createByRef(data1, data2, data3, data4) + + # Create from objects taking them by reference + @staticmethod + def createByRef(data1, data2, data3, data4): + guid = GUID() + + guid._data1 = data1 # pylint: disable=protected-access + guid._data2 = data2 # pylint: disable=protected-access + guid._data3 = data3 # pylint: disable=protected-access + guid._data4 = data4 # pylint: disable=protected-access + + guid.verify() + return guid + + # Create from objects taking them by copy + @staticmethod + def createByCopy(data1, data2, data3, data4): + return GUID.createByRef(data1, data2, data3, data4.copy()) + + # Copy object + def copy(self): + return GUID.createByCopy(self._data1, self._data2, self._data3, self._data4) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + bytesArr += Utility.splitNBytes(self._data1, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(self._data2, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._data3, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + for i in self._data4: + bytesArr += Utility.splitNBytes(i, RuntimeTypeSizes.char) + index += RuntimeTypeSizes.char + + if (index != GUID.Size): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# Module version. +# +# Corresponds to "class ModuleVersion" from multicorejitimpl.h +# +# To create from bytes: ModuleVersion.createFromBytes(bytes) +# To create from data structures using references: ModuleVersion.createByRef(...) +# To create from data structures using copy: ModuleVersion.createByCopy(...) +# +class ModuleVersion: + Alignment = RuntimeTypeSizes.int + + Size = 4 * RuntimeTypeSizes.short + RuntimeTypeSizes.int + GUID.Size + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._major = None + self._minor = None + self._build = None + self._revision = None + self._versionFlags = None + self._hasNativeImage = None + self._mvid = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), ModuleVersion)): + return False + return self._major == rhs._major and self._minor == rhs._minor and self._build == rhs._build \ + and self._revision == rhs._revision and self._versionFlags == rhs._versionFlags \ + and self._hasNativeImage == rhs._hasNativeImage and self._mvid == rhs._mvid + + # Verify consistency + def verify(self): + if (self._major is None or self._minor is None or self._build is None or self._revision is None + or self._versionFlags is None or self._hasNativeImage is None or self._mvid is None): + raise InternalError() + + if (not issubclass(type(self._major), int) or not issubclass(type(self._minor), int) + or not issubclass(type(self._build), int) or not issubclass(type(self._revision), int) + or not issubclass(type(self._versionFlags), int) or not issubclass(type(self._hasNativeImage), int) + or not issubclass(type(self._mvid), GUID)): + raise InternalError() + + if (self._major < 0 or self._major > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._minor < 0 or self._minor > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._build < 0 or self._build > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._revision < 0 or self._revision > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._versionFlags < 0 or self._versionFlags > Utility.getMaxForNBits(RuntimeTypeSizes.int * 8 - 1) + or self._hasNativeImage < 0 or self._hasNativeImage > Utility.getMaxForNBits(1)): + raise InternalError() + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> Raw ModuleVersion:") + print(offsetStr + "major = " + str(self._major)) + print(offsetStr + "minor = " + str(self._minor)) + print(offsetStr + "build = " + str(self._build)) + print(offsetStr + "revision = " + str(self._revision)) + print(offsetStr + "versionFlags = " + str(self._versionFlags)) + print(offsetStr + "hasNativeImage = " + str(self._hasNativeImage)) + print(offsetStr + "mvid = {") + self._mvid.printRaw(offsetStr + " ") + print(offsetStr + "}") + + # Print pretty + def print(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + "ModuleVersion: " + str(self._major) + + "." + str(self._minor) + + "." + str(self._build) + + "." + str(self._revision) + + ", " + str(self._versionFlags) + + ", " + str(self._hasNativeImage)) + self._mvid.print(offsetStr) + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None or not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) != ModuleVersion.Size): + raise ProfileInconsistencyError() + + index = 0 + + major = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + minor = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + build = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + revision = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + versionFlags = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + hasNativeImage = versionFlags & 0x01 + versionFlags = versionFlags & 0xfffffffe + index += RuntimeTypeSizes.int + + mvid = GUID.createFromBytes(bytesArr[index:index+GUID.Size]) + index += GUID.Size + + return ModuleVersion.createByRef(major, minor, build, revision, versionFlags, hasNativeImage, mvid) + + # Create from objects taking them by reference + @staticmethod + def createByRef(major, minor, build, revision, versionFlags, hasNativeImage, mvid): + moduleVersion = ModuleVersion() + + moduleVersion._major = major # pylint: disable=protected-access + moduleVersion._minor = minor # pylint: disable=protected-access + moduleVersion._build = build # pylint: disable=protected-access + moduleVersion._revision = revision # pylint: disable=protected-access + moduleVersion._versionFlags = versionFlags # pylint: disable=protected-access + moduleVersion._hasNativeImage = hasNativeImage # pylint: disable=protected-access + moduleVersion._mvid = mvid # pylint: disable=protected-access + + moduleVersion.verify() + return moduleVersion + + # Create from objects taking them by copy + @staticmethod + def createByCopy(major, minor, build, revision, versionFlags, hasNativeImage, mvid): + return ModuleVersion.createByRef(major, minor, build, revision, versionFlags, hasNativeImage, mvid.copy()) + + # Copy object + def copy(self): + return ModuleVersion.createByCopy(self._major, self._minor, self._build, self._revision, + self._versionFlags, self._hasNativeImage, self._mvid) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + bytesArr += Utility.splitNBytes(self._major, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._minor, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._build, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._revision, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + versionFlags = self._versionFlags | self._hasNativeImage + bytesArr += Utility.splitNBytes(versionFlags, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += self._mvid.convertToBytes() + index += GUID.Size + + if (index != ModuleVersion.Size): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# Record with information about used module. +# +# Corresponds to "struct ModuleRecord" from multicorejitimpl.h +# ModuleRecord in runtime implicitly "contains" module name and assembly name strings after itself, +# here they are a part of ModuleRecordExtended. +# +# To create from bytes: ModuleRecord.createFromBytes(bytes) +# To create from data structures using references: ModuleRecord.createByRef(...) +# To create from data structures using copy: ModuleRecord.createByCopy(...) +# +class ModuleRecord: # pylint: disable=too-many-public-methods + Alignment = RuntimeTypeSizes.int + + Size = RuntimeTypeSizes.int + ModuleVersion.Size + Utility.alignUp(5 * RuntimeTypeSizes.short, Alignment) + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._recordId = None + self._version = None + self._jitMethodCount = None + self._flags = None + self._wLoadLevel = None + self._lenModuleName = None + self._lenAssemblyName = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), ModuleRecord)): + return False + return self._recordId == rhs._recordId and self._version == rhs._version \ + and self._jitMethodCount == rhs._jitMethodCount and self._flags == rhs._flags \ + and self._wLoadLevel == rhs._wLoadLevel and self._lenModuleName == rhs._lenModuleName \ + and self._lenAssemblyName == rhs._lenAssemblyName + + # Get recordId + def getRecordId(self): + return self._recordId + + # Get version + def getVersion(self): + return self._version + + # Get jitMethodCount + def getJitMethodCount(self): + return self._jitMethodCount + + # Get flags + def getFlags(self): + return self._flags + + # Get wLoadLevel + def getLoadLevel(self): + return self._wLoadLevel + + # Get lenModuleName + def getLenModuleName(self): + return self._lenModuleName + + # Get lenAssemblyName + def getLenAssemblyName(self): + return self._lenAssemblyName + + # Set jitMethodCount + def setJitMethodCount(self, count): + self._jitMethodCount = count + self.verify() + + # Set wLoadLevel + def setLoadLevel(self, level): + self._wLoadLevel = level + self.verify() + + # Verify consistency + def verify(self): + if (self._recordId is None or self._version is None or self._jitMethodCount is None or self._flags is None + or self._wLoadLevel is None or self._lenModuleName is None or self._lenAssemblyName is None): + raise InternalError() + + if (not issubclass(type(self._recordId), int) or not issubclass(type(self._version), ModuleVersion) + or not issubclass(type(self._jitMethodCount), int) or not issubclass(type(self._flags), int) + or not issubclass(type(self._wLoadLevel), int) or not issubclass(type(self._lenModuleName), int) + or not issubclass(type(self._lenAssemblyName), int)): + raise InternalError() + + if (self._recordId < 0 or self._recordId > Utility.getMaxForNBytes(RuntimeTypeSizes.int) + or self._jitMethodCount < 0 or self._jitMethodCount > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._flags < 0 or self._flags > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._wLoadLevel < 0 or self._wLoadLevel > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._lenModuleName < 0 or self._lenModuleName > Utility.getMaxForNBytes(RuntimeTypeSizes.short) + or self._lenAssemblyName < 0 or self._lenAssemblyName > Utility.getMaxForNBytes(RuntimeTypeSizes.short)): + raise InternalError() + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> Raw ModuleRecord:") + print(offsetStr + "recordId = " + str(self._recordId)) + print(offsetStr + "version = {") + self._version.printRaw(offsetStr + " ") + print(offsetStr + "}") + print(offsetStr + "jitMethodCount = " + str(self._jitMethodCount)) + print(offsetStr + "flags = " + str(self._flags)) + print(offsetStr + "wLoadLevel = " + str(self._wLoadLevel)) + print(offsetStr + "lenModuleName = " + str(self._lenModuleName)) + print(offsetStr + "lenAssemblyName = " + str(self._lenAssemblyName)) + + # Print pretty + def print(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + self._version.print(offsetStr) + print(offsetStr + "Number of used methods from module: " + str(self._jitMethodCount)) + print(offsetStr + "Final load level for module: " + str(self._wLoadLevel)) + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None or not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) != ModuleRecord.Size): + raise ProfileInconsistencyError() + + index = 0 + + recordId = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + version = ModuleVersion.createFromBytes(bytesArr[index:index+ModuleVersion.Size]) + index += ModuleVersion.Size + + jitMethodCount = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + flags = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + wLoadLevel = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + lenModuleName = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + lenAssemblyName = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + return ModuleRecord.createByRef(recordId, version, jitMethodCount, flags, wLoadLevel, + lenModuleName, lenAssemblyName) + + # Create from objects taking them by reference + @staticmethod + def createByRef(recordId, version, jitMethodCount, flags, wLoadLevel, lenModuleName, lenAssemblyName): + moduleRecord = ModuleRecord() + + moduleRecord._recordId = recordId # pylint: disable=protected-access + moduleRecord._version = version # pylint: disable=protected-access + moduleRecord._jitMethodCount = jitMethodCount # pylint: disable=protected-access + moduleRecord._flags = flags # pylint: disable=protected-access + moduleRecord._wLoadLevel = wLoadLevel # pylint: disable=protected-access + moduleRecord._lenModuleName = lenModuleName # pylint: disable=protected-access + moduleRecord._lenAssemblyName = lenAssemblyName # pylint: disable=protected-access + + moduleRecord.verify() + return moduleRecord + + # Create from objects taking them by copy + @staticmethod + def createByCopy(recordId, version, jitMethodCount, flags, wLoadLevel, lenModuleName, lenAssemblyName): + return ModuleRecord.createByRef(recordId, version.copy(), jitMethodCount, flags, wLoadLevel, + lenModuleName, lenAssemblyName) + + # Copy object + def copy(self): + return ModuleRecord.createByCopy(self._recordId, self._version, self._jitMethodCount, + self._flags, self._wLoadLevel, + self._lenModuleName, self._lenAssemblyName) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + bytesArr += Utility.splitNBytes(self._recordId, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += self._version.convertToBytes() + index += ModuleVersion.Size + + bytesArr += Utility.splitNBytes(self._jitMethodCount, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._flags, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._wLoadLevel, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._lenModuleName, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += Utility.splitNBytes(self._lenAssemblyName, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + padding = Utility.alignUp(index, HeaderRecord.Alignment) - index + bytesArr += [0] * padding + index += padding + + if (index != ModuleRecord.Size): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# Record with information about used module. +# +# This is ModuleRecord combined with actual module/assembly name strings. +# +# To create from bytes: ModuleRecordExtended.createFromBytes(bytes) +# To create from data structures using references: ModuleRecordExtended.createByRef(...) +# To create from data structures using copy: ModuleRecordExtended.createByCopy(...) +# +class ModuleRecordExtended: + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._moduleRecord = None + self._moduleName = None + self._assemblyName = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), ModuleRecordExtended)): + return False + + return self._moduleRecord == rhs._moduleRecord and \ + self._moduleName == rhs._moduleName and self._assemblyName == rhs._assemblyName + + # Get moduleRecord + def getModuleRecord(self): + return self._moduleRecord + + # Get moduleName + def getModuleName(self): + return self._moduleName + + # Get assemblyName + def getAssemblyName(self): + return self._assemblyName + + # Verify consistency + def verify(self): + if (self._moduleRecord is None or self._moduleName is None or self._assemblyName is None): + raise InternalError() + + if (not issubclass(type(self._moduleRecord), ModuleRecord) + or not issubclass(type(self._moduleName), str) + or not issubclass(type(self._assemblyName), str)): + raise InternalError() + + if (len(self._moduleName) != self._moduleRecord.getLenModuleName() + or len(self._assemblyName) != self._moduleRecord.getLenAssemblyName()): + raise InternalError() + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + "moduleRecord = {") + self._moduleRecord.printRaw(offsetStr + " ") + print(offsetStr + "}") + print(offsetStr + "moduleName = " + self._moduleName) + print(offsetStr + "assemblyName = " + self._assemblyName) + + # Print pretty + def print(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> Module: " + self._moduleName + "; Assembly: " + self._assemblyName) + print(offsetStr + "") + self._moduleRecord.print(offsetStr) + print(offsetStr + "") + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None or not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) < ModuleRecord.Size): + raise ProfileInconsistencyError() + + index = 0 + + moduleRecord = ModuleRecord.createFromBytes(bytesArr[index:index+ModuleRecord.Size]) + index += ModuleRecord.Size + + lenModuleName = moduleRecord.getLenModuleName() + lenModuleNameAligned = Utility.alignUp(lenModuleName, RuntimeTypeSizes.int) + + lenAssemblyName = moduleRecord.getLenAssemblyName() + lenAssemblyNameAligned = Utility.alignUp(lenAssemblyName, RuntimeTypeSizes.int) + + if (len(bytesArr) != (ModuleRecord.Size + lenModuleNameAligned + lenAssemblyNameAligned)): + raise ProfileInconsistencyError() + + moduleName = "" + for i in bytesArr[index:index+lenModuleName]: + moduleName += chr(i) + index += lenModuleNameAligned + + assemblyName = "" + for i in bytesArr[index:index+lenAssemblyName]: + assemblyName += chr(i) + index += lenAssemblyNameAligned + + if (index != (ModuleRecord.Size + lenModuleNameAligned + lenAssemblyNameAligned)): + raise InternalError() + + return ModuleRecordExtended.createByRef(moduleRecord, moduleName, assemblyName) + + # Create from objects taking them by reference + @staticmethod + def createByRef(moduleRecord, moduleName, assemblyName): + moduleRecordExtended = ModuleRecordExtended() + + moduleRecordExtended._moduleRecord = moduleRecord # pylint: disable=protected-access + moduleRecordExtended._moduleName = moduleName # pylint: disable=protected-access + moduleRecordExtended._assemblyName = assemblyName # pylint: disable=protected-access + + moduleRecordExtended.verify() + return moduleRecordExtended + + # Create from objects taking them by copy + @staticmethod + def createByCopy(moduleRecord, moduleName, assemblyName): + return ModuleRecordExtended.createByRef(moduleRecord.copy(), moduleName, assemblyName) + + # Copy object + def copy(self): + return ModuleRecordExtended.createByCopy(self._moduleRecord, self._moduleName, self._assemblyName) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + bytesArr += self._moduleRecord.convertToBytes() + index += ModuleRecord.Size + + lenModuleName = self._moduleRecord.getLenModuleName() + lenModuleNameAligned = Utility.alignUp(lenModuleName, RuntimeTypeSizes.int) + bytesArr += list(bytearray(self._moduleName, "ascii")) + index += lenModuleName + + padding = lenModuleNameAligned - lenModuleName + bytesArr += [0] * padding + index += padding + + lenAssemblyName = self._moduleRecord.getLenAssemblyName() + lenAssemblyNameAligned = Utility.alignUp(lenAssemblyName, RuntimeTypeSizes.int) + bytesArr += list(bytearray(self._assemblyName, "ascii")) + index += lenAssemblyName + + padding = lenAssemblyNameAligned - lenAssemblyName + bytesArr += [0] * padding + index += padding + + if (index != (ModuleRecord.Size + lenModuleNameAligned + lenAssemblyNameAligned)): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# Detailed info on types +# +# Semantically corresponds to CorTypeInfo +# +class CorTypeInfo: + + # Kinds of types + # + # Semantically corresponds to CorElementType + # TODO: verify these constants somehow, see above + class CorElementType: # pylint: disable=too-few-public-methods + + ELEMENT_TYPE_END = 0x00 + ELEMENT_TYPE_VOID = 0x01 + ELEMENT_TYPE_BOOLEAN = 0x02 + ELEMENT_TYPE_CHAR = 0x03 + ELEMENT_TYPE_I1 = 0x04 + ELEMENT_TYPE_U1 = 0x05 + ELEMENT_TYPE_I2 = 0x06 + ELEMENT_TYPE_U2 = 0x07 + ELEMENT_TYPE_I4 = 0x08 + ELEMENT_TYPE_U4 = 0x09 + ELEMENT_TYPE_I8 = 0x0a + ELEMENT_TYPE_U8 = 0x0b + ELEMENT_TYPE_R4 = 0x0c + ELEMENT_TYPE_R8 = 0x0d + ELEMENT_TYPE_STRING = 0x0e + + ELEMENT_TYPE_PTR = 0x0f + ELEMENT_TYPE_BYREF = 0x10 + + ELEMENT_TYPE_VALUETYPE = 0x11 + ELEMENT_TYPE_CLASS = 0x12 + ELEMENT_TYPE_VAR = 0x13 + ELEMENT_TYPE_ARRAY = 0x14 + ELEMENT_TYPE_GENERICINST = 0x15 + ELEMENT_TYPE_TYPEDBYREF = 0x16 + ELEMENT_TYPE_VALUEARRAY_UNSUPPORTED = 0x17 + ELEMENT_TYPE_I = 0x18 + ELEMENT_TYPE_U = 0x19 + ELEMENT_TYPE_R_UNSUPPORTED = 0x1a + ELEMENT_TYPE_FNPTR = 0x1b + ELEMENT_TYPE_OBJECT = 0x1c + ELEMENT_TYPE_SZARRAY = 0x1d + ELEMENT_TYPE_MVAR = 0x1e + + ELEMENT_TYPE_CMOD_REQD = 0x1f + ELEMENT_TYPE_CMOD_OPT = 0x20 + + ELEMENT_TYPE_INTERNAL = 0x21 + + ELEMENT_TYPE_MAX = 0x22 + + ELEMENT_TYPE_VAR_ZAPSIG = 0x3b + ELEMENT_TYPE_NATIVE_VALUETYPE_ZAPSIG = 0x3d + ELEMENT_TYPE_CANON_ZAPSIG = 0x3e + ELEMENT_TYPE_MODULE_ZAPSIG = 0x3f + + ELEMENT_TYPE_MODIFIER = 0x40 + ELEMENT_TYPE_SENTINEL = 0x01 | ELEMENT_TYPE_MODIFIER + ELEMENT_TYPE_PINNED = 0x05 | ELEMENT_TYPE_MODIFIER + + # This constant is not defined in runtime + X_ELEMENT_TYPE_LAST = ELEMENT_TYPE_PINNED + 1 + + # Entry in map of types info + class Entry: + + # Default constructor + def __init__(self, elementKind, name, isPrim): + self._elementKind = elementKind + self._name = name + self._isPrim = isPrim + + # Get kind of record + def getKind(self): + return self._elementKind + + # Get name of record + def getName(self): + return self._name + + # Get flag whether type is primitive + def getIsPrim(self): + return self._isPrim + + # Fill map with types info + @staticmethod + def fillMap(): + tmpmap = [None] * CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_END] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_END, "ELEMENT_TYPE_END", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_VOID] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_VOID, "void", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_BOOLEAN] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_BOOLEAN, "bool", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_CHAR] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_CHAR, "char", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_I1] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_I1, "i1", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_U1] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_U1, "u1", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_I2] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_I2, "i2", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_U2] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_U2, "u2", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_I4] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_I4, "i4", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_U4] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_U4, "u4", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_I8] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_I8, "i8", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_U8] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_U8, "u8", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_R4] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_R4, "r4", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_R8] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_R8, "r8", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_STRING] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_STRING, "string", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_PTR] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_PTR, "ptr", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_BYREF] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_BYREF, "byref", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_VALUETYPE] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_VALUETYPE, "valuetype", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_CLASS] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_CLASS, "class", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR, "var", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_ARRAY] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_ARRAY, "array", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_GENERICINST] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_GENERICINST, "generic", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_TYPEDBYREF] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_TYPEDBYREF, "typedbyref", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_VALUEARRAY_UNSUPPORTED] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_VALUEARRAY_UNSUPPORTED, + "valuearray_unsupported", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_I] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_I, "i", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_U] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_U, "u", True) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_R_UNSUPPORTED] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_R_UNSUPPORTED, "r_unsupported", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_FNPTR] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_FNPTR, "fnptr", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_OBJECT] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_OBJECT, "object", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_SZARRAY] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_SZARRAY, "szarray", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_MVAR] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_MVAR, "mvar", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_CMOD_REQD] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_CMOD_REQD, "cmod_reqd", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_CMOD_OPT] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_CMOD_OPT, "cmod_opt", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_INTERNAL] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_INTERNAL, "internal", False) + #tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_MAX] = \ + # CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_MAX, "ELEMENT_TYPE_MAX", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR_ZAPSIG] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR_ZAPSIG, "vap_zapsig", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_NATIVE_VALUETYPE_ZAPSIG] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_NATIVE_VALUETYPE_ZAPSIG, + "native_valuetype_zapsig", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_CANON_ZAPSIG] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_CANON_ZAPSIG, "canon_zapsig", False) + tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_MODULE_ZAPSIG] = \ + CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_MODULE_ZAPSIG, "module_zapsig", False) + #tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_MODIFIER] = \ + # CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_MODIFIER, "ELEMENT_TYPE_MODIFIER", False) + #tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_SENTINEL] = \ + # CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_SENTINEL, "ELEMENT_TYPE_SENTINEL", False) + #tmpmap[CorTypeInfo.CorElementType.ELEMENT_TYPE_PINNED] = \ + # CorTypeInfo.Entry(CorTypeInfo.CorElementType.ELEMENT_TYPE_PINNED, "ELEMENT_TYPE_PINNED", False) + + # verify generated map + for index, record in enumerate(tmpmap): + if (not record is None and record.getKind() != index): + raise InternalError() + + return tmpmap + + # Map with types info, is filled explicitly below. + # Records that are absent from CorTypeInfo.CorElementType are None. + # Use getTypeMapEntry to access typemap. + typemap = None + + # Get info for type + @staticmethod + def getTypeMapEntry(elementKind): + if (elementKind is None or not issubclass(type(elementKind), int)): + raise InternalError() + record = CorTypeInfo.typemap[elementKind] # pylint: disable=unsubscriptable-object + if (record is None): + raise InternalError() + if (record.getKind() != elementKind): + raise InternalError() + return record + +# Fill map of types info +CorTypeInfo.typemap = CorTypeInfo.fillMap() + +# ---------------------------------------------------------------------------------------------------------------------- +# These are different kinds of types corresponding to CorTypeInfo.CorElementType + +# Base class for all types +class IType(ABC): + + # Get element kind + @abstractmethod + def getElementKind(self): + pass + + # Get string representation of type + @abstractmethod + def getStr(self, modules=None): + pass + + # Update all module indexes according to map old_index->new_index + @abstractmethod + def updateModuleIndex(self, moduleIndexMap): + pass + + # Get all module indexes used in related types + @abstractmethod + def getAllModuleIndexes(self): + pass + + # Copy type + @abstractmethod + def copy(self): + pass + + +# Corresponding to simple types that are fully described by their CorElementType +# +# To create from data structures using references: TypeSimple.createByRef(...) +# To create from data structures using copy: TypeSimple.createByCopy(...) +# +class TypeSimple(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeSimple)): + return False + return self._elementKind == rhs._elementKind + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + return CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return [] + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind): + handle = TypeSimple() + + handle._elementKind = elementKind # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind): + return TypeSimple.createByRef(elementKind) + + # Copy object + def copy(self): + return TypeSimple.createByCopy(self._elementKind) + + +# Corresponding to types that are fully described by their CorElementType, module index and type token +# +# To create from data structures using references: TypeToken.createByRef(...) +# To create from data structures using copy: TypeToken.createByCopy(...) +# +class TypeToken(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + self._moduleIndex = None + self._typeToken = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeToken)): + return False + return self._elementKind == rhs._elementKind and self._moduleIndex == rhs._moduleIndex \ + and self._typeToken == rhs._typeToken + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + moduleStr = str(self._moduleIndex) + if (not modules is None): + moduleStr = modules[self._moduleIndex].getModuleName() + + return CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + "{" + str(self._typeToken) \ + + ":" + moduleStr + "}" + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get moduleIndex + def getModuleIndex(self): + return self._moduleIndex + + # Get typeToken + def getTypeToken(self): + return self._typeToken + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return [self._moduleIndex] + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._moduleIndex = moduleIndexMap[self._moduleIndex] + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None or self._moduleIndex is None or self._typeToken is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int) or not issubclass(type(self._moduleIndex), int) + or not issubclass(type(self._typeToken), int)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + if (self._moduleIndex < 0 or self._moduleIndex >= RuntimeConstants.MAX_MODULES): + raise InternalError() + + if (self._typeToken < 0 or self._typeToken > Utility.getMaxForNBytes(RuntimeTypeSizes.int)): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind, moduleIndex, typeToken): + handle = TypeToken() + + handle._elementKind = elementKind # pylint: disable=protected-access + handle._moduleIndex = moduleIndex # pylint: disable=protected-access + handle._typeToken = typeToken # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind, moduleIndex, typeToken): + return TypeToken.createByRef(elementKind, moduleIndex, typeToken) + + # Copy object + def copy(self): + return TypeToken.createByCopy(self._elementKind, self._moduleIndex, self._typeToken) + + +# Corresponding to simple types with param type +# +# To create from data structures using references: TypeParamType.createByRef(...) +# To create from data structures using copy: TypeParamType.createByCopy(...) +# +class TypeParamType(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + self._paramType = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeParamType)): + return False + return self._elementKind == rhs._elementKind and self._paramType == rhs._paramType + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + return CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + "{" + self._paramType.getStr(modules) + "}" + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get paramType + def getParamType(self): + return self._paramType + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return self._paramType.getAllModuleIndexes() + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._paramType.updateModuleIndex(moduleIndexMap) + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None or self._paramType is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int) or not issubclass(type(self._paramType), IType)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind, paramType): + handle = TypeParamType() + + handle._elementKind = elementKind # pylint: disable=protected-access + handle._paramType = paramType # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind, paramType): + return TypeParamType.createByRef(elementKind, paramType.copy()) + + # Copy object + def copy(self): + return TypeParamType.createByCopy(self._elementKind, self._paramType) + + +# Corresponding to array type +# +# To create from data structures using references: TypeArray.createByRef(...) +# To create from data structures using copy: TypeArray.createByCopy(...) +# +class TypeArray(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + self._arrayElementType = None + self._rank = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeArray)): + return False + return self._elementKind == rhs._elementKind and self._arrayElementType == rhs._arrayElementType \ + and self._rank == rhs._rank + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + return CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + "{[" \ + + self._arrayElementType.getStr(modules) + " ]" + str(self._rank) + "}" + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get arrayElementType + def getArrayElementType(self): + return self._arrayElementType + + # Get rank + def getRank(self): + return self._rank + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return self._arrayElementType.getAllModuleIndexes() + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._arrayElementType.updateModuleIndex(moduleIndexMap) + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None or self._arrayElementType is None or self._rank is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int) or not issubclass(type(self._arrayElementType), IType) + or not issubclass(type(self._rank), int)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + if (self._rank < 0 or self._rank > Utility.getMaxForNBytes(RuntimeTypeSizes.char)): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind, arrayElementType, rank): + handle = TypeArray() + + handle._elementKind = elementKind # pylint: disable=protected-access + handle._arrayElementType = arrayElementType # pylint: disable=protected-access + handle._rank = rank # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind, arrayElementType, rank): + return TypeArray.createByRef(elementKind, arrayElementType.copy(), rank) + + # Copy object + def copy(self): + return TypeArray.createByCopy(self._elementKind, self._arrayElementType, self._rank) + + +# Corresponding to szarray type +# +# To create from data structures using references: TypeSZArray.createByRef(...) +# To create from data structures using copy: TypeSZArray.createByCopy(...) +# +class TypeSZArray(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + self._arrayElementType = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeSZArray)): + return False + return self._elementKind == rhs._elementKind and self._arrayElementType == rhs._arrayElementType + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + return CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + "{[" \ + + self._arrayElementType.getStr(modules) + "]}" + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get arrayElementType + def getArrayElementType(self): + return self._arrayElementType + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return self._arrayElementType.getAllModuleIndexes() + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._arrayElementType.updateModuleIndex(moduleIndexMap) + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None or self._arrayElementType is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int) or not issubclass(type(self._arrayElementType), IType)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind, arrayElementType): + handle = TypeSZArray() + + handle._elementKind = elementKind # pylint: disable=protected-access + handle._arrayElementType = arrayElementType # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind, arrayElementType): + return TypeSZArray.createByRef(elementKind, arrayElementType.copy()) + + # Copy object + def copy(self): + return TypeSZArray.createByCopy(self._elementKind, self._arrayElementType) + + +# Corresponding to generic type +# +# To create from data structures using references: TypeGeneric.createByRef(...) +# To create from data structures using copy: TypeGeneric.createByCopy(...) +# +class TypeGeneric(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + self._genericBaseType = None + self._instArgs = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeGeneric)): + return False + return self._elementKind == rhs._elementKind and self._genericBaseType == rhs._genericBaseType \ + and self._instArgs == rhs._instArgs + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + output = CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + "{" \ + + self._genericBaseType.getStr(modules) + "<" + + for index in range(len(self._instArgs)): + output += self._instArgs[index].getStr(modules) + if (index != (len(self._instArgs) - 1)): + output += "," + + output += ">}" + return output + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get genericBaseType + def getGenericBaseType(self): + return self._genericBaseType + + # Get args + def getInstArgs(self): + return self._instArgs + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + indexes = self._genericBaseType.getAllModuleIndexes() + for i in self._instArgs: + indexes += i.getAllModuleIndexes() + return sorted(set(indexes)) + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._genericBaseType.updateModuleIndex(moduleIndexMap) + for i in self._instArgs: + i.updateModuleIndex(moduleIndexMap) + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None or self._genericBaseType is None or self._instArgs is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int) or not issubclass(type(self._genericBaseType), TypeToken) + or not issubclass(type(self._instArgs), list)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + for i in self._instArgs: + if (not issubclass(type(i), IType)): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind, genericBaseType, instArgs): + handle = TypeGeneric() + + handle._elementKind = elementKind # pylint: disable=protected-access + handle._genericBaseType = genericBaseType # pylint: disable=protected-access + handle._instArgs = instArgs # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind, genericBaseType, instArgs): + copiedArgs = [] + for i in instArgs: + copiedArgs.append(i.copy()) + + return TypeGeneric.createByRef(elementKind, genericBaseType.copy(), copiedArgs) + + # Copy object + def copy(self): + return TypeGeneric.createByCopy(self._elementKind, self._genericBaseType, self._instArgs) + + +# Corresponding to fnptr type +# +# To create from data structures using references: TypeFNPtr.createByRef(...) +# To create from data structures using copy: TypeFNPtr.createByCopy(...) +# +class TypeFNPtr(IType): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._elementKind = None + self._callConv = None + self._retAndArgs = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), TypeFNPtr)): + return False + return self._elementKind == rhs._elementKind and self._callConv == rhs._callConv \ + and self._retAndArgs == rhs._retAndArgs + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + output = CorTypeInfo.getTypeMapEntry(self._elementKind).getName() + "{" + str(self._callConv) + "(" + + for index in range(len(self._retAndArgs)): + output += self._retAndArgs[index].getStr(modules) + if (index != (len(self._retAndArgs) - 1)): + output += "," + + output += ")}" + return output + + # Get elementKind + def getElementKind(self): + return self._elementKind + + # Get callConv + def getCallConv(self): + return self._callConv + + # Get retAndArgs + def getRetAndArgs(self): + return self._retAndArgs + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + indexes = [] + for i in self._retAndArgs: + indexes += i.getAllModuleIndexes() + return sorted(set(indexes)) + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + for i in self._retAndArgs: + i.updateModuleIndex(moduleIndexMap) + self.verify() + + # Verify consistency + def verify(self): + if (self._elementKind is None or self._callConv is None or self._retAndArgs is None): + raise InternalError() + + if (not issubclass(type(self._elementKind), int) or not issubclass(type(self._callConv), int) + or not issubclass(type(self._retAndArgs), list)): + raise InternalError() + + if (self._elementKind < 0 or self._elementKind >= CorTypeInfo.CorElementType.X_ELEMENT_TYPE_LAST): + raise InternalError() + + if (self._callConv < 0 or self._callConv > Utility.getMaxForNBytes(RuntimeTypeSizes.char)): + raise InternalError() + + for i in self._retAndArgs: + if (not issubclass(type(i), IType)): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(elementKind, callConv, retAndArgs): + handle = TypeFNPtr() + + handle._elementKind = elementKind # pylint: disable=protected-access + handle._callConv = callConv # pylint: disable=protected-access + handle._retAndArgs = retAndArgs # pylint: disable=protected-access + + handle.verify() + return handle + + # Create from objects taking them by copy + @staticmethod + def createByCopy(elementKind, callConv, retAndArgs): + copiedRetAndArgs = [] + for i in retAndArgs: + copiedRetAndArgs.append(i.copy()) + + return TypeFNPtr.createByRef(elementKind, callConv, copiedRetAndArgs) + + # Copy object + def copy(self): + return TypeFNPtr.createByCopy(self._elementKind, self._callConv, self._retAndArgs) + + +# Corresponding to generic method +# +# To create from data structures using references: GenericMethod.createByRef(...) +# To create from data structures using copy: GenericMethod.createByCopy(...) +# +class GenericMethod: + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._methodFlags = None + self._moduleIndex = None + self._typeHandle = None + self._methodToken = None + self._methodInstArgs = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), GenericMethod)): + return False + return self._methodFlags == rhs._methodFlags and self._moduleIndex == rhs._moduleIndex \ + and self._typeHandle == rhs._typeHandle and self._methodToken == rhs._methodToken \ + and self._methodInstArgs == rhs._methodInstArgs + + # Convert to string + def getStr(self, modules=None): + if (not modules is None): + if (not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + moduleStr = str(self._moduleIndex) + if (not modules is None): + moduleStr = modules[self._moduleIndex].getModuleName() + + output = "method{token{" + str(self._methodToken) + ":" + moduleStr + "}" + + if (not self._methodInstArgs is None): + output += "<" + for index in range(len(self._methodInstArgs)): + output += self._methodInstArgs[index].getStr(modules) + if (index != (len(self._methodInstArgs) - 1)): + output += "," + output += ">" + + output += "@type{" + self._typeHandle.getStr(modules) + "}}" + return output + + # Get methodFlags + def getMethodFlags(self): + return self._methodFlags + + # Get moduleIndex + def getModuleIndex(self): + return self._moduleIndex + + # Get typeHandle + def getTypeHandle(self): + return self._typeHandle + + # Get methodToken + def getMethodToken(self): + return self._methodToken + + # Get methodInstArgs + def getMethodInstArgs(self): + return self._methodInstArgs + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + indexes = [self._moduleIndex] + indexes += self._typeHandle.getAllModuleIndexes() + if (not self._methodInstArgs is None): + for i in self._methodInstArgs: + indexes += i.getAllModuleIndexes() + return sorted(set(indexes)) + + # Update all module indexes according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + + self._moduleIndex = moduleIndexMap[self._moduleIndex] + self._typeHandle.updateModuleIndex(moduleIndexMap) + + if (not self._methodInstArgs is None): + for i in self._methodInstArgs: + i.updateModuleIndex(moduleIndexMap) + + self.verify() + + # Verify consistency + def verify(self): + if (self._methodFlags is None or self._moduleIndex is None or self._typeHandle is None + or self._methodToken is None): + raise InternalError() + + if (not issubclass(type(self._methodFlags), int) or not issubclass(type(self._moduleIndex), int) + or not issubclass(type(self._typeHandle), IType) or not issubclass(type(self._methodToken), int)): + raise InternalError() + + if (self._methodFlags < 0 or self._methodFlags > Utility.getMaxForNBytes(RuntimeTypeSizes.int)): + raise InternalError() + + if (self._moduleIndex < 0 or self._moduleIndex >= RuntimeConstants.MAX_MODULES): + raise InternalError() + + if (self._methodToken < 0 or self._methodToken > Utility.getMaxForNBytes(RuntimeTypeSizes.int)): + raise InternalError() + + if (not self._methodInstArgs is None): + if (not issubclass(type(self._methodInstArgs), list)): + raise InternalError() + + for i in self._methodInstArgs: + if (not issubclass(type(i), IType)): + raise InternalError() + + # Create from objects taking them by reference + @staticmethod + def createByRef(methodFlags, moduleIndex, typeHandle, methodToken, methodInstArgs): + method = GenericMethod() + + method._methodFlags = methodFlags # pylint: disable=protected-access + method._moduleIndex = moduleIndex # pylint: disable=protected-access + method._typeHandle = typeHandle # pylint: disable=protected-access + method._methodToken = methodToken # pylint: disable=protected-access + method._methodInstArgs = methodInstArgs # pylint: disable=protected-access + + method.verify() + return method + + # Create from objects taking them by copy + @staticmethod + def createByCopy(methodFlags, moduleIndex, typeHandle, methodToken, methodInstArgs): + copiedInstArgs = None + if (not methodInstArgs is None): + copiedInstArgs = [] + for i in methodInstArgs: + copiedInstArgs.append(i.copy()) + + return GenericMethod.createByRef(methodFlags, moduleIndex, typeHandle.copy(), methodToken, copiedInstArgs) + + # Copy object + def copy(self): + return GenericMethod.createByCopy(self._methodFlags, self._moduleIndex, self._typeHandle, + self._methodToken, self._methodInstArgs) + +#---------------------------------------------------------------------------------------------------------------------- +# Utilities for binary signature encoding/decoding +class SignatureDecoderUtil: + + # Corresponds to EncodeMethodSigFlags + # TODO: verify these constants somehow, see above + class EncodeMethodSigFlags: # pylint: disable=too-few-public-methods + ENCODE_METHOD_SIG_UnboxingStub = 0x01 + ENCODE_METHOD_SIG_InstantiatingStub = 0x02 + ENCODE_METHOD_SIG_MethodInstantiation = 0x04 + ENCODE_METHOD_SIG_SlotInsteadOfToken = 0x08 + ENCODE_METHOD_SIG_MemberRefToken = 0x10 + ENCODE_METHOD_SIG_Constrained = 0x20 + ENCODE_METHOD_SIG_OwnerType = 0x40 + ENCODE_METHOD_SIG_UpdateContext = 0x80 + + # Corresponds to CorTokenType + # TODO: verify these constants somehow, see above + class CorTokenType: # pylint: disable=too-few-public-methods + mdtModule = 0x00000000 + mdtTypeRef = 0x01000000 + mdtTypeDef = 0x02000000 + mdtFieldDef = 0x04000000 + mdtMethodDef = 0x06000000 + mdtParamDef = 0x08000000 + mdtInterfaceImpl = 0x09000000 + mdtMemberRef = 0x0a000000 + mdtCustomAttribute = 0x0c000000 + mdtPermission = 0x0e000000 + mdtSignature = 0x11000000 + mdtEvent = 0x14000000 + mdtProperty = 0x17000000 + mdtMethodImpl = 0x19000000 + mdtModuleRef = 0x1a000000 + mdtTypeSpec = 0x1b000000 + mdtAssembly = 0x20000000 + mdtAssemblyRef = 0x23000000 + mdtFile = 0x26000000 + mdtExportedType = 0x27000000 + mdtManifestResource = 0x28000000 + mdtNestedClass = 0x29000000 + mdtGenericParam = 0x2a000000 + mdtMethodSpec = 0x2b000000 + mdtGenericParamConstraint = 0x2c000000 + mdtString = 0x70000000 + mdtName = 0x71000000 + mdtBaseType = 0x72000000 + + # Get token from rid and type + @staticmethod + def getTokenFromRid(rid, typ): + return rid | typ + + # Get rid from token + @staticmethod + def getRidFromToken(token): + return token & 0x00ffffff + + # Get type from token + @staticmethod + def getTypeFromToken(token): + return token & 0xff000000 + +#---------------------------------------------------------------------------------------------------------------------- +# Decoder for binary signature of method that is created by MCJ +# +# Conceptually corresponds to ZapSig::DecodeMethod +# +class MethodBinarySignatureDecoder: # pylint: disable=too-few-public-methods + + # Constructor with module index for binary signature and binary signature itself + def __init__(self, moduleIndex, bytesArr): + if (moduleIndex is None or bytesArr is None): + raise InternalError() + if (not issubclass(type(moduleIndex), int) or not issubclass(type(bytesArr), list)): + raise InternalError() + + for i in bytesArr: + if (i < 0 or i > Utility.getMaxForNBytes(RuntimeTypeSizes.char)): + raise InternalError() + + self._moduleIndex = moduleIndex + self._bytesArr = bytesArr + self._index = 0 + + # Decode one byte from binary signature + # + # See SigBuilder::AppendByte + def _decodeByte(self): + value = self._bytesArr[self._index] + self._index += 1 + return value + + # Decode value (1, 2 or 4 bytes) from binary signature + # + # See SigBuilder::AppendData + def _decodeValue(self): + value = 0 + + if ((self._bytesArr[self._index] & 0x80) == 0): + # 1 byte case + length = 1 + if (self._index + length > len(self._bytesArr)): + raise ProfileInconsistencyError() + value = self._bytesArr[self._index] + if (value > 0x7f): + raise InternalError() + elif ((self._bytesArr[self._index] & 0xc0) == 0x80): + # 2 byte case + length = 2 + if (self._index + length > len(self._bytesArr)): + raise ProfileInconsistencyError() + value = ((self._bytesArr[self._index] & 0x3f) << 8) | self._bytesArr[self._index + 1] + if (value > 0x3fff): + raise InternalError() + elif ((self._bytesArr[self._index] & 0xe0) == 0xc0): + # 4 byte case + length = 4 + if (self._index + length > len(self._bytesArr)): + raise ProfileInconsistencyError() + value = ((self._bytesArr[self._index] & 0x1f) << 24) | (self._bytesArr[self._index + 1] << 16) + value = value | (self._bytesArr[self._index + 2] << 8) | self._bytesArr[self._index + 3] + if (value > 0x1fffffff): + raise InternalError() + else: + raise ProfileInconsistencyError() + + self._index += length + + return value + + # Decode token from binary signature + # + # See SigBuilder::AppendToken + def _decodeToken(self): + value = self._decodeValue() + + # get encode type and rid + encodedType = value & 0x3 + rid = value >> 2 + typ = None + + if (encodedType == 0x0): + typ = SignatureDecoderUtil.CorTokenType.mdtTypeDef + elif (encodedType == 0x1): + typ = SignatureDecoderUtil.CorTokenType.mdtTypeRef + elif (encodedType == 0x2): + typ = SignatureDecoderUtil.CorTokenType.mdtTypeSpec + elif (encodedType == 0x3): + typ = SignatureDecoderUtil.CorTokenType.mdtBaseType + else: + raise ProfileInconsistencyError() + + return SignatureDecoderUtil.getTokenFromRid(rid, typ) + + # Decode type from binary signature + # + # See ZapSig::GetSignatureForTypeHandle + def _decodeSignatureForTypeHandle(self, moduleIndex): # pylint: disable=too-many-locals, too-many-return-statements, too-many-statements + if (moduleIndex is None or not issubclass(type(moduleIndex), int)): + raise InternalError() + + # I. Decode type + typ = self._decodeByte() + + # II. Decode primitive type + if (typ < CorTypeInfo.CorElementType.ELEMENT_TYPE_MAX + and (CorTypeInfo.getTypeMapEntry(typ).getIsPrim() + or typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_STRING + or typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_OBJECT)): + return TypeSimple.createByRef(typ) + + # III. Decode non-primitive type + if (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_TYPEDBYREF): # pylint: disable=no-else-raise + # TODO: support this? + raise UnsupportedError() + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_NATIVE_VALUETYPE_ZAPSIG): + paramType = self._decodeSignatureForTypeHandle(moduleIndex) + return TypeParamType.createByRef(typ, paramType) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_CANON_ZAPSIG): + return TypeSimple.createByRef(typ) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_MODULE_ZAPSIG): + nextModuleIndex = self._decodeValue() + return self._decodeSignatureForTypeHandle(nextModuleIndex) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR_ZAPSIG): + rid = self._decodeValue() + typeToken = SignatureDecoderUtil.getTokenFromRid(rid, SignatureDecoderUtil.CorTokenType.mdtGenericParam) + return TypeToken.createByRef(typ, moduleIndex, typeToken) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR): # pylint: disable=no-else-raise + # TODO: support this? + raise UnsupportedError() + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_MVAR): # pylint: disable=no-else-raise + # TODO: support this? + raise UnsupportedError() + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_GENERICINST): + nextTyp = self._decodeByte() + + if (nextTyp == CorTypeInfo.CorElementType.ELEMENT_TYPE_INTERNAL): # pylint: disable=no-else-raise + # TODO: support this? + raise UnsupportedError() + else: + typeToken = self._decodeToken() + tid = SignatureDecoderUtil.getTypeFromToken(typeToken) + + if (not tid in (SignatureDecoderUtil.CorTokenType.mdtTypeRef, + SignatureDecoderUtil.CorTokenType.mdtTypeDef)): + raise ProfileInconsistencyError() + + numArgs = self._decodeValue() + args = [] + for i in range(numArgs): # pylint: disable=unused-variable + args.append(self._decodeSignatureForTypeHandle(moduleIndex)) + + return TypeGeneric.createByRef(typ, TypeToken.createByRef(nextTyp, moduleIndex, typeToken), args) + elif (typ in (CorTypeInfo.CorElementType.ELEMENT_TYPE_CLASS, + CorTypeInfo.CorElementType.ELEMENT_TYPE_VALUETYPE)): + typeToken = self._decodeToken() + tid = SignatureDecoderUtil.getTypeFromToken(typeToken) + + if (not tid in (SignatureDecoderUtil.CorTokenType.mdtTypeRef, + SignatureDecoderUtil.CorTokenType.mdtTypeDef)): + raise ProfileInconsistencyError() + + return TypeToken.createByRef(typ, moduleIndex, typeToken) + elif (typ in (CorTypeInfo.CorElementType.ELEMENT_TYPE_ARRAY, CorTypeInfo.CorElementType.ELEMENT_TYPE_SZARRAY)): + elemType = self._decodeSignatureForTypeHandle(moduleIndex) + + if (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_ARRAY): # pylint: disable=no-else-return + rank = self._decodeValue() + # Next two values are always written as 0 (see ZapSig::GetSignatureForTypeHandle) + nsizes = self._decodeValue() + nlbounds = self._decodeValue() + + if (nsizes != 0 or nlbounds != 0): + raise UnsupportedError() + + return TypeArray.createByRef(typ, elemType, rank) + else: + return TypeSZArray.createByRef(typ, elemType) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_PINNED): + nextType = self._decodeSignatureForTypeHandle(moduleIndex) + return TypeParamType.createByRef(typ, nextType) + elif (typ in (CorTypeInfo.CorElementType.ELEMENT_TYPE_BYREF, CorTypeInfo.CorElementType.ELEMENT_TYPE_PTR)): + paramType = self._decodeSignatureForTypeHandle(moduleIndex) + return TypeParamType.createByRef(typ, paramType) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_FNPTR): + callConv = self._decodeByte() + cArgs = self._decodeValue() + retAndArgs = [] + for i in range(cArgs): + retAndArgs.append(self._decodeSignatureForTypeHandle(moduleIndex)) + return TypeFNPtr.createByRef(typ, callConv, retAndArgs) + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_INTERNAL): # pylint: disable=no-else-raise + # TODO: support this? + raise UnsupportedError() + elif (typ == CorTypeInfo.CorElementType.ELEMENT_TYPE_SENTINEL): # pylint: disable=no-else-raise + # TODO: support this? + raise UnsupportedError() + else: + raise InternalError() + + # Decode binary signature for method, result can be used by reference + # + # See ZapSig::DecodeMethod + def decodeMethod(self): + typeHandle = None + methodToken = None + numInstArgs = None + methodInstArgs = None + + # I. Decode method flags + methodFlags = self._decodeValue() + + # II. Decode type + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_UpdateContext != 0): + # Should not be encoded in mcj profile + raise UnsupportedError() + + if ((methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_OwnerType) != 0): + typeHandle = self._decodeSignatureForTypeHandle(self._moduleIndex) + else: + raise UnsupportedError() + + # III. Decode method token + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_SlotInsteadOfToken != 0): # pylint: disable=no-else-raise + raise UnsupportedError() + else: + rid = self._decodeValue() + + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_MemberRefToken): + methodToken = SignatureDecoderUtil.getTokenFromRid(rid, SignatureDecoderUtil.CorTokenType.mdtMemberRef) + else: + methodToken = SignatureDecoderUtil.getTokenFromRid(rid, SignatureDecoderUtil.CorTokenType.mdtMethodDef) + + # IV. Decode method instantiation args + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_MethodInstantiation != 0): + numInstArgs = self._decodeValue() + + methodInstArgs = [] + for i in range(numInstArgs): # pylint: disable=unused-variable + methodInstArgs.append(self._decodeSignatureForTypeHandle(self._moduleIndex)) + + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_Constrained != 0): + raise UnsupportedError() + + ret = GenericMethod.createByRef(methodFlags, self._moduleIndex, typeHandle, methodToken, methodInstArgs) + + self._moduleIndex = None + self._bytesArr = None + self._index = None + + return ret + +#---------------------------------------------------------------------------------------------------------------------- +# Encoder for binary signature of method that is stored in MCJ profile +# +# Conceptually corresponds to ZapSig::EncodeMethod +# +class MethodBinarySignatureEncoder: # pylint: disable=too-few-public-methods + + # Constructor with generic method handle + def __init__(self, methodHandle): + if (methodHandle is None or not issubclass(type(methodHandle), GenericMethod)): + raise InternalError() + + self._methodHandle = methodHandle + self._bytesArr = [] + + # Encode one byte to binary signature + # + # See SigBuilder::AppendByte + def _encodeByte(self, value): + if (value is None or not issubclass(type(value), int)): + raise InternalError() + if (value < 0 or value > Utility.getMaxForNBytes(RuntimeTypeSizes.char)): + raise InternalError() + + self._bytesArr.append(value) + + # Encode value (1, 2 or 4 bytes) to binary signature + # + # See SigBuilder::AppendData + def _encodeValue(self, value): + if (value is None or not issubclass(type(value), int)): + raise InternalError() + if (value < 0 or value > Utility.getMaxForNBytes(RuntimeTypeSizes.int)): + raise InternalError() + + if (value <= 0x7f): + self._bytesArr.append(value) + elif (value <= 0x3fff): + self._bytesArr.append((value >> 8) | 0x80) + self._bytesArr.append(value & 0xff) + elif (value <= 0x1fffffff): + self._bytesArr.append((value >> 24) | 0xc0) + self._bytesArr.append((value >> 16) & 0xff) + self._bytesArr.append((value >> 8) & 0xff) + self._bytesArr.append(value & 0xff) + else: + raise InternalError() + + # Encode token to binary signature + # + # See SigBuilder::AppendToken + def _encodeToken(self, value): + if (value is None or not issubclass(type(value), int)): + raise InternalError() + if (value < 0 or value > Utility.getMaxForNBytes(RuntimeTypeSizes.int)): + raise InternalError() + + rid = SignatureDecoderUtil.getRidFromToken(value) + typ = SignatureDecoderUtil.getTypeFromToken(value) + + if (rid > 0x3ffffff): + raise InternalError() + + rid = rid << 2 + + if (typ == SignatureDecoderUtil.CorTokenType.mdtTypeDef): + pass + elif (typ == SignatureDecoderUtil.CorTokenType.mdtTypeRef): + rid |= 0x1 + elif (typ == SignatureDecoderUtil.CorTokenType.mdtTypeSpec): + rid |= 0x2 + elif (typ == SignatureDecoderUtil.CorTokenType.mdtBaseType): + rid |= 0x3 + + self._encodeValue(rid) + + # Encode type to binary signature + # + # See ZapSig::GetSignatureForTypeHandle + def _encodeSignatureForTypeHandle(self, typeHandle): + if (typeHandle is None or not issubclass(type(typeHandle), IType)): + raise InternalError() + + # I. Encode element type + if (issubclass(type(typeHandle), TypeSimple)): + self._encodeByte(typeHandle.getElementKind()) + elif (issubclass(type(typeHandle), TypeToken)): + if (typeHandle.getModuleIndex() != self._methodHandle.getModuleIndex()): + self._encodeByte(CorTypeInfo.CorElementType.ELEMENT_TYPE_MODULE_ZAPSIG) + self._encodeValue(typeHandle.getModuleIndex()) + + self._encodeByte(typeHandle.getElementKind()) + + if (typeHandle.getElementKind() == CorTypeInfo.CorElementType.ELEMENT_TYPE_VAR_ZAPSIG): + self._encodeValue(SignatureDecoderUtil.getRidFromToken(typeHandle.getTypeToken())) + else: + self._encodeToken(typeHandle.getTypeToken()) + elif (issubclass(type(typeHandle), TypeParamType)): + self._encodeByte(typeHandle.getElementKind()) + self._encodeSignatureForTypeHandle(typeHandle.getParamType()) + elif (issubclass(type(typeHandle), TypeArray)): + self._encodeByte(typeHandle.getElementKind()) + self._encodeSignatureForTypeHandle(typeHandle.getArrayElementType()) + self._encodeValue(typeHandle.getRank()) + self._encodeValue(0) + self._encodeValue(0) + elif (issubclass(type(typeHandle), TypeSZArray)): + self._encodeByte(typeHandle.getElementKind()) + self._encodeSignatureForTypeHandle(typeHandle.getArrayElementType()) + elif (issubclass(type(typeHandle), TypeGeneric)): + genericBaseType = typeHandle.getGenericBaseType() + if (genericBaseType.getModuleIndex() != self._methodHandle.getModuleIndex()): + self._encodeByte(CorTypeInfo.CorElementType.ELEMENT_TYPE_MODULE_ZAPSIG) + self._encodeValue(genericBaseType.getModuleIndex()) + + self._encodeByte(typeHandle.getElementKind()) + self._encodeByte(genericBaseType.getElementKind()) + self._encodeToken(genericBaseType.getTypeToken()) + + self._encodeValue(len(typeHandle.getInstArgs())) + for i in typeHandle.getInstArgs(): + self._encodeSignatureForTypeHandle(i) + elif (issubclass(type(typeHandle), TypeFNPtr)): + self._encodeByte(typeHandle.getElementKind()) + self._encodeByte(typeHandle.getCallConv()) + + self._encodeValue(len(typeHandle.getRetAndArgs())) + for i in typeHandle.getRetAndArgs(): + self._encodeSignatureForTypeHandle(i) + + # Encode binary signature for method, result can be used by reference + # + # See ZapSig::EncodeMethod + def encodeMethod(self): + methodFlags = self._methodHandle.getMethodFlags() + + # I. Encode method flags + self._encodeValue(methodFlags) + + # II. Encode type + if ((methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_OwnerType) != 0): + self._encodeSignatureForTypeHandle(self._methodHandle.getTypeHandle()) + else: + raise UnsupportedError() + + # III. Encode method token + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_SlotInsteadOfToken != 0): # pylint: disable=no-else-raise + raise UnsupportedError() + else: + self._encodeValue(SignatureDecoderUtil.getRidFromToken(self._methodHandle.getMethodToken())) + + # IV. Encode method instantiation args + if (methodFlags & SignatureDecoderUtil.EncodeMethodSigFlags.ENCODE_METHOD_SIG_MethodInstantiation != 0): + methodInstArgs = self._methodHandle.getMethodInstArgs() + + if (methodInstArgs is None): + raise InternalError() + + self._encodeValue(len(methodInstArgs)) + for i in methodInstArgs: + self._encodeSignatureForTypeHandle(i) + + ret = self._bytesArr + + self._bytesArr = None + self._methodHandle = None + + return ret + +# ---------------------------------------------------------------------------------------------------------------------- +# Base class for records with module dependency or method +# +class Info(ABC): + + # Get record type + @abstractmethod + def getRecordType(self): + pass + + # Get module index + @abstractmethod + def getModuleIndex(self): + pass + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + pass + + # Check if info is related to generic method + @abstractmethod + def isGenericMethodInfo(self): + pass + + # Check if info is related to non-generic method + @abstractmethod + def isNonGenericMethodInfo(self): + pass + + # Check if info is related to method + @abstractmethod + def isMethodInfo(self): + pass + + # Check if info is related to module + @abstractmethod + def isModuleInfo(self): + pass + +# ---------------------------------------------------------------------------------------------------------------------- +# Record with module dependency +# +# Conceptually corresponds to "struct RecorderInfo" from multicorejitimpl.h, but has slight differences +# +# To create from bytes: InfoModuleDependency.createFromBytes(bytes) +# To create from data structures using references: InfoModuleDependency.createByRef(...) +# To create from data structures using copy: InfoModuleDependency.createByCopy(...) +# +class InfoModuleDependency(Info): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._moduleIndex = None + self._moduleLoadLevel = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None or not issubclass(type(rhs), InfoModuleDependency)): + return False + return self._moduleIndex == rhs._moduleIndex and self._moduleLoadLevel == rhs._moduleLoadLevel + + # Get record type + def getRecordType(self): # pylint: disable=no-self-use + return RuntimeConstants.MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID + + # Check if info is related to generic method + def isGenericMethodInfo(self): # pylint: disable=no-self-use + return False + + # Check if info is related to non-generic method + def isNonGenericMethodInfo(self): # pylint: disable=no-self-use + return False + + # Check if info is related to method + def isMethodInfo(self): # pylint: disable=no-self-use + return False + + # Check if info is related to module + def isModuleInfo(self): # pylint: disable=no-self-use + return True + + # Get module index + def getModuleIndex(self): + return self._moduleIndex + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return [self._moduleIndex] + + # Get module load level + def getModuleLoadLevel(self): + return self._moduleLoadLevel + + # Update module index according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._moduleIndex = moduleIndexMap[self._moduleIndex] + self.verify() + + # Verify consistency + def verify(self): + if (self._moduleIndex is None or self._moduleLoadLevel is None): + raise InternalError() + + if (not issubclass(type(self._moduleIndex), int) or not issubclass(type(self._moduleLoadLevel), int)): + raise InternalError() + + if (self._moduleIndex < 0 or self._moduleIndex >= RuntimeConstants.MAX_MODULES): + raise InternalError() + + if (self._moduleLoadLevel < 0 or self._moduleLoadLevel >= RuntimeConstants.MAX_MODULE_LEVELS): + raise InternalError() + + # Encode info + def _encodeInfo(self): + data1 = 0 + # high byte is record id + data1 |= RuntimeConstants.MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID << RuntimeConstants.RECORD_TYPE_OFFSET + # next byte is load level + data1 |= self._moduleLoadLevel << RuntimeConstants.MODULE_LEVEL_OFFSET + # two low bytes are module index + data1 |= self._moduleIndex + return data1 + + # Decode type of this record, module index and module load level + @staticmethod + def _decodeInfo(data1): + recordType = data1 >> RuntimeConstants.RECORD_TYPE_OFFSET + loadLevel = (data1 >> RuntimeConstants.MODULE_LEVEL_OFFSET) & (RuntimeConstants.MAX_MODULE_LEVELS - 1) + moduleIndex = data1 & RuntimeConstants.MODULE_MASK + return (recordType, loadLevel, moduleIndex) + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + data1 = self._encodeInfo() + + print(offsetStr + ">>> Raw ModuleDependency:") + print(offsetStr + "data1 = " + str(data1)) + + # Print pretty + def print(self, offsetStr="", modules=None): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + # Get module name if it is available + moduleName = "" + if (not modules is None): + moduleName = modules[self.getModuleIndex()].getModuleName() + + print(offsetStr + ">>> Record, Module Dependency:") + print(offsetStr + "") + print(offsetStr + "Module index: " + str(self.getModuleIndex()) + + ((" (" + moduleName + ")") if moduleName != "" else "")) + print(offsetStr + "Module load level: " + str(self.getModuleLoadLevel())) + print(offsetStr + "") + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None): + raise InternalError() + + if (not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) != RuntimeTypeSizes.int): + raise ProfileInconsistencyError() + + index = 0 + + data1 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + if (index != RuntimeTypeSizes.int): + raise InternalError() + + recordType, moduleLoadLevel, moduleIndex = InfoModuleDependency._decodeInfo(data1) + + if (recordType != RuntimeConstants.MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID): + raise InternalError() + + return InfoModuleDependency.createByRef(moduleIndex, moduleLoadLevel) + + # Create from objects taking them by reference + @staticmethod + def createByRef(moduleIndex, moduleLoadLevel): + moduleDependency = InfoModuleDependency() + + moduleDependency._moduleIndex = moduleIndex # pylint: disable=protected-access + moduleDependency._moduleLoadLevel = moduleLoadLevel # pylint: disable=protected-access + + moduleDependency.verify() + return moduleDependency + + # Create from objects taking them by copy + @staticmethod + def createByCopy(moduleIndex, moduleLoadLevel): + return InfoModuleDependency.createByRef(moduleIndex, moduleLoadLevel) + + # Copy object + def copy(self): + return InfoModuleDependency.createByCopy(self._moduleIndex, self._moduleLoadLevel) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + data1 = self._encodeInfo() + + bytesArr += Utility.splitNBytes(data1, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + if (index != RuntimeTypeSizes.int): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# Record with non-generic method +# +# Conceptually corresponds to "struct RecorderInfo" from multicorejitimpl.h, but has slight differences +# +# To create from bytes: InfoNonGenericMethod.createFromBytes(bytes) +# To create from data structures using references: InfoNonGenericMethod.createByRef(...) +# To create from data structures using copy: InfoNonGenericMethod.createByCopy(...) +# +class InfoNonGenericMethod(Info): + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._moduleIndex = None + self._methodFlags = None + self._methodToken = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None or not issubclass(type(rhs), InfoNonGenericMethod)): + return False + return self._moduleIndex == rhs._moduleIndex and self._methodFlags == rhs._methodFlags \ + and self._methodToken == rhs._methodToken + + # Get record type + def getRecordType(self): # pylint: disable=no-self-use + return RuntimeConstants.MULTICOREJIT_METHOD_RECORD_ID + + # Check if info is related to generic method + def isGenericMethodInfo(self): # pylint: disable=no-self-use + return False + + # Check if info is related to non-generic method + def isNonGenericMethodInfo(self): # pylint: disable=no-self-use + return True + + # Check if info is related to method + def isMethodInfo(self): # pylint: disable=no-self-use + return True + + # Check if info is related to module + def isModuleInfo(self): # pylint: disable=no-self-use + return False + + # Get module index + def getModuleIndex(self): + return self._moduleIndex + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return [self._moduleIndex] + + # Get method flags + def getMethodFlags(self): + return self._methodFlags + + # Get method token + def getMethodToken(self): + return self._methodToken + + # Update module index according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._moduleIndex = moduleIndexMap[self._moduleIndex] + self.verify() + + # Verify consistency + def verify(self): + if (self._moduleIndex is None or self._methodFlags is None or self._methodToken is None): + raise InternalError() + + if (not issubclass(type(self._moduleIndex), int) or not issubclass(type(self._methodFlags), int) + or not issubclass(type(self._methodToken), int)): + raise InternalError() + + if (self._moduleIndex < 0 or self._moduleIndex >= RuntimeConstants.MAX_MODULES): + raise InternalError() + + if (self._methodFlags < 0 + or ((self._methodFlags | RuntimeConstants.METHOD_FLAGS_MASK) ^ RuntimeConstants.METHOD_FLAGS_MASK) != 0): + raise InternalError() + + if (self._methodToken < 0 or self._methodToken > Utility.getMaxForNBytes(RuntimeTypeSizes.int)): + raise InternalError() + + # Encode info + def _encodeInfo(self): + data1 = 0 + # high byte is record id + data1 |= RuntimeConstants.MULTICOREJIT_METHOD_RECORD_ID << RuntimeConstants.RECORD_TYPE_OFFSET + # next byte is method flags + data1 |= self._methodFlags + # two low bytes are module index + data1 |= self._moduleIndex + + # this is simply token + data2 = self._methodToken + + return (data1, data2) + + # Decode type of this record, module index and module load level + @staticmethod + def _decodeInfo(data1, data2): + recordType = data1 >> RuntimeConstants.RECORD_TYPE_OFFSET + methodFlags = data1 & RuntimeConstants.METHOD_FLAGS_MASK + moduleIndex = data1 & RuntimeConstants.MODULE_MASK + methodToken = data2 + return (recordType, methodFlags, moduleIndex, methodToken) + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + data1, data2 = self._encodeInfo() + + print(offsetStr + ">>> Raw Non-Generic MethodRecord:") + print(offsetStr + "data1 = " + str(data1)) + print(offsetStr + "data2 = " + str(data2)) + + # Print pretty + def print(self, offsetStr="", modules=None): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + # Get module name if it is available + moduleName = "" + if (not modules is None): + moduleName = modules[self.getModuleIndex()].getModuleName() + + flagsStr = "jitted by background thread" + if (self._methodFlags & RuntimeConstants.JIT_BY_APP_THREAD_TAG != 0): + flagsStr = "jitted by foreground (app) thread" + + print(offsetStr + ">>> Record, Non-generic Method:") + print(offsetStr + "") + print(offsetStr + "Module index: " + str(self.getModuleIndex()) + + ((" (" + moduleName + ")") if moduleName != "" else "")) + print(offsetStr + "Method flags: " + str(self.getMethodFlags()) + " (" + flagsStr + ")") + print(offsetStr + "Method token: " + str(self.getMethodToken())) + print(offsetStr + "") + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None): + raise InternalError() + + if (not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) != 2 * RuntimeTypeSizes.int): + raise ProfileInconsistencyError() + + index = 0 + + data1 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + data2 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + if (index != 2 * RuntimeTypeSizes.int): + raise InternalError() + + recordType, methodFlags, moduleIndex, methodToken = InfoNonGenericMethod._decodeInfo(data1, data2) + + if (recordType != RuntimeConstants.MULTICOREJIT_METHOD_RECORD_ID): + raise InternalError() + + return InfoNonGenericMethod.createByRef(moduleIndex, methodFlags, methodToken) + + # Create from objects taking them by reference + @staticmethod + def createByRef(moduleIndex, methodFlags, methodToken): + method = InfoNonGenericMethod() + + method._moduleIndex = moduleIndex # pylint: disable=protected-access + method._methodFlags = methodFlags # pylint: disable=protected-access + method._methodToken = methodToken # pylint: disable=protected-access + + method.verify() + return method + + # Create from objects taking them by copy + @staticmethod + def createByCopy(moduleIndex, methodFlags, methodToken): + return InfoNonGenericMethod.createByRef(moduleIndex, methodFlags, methodToken) + + # Copy object + def copy(self): + return InfoNonGenericMethod.createByCopy(self._moduleIndex, self._methodFlags, self._methodToken) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + data1, data2 = self._encodeInfo() + + bytesArr += Utility.splitNBytes(data1, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(data2, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + if (index != 2 * RuntimeTypeSizes.int): + raise InternalError() + + return bytesArr +# ---------------------------------------------------------------------------------------------------------------------- +# Record with generic method +# +# Conceptually corresponds to "struct RecorderInfo" from multicorejitimpl.h, but has slight differences +# +# To create from bytes: InfoGenericMethod.createFromBytes(bytes) +# To create from data structures using references: InfoGenericMethod.createByRef(...) +# To create from data structures using copy: InfoGenericMethod.createByCopy(...) +# +class InfoGenericMethod(Info): # pylint: disable=too-many-public-methods + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._methodFlags = None + self._methodHandle = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None or not issubclass(type(rhs), InfoGenericMethod)): + return False + return self._methodFlags == rhs._methodFlags and self._methodHandle == rhs._methodHandle + + # Get record type + def getRecordType(self): # pylint: disable=no-self-use + return RuntimeConstants.MULTICOREJIT_GENERICMETHOD_RECORD_ID + + # Check if info is related to generic method + def isGenericMethodInfo(self): # pylint: disable=no-self-use + return True + + # Check if info is related to non-generic method + def isNonGenericMethodInfo(self): # pylint: disable=no-self-use + return False + + # Check if info is related to method + def isMethodInfo(self): # pylint: disable=no-self-use + return True + + # Check if info is related to module + def isModuleInfo(self): # pylint: disable=no-self-use + return False + + # Get module index + def getModuleIndex(self): + return self._methodHandle.getModuleIndex() + + # Get all module indexes used in related types + def getAllModuleIndexes(self): + return self._methodHandle.getAllModuleIndexes() + + # Get method flags + def getMethodFlags(self): + return self._methodFlags + + # Get method handle + def getMethodHandle(self): + return self._methodHandle + + # Get method token + def getMethodToken(self): + return self._methodHandle.getMethodToken() + + # Update module index according to map old_index->new_index + def updateModuleIndex(self, moduleIndexMap): + if (moduleIndexMap is None or not issubclass(type(moduleIndexMap), list)): + raise InternalError() + self._methodHandle.updateModuleIndex(moduleIndexMap) + self.verify() + + # Verify consistency + def verify(self): + if (self._methodFlags is None or self._methodHandle is None): + raise InternalError() + + if (not issubclass(type(self._methodFlags), int) or not issubclass(type(self._methodHandle), GenericMethod)): + raise InternalError() + + if (self._methodFlags < 0 + or ((self._methodFlags | RuntimeConstants.METHOD_FLAGS_MASK) ^ RuntimeConstants.METHOD_FLAGS_MASK) != 0): + raise InternalError() + + # Encode info + def _encodeInfo(self): + binarySignature = MethodBinarySignatureEncoder(self._methodHandle).encodeMethod() + + data1 = 0 + # high byte is record id + data1 |= RuntimeConstants.MULTICOREJIT_GENERICMETHOD_RECORD_ID << RuntimeConstants.RECORD_TYPE_OFFSET + # next byte is method flags + data1 |= self._methodFlags + # two low bytes are module index + data1 |= self._methodHandle.getModuleIndex() + + # this is simply length of binary signature + data2 = len(binarySignature) + + # this is simply binary signature + ptr = binarySignature + + return (data1, data2, ptr) + + # Decode type of this record, module index and module load level + @staticmethod + def _decodeInfo(data1, data2, ptr): # pylint: disable=unused-argument + recordType = data1 >> RuntimeConstants.RECORD_TYPE_OFFSET + methodFlags = data1 & RuntimeConstants.METHOD_FLAGS_MASK + moduleIndex = data1 & RuntimeConstants.MODULE_MASK + + method = MethodBinarySignatureDecoder(moduleIndex, ptr).decodeMethod() + + return (recordType, methodFlags, method) + + # Print raw + def printRaw(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + data1, data2, ptr = self._encodeInfo() + + size = RuntimeTypeSizes.int + RuntimeTypeSizes.short + data2 + padding = Utility.alignUp(size, RuntimeTypeSizes.int) - size + + print(offsetStr + ">>> Raw Generic MethodRecord:") + print(offsetStr + "data1 = " + str(data1)) + print(offsetStr + "data2 = " + str(data2)) + print(offsetStr + "ptr = " + str(ptr)) + print(offsetStr + "alignment padding = " + str(padding) + " bytes") + + # Print pretty + def print(self, offsetStr="", modules=None): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + # Get module name if it is available + moduleName = "" + if (not modules is None): + if (modules is None or not issubclass(type(modules), list)): + raise InternalError() + for i in modules: + if (i is None or not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + moduleName = modules[self.getModuleIndex()].getModuleName() + + flagsStr = "jitted by background thread" + if (self._methodFlags & RuntimeConstants.JIT_BY_APP_THREAD_TAG != 0): + flagsStr = "jitted by foreground (app) thread" + + print(offsetStr + ">>> Record, Generic Method:") + print(offsetStr + "") + print(offsetStr + "Module index: " + str(self.getModuleIndex()) + + ((" (" + moduleName + ")") if moduleName != "" else "")) + print(offsetStr + "Method flags: " + str(self.getMethodFlags()) + " (" + flagsStr + ")") + print(offsetStr + "Decoded binary signature: " + self.getMethodHandle().getStr(modules)) + print(offsetStr + "") + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None): + raise InternalError() + + if (not issubclass(type(bytesArr), list)): + raise InternalError() + + if (len(bytesArr) < (RuntimeTypeSizes.int + RuntimeTypeSizes.short)): + raise ProfileInconsistencyError() + + index = 0 + + data1 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + index += RuntimeTypeSizes.int + + data2 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.short]) + index += RuntimeTypeSizes.short + + if (len(bytesArr) != Utility.alignUp(RuntimeTypeSizes.int + RuntimeTypeSizes.short + data2, + RuntimeTypeSizes.int)): + raise ProfileInconsistencyError() + + ptr = bytesArr[index:index+data2] + index += data2 + + if (index != RuntimeTypeSizes.int + RuntimeTypeSizes.short + data2): + raise InternalError() + + recordType, methodFlags, methodHandle = InfoGenericMethod._decodeInfo(data1, data2, ptr) + + if (recordType != RuntimeConstants.MULTICOREJIT_GENERICMETHOD_RECORD_ID): + raise InternalError() + + return InfoGenericMethod.createByRef(methodFlags, methodHandle) + + # Create from objects taking them by reference + @staticmethod + def createByRef(methodFlags, methodHandle): + method = InfoGenericMethod() + + method._methodFlags = methodFlags # pylint: disable=protected-access + method._methodHandle = methodHandle # pylint: disable=protected-access + + method.verify() + return method + + # Create from objects taking them by copy + @staticmethod + def createByCopy(methodFlags, methodHandle): + return InfoGenericMethod.createByRef(methodFlags, methodHandle.copy()) + + # Copy object + def copy(self): + return InfoGenericMethod.createByCopy(self._methodFlags, self._methodHandle) + + # Convert object to list of bytes + def convertToBytes(self): + index = 0 + + bytesArr = [] + + data1, data2, ptr = self._encodeInfo() + + bytesArr += Utility.splitNBytes(data1, RuntimeTypeSizes.int) + index += RuntimeTypeSizes.int + + bytesArr += Utility.splitNBytes(data2, RuntimeTypeSizes.short) + index += RuntimeTypeSizes.short + + bytesArr += ptr + index += len(ptr) + + padding = Utility.alignUp(index, RuntimeTypeSizes.int) - index + bytesArr += [0] * padding + index += padding + + if (index != Utility.alignUp(RuntimeTypeSizes.int + RuntimeTypeSizes.short + data2, + RuntimeTypeSizes.int)): + raise InternalError() + + return bytesArr + +# ---------------------------------------------------------------------------------------------------------------------- +# MultiCoreJit profile +# +# _moduleOrMethodInfo contains records with module deps and methods, both generic and non-generic, +# records' order is the same as in profile +# +# To create from bytes: MCJProfile.createFromBytes(bytes) +# To create from data structures using references: MCJProfile.createByRef(...) +# To create from data structures using copy: MCJProfile.createByCopy(...) +# +class MCJProfile: # pylint: disable=too-many-public-methods + + # Empty constructor, do not use it directly + # Create new objects by createByCopy or createByRef + def __init__(self): + self._header = None + self._modules = None + self._moduleOrMethodInfo = None + + # Equality comparison operator + def __eq__(self, rhs): + if (rhs is None): + return False + if (not issubclass(type(rhs), MCJProfile)): + return False + + if (self._header != rhs._header + or len(self._modules) != len(rhs._modules) + or len(self._moduleOrMethodInfo) != len(rhs._moduleOrMethodInfo)): + return False + + for index, module in enumerate(self._modules): + if (module != rhs._modules[index]): + return False + + for index, info in enumerate(self._moduleOrMethodInfo): + if (info != rhs._moduleOrMethodInfo[index]): + return False + + return True + + # Get header + def getHeader(self): + return self._header + + # Get modules + def getModules(self): + return self._modules + + # Get moduleOrMethodInfo + def getModuleOrMethodInfo(self): + return self._moduleOrMethodInfo + + # Verify consistency + def verify(self): + if (self._header is None or self._modules is None or self._moduleOrMethodInfo is None): + raise InternalError() + + if (not issubclass(type(self._header), HeaderRecord) + or not issubclass(type(self._modules), list) or not issubclass(type(self._moduleOrMethodInfo), list)): + raise InternalError() + + for i in self._modules: + if (i is None): + raise InternalError() + if (not issubclass(type(i), ModuleRecordExtended)): + raise InternalError() + + for i in self._moduleOrMethodInfo: + if (i is None): + raise InternalError() + if (not issubclass(type(i), InfoModuleDependency) + and not issubclass(type(i), InfoNonGenericMethod) + and not issubclass(type(i), InfoGenericMethod)): + raise InternalError() + + if (self._header.getModuleCount() != len(self._modules)): + raise ProfileInconsistencyError() + + if ((self._header.getMethodCount() + self._header.getModuleDepCount()) != len(self._moduleOrMethodInfo)): + raise ProfileInconsistencyError() + + for info in self._moduleOrMethodInfo: + indexes = info.getAllModuleIndexes() + for i in indexes: + if (i >= len(self._modules)): + raise InternalError() + + # Print raw header + def printRawHeader(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + ">>> Raw MCJProfile:") + print(offsetStr + "header = {") + self._header.printRaw(offsetStr + " ") + print(offsetStr + "}") + + # Print raw modules + def printRawModules(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + print(offsetStr + "modules = [") + for i in self._modules: + print(offsetStr + " " + "{") + i.printRaw(offsetStr + " ") + print(offsetStr + " " + "},") + print(offsetStr + "]") + + # Print raw module dependencies and methods + def printRawModuleOrMethodInfo(self, offsetStr="", printModuleDeps=True, + printGenericMethods=True, printNonGenericMethods=True): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + if (printModuleDeps is None or printGenericMethods is None or printNonGenericMethods is None): + raise InternalError() + + if (not issubclass(type(printModuleDeps), bool) or not issubclass(type(printGenericMethods), bool) + or not issubclass(type(printNonGenericMethods), bool)): + raise InternalError() + + name = "" + if (printModuleDeps and printGenericMethods and printNonGenericMethods): + name = "moduleOrMethodInfo" + else: + if (printModuleDeps): + if (name != ""): + name += ", " + name += "module deps" + if (printGenericMethods): + if (name != ""): + name += ", " + name += "generic methods" + if (printNonGenericMethods): + if (name != ""): + name += ", " + name += "non generic methods" + + print(offsetStr + name + " = [") + for i in self._moduleOrMethodInfo: + doPrint = (i.isModuleInfo() and printModuleDeps) or (i.isGenericMethodInfo() and printGenericMethods) \ + or (i.isNonGenericMethodInfo() and printNonGenericMethods) + + if (doPrint): + print(offsetStr + " " + "{") + i.printRaw(offsetStr + " ") + print(offsetStr + " " + "},") + print(offsetStr + "]") + + # Print raw + def printRaw(self, offsetStr=""): + self.printRawHeader(offsetStr) + self.printRawModules(offsetStr) + self.printRawModuleOrMethodInfo(offsetStr) + + # Print pretty header + def printHeader(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + self._header.print(offsetStr) + + # Print pretty modules + def printModules(self, offsetStr=""): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + for i in self._modules: + i.print(offsetStr) + + # Print pretty module dependencies and methods + def printModuleOrMethodInfo(self, offsetStr="", printModuleDeps=True, + printGenericMethods=True, printNonGenericMethods=True): + if (offsetStr is None or not issubclass(type(offsetStr), str)): + raise InternalError() + + if (printModuleDeps is None or printGenericMethods is None or printNonGenericMethods is None): + raise InternalError() + + if (not issubclass(type(printModuleDeps), bool) or not issubclass(type(printGenericMethods), bool) + or not issubclass(type(printNonGenericMethods), bool)): + raise InternalError() + + for i in self._moduleOrMethodInfo: + doPrint = (i.isModuleInfo() and printModuleDeps) or (i.isGenericMethodInfo() and printGenericMethods) \ + or (i.isNonGenericMethodInfo() and printNonGenericMethods) + + if (doPrint): + i.print(offsetStr, self._modules) + + # Print pretty + def print(self, offsetStr=""): + self.printHeader(offsetStr) + self.printModules(offsetStr) + self.printModuleOrMethodInfo(offsetStr) + + # Get length and type of record at specified byte + @staticmethod + def _getRecordTypeAndLen(bytesArr, index): + if (bytesArr is None or index is None): + raise InternalError() + + if (not issubclass(type(bytesArr), list) or not issubclass(type(index), int)): + raise InternalError() + + if (index + RuntimeTypeSizes.int > len(bytesArr)): + raise ProfileInconsistencyError() + + data1 = Utility.mergeNBytes(bytesArr[index:index+RuntimeTypeSizes.int]) + rcdTyp = data1 >> RuntimeConstants.RECORD_TYPE_OFFSET + rcdLen = 0 + + if (rcdTyp == RuntimeConstants.MULTICOREJIT_MODULE_RECORD_ID): + rcdLen = data1 & RuntimeConstants.X_MODULE_RECORD_LEN_MASK + elif (rcdTyp == RuntimeConstants.MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID): + rcdLen = RuntimeTypeSizes.int + elif (rcdTyp == RuntimeConstants.MULTICOREJIT_METHOD_RECORD_ID): + rcdLen = 2 * RuntimeTypeSizes.int + elif (rcdTyp == RuntimeConstants.MULTICOREJIT_GENERICMETHOD_RECORD_ID): + if (index + RuntimeTypeSizes.int + RuntimeTypeSizes.short > len(bytesArr)): + raise ProfileInconsistencyError() + + tmpindex = index+RuntimeTypeSizes.int + signatureLength = Utility.mergeNBytes(bytesArr[tmpindex:tmpindex+RuntimeTypeSizes.short]) + dataSize = signatureLength + RuntimeTypeSizes.int + RuntimeTypeSizes.short + dataSize = Utility.alignUp(dataSize, RuntimeTypeSizes.int) + rcdLen = dataSize + else: + raise ProfileInconsistencyError() + + if ((index + rcdLen > len(bytesArr)) or ((rcdLen & 3) != 0)): + raise ProfileInconsistencyError() + + return rcdTyp, rcdLen + + # Create from bytes + @staticmethod + def createFromBytes(bytesArr): + if (bytesArr is None or not issubclass(type(bytesArr), list)): + raise InternalError() + + index = 0 + header = HeaderRecord.createFromBytes(bytesArr[index:index+HeaderRecord.Size]) + index += HeaderRecord.Size + + modules = [] + moduleOrMethodInfo = [] + + while (index <= len(bytesArr) - RuntimeTypeSizes.int): + rcdTyp, rcdLen = MCJProfile._getRecordTypeAndLen(bytesArr, index) + + if (rcdTyp == RuntimeConstants.MULTICOREJIT_MODULE_RECORD_ID): + modules.append(ModuleRecordExtended.createFromBytes(bytesArr[index:index+rcdLen])) + elif (rcdTyp == RuntimeConstants.MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID): + moduleOrMethodInfo.append(InfoModuleDependency.createFromBytes(bytesArr[index:index+rcdLen])) + elif (rcdTyp == RuntimeConstants.MULTICOREJIT_METHOD_RECORD_ID): + moduleOrMethodInfo.append(InfoNonGenericMethod.createFromBytes(bytesArr[index:index+rcdLen])) + elif (rcdTyp == RuntimeConstants.MULTICOREJIT_GENERICMETHOD_RECORD_ID): + moduleOrMethodInfo.append(InfoGenericMethod.createFromBytes(bytesArr[index:index+rcdLen])) + else: + raise InternalError() + + index += rcdLen + + if (index != len(bytesArr)): + raise ProfileInconsistencyError() + + return MCJProfile.createByRef(header, modules, moduleOrMethodInfo) + + # Create from objects taking them by reference + @staticmethod + def createByRef(header, modules, moduleOrMethodInfo): + profile = MCJProfile() + + profile._header = header # pylint: disable=protected-access + profile._modules = modules # pylint: disable=protected-access + profile._moduleOrMethodInfo = moduleOrMethodInfo # pylint: disable=protected-access + + profile.verify() + return profile + + # Create from objects taking them by copy + @staticmethod + def createByCopy(header, modules, moduleOrMethodInfo): + newModules = [] + for i in modules: + newModules.append(i.copy()) + + newModuleOrMethodInfo = [] + for i in moduleOrMethodInfo: + newModuleOrMethodInfo.append(i.copy()) + + return MCJProfile.createByRef(header.copy(), newModules, newModuleOrMethodInfo) + + # Copy object + def copy(self): + return MCJProfile.createByCopy(self._header, self._modules, self._moduleOrMethodInfo) + + # Convert object to list of bytes + def convertToBytes(self): + bytesArr = [] + + bytesArr += self._header.convertToBytes() + + for module in self._modules: + bytesArr += module.convertToBytes() + + for info in self._moduleOrMethodInfo: + bytesArr += info.convertToBytes() + + return bytesArr + + # Split mcj profile in two: app-dependent (app) and app-independent (system) + def split(self, systemModules): # pylint: disable=too-many-locals,too-many-statements,too-many-branches + if (systemModules is None or not issubclass(type(systemModules), list)): + raise InternalError() + + for i in systemModules: + if (i is None): + raise InternalError() + if (not issubclass(type(i), str)): + raise InternalError() + + cIdxApp = 0 + cIdxSys = 1 + + # App header at [0], sys header at [1] + header = [self._header.copy(), self._header.copy()] + # App modules at [0], sys modules at [1] + modules = [[], []] + # App moduleOrMethodInfo at [0], sys moduleOrMethodInfo at [1] + moduleOrMethodInfo = [[], []] + # App methodCount at [0], sys methodCount at [1] + methodCount = [0, 0] + # App moduleDepCount at [0], sys moduleDepCount at [1] + moduleDepCount = [0, 0] + + # Map from old module index to flag, whether module should be added to: + # app profile at [0], system profile at [1] + moduleMap = [[False] * len(self._modules), [False] * len(self._modules)] + # Map from old method info index to flag, whether method info should be added to: + # app profile (True) or to sys profile (False) + methodInfoAppMap = [False] * len(self._moduleOrMethodInfo) + # Map from old module index to new module index in: + # app profile at [0], system profile at [1] + moduleIndexMap = [[None] * len(self._modules), [None] * len(self._modules)] + + # I. ==== Create map of system modules ==== + + # Map from system module name to flag True + systemModulesMap = {} + for sysmodule in systemModules: + systemModulesMap[sysmodule] = True + + # II. ==== Go through all modules and mark system modules ==== + + for index, module in enumerate(self._modules): + moduleMap[cIdxSys][index] = module.getModuleName() in systemModulesMap + + # III. ==== Go through all method infos and create lists of module indexes for system and app profiles + + for index, value in enumerate(moduleMap[cIdxSys]): + moduleMap[cIdxApp][index] = not value + + for index, info in enumerate(self._moduleOrMethodInfo): + if (info.isMethodInfo()): + indexes = info.getAllModuleIndexes() + + for i in indexes: + if (not moduleMap[cIdxSys][i]): + methodInfoAppMap[index] = True + break + + if (methodInfoAppMap[index]): + # mark all modules as requiremens for app profile + for i in indexes: + moduleMap[cIdxApp][i] = True + + # IV. === Go through all modules again and add to profiles accordingly ==== + + for index, module in enumerate(self._modules): + + isAdded = False + + # add to app and system profiles + for i in range(2): + if (moduleMap[i][index]): + newModule = module.copy() + moduleIndexMap[i][index] = len(modules[i]) + modules[i].append(newModule) + isAdded = True + + if (not isAdded): + raise InternalError() + + # V. ==== Go through all infos again and add to profiles accordingly ==== + + for index, info in enumerate(self._moduleOrMethodInfo): + isAdded = False + + # add to app and system profiles + for i in range(2): + doAddModule = info.isModuleInfo() and moduleMap[i][info.getModuleIndex()] + doAddMethodApp = (i == cIdxApp) and methodInfoAppMap[index] + doAddMethodSys = (i == cIdxSys) and not methodInfoAppMap[index] + + doAdd = doAddModule or (info.isMethodInfo() and (doAddMethodApp or doAddMethodSys)) + + if (doAdd): + if (info.isMethodInfo() and isAdded): + raise InternalError() + + newInfo = info.copy() + newInfo.updateModuleIndex(moduleIndexMap[i]) + moduleOrMethodInfo[i].append(newInfo) + + if (info.isModuleInfo()): + moduleDepCount[i] += 1 + elif (info.isMethodInfo()): + methodCount[i] += 1 + else: + raise InternalError() + + isAdded = True + + if (not isAdded): + raise InternalError() + + # VI. ==== Recalculate jitMethodCount ==== + + for index in range(2): + for module in modules[index]: + module.getModuleRecord().setJitMethodCount(0) + + for info in moduleOrMethodInfo[index]: + if (info.isMethodInfo()): + indexes = info.getAllModuleIndexes() + for i in indexes: + moduleRecord = modules[index][i].getModuleRecord() + count = moduleRecord.getJitMethodCount() + moduleRecord.setJitMethodCount(count + 1) + + # VII. ==== Initialize new headers ==== + + for index in range(2): + header[index].setModuleCount(len(modules[index])) + header[index].setMethodCount(methodCount[index]) + header[index].setModuleDepCount(moduleDepCount[index]) + header[index].dropGlobalUsageStats() + + # VIII. ==== Perform some consistency checks ==== + + if (methodCount[0] + methodCount[1] != self._header.getMethodCount()): + raise InternalError() + + # IX. ==== Create new profiles ==== + + mcjProfileApp = MCJProfile.createByRef(header[0], modules[0], moduleOrMethodInfo[0]) + mcjProfileSys = MCJProfile.createByRef(header[1], modules[1], moduleOrMethodInfo[1]) + + return mcjProfileApp, mcjProfileSys + + # Merge new mcj profile with existing one + def merge(self, mcjProfile): # pylint: disable=too-many-locals,too-many-statements + if (mcjProfile is None or not issubclass(type(mcjProfile), MCJProfile)): + raise InternalError() + + # Map from module name in self to module index + moduleNameMap = {} + # Map from str method representation to info index + methodMap = [] + # Map from old module index in mcjProfile to previous max module level in self (if existed, otherwise None) + moduleLoadLevelMap = [None] * len(mcjProfile.getModules()) + # Map from old module index in mcjProfile to new module index in self profile + moduleIndexMap = [None] * len(mcjProfile.getModules()) + + # I. ==== Create map of existing module names ==== + + for index, module in enumerate(self._modules): + moduleNameMap[module.getModuleName()] = index + + # II. ==== Merge modules ==== + + # Merge modules: add new modules if needed, update max load level for existing modules + for index, newmodule in enumerate(mcjProfile.getModules()): + # check if modules match by simple name, because module search is performed by it + existingModuleIndex = moduleNameMap.get(newmodule.getModuleName()) + + if (not existingModuleIndex is None): + # exists, update max load level (if load levels do not match, use max one) + existingModule = self._modules[existingModuleIndex] + + if (existingModule.getAssemblyName() != newmodule.getAssemblyName()): + print("Can't merge profiles: two modules with same names but different assembly names!") + raise UnsupportedError() + if (existingModule.getModuleRecord().getVersion() != newmodule.getModuleRecord().getVersion()): + print("Can't merge profiles: two modules with same names but different versions!") + raise UnsupportedError() + if (existingModule.getModuleRecord().getFlags() != newmodule.getModuleRecord().getFlags()): + print("Can't merge profiles: two modules with same names but different flags!") + raise UnsupportedError() + + existingModuleLoadLevel = existingModule.getModuleRecord().getLoadLevel() + moduleLoadLevelMap[index] = existingModuleLoadLevel + moduleIndexMap[index] = existingModuleIndex + existingModule.getModuleRecord().setLoadLevel(max(newmodule.getModuleRecord().getLoadLevel(), + existingModuleLoadLevel)) + else: + # simple module names do not match, safe to add new module + moduleLoadLevelMap[index] = None + moduleIndexMap[index] = len(self._modules) + newmoduleToAdd = newmodule.copy() + self._modules.append(newmoduleToAdd) + + # III. ==== Fill method map ==== + + for module in self._modules: + methodMap.append({}) + + # Prepare map from method str representation to index of info + for index, info in enumerate(self._moduleOrMethodInfo): + moduleIndex = info.getModuleIndex() + + if (info.isNonGenericMethodInfo()): + methodMap[moduleIndex][str(info.getMethodToken())] = index + elif (info.isGenericMethodInfo()): + methodMap[moduleIndex][info.getMethodHandle().getStr(self._modules)] = index + + # IV. ==== Merge infos ==== + + # Merge module and method info (existing part of self doesn't change, simply add mcjProfile after self) + for newinfo in mcjProfile.getModuleOrMethodInfo(): + if (newinfo.isMethodInfo()): + infoIndex = None + + newModuleIndex = moduleIndexMap[newinfo.getModuleIndex()] + + if (newinfo.isNonGenericMethodInfo()): + infoIndex = methodMap[newModuleIndex].get(str(newinfo.getMethodToken())) + elif (newinfo.isGenericMethodInfo()): + infoIndex = methodMap[newModuleIndex].get(newinfo.getMethodHandle().getStr(mcjProfile.getModules())) + else: + raise InternalError() + + if (infoIndex is None): + newinfoToAdd = newinfo.copy() + newinfoToAdd.updateModuleIndex(moduleIndexMap) + self._moduleOrMethodInfo.append(newinfoToAdd) + else: + if (newinfo.getMethodFlags() != self._moduleOrMethodInfo[infoIndex].getMethodFlags()): + print("Can't merge profiles: two methods with same token/signature but different flags!") + raise UnsupportedError() + else: + oldExistingLoadLevel = moduleLoadLevelMap[newinfo.getModuleIndex()] + + # module dependency + if ((not oldExistingLoadLevel is None) and oldExistingLoadLevel >= newinfo.getModuleLoadLevel()): + # nothing to do, don't add module dependency, already loaded at required level + pass + else: + newinfoToAdd = newinfo.copy() + newinfoToAdd.updateModuleIndex(moduleIndexMap) + self._moduleOrMethodInfo.append(newinfoToAdd) + + # IV. ==== Set stats ==== + + methodCount = 0 + moduleDepCount = 0 + + # Reset method count + for module in self._modules: + module.getModuleRecord().setJitMethodCount(0) + + # Update stats in modules + for info in self._moduleOrMethodInfo: + if (info.isMethodInfo()): + methodCount += 1 + + indexes = info.getAllModuleIndexes() + for i in indexes: + moduleRecord = self._modules[i].getModuleRecord() + count = moduleRecord.getJitMethodCount() + moduleRecord.setJitMethodCount(count + 1) + else: + moduleDepCount += 1 + + # update stats in headers + self._header.setModuleCount(len(self._modules)) + self._header.setMethodCount(methodCount) + self._header.setModuleDepCount(moduleDepCount) + # new profile was not used yet, so drop usage stats + self._header.dropGlobalUsageStats() + + self.verify() + + # Find module by name + def findModuleByName(self, moduleName): + if (moduleName is None or not issubclass(type(moduleName), str)): + raise InternalError() + for module in self._modules: + if (moduleName == module.getModuleName()): + return module + return None + + # Find method by token + def findMethodByToken(self, methodToken): + if (methodToken is None or not issubclass(type(methodToken), int)): + raise InternalError() + for moduleOrMethodInfo in self._moduleOrMethodInfo: + if (moduleOrMethodInfo.isMethodInfo() and moduleOrMethodInfo.getMethodToken() == methodToken): + return moduleOrMethodInfo + return None + + # Read MCJ profile from file + @staticmethod + def readFromFile(inputFile): + file = open(inputFile, "rb") + bytesArr = list(file.read()) + file.close() + + mcjProfile = MCJProfile.createFromBytes(bytesArr) + + return mcjProfile + + # Write MCJ profile to file + @staticmethod + def writeToFile(outputFile, mcjProfile): + file = open(outputFile, "wb") + file.write(bytearray(mcjProfile.convertToBytes())) + file.close() + +# ---------------------------------------------------------------------------------------------------------------------- +class CLI: + + # Constructor with command line arguments + def __init__(self, args): + self._args = args + + # Split mcj profiles in two: app-dependent (app) and app-independent (system) + def commandSplit(self): + systemModulesFile = open(self._args.system_modules_list, "r") + systemModules = systemModulesFile.readlines() + systemModulesFile.close() + + for i, module in enumerate(systemModules): + systemModules[i] = module.rstrip("\n") + + for filepath in self._args.input: + outFilepathApp = filepath + ".app" + outFilepathSys = filepath + ".sys" + + mcjProfile = MCJProfile.readFromFile(filepath) + mcjProfileApp, mcjProfileSys = mcjProfile.split(systemModules) + + MCJProfile.writeToFile(outFilepathApp, mcjProfileApp) + MCJProfile.writeToFile(outFilepathSys, mcjProfileSys) + + print("MCJ profile " + filepath + " was split in:") + print(" 1. app-dependent " + outFilepathApp) + print(" 2. app-independent " + outFilepathSys) + print("") + + # Merge mcj profiles + def commandMerge(self): + mcjProfileBase = MCJProfile.readFromFile(self._args.input[0]) + + for index in range(1, len(self._args.input)): + mcjProfile = MCJProfile.readFromFile(self._args.input[index]) + mcjProfileBase.merge(mcjProfile) + + MCJProfile.writeToFile(self._args.output, mcjProfileBase) + + print("MCJ profiles " + str(self._args.input) + " were merged in: " + self._args.output) + print("") + + # Verify mcj profiles format consistency + def commandVerify(self): + for filepath in self._args.input: + mcjProfile = MCJProfile.readFromFile(filepath) # pylint: disable=unused-variable + print("MCJ profile " + filepath + " is correct!") + + # Find module or method in mcj profile + def commandFind(self): + for filepath in self._args.input: + mcjProfile = MCJProfile.readFromFile(filepath) + + if (not self._args.module is None): + for module in self._args.module: + if (not mcjProfile.findModuleByName(module) is None): + print("MCJ profile " + filepath + " contains module " + module) + else: + print("MCJ profile " + filepath + " does not contain module " + module) + + if (not self._args.method_token is None): + for method in self._args.method_token: + methodToken = int(method) + if (not mcjProfile.findMethodByToken(methodToken) is None): + print("MCJ profile " + filepath + " contains method with token " + method) + else: + print("MCJ profile " + filepath + " does not contain method with token " + method) + + # Compare multiple mcj profiles for equality + def commandCompare(self): + if (self._args.sha256): + resBase = subprocess.run(["sha256sum", self._args.input[0]], check=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + shaBase = resBase.stdout.decode("ascii").split(" ")[0] + for index in range(1, len(self._args.input)): + res = subprocess.run(["sha256sum", self._args.input[index]], check=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + sha = res.stdout.decode("ascii").split(" ")[0] + resultStr = " are equal by hash." if sha == shaBase else " are not equal by hash." + print("MCJ profiles " + self._args.input[0] + " and " + self._args.input[index] + resultStr) + else: + mcjProfileBase = MCJProfile.readFromFile(self._args.input[0]) + for index in range(1, len(self._args.input)): + mcjProfile = MCJProfile.readFromFile(self._args.input[index]) + resultStr = " are equal." if mcjProfile == mcjProfileBase else " are not equal." + print("MCJ profiles " + self._args.input[0] + " and " + self._args.input[index] + resultStr) + + # Clean usage stats in profile + def commandCleanStats(self): + for filepath in self._args.input: + outfilepath = filepath + ".cleaned" + + mcjProfile = MCJProfile.readFromFile(filepath) + mcjProfile.getHeader().dropGlobalUsageStats() + MCJProfile.writeToFile(outfilepath, mcjProfile) + + print("Cleaned usage stats: " + filepath + " is saved as " + outfilepath) + print("") + + def commandPrint(self): + for filepath in self._args.input: + print("============ MCJ profile " + filepath + " ============") + mcjProfile = MCJProfile.readFromFile(filepath) + + doPrintModuleDeps = self._args.module_deps + doPrintGenericMethods = self._args.methods or self._args.generics + doPrintNonGenericMethods = self._args.methods or self._args.non_generics + doPrintModuleOrMethodInfo = doPrintModuleDeps or doPrintGenericMethods or doPrintNonGenericMethods + + if (self._args.raw): + # Print raw mcj profile + if (self._args.header): + mcjProfile.printRawHeader() + if (self._args.modules): + mcjProfile.printRawModules() + if (doPrintModuleOrMethodInfo): + mcjProfile.printRawModuleOrMethodInfo("", doPrintModuleDeps, doPrintGenericMethods, + doPrintNonGenericMethods) + if (not self._args.header and not self._args.modules and not doPrintModuleOrMethodInfo): + mcjProfile.printRaw() + else: + # Pretty-print mcj profile + if (self._args.header): + mcjProfile.printHeader() + if (self._args.modules): + mcjProfile.printModules() + if (doPrintModuleOrMethodInfo): + mcjProfile.printModuleOrMethodInfo("", doPrintModuleDeps, doPrintGenericMethods, + doPrintNonGenericMethods) + if (not self._args.header and not self._args.modules and not doPrintModuleOrMethodInfo): + mcjProfile.print() + + # Show short summary on mcj profile format + def commandHelp(self): # pylint: disable=no-self-use,too-many-statements + print("! To show command options help use --help option !") + print("") + + if (self._args.mcj_format): + print(">>> MCJ format help.") + print("") + print("MCJ file is a binary file with next sections:") + print("1. Header, i.e. 'HeaderRecord'") + print("2. Modules, i.e. multiple 'ModuleRecord'") + print("3. Methods and Module dependencies, i.e. multiple 'JitInfRecord':") + print(" 'ModuleDependency', 'GenericMethod', 'NonGenericMethod'") + print("") + print("I. Header.") + print("") + print("Header contains:") + print("1. MULTICOREJIT_HEADER_RECORD_ID tag") + print("2. Version of profile, MULTICOREJIT_PROFILE_VERSION") + print("3. Number of used modules, module dependencies and methods") + print("4. Profile usage stats (0 if profile was not written after usage)") + print("") + print("II. Modules.") + print("") + print("These section contains multiple ModuleRecord, each ModuleRecord contains:") + print("1. MULTICOREJIT_MODULE_RECORD_ID tag") + print("2. Module version") + print("3. Number of methods from module in profile") + print("4. Final load level for module") + print("5. Simple name of module") + print("6. Assembly name") + print("") + print("III. Modules and Methods dependencies") + print("") + print("This section contains multiple JitInfRecord, each JitInfRecord can be:") + print(" Module dependency, non-generic Method or generic Method.") + print("") + print("Each Module dependency contains:") + print("1. Module index") + print("2. Current module load level") + print("3. MULTICOREJIT_MODULEDEPENDENCY_RECORD_ID tag") + print("") + print("Each non-generic Method contains:") + print("1. Module index") + print("2. Method flags") + print("3. MULTICOREJIT_METHOD_RECORD_ID tag") + print("4. Method token from dll") + print("") + print("Each generic Method contains:") + print("1. Module index") + print("2. Method flags") + print("3. MULTICOREJIT_GENERICMETHOD_RECORD_ID tag") + print("4. Length of binary signature for method") + print("5. Binary signature for method") + elif (self._args.binary_signature_format): + print(">>> Binary signature format help.") + print("") + print("I. Value encoding") + print("") + print("Binary signature for method (as well as binary signature for type) is a byte array.") + print("") + print("Each value that is stored in this byte array is compacted (see _encodeValue) (except plain bytes):") + print(" - values '<= 0x7f' are stored as 1 byte without changes") + print(" - values '> 0x7f and <= 0x3fff' are stored as 2 bytes in big endian order (1st byte | 0x80)") + print(" - values '> 0x3fff and <= 0x1fffffff' are stored as 4 bytes in big endian order (1st byte | 0xc0)") + print("As a result of this 3 high bits of 1st byte determine length of value:") + print(" - 0yz mean 1 byte") + print(" - 10y mean 2 bytes") + print(" - 110 mean 4 bytes") + print("As mentioned above, sometimes plain bytes are saved (see _encodeByte).") + print("") + print("Tokens (method, type, etc.) are stored as plain values (see above) or tokens (see _encodeToken):") + print(" 1. rid and typ are obtained from token") + print(" 2. rid is shifted left on 2 bits") + print(" 3. rid is ORed with special 2bit constant determined by typ") + print(" 4. resulting modified rid goes through value encoding procedure above") + print("As a result of this typ of token is determined by 2 lowest bits of saved value.") + print("") + print("II. Type signature encoding") + print("") + print("Each type is encoded using logic mentioned in previous section.") + print("") + print("This logic is pretty complex, but main building blocks of all types are:") + print(" - other types") + print(" - type tokens") + print(" - calling convention") + print(" - number of func args") + print(" - number of generic args") + print(" - array rank") + print(" - module index <-- this is module index in array of modules that are processed by MCJ!") + print("For more details see source code (see _encodeSignatureForTypeHandle).") + print("") + print("III. Method signature encoding") + print("") + print("Each method is encoded using logic mentioned in previous section.") + print("") + print("Method encoding order:") + print(" 1. Method flags") + print(" 2. Type from which this method comes from") + print(" 3. Method token") + print(" 4*. Number of method instantion args (for generic methods)") + print(" 5*. Types of all method instantion args (for generic methods)") + print("For more details see source code (see encodeMethod).") + else: + print("! To show format help use --mcj-format or --binary-signature-format options !") + + # Perform various self testing + def commandSelfTest(self): # pylint: disable=too-many-locals,too-many-statements + if (self._args.rw): + # Read mcj profile, write it to file, read it back again and compare + for filepath in self._args.input: + outFilepath = filepath + ".__tmp" + mcjProfileIn = MCJProfile.readFromFile(filepath) + MCJProfile.writeToFile(outFilepath, mcjProfileIn) + mcjProfileOut = MCJProfile.readFromFile(outFilepath) + if (mcjProfileIn == mcjProfileOut): + print("Read-write self test passed for " + filepath) + else: + print("Read-write self test failed for " + filepath) + elif (self._args.rw_sha256): + # Read mcj profile, write it to file, compare with sha256sum with original file + for filepath in self._args.input: + outFilepath = filepath + ".__tmp" + mcjProfile = MCJProfile.readFromFile(filepath) + MCJProfile.writeToFile(outFilepath, mcjProfile) + + inputRes = subprocess.run(["sha256sum", filepath], check=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + outputRes = subprocess.run(["sha256sum", outFilepath], check=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + + inputSha = inputRes.stdout.decode("ascii").split(" ")[0] + outputSha = outputRes.stdout.decode("ascii").split(" ")[0] + + if (inputSha == outputSha): + print("Read-write sha256sum self test passed for " + filepath) + else: + print("Read-write sha256sum self test failed for " + filepath) + elif (self._args.sm): + # Split mcj profile, merge two resulting profiles, split again and compare + systemModulesFile = open(self._args.system_modules_list, "r") + systemModules = systemModulesFile.readlines() + systemModulesFile.close() + + for i, moduleName in enumerate(systemModules): + systemModules[i] = moduleName.rstrip("\n") + + for filepath in self._args.input: + mcjProfile = MCJProfile.readFromFile(filepath) + mcjProfileApp, mcjProfileSys = mcjProfile.split(systemModules) + + prevMergedAppFirst = mcjProfile + prevMergedSysFirst = mcjProfile + prevSplitAppFirstApp = mcjProfileApp + prevSplitAppFirstSys = mcjProfileSys + prevSplitSysFirstApp = mcjProfileApp + prevSplitSysFirstSys = mcjProfileSys + + isCorrect = True + depthCheck = 1 + + # Note: first split&merge won't match because order changes, but it stabilizes on 2nd merge + for depth in range(1,5): + # Merge two prev profiles + mergedAppFirst = prevSplitAppFirstApp.copy() + mergedAppFirst.merge(prevSplitAppFirstSys) + + mergedSysFirst = prevSplitSysFirstSys.copy() + mergedSysFirst.merge(prevSplitSysFirstApp) + + if (depth > depthCheck + and (mergedAppFirst != prevMergedAppFirst or mergedSysFirst != prevMergedSysFirst)): + isCorrect = False + break + + prevMergedAppFirst = mergedAppFirst + prevMergedSysFirst = mergedSysFirst + + mergedAppFirst2 = mergedAppFirst.copy() + mergedAppFirst2.merge(mergedSysFirst) + + mergedSysFirst2 = mergedSysFirst.copy() + mergedSysFirst2.merge(mergedAppFirst) + + if (mergedAppFirst2 != mergedAppFirst or mergedSysFirst2 != mergedSysFirst): + isCorrect = False + break + + # Split cur profiles + splitAppFirstApp, splitAppFirstSys = mergedAppFirst.split(systemModules) + splitSysFirstApp, splitSysFirstSys = mergedSysFirst.split(systemModules) + + if (depth > depthCheck + and (splitAppFirstApp != prevSplitAppFirstApp or splitAppFirstSys != prevSplitAppFirstSys + or splitSysFirstApp != prevSplitSysFirstApp or splitSysFirstSys != prevSplitSysFirstSys)): + isCorrect = False + break + + prevSplitAppFirstApp = splitAppFirstApp + prevSplitAppFirstSys = splitAppFirstSys + prevSplitSysFirstApp = splitSysFirstApp + prevSplitSysFirstSys = splitSysFirstSys + + splitAppFirstApp2, splitAppFirstSys2 = splitAppFirstApp.split(systemModules) + + if (splitAppFirstApp2 != splitAppFirstApp): + isCorrect = False + break + + splitAppFirstApp2, splitAppFirstSys2 = splitAppFirstSys.split(systemModules) + + if (splitAppFirstSys2 != splitAppFirstSys): + isCorrect = False + break + + splitSysFirstApp2, splitSysFirstSys2 = splitSysFirstApp.split(systemModules) + + if (splitSysFirstApp2 != splitSysFirstApp): + isCorrect = False + break + + splitSysFirstApp2, splitSysFirstSys2 = splitSysFirstSys.split(systemModules) + + if (splitSysFirstSys2 != splitSysFirstSys): + isCorrect = False + break + + if (isCorrect): + print("Split-merge self test passed for " + filepath) + else: + print("Split-merge self test failed for " + filepath) + elif (self._args.unit): + # TODO: add unit tests + pass + else: + pass + +# ---------------------------------------------------------------------------------------------------------------------- + +def main(): + parser = argparse.ArgumentParser() + + commands = "split, merge, verify, find, compare, clean-stats, print, help, self-test" + + parser.add_argument("command", help="Command to execute: " + commands) + + # Overall options + parser.add_argument("-i", "--input", help="Input mcj profiles", action="append") + + # Split options + parser.add_argument("--system-modules-list", help="[split], file with app-independent (i.e. system) module names") + + # Merge options + parser.add_argument("-o", "--output", help="[merge], output mcj profile") + + # Find options + parser.add_argument("--module", help="[find], name of module to find in mcj profile", action="append") + parser.add_argument("--method-token", help="[find], token of method to find in mcj profile", action="append") + + # Compare options + parser.add_argument("--sha256", help="[compare], compare mcj profiles using sha256sum", action="store_true") + + # Print options + parser.add_argument("--raw", help="[print], print raw mcj profile", action="store_true") + parser.add_argument("--header", help="[print], print header of mcj profile", action="store_true") + parser.add_argument("--modules", help="[print], print modules in mcj profile", action="store_true") + + parser.add_argument("--methods", help="[print], print methods in mcj profile", action="store_true") + parser.add_argument("--generics", help="[print], print generic methods in mcj profile", action="store_true") + parser.add_argument("--non-generics", help="[print], print non-generic methods in mcj profile", action="store_true") + parser.add_argument("--module-deps", help="[print], print module dependencies in mcj profile", action="store_true") + + # Help options + parser.add_argument("--mcj-format", help="[help], show help on mcj format", action="store_true") + parser.add_argument("--binary-signature-format", help="[help], show help on binary signature format", + action="store_true") + + # Self test option + parser.add_argument("--rw", help="[self-test], perform read-write self-test", action="store_true") + parser.add_argument("--rw-sha256", + help="[self-test], perform read-write self-test using sha256sum", action="store_true") + parser.add_argument("--sm", help="[self-test], perform split-merge self-test", action="store_true") + parser.add_argument("--unit", help="TODO, [self-test], perform unit testing self-test", action="store_true") + + args = parser.parse_args() + + cli = CLI(args) + + if (args.command == "split"): + cli.commandSplit() + elif (args.command == "merge"): + cli.commandMerge() + elif (args.command == "verify"): + cli.commandVerify() + elif (args.command == "find"): + cli.commandFind() + elif (args.command == "compare"): + cli.commandCompare() + elif (args.command == "clean-stats"): + cli.commandCleanStats() + elif (args.command == "print"): + cli.commandPrint() + elif (args.command == "help"): + cli.commandHelp() + elif (args.command == "self-test"): + cli.commandSelfTest() + +if __name__ == "__main__": + main()