From: Andon Andonov Date: Wed, 27 Jun 2018 21:43:36 +0000 (-0700) Subject: Windows CoreFX CI (dotnet/coreclr#18365) X-Git-Tag: submit/tizen/20210909.063632~11030^2~4500 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=8f0fd82ea8b80e2dd952e7b727b6adf92f2e196c;p=platform%2Fupstream%2Fdotnet%2Fruntime.git Windows CoreFX CI (dotnet/coreclr#18365) * Add test list CL switch * End-To-End Test Run on Windows * Cleanup * MAX_PATH Workaround * Set Execution directory for CoreFX tests * Add All CoreFX PR Tests * Add test dependencies * Add extra dependencies * Add parallel test execution * Disable OuterLoop tests and System.Data.SqlClient.* tests * Initialize maximum degree of parallelization to Environment.ProcessCount * Remove unnecessary cli option * Update Dependencies * Add "enabled" property to tests * Remove exclusions due to TestUtilities mismatch * Add capability to run all tests for running Helix test lists directly * Refactor build script to build testhost when skipping managed tests * Disable failing System.Threading.Tests.EventWaitHandleTests.Ctor_InvalidMode * Add switch to skip native test build * Add testing documentation * Don't run tests marked as "disabled" when running all available tests * Add switch to build only testhost and remove Core_Root_Stage * Clean up TopN.CoreFX.Windows.issues.json * Refactor build-test.cmd * PR feedback - build pipeline and documentation * PR Feedback - Test Helper headers and comments * Fix buildtesthost option for only building CoreFX test dependencies * Disable intermittently failing test DrawBezier_PointFs Commit migrated from https://github.com/dotnet/coreclr/commit/d3772d9efc39591d15ffdfbcf60f5b5936c25d0e --- diff --git a/docs/coreclr/building/testing-with-corefx.md b/docs/coreclr/building/testing-with-corefx.md index a400d14..0f7beef 100644 --- a/docs/coreclr/building/testing-with-corefx.md +++ b/docs/coreclr/building/testing-with-corefx.md @@ -3,6 +3,7 @@ Testing with CoreFX It may be valuable to use CoreFX tests to validate your changes to CoreCLR or mscorlib. +## Building CoreFX against CoreCLR **NOTE:** The `BUILDTOOLS_OVERRIDE_RUNTIME` property no longer works. To run CoreFX tests with an updated System.Private.Corelib.dll, [use these instructions](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits). @@ -27,3 +28,74 @@ Use the following instructions to test a change to the dotnet/coreclr repo using [run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) will clone dotnet/corefx and run steps 2-4 above automatically. It is primarily intended to be run by the dotnet/coreclr CI system, but it might provide a useful reference or shortcut for individuals running the tests locally. +## Using the built CoreCLR testhost +**These instructions are currently Windows only.** + +Instead of copying CoreCLR binaries you can also test your changes with an existing CoreFX build or CoreCLR's CI assemblies + +### Locally-built CoreFX +Once you have finished steps 1, 2. and 4. above execute the following instructions to test your local CLR changes with the built-CoreFX changes. + +1. From `` run `build-test.cmd skipmanaged` to generate the test host. +2. Navigate to `\bin\tests\` and then the test you would like to run +3. Run + +```cmd +\bin\..\testhost\dotnet.exe \bin\tests\\xunit.console.netcore.exe .dll +``` +followed by any extra command-line arguments. + +For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR. +``` +pushd C:\corefx\bin\tests\System.Collections.Tests +C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonwindowstests +``` + +### CI Script +CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs. To run tests against pre-built binaries you can execute the following from the CoreCLR repo root: + +1. `.\build.cmd ` +2. `.\build-test.cmd skipmanaged` - generates the test host +3. `.\tests\runtest.cmd corefxtests|corefxtestsall` - runs CoreFX tests + +CoreFXTests - runs all tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList` +CoreFXTestsAll - runs all tests available in the test list found at the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt`. + +### Helix Testing +To use Helix-built binaries, substitute the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt` with one acquired from a Helix test run and run the commands above. + +### Test List Format +The tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList` should conform to the following format - +```json + { + "name": "", //e.g. System.Collections.Concurrent.Tests + "enabled": true|false, // Defines whether a test assembly should be run. If set to false any tests with the same name will not be run even if corefxtestsall is specified + "exclusions": { + "namespaces": // Can be null + [ + { + "name": "System.Collections.Concurrent.Tests", // All test methods under this namespace will be skipped + "reason": "" // This should be a link to the GitHub issue describing the problem + } + ] + "classes": // Can be null + [ + { + "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests", // All test methods in this class will be skipped + "reason": "" + } + ] + "methods": // Can be null + [ + { + "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ReferenceType_null", + "reason": "" + }, + { + "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ValueType_null_add", + "reason": "" + } + ] + } + } +``` \ No newline at end of file diff --git a/src/coreclr/build-test.cmd b/src/coreclr/build-test.cmd index e65cbca..58c6d32 100644 --- a/src/coreclr/build-test.cmd +++ b/src/coreclr/build-test.cmd @@ -46,6 +46,9 @@ set processedArgs= set __unprocessedBuildArgs= set __RunArgs= set __BuildAgainstPackagesArg= +set __SkipRestorePackages= +set __SkipManaged= +set __SkipNative= set __RuntimeId= set __ZipTests= set __TargetsWindows=1 @@ -74,8 +77,11 @@ if /i "%1" == "release" (set __BuildType=Release&set processedArgs if /i "%1" == "checked" (set __BuildType=Checked&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) if /i "%1" == "skipmanaged" (set __SkipManaged=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) +if /i "%1" == "skipnative" (set __SkipNative=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) +if /i "%1" == "buildtesthostonly" (set __SkipNative=1&set __SkipManaged=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) if /i "%1" == "toolset_dir" (set __ToolsetDir=%2&set __PassThroughArgs=%__PassThroughArgs% %2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop) if /i "%1" == "buildagainstpackages" (set __ZipTests=1&set __BuildAgainstPackagesArg=-BuildTestsAgainstPackages&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) +if /i "%1" == "skiprestorepackages" (set __SkipRestorePackages=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) if /i "%1" == "ziptests" (set __ZipTests=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) if /i "%1" == "crossgen" (set __DoCrossgen=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop) if /i "%1" == "runtimeid" (set __RuntimeId=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop) @@ -160,6 +166,8 @@ REM ============================================================================ call "%__TestDir%\setup-stress-dependencies.cmd" /arch %__BuildArch% /outputdir %__BinDir% @if defined _echo @echo on +if defined __SkipNative goto skipnative + REM ========================================================================================= REM === REM === Native test build section @@ -224,8 +232,7 @@ if errorlevel 1 ( :skipnative set "__TestWorkingDir=%__RootBinDir%\tests\%__BuildOS%.%__BuildArch%.%__BuildType%" - -if not defined __BuildAgainstPackagesArg goto SkipRestoreProduct +if "%__SkipRestorePackages%" == 1 goto SkipRestoreProduct REM ========================================================================================= REM === REM === Restore product binaries from packages @@ -245,6 +252,7 @@ set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%" call "%__ProjectDir%\run.cmd" build -Project=%__ProjectDir%\tests\build.proj -BatchRestorePackages -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %__unprocessedBuildArgs% +if not defined __BuildAgainstPackagesArg goto SkipRestoreProduct set __BuildLogRootName=Tests_GenerateRuntimeLayout set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn @@ -263,7 +271,7 @@ echo %__MsgPrefix% Restored CoreCLR product from packages :SkipRestoreProduct -if defined __SkipManaged exit /b 0 +if defined __SkipManaged goto SkipManagedBuild REM ========================================================================================= REM === @@ -316,6 +324,7 @@ for /l %%G in (1, 1, %__BuildLoopCount%) do ( set __AppendToLog=true ) +:SkipManagedBuild REM Prepare the Test Drop REM Cleans any NI from the last run powershell -NoProfile "Get-ChildItem -path %__TestWorkingDir% -Include '*.ni.*' -Recurse -Force | Remove-Item -force" @@ -344,22 +353,14 @@ if defined __BuildAgainstPackagesArg ( ) ) -echo %__MsgPrefix%Creating test wrappers... - -set RuntimeIdArg= -set TargetsWindowsArg= - -if defined __RuntimeId ( - set RuntimeIdArg=-RuntimeID="%__RuntimeId%" -) - -if "%__TargetsWindows%"=="1" ( - set TargetsWindowsArg=-TargetsWindows=true -) else if "%__TargetsWindows%"=="0" ( - set TargetsWindowsArg=-TargetsWindows=false -) +REM ========================================================================================= +REM === +REM === Create the test overlay +REM === +REM ========================================================================================= +echo %__MsgPrefix%Creating test overlay... -set __BuildLogRootName=Tests_XunitWrapper +set __BuildLogRootName=Tests_Overlay_Managed set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err @@ -367,15 +368,21 @@ set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%" set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%" set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%" -call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -BuildWrappers -MsBuildEventLogging=" " -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %TargetsWindowsArg% %__unprocessedBuildArgs% +call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -testOverlay -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %RuntimeIdArg% %__unprocessedBuildArgs% if errorlevel 1 ( - echo Xunit Wrapper build failed + echo %__MsgPrefix%Error: build failed. Refer to the build log files for details: + echo %__BuildLog% + echo %__BuildWrn% + echo %__BuildErr% exit /b 1 ) -echo %__MsgPrefix%Creating test overlay... +xcopy /s /y "%CORE_ROOT_STAGE%" "%CORE_ROOT%" -set __BuildLogRootName=Tests_Overlay_Managed +REM Create the test host necessary for running CoreFX tests +REM The test host includes a dotnet executable, system libraries and CoreCLR assemblies found in CoreRoot + +set __BuildLogRootName=Tests_CoreFX_Testhost set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err @@ -383,7 +390,7 @@ set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%" set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%" set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%" -call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -testOverlay -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %RuntimeIdArg% %__unprocessedBuildArgs% +call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -testHost -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %RuntimeIdArg% %__unprocessedBuildArgs% if errorlevel 1 ( echo %__MsgPrefix%Error: build failed. Refer to the build log files for details: echo %__BuildLog% @@ -392,7 +399,37 @@ if errorlevel 1 ( exit /b 1 ) -xcopy /s /y "%CORE_ROOT_STAGE%" "%CORE_ROOT%" +if defined __SkipManaged goto SkipBuildingWrappers + +echo %__MsgPrefix%Creating test wrappers... + +set RuntimeIdArg= +set TargetsWindowsArg= + +if defined __RuntimeId ( + set RuntimeIdArg=-RuntimeID="%__RuntimeId%" +) + +if "%__TargetsWindows%"=="1" ( + set TargetsWindowsArg=-TargetsWindows=true +) else if "%__TargetsWindows%"=="0" ( + set TargetsWindowsArg=-TargetsWindows=false +) + +set __BuildLogRootName=Tests_XunitWrapper +set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log +set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn +set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err +set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%" +set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%" +set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%" + +call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -BuildWrappers -MsBuildEventLogging=" " -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %TargetsWindowsArg% %__unprocessedBuildArgs% +if errorlevel 1 ( + echo Xunit Wrapper build failed + exit /b 1 +) +:SkipBuildingWrappers set __CrossgenArg = "" if defined __DoCrossgen ( diff --git a/src/coreclr/config.json b/src/coreclr/config.json index 406ea51..65ebfa1 100644 --- a/src/coreclr/config.json +++ b/src/coreclr/config.json @@ -336,6 +336,12 @@ "values": [], "defaultValue": "" }, + "CreateTestHost": { + "description": "Place test dependencies in the /bin/testhost folder to be used to run CoreFX tests", + "valueType": "target", + "values": [], + "defaultValue": "" + }, "Verbosity": { "description": "Sets build verbosity.", "valueType": "passThrough", @@ -504,6 +510,12 @@ "CreateTestOverlay": "default" } }, + "testHost": { + "description": "Runs test host target, which creates a dotnet executable from the built CoreCLR repo to run pre-built CoreFX test assemblies", + "settings": { + "CreateTestHost": "default" + } + }, "priority": { "description": "Sets CLRTestPriorityToBuild property.", "settings": { diff --git a/src/coreclr/dependencies.props b/src/coreclr/dependencies.props index 8024ae0..ed8514b 100644 --- a/src/coreclr/dependencies.props +++ b/src/coreclr/dependencies.props @@ -36,7 +36,7 @@ 99.99.99-master-20180621-0050 2.2.0-preview1-26621-01 2.2.0-beta2-build3300 - 1.0.2-prerelease-00177 + 2.2.0-preview1-02830-02 1.0.0-beta-build0015 2.0.4 2.2.0 @@ -137,6 +137,7 @@ + diff --git a/src/coreclr/tests/CoreFX/CoreFXTestListURL.txt b/src/coreclr/tests/CoreFX/CoreFXTestListURL.txt new file mode 100644 index 0000000..926cf3c --- /dev/null +++ b/src/coreclr/tests/CoreFX/CoreFXTestListURL.txt @@ -0,0 +1 @@ +https://cloudcijobs.blob.core.windows.net/coreclrci/CoreFXArchives/TestList.json \ No newline at end of file diff --git a/src/coreclr/tests/CoreFX/CoreFXTestListURL_Linux.txt b/src/coreclr/tests/CoreFX/CoreFXTestListURL_Linux.txt new file mode 100644 index 0000000..e7b7c74 --- /dev/null +++ b/src/coreclr/tests/CoreFX/CoreFXTestListURL_Linux.txt @@ -0,0 +1 @@ +https://cloudcijobs.blob.core.windows.net/corertci/TestList_Linux.json \ No newline at end of file diff --git a/src/coreclr/tests/CoreFX/CoreFXTestListURL_OSX.txt b/src/coreclr/tests/CoreFX/CoreFXTestListURL_OSX.txt new file mode 100644 index 0000000..e201a86 --- /dev/null +++ b/src/coreclr/tests/CoreFX/CoreFXTestListURL_OSX.txt @@ -0,0 +1 @@ +https://cloudcijobs.blob.core.windows.net/corertci/TestList_OSX.json \ No newline at end of file diff --git a/src/coreclr/tests/CoreFX/TopN.CoreFX.x64.Unix.issues.json b/src/coreclr/tests/CoreFX/TopN.CoreFX.x64.Unix.issues.json new file mode 100644 index 0000000..7076613 --- /dev/null +++ b/src/coreclr/tests/CoreFX/TopN.CoreFX.x64.Unix.issues.json @@ -0,0 +1,48 @@ +// The expected format of this file can be found at https://github.com/dotnet/coreclr/blob/master/Documentation/building/testing-with-corefx.md +[ + { + "name": "System.Collections.Tests", + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Collections.Tests.SortedList_Generic_Tests_string_string.ICollection_Generic_Remove_DefaultValueContainedInCollection", + "reason": "Exclusion Sample" + } + ] + } + }, + { + "name": "System.Runtime.Extensions.Tests", + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [] + } + }, + { + "name": "System.Runtime.Tests", + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [] + } + }, + { + "name": "System.Threading.Tasks.Tests", + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [] + } + }, + { + "name": "System.Threading.Tests", + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [] + } + } +] \ No newline at end of file diff --git a/src/coreclr/tests/CoreFX/TopN.CoreFX.x64.Windows.issues.json b/src/coreclr/tests/CoreFX/TopN.CoreFX.x64.Windows.issues.json new file mode 100644 index 0000000..5518056 --- /dev/null +++ b/src/coreclr/tests/CoreFX/TopN.CoreFX.x64.Windows.issues.json @@ -0,0 +1,1268 @@ +// The expected format of this file can be found at https://github.com/dotnet/coreclr/blob/master/Documentation/building/testing-with-corefx.md +[ + { + "name": "Microsoft.Win32.SystemEvents.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "Microsoft.Win32.SystemEventsTests.PowerModeTests.SignalsPowerModeChanged", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.SessionEndingTests.CancelSessionEnding", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: 0\\r\\nActual: 1" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.SessionEndingTests.SignalsSessionEnding", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.PaletteChangedTests.SignalsEventsAsynchronouslyOnMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.PaletteChangedTests.SignalsEventsSynchronouslyOnReflectedMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.InvokeOnEventsThreadTests.InvokeOnEventsThreadRunsOnSameThreadAsOtherEvents", + "reason": "Xunit.Sdk.NotEqualException Assert.NotEqual() Failure\\r\\nExpected: Not -1\\r\\nActual: -1" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.LowMemoryTests.SignalsEventsAsynchronouslyOnMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.LowMemoryTests.SignalsEventsSynchronouslyOnReflectedMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.UserPreferenceTests.SignalsUserPreferenceEventsAsynchronouslyOnThemeChanged", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.UserPreferenceTests.SignalsUserPreferenceEventsSynchronously", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.SessionSwitchTests.SignalsSessionSwitch", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.TimeChangedTests.SignalsEventsAsynchronouslyOnMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.TimeChangedTests.SignalsEventsSynchronouslyOnReflectedMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.SessionEndedTests.SignalsSessionEnded", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.InstalledFontsChangedTests.SignalsEventsSynchronouslyOnReflectedMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.InstalledFontsChangedTests.SignalsEventsAsynchronouslyOnMessage", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.DisplaySettingsTests.SignalsDisplayEventsAsynchronouslyOnDISPLAYCHANGE", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "Microsoft.Win32.SystemEventsTests.DisplaySettingsTests.SignalsDisplayEventsSynchronouslyOnREFLECTDISPLAYCHANGE", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + } + ] + } + }, + { + "name": "System.Collections.Concurrent.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ReferenceType_null", + "reason": "System.ArgumentException System.ArgumentException : The value was of an incorrect type for this dictionary." + }, + { + "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ValueType_null_add", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentException)\\r\\nActual: typeof(System.NullReferenceException): Object reference not set to an instance of an object." + } + ] + } + }, + { + "name": "System.Collections.Immutable.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Collections.Immutable.Tests.ImmutableArrayTest.CreateRangeSliceWithSelector", + "reason": "System.IndexOutOfRangeException System.IndexOutOfRangeException : Index was outside the bounds of the array." + }, + { + "name": "System.Collections.Immutable.Tests.ImmutableArrayTest.CreateRangeWithSelector", + "reason": "System.IndexOutOfRangeException System.IndexOutOfRangeException : Index was outside the bounds of the array." + }, + { + "name": "System.Collections.Immutable.Tests.ImmutableArrayTest.CreateRangeWithSelectorAndArgument", + "reason": "System.IndexOutOfRangeException System.IndexOutOfRangeException : Index was outside the bounds of the array." + }, + { + "name": "System.Collections.Immutable.Tests.ImmutableArrayTest.CreateRangeSliceWithSelectorAndArgument", + "reason": "System.IndexOutOfRangeException System.IndexOutOfRangeException : Index was outside the bounds of the array." + } + ] + } + }, + { + "name": "System.ComponentModel.TypeConverter.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.ComponentModel.ToolboxItemAttributeTests.Ctor_NullToolboxItemTypeName_ThrowsArgumentNullException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.NullReferenceException): Object reference not set to an instance of an object." + }, + { + "name": "System.ComponentModel.Tests.EditorAttributeTests.Ctor_NullEditorTypeName_ThrowsArgumentNullException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.NullReferenceException): Object reference not set to an instance of an object." + }, + { + "name": "System.ComponentModel.Tests.DesignerAttributeTests.Ctor_NullDesignerTypeName_ThrowsArgumentNullExceptionException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.NullReferenceException): Object reference not set to an instance of an object." + } + ] + } + }, + { + "name": "System.Console.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "WindowAndCursorProps.WindowLeftTop_Windows", + "reason": "Assert.Throws() Failure\r\nExpected: typeof(System.IO.IOException)\r\nActual: (No exception was thrown)" + }, + { + "name": "WindowAndCursorProps.WindowWidth_WindowHeight_InvalidSize", + "reason": "Assert.Throws() Failure\r\nExpected: typeof(System.IO.IOException)\r\nActual: typeof(System.ArgumentOutOfRangeException): Positive number required.\r\nParameter name: width\r\nActual value was 0." + }, + { + "name": "WindowAndCursorProps.SetWindowPosition_GetWindowPosition_ReturnsExpected", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: top\\r\\nActual: left\\r\\n ↑ (pos 0)" + }, + { + "name": "WindowAndCursorProps.GetCursorPosition", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: 2886\\r\\nActual: 2887" + } + ] + } + }, + { + "name": "System.Data.DataSetExtensions.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Data.Tests.DataRowComparerTests.Equals_NullStringValueInStringArray_CanBeCompared", + "reason": "System.NullReferenceException : Object reference not set to an instance of an object." + } + ] + } + }, + { + "name": "System.Drawing.Common.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Drawing.Tests.GraphicsTests.InterpolationMode_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'InterpolationMode'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.GraphicsTests.CompositingQuality_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'CompositingQuality'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.GraphicsTests.PixelOffsetMode_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'PixelOffsetMode'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.GraphicsTests.PageUnit_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-1) is invalid for Enum type 'GraphicsUnit'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.GraphicsTests.SmoothingMode_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'SmoothingMode'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.GraphicsTests.CompositingMode_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-1) is invalid for Enum type 'CompositingMode'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.GraphicsTests.TextRenderingHint_SetInvalid_ThrowsInvalidEnumArgumentException", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\r\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-1) is invalid for Enum type 'TextRenderingHint'.\\r\\nParameter name: value" + }, + { + "name": "System.Drawing.Tests.Graphics_DrawBezierTests.DrawBezier_PointFs", + "reason": "Intermittent Failure" + } + ] + } + }, + { + "name": "System.IO.Compression.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.IO.Compression.GzipStreamUnitTests.ConcatenatedGzipStreams", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: Stream 2\\r\\nActual: (null)" + }, + { + "name": "System.IO.Compression.GzipStreamUnitTests.ManyConcatenatedGzipStreams", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: 1000\\r\\nActual: 1" + } + ] + } + }, + { + "name": "System.IO.FileSystem.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.IO.Tests.Enumeration.RemovedDirectoryTests.RemoveDirectoryBeforeHandleCreationAndReplaceWithFile", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.RemovedDirectoryTests.RemoveDirectoryBeforeHandleCreation", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.ExampleTests.TestCountFiles", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.ExampleTests.TestGetFilesWithExtensions", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.ExampleTests.TestCountFileBytes", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.ExampleTests.GetFileNamesEnumerable", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.DirectoryInfo_CreateSubDirectory.ValidPathWithoutTrailingSlash", + "reason": "System.ArgumentException System.ArgumentException : The directory specified, 'oq3eklx3.oeu', is not a subdirectory of 'C:\\\\Users\\\\anandono\\\\AppData\\\\Local\\\\Temp\\\\DirectoryInfo_CreateSubDirectory_vv4tbdki.2q5\\\\ValidPathWithoutTrailingSlash_142_98959e38\\\\'.\\r\\nParameter name: path" + }, + { + "name": "System.IO.Tests.DirectoryInfo_CreateSubDirectory.ValidPathWithTrailingSlash", + "reason": "System.ArgumentException System.ArgumentException : The directory specified, '3tw22dam.r5w\\\\', is not a subdirectory of 'C:\\\\Users\\\\anandono\\\\AppData\\\\Local\\\\Temp\\\\DirectoryInfo_CreateSubDirectory_xiv2oluq.bps\\\\ValidPathWithTrailingSlash_121_6652a2bc\\\\'.\\r\\nParameter name: path" + }, + { + "name": "System.IO.Tests.Enumeration.TrimmedPaths.TrimmedPathsText_Windows", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.TrimmedPaths.TrimmedPathsCopyTo_Windows", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.TrimmedPaths.TrimmedPathsOpen_Windows", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.TrimmedPaths.TrimmedPathsMoveTo_Windows", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String, System.String)'." + }, + { + "name": "System.IO.Tests.Enumeration.TrimmedPaths.TrimmedPathsReplace_Windows", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + } + ] + } + }, + { + "name": "System.Linq.Expressions.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Linq.Expressions.Tests.GetDelegateTypeTests.CantBeFunc", + "reason": "System.TypeLoadException System.TypeLoadException : The generic type 'System.Func`1' was used with an invalid instantiation in assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'." + }, + { + "name": "System.Linq.Expressions.Tests.GetDelegateTypeTests.CantBeAction", + "reason": "System.TypeLoadException System.TypeLoadException : The generic type 'System.Action`1' was used with an invalid instantiation in assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'." + } + ] + } + }, + { + "name": "System.Net.Http.Functional.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.GetAsync_ExpectContinueTrue_NoContent_StillSendsHeader", + "reason": "System.Net.Http.HttpRequestException System.Net.Http.HttpRequestException : Response status code does not indicate success: 100 (Continue)." + }, + { + "name": "System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpProtocolTests_Dribble.CustomMethod_SentUppercasedIfKnown", + "reason": "Xunit.Sdk.StartsWithException Assert.StartsWith() Failure:\\r\\nExpected: GET \\r\\nActual: get ..." + }, + { + "name": "System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpProtocolTests.CustomMethod_SentUppercasedIfKnown_Additional", + "reason": "Xunit.Sdk.StartsWithException Assert.StartsWith() Failure:\\r\\nExpected: DELETE \\r\\nActual: delete ..." + }, + { + "name": "System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpProtocolTests.CustomMethod_SentUppercasedIfKnown", + "reason": "Xunit.Sdk.StartsWithException Assert.StartsWith() Failure:\\r\\nExpected: GET \\r\\nActual: get ..." + } + ] + } + }, + { + "name": "System.Net.Primitives.Functional.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Net.Primitives.Functional.Tests.CookieCollectionTest.Remove_NonExistantCookie_ReturnsFalse", + "reason": "System.EntryPointNotFoundException System.EntryPointNotFoundException : Entry point was not found." + }, + { + "name": "System.Net.Primitives.Functional.Tests.CookieCollectionTest.Contains_Success", + "reason": "System.EntryPointNotFoundException System.EntryPointNotFoundException : Entry point was not found." + }, + { + "name": "System.Net.Primitives.Functional.Tests.CookieCollectionTest.Remove_Success", + "reason": "System.EntryPointNotFoundException System.EntryPointNotFoundException : Entry point was not found." + }, + { + "name": "System.Net.Primitives.Functional.Tests.CookieCollectionTest.Clear_Success", + "reason": "System.EntryPointNotFoundException System.EntryPointNotFoundException : Entry point was not found." + } + ] + } + }, + + { + "name": "System.Net.Security.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Net.Security.Tests.SslStreamCredentialCacheTest.SslStream_SameCertUsedForClientAndServer_Ok", + "reason": "Unreliable" + } + ] + } + }, + { + "name": "System.Net.Security.Unit.Tests", + "enabled": false, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": null + } + }, + { + "name": "System.Net.WebSockets.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Net.WebSockets.Tests.WebSocketTests.ReceiveAsync_ServerSplitHeader_ValidDataReceived", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: 84\\r\\nActual: 0" + } + ] + } + }, + { + "name": "System.Net.WebSockets.WebSocketProtocol.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Net.WebSockets.Tests.WebSocketProtocolCreateTests.ReceiveAsync_ServerSplitHeader_ValidDataReceived", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: 84\\r\\nActual: 0" + } + ] + } + }, + { + "name": "System.Reflection.Context.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.Invoke_NullObject_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.GetCustomAttributes_NoType_Test", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.GetCustomAttributes_WithType_Test", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.Invoke_NotEmptyParameter_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.ProjectionTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.GetCustomAttributesDataTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.IsDefinedTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.Invoke_NullParameters_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.Invoke_EmptyParameters_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertyGetter_Tests.Invoke_WrongObject_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.SetValue_HasSetter_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.Ctor_NullPropertyName_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.ReflectedTypeTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.AttributesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetCustomAttributes_NoType_Test", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.CanReadTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.Ctor_NullPropertyType_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.ModuleTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.ToStringTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetAccessorsTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetConstantValueTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetHashCodeTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetOptionalCustomModifiersTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetRawConstantValueTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetValue_NoGetter_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.ProjectionTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.SetValue_NoSetter_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.Ctor_WrongPropertyType_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.SetValue_HasSetterWithIndex_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetCustomAttributesDataTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetRequiredCustomModifiersTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetValue_HasGetter_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.CanWriteTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.MetadataTokenTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.Ctor_GetterAndSetterNull_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetCustomAttributes_WithType_Test", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.Ctor_EmptyPropertyName_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.GetIndexParametersTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfoTests.IsDefinedTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.IsDefinedTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetTypesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.ProjectAssemblyTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetTypeTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetCustomAttributesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetExportedTypesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetModuleTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetManifestResourceInfoTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.EntryPoint", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetModulesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetCustomAttributesDataTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetHashCodeTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.EqualsTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetSatelliteAssemblyTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.ManifestModuleTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomAssemblyTests.GetLoadedModulesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetCustomAttributes_NoType_Test", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.MakeGenericMethodTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.Invoke_WrongObject_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetGenericMethodDefinitionTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.Invoke_NotSingleParameter_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.IsGenericMethodDefinitionTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetMethodImplementationFlagsTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetBaseDefinition", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.MethodHandleTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.ProjectionTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.Invoke_NullParameter_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.ReturnTypeCustomAttributes", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.Invoke_ValidArguments_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.CallingConventionTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.ModuleTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetParametersTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetCustomAttributesTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.IsGenericMethodTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.Invoke_NullObject_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.IsDefinedTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetCustomAttributes_WithType_Test", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.ReturnParameterTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetGenericArgumentsTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.ContainsGenericParametersTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.VirtualPropertyInfo_PropertySetter_Tests.GetCustomAttributesDataTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomPropertyInfoTests.GetCustomAttributesDataTest", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomReflectionContextTests.MapType_ParameterAttributes_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomReflectionContextTests.MapType_MemberAttributes_Success", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomReflectionContextTests.MapType_Null_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomReflectionContextTests.MapAssembly_Null_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomReflectionContextTests.Ctor_Null_Throws", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.PlatformNotSupportedException): Customized reflection contexts are only supported on .NET Framework." + }, + { + "name": "System.Reflection.Context.Tests.CustomReflectionContextTests.MapType_Interface_Throws", + "reason": "System.PlatformNotSupportedException System.PlatformNotSupportedException : Customized reflection contexts are only supported on .NET Framework." + } + ] + } + }, + { + "name": "System.Runtime.Extensions.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.IO.Tests.PathTests_Join.JoinTwoPaths", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String)'." + }, + { + "name": "System.IO.Tests.PathTests_Join.JoinThreePaths", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'System.String System.IO.Path.Join(System.String, System.String, System.String)'." + } + ] + } + }, + { + "name": "System.Runtime.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Tests.DecimalTests.GetHashCode", + "reason": "Xunit.Sdk.XunitException Decimal 3 has multiple hash codes: 1074266112 (3) and -1074266097 (3.000000000000000000000000000)" + }, + { + "name": "System.Runtime.CompilerServices.Tests.CallerArgumentExpressionAttributeTests.ArgumentToCallerArgumentExpressionSetsParameterNameProperty", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Runtime.CompilerServices.CallerArgumentExpressionAttribute' from assembly 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Tests.ArrayTests.Copy", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: Object[,] [1, 2, 3, 4, 5, ...]\\r\\nActual: Object[,] [1, 2, 3, 4, 5, ...]" + } + ] + } + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.ProcessResponse_Symantec_NoCerts_WithNonce", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.NegativeNonceIsMadePositive", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.BuildFromSignerInfo", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.BuildExpectedRequest_FromHashAndName", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.NonceLeadingZerosIgnored", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.BuildExpectedRequest_FromData", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.NoncePaddingZerosIgnored", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.BuildFromNullSignerInfo", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.TypeLoadException): Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.TryDecode_WithExtensions", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.BuildExpectedRequest_FromHashAndOid", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.ProcessResponse_FreeTsa_WithCerts_NoNonce", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.BuildWithAllOptions", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampRequestTests.EmptyNonce", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampRequest' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.EnsureExtraCertsAdded", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Unknown algorithm '1.2.840.10040.4.3'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignCmsUsingECDsaCertWithNotMatchingKeyThrows", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.Security.Cryptography.CryptographicException)\\r\\nActual: (No exception was thrown)" + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.CheckSignedEncrypted_IssuerSerial_FromNetFx", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Unable to set field EncapContentInfo on type System.Security.Cryptography.Pkcs.Asn1.SignedDataAsn.\\r\\n---- System.Security.Cryptography.CryptographicException : Unable to set field Content on type System.Security.Cryptography.Pkcs.Asn1.EncapsulatedContentInfoAsn.\\r\\n-------- System.Security.Cryptography.CryptographicException : ASN1 corrupted data." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.AddFirstSigner_DSA", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Could not determine signature algorithm for the signer certificate." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.CounterSignCmsUsingExplicitDSAKeyForFirstSignerAndECDsaForCounterSignature", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Could not determine signature algorithm for the signer certificate." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.CounterSignCmsUsingExplicitRSAKeyForFirstSignerAndDSAForCounterSignature", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Could not determine signature algorithm for the signer certificate." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignEnveloped", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\nExpected: 2\\r\\nActual: 0" + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignIdentifiedContent_BadOid", + "reason": "System.ArgumentNullException System.ArgumentNullException : Value cannot be null.\\r\\nParameter name: oidValue" + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignCmsUsingExplicitDSAKey", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Could not determine signature algorithm for the signer certificate." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignedCmsTests.SignerInfoCollection_Indexer_MinusOne", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentOutOfRangeException)\\r\\nActual: typeof(System.IndexOutOfRangeException): Index was outside the bounds of the array." + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedEmptyOctetStringWithIndefiniteLength", + "reason": "Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : Unexpected cryptographic message encoding." + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedDataWithNonPkcs7Oid", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: 3003010203\\r\\nActual: 010203\\r\\n ↑ (pos 0)" + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedOctetStringWithExtraData", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: 300102\\r\\nActual: 02\\r\\n ↑ (pos 0)" + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedEmptyArray", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: 3000\\r\\nActual: \\r\\n ↑ (pos 0)" + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedOctetStringWithDefiniteLength", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: 3003010203\\r\\nActual: 010203\\r\\n ↑ (pos 0)" + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedOctetStringWithIndefiniteLength", + "reason": "Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : Unexpected cryptographic message encoding." + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedEmptyOctetString", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: 3000\\r\\nActual: \\r\\n ↑ (pos 0)" + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingExplicitPrivateKey.DecryptEnvelopedOctetStringWithInefficientlyEncodedLength", + "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\r\\n ↓ (pos 0)\\r\\nExpected: 3003010203\\r\\nActual: 010203\\r\\n ↑ (pos 0)" + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignerInfoTests.SignerInfo_AddRemoveUnsignedAttributes_JoinCounterSignaturesAttributesIntoOne", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Could not determine signature algorithm for the signer certificate." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignerInfoTests.SignerInfo_AddUnsignedAttribute_Adds", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignerInfoTests.EnsureExtraCertsAdded", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Unknown algorithm '1.2.840.10040.4.3'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignerInfoTests.AddCounterSigner_DSA", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Could not determine signature algorithm for the signer certificate." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignerInfoTests.SignerInfo_RemoveUnsignedAttributes_MultipleAttributeValues", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.SignerInfoTests.RemoveCounterSignature_EncodedInSingleAttribute", + "reason": "System.Security.Cryptography.CryptographicException System.Security.Cryptography.CryptographicException : Unknown algorithm '1.2.840.10040.4.3'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TryDecode_Fails_Empty", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.NoEkuExtension", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.CertHashMismatchV2", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.MatchV2", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TimestampTooOld", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TryDecode_Fails_SignedCmsOfData", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TwoEkuExtensions", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TryDecode_Fails_MalformedToken", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.CertMismatchIssuerAndSerialV2", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.CertHashMismatchV1", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TimestampTooNew", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.MatchV1", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.CertMismatchIssuerAndSerialV1", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.CertMismatchV1OrV2", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.CertMatchV1AndV2", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.TryDecode_Fails_EnvelopedCms", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.NonCriticalEkuExtension", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.ParseDocument_ExcessData", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.NoTsaEku", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenTests.ParseDocument", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampToken' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingCertWithPrivateKey.DecryptEnvelopedOctetStringWithIndefiniteLength", + "reason": "Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : Unexpected cryptographic message encoding." + }, + { + "name": "System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingCertWithPrivateKey.DecryptEnvelopedEmptyOctetStringWithIndefiniteLength", + "reason": "Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : Unexpected cryptographic message encoding." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.BuilderCtor_TsaNameOptional", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.BuilderCtor_IsOrdering_Roundtrips", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.BuilderCtor_HashAlgorithmIdRequired", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.TypeLoadException): Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.Accuracy_Bounds_ParsesAsExpected", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.NegativeAccuracyThrows", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentOutOfRangeException)\\r\\nActual: typeof(System.TypeLoadException): Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.ExtensionsRoundtrips", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.AccuracyRoundtrips", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.CreateFromValue", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.BuilderCtor_AccuracyOptional", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.TryDecode_Invalid", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.CreateFromParameters", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.BuilderCtor_Timestamp_KeepsSubSeconds", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.TryDecode_LongerThanNeeded", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.BuilderCtor_PolicyIdRequired", + "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\r\\nExpected: typeof(System.ArgumentNullException)\\r\\nActual: typeof(System.TypeLoadException): Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + }, + { + "name": "System.Security.Cryptography.Pkcs.Tests.TimestampTokenInfoTests.TsaName_SameDataSecondInvocation", + "reason": "System.TypeLoadException System.TypeLoadException : Could not load type 'System.Security.Cryptography.Pkcs.Rfc3161TimestampTokenInfo' from assembly 'System.Security.Cryptography.Pkcs, Version=4.0.4.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'." + } + ] + } + }, + { + "name": "System.Security.Cryptography.Xml.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Security.Cryptography.Xml.Tests.EncryptionMethodTests.KeySize_SetNegativeValue_ThrowsArgumentOutOfRangeException", + "reason": "System.MissingMethodException System.MissingMethodException : Method not found: 'Void System.AssertExtensions.Throws(System.String, System.String, System.Func`1)'." + } + ] + } + }, + { + "name": "System.Data.SqlClient.ManualTesting.Tests", + "enabled": false, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": null + } + }, + { + "name": "System.Data.SqlClient.Stress.Tests", + "enabled": false, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": null + } + }, + { + "name": "System.Data.SqlClient.Tests", + "enabled": false, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": null + } + }, + { + "name": "System.Text.Encoding.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Text.Tests.DecoderFallbackTests.TestDecoderFallbackIndex", + "reason": "System.Text.DecoderFallbackException System.Text.DecoderFallbackException : Encountered a negative index during Utf8 decoding fallback " + } + ] + } + }, + { + "name": "System.Text.RegularExpressions.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Text.RegularExpressions.Tests.GroupCollectionReadOnlyDictionaryTests.IReadOnlyDictionary_GetEnumerator", + "reason": "System.EntryPointNotFoundException System.EntryPointNotFoundException : Entry point was not found." + }, + { + "name": "System.Text.RegularExpressions.Tests.GroupCollectionReadOnlyDictionaryTests.GetEnumerator_Invalid", + "reason": "System.EntryPointNotFoundException System.EntryPointNotFoundException : Entry point was not found." + } + ] + } + }, + { + "name": "System.Threading.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Threading.Tests.EventWaitHandleTests.OpenExisting_Windows", + "reason": "System.ArgumentException System.ArgumentException : The name 'fac694268eb248e9990777825b5aaab9aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' can be no more than 260 characters in length.\\r\\nParameter name: name" + }, + { + "name": "System.Threading.Tests.EventWaitHandleTests.Ctor_ValidNames", + "reason": "System.ArgumentException System.ArgumentException : The name '8503e07c28f04a8e8ce1c4604f6adbb2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' can be no more than 260 characters in length.\\r\\nParameter name: name" + }, + { + "name": "System.Threading.Tests.MutexTests.Ctor_ValidName", + "reason": "System.ArgumentException System.ArgumentException : The name '998ce9564d1745d8b901082f7ad17183aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' can be no more than 260 characters in length.\\r\\nParameter name: name" + }, + { + "name": "System.Threading.Tests.MutexTests.OpenExisting", + "reason": "System.ArgumentException System.ArgumentException : The name '9cc4ad4ac10046cdb05f91bf0f81ad13aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' can be no more than 260 characters in length.\\r\\nParameter name: name" + }, + { + "name": "System.Threading.Tests.SemaphoreTests.OpenExisting_SameAsOriginal_Windows", + "reason": "System.ArgumentException System.ArgumentException : The name '03ded9d0ccd74eefa101daae32e80ef8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' can be no more than 260 characters in length.\\r\\nParameter name: name" + }, + { + "name": "System.Threading.Tests.SemaphoreTests.Ctor_ValidName_Windows", + "reason": "System.ArgumentException System.ArgumentException : The name '27095bdc00264a159609fd12b7052b03aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' can be no more than 260 characters in length.\\r\\nParameter name: name" + }, + { + "name": "System.Threading.Tests.EventWaitHandleTests.Ctor_InvalidMode", + "reason": "Assert.Equal() Failure Expected: (null) Actual: mode" + } + ] + } + }, + { + "name": "System.Xml.Xsl.XslTransformApi.Tests", + "enabled": true, + "exclusions": { + "namespaces": null, + "classes": null, + "methods": [ + { + "name": "System.Xml.Tests.CXmlResolverTest.TC_AbsolutePath_Transform", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + }, + { + "name": "System.Xml.Tests.CTransformResolverTest.TC_AbsolutePath_Transform", + "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\r\\nExpected: True\\r\\nActual: False" + } + ] + } + } +] \ No newline at end of file diff --git a/src/coreclr/tests/runtest.cmd b/src/coreclr/tests/runtest.cmd index 89a5acd..368451a 100644 --- a/src/coreclr/tests/runtest.cmd +++ b/src/coreclr/tests/runtest.cmd @@ -41,6 +41,7 @@ set __DoCrossgen= set __CrossgenAltJit= set __PerfTests= set __CoreFXTests= +set __CoreFXTestsRunAllAvailable= :Arg_Loop if "%1" == "" goto ArgsDone @@ -79,7 +80,9 @@ if /i "%1" == "jitdisasm" (set __JitDisasm=1&shift&goto Arg_Loop) if /i "%1" == "ilasmroundtrip" (set __IlasmRoundTrip=1&shift&goto Arg_Loop) if /i "%1" == "GenerateLayoutOnly" (set __GenerateLayoutOnly=1&shift&goto Arg_Loop) if /i "%1" == "PerfTests" (set __PerfTests=true&shift&goto Arg_Loop) -if /i "%1" == "CoreFXTests" (exit /b 0) +if /i "%1" == "CoreFXTests" (set __CoreFXTests=true&shift&goto Arg_Loop) +if /i "%1" == "CoreFXTestsAll" (set __CoreFXTests=true&set __CoreFXTestsRunAllAvailable=true&shift&goto Arg_Loop) +if /i "%1" == "CoreFXTestList" (set __CoreFXTests=true&set __CoreFXTestList=%2&shift&shift&goto Arg_Loop) if /i "%1" == "runcrossgentests" (set RunCrossGen=true&shift&goto Arg_Loop) if /i "%1" == "link" (set DoLink=true&set ILLINK=%2&shift&shift&goto Arg_Loop) if /i "%1" == "tieredcompilation" (set COMPLUS_TieredCompilation=1&shift&goto Arg_Loop) @@ -107,7 +110,15 @@ shift :: Done with argument processing. Check argument values for validity. if defined __TestEnv (if not exist %__TestEnv% echo %__MsgPrefix%Error: Test Environment script %__TestEnv% not found && exit /b 1) -if "%__PerfTests%"=="true" (if defined __GenerateLayoutOnly echo %__MsgPrefix%Error: Don't specify both "PerfTests" and "GenerateLayoutOnly" && exit /b 1) +if "%__PerfTests%"=="true" (if defined __GenerateLayoutOnly echo %__MsgPrefix%Error: Don't specify both "PerfTests" and "GenerateLayoutOnly" && exit /b 1) +if "%__CoreFXTests%"=="true" (if defined __GenerateLayoutOnly echo %__MsgPrefix%Error: Don't specify both "CoreFXTests" and "GenerateLayoutOnly" && exit /b 1) + +if defined __CoreFXTestList ( + if not exist "%__CoreFXTestList%" ( + echo %__MsgPrefix%Error: Couldn't find CoreFX Test List "%__CoreFXTestList%". + exit /b 1 + ) +) :: Set the remaining variables based upon the determined configuration set "__BinDir=%__RootBinDir%\Product\%__BuildOS%.%__BuildArch%.%__BuildType%" @@ -203,6 +214,8 @@ if not exist %CORE_ROOT%\coreclr.dll ( exit /b 1 ) +if "%__CoreFXTests%"=="true" goto RunCoreFXTests + if "%__PerfTests%"=="true" goto RunPerfTests REM ========================================================================================= @@ -285,7 +298,63 @@ if errorlevel 1 ( echo %__MsgPrefix%Test Run failed. Refer to the following: echo Html report: %__TestRunHtmlLog% ) +goto TestsDone + +REM ========================================================================================= +REM === +REM === Run CoreFX tests +REM === +REM ========================================================================================= +:RunCoreFXTests +set _CoreFXTestHost=%XunitTestBinBase%\testhost +set _toolsDir=%__ProjectDir%\..\Tools +set _dotnet=%_toolsDir%\dotnetcli\dotnet.exe + +set _RootCoreFXTestPath=%__TestWorkingDir%\CoreFX +set _CoreFXTestUtilitiesOutputPath=%_RootCoreFXTestPath%\CoreFXTestUtilities +set _CoreFXTestBinariesPath=%_RootCoreFXTestPath%\tests_downloaded +set _CoreFXLogsDir=%__LogsDir%\CoreFX\%__BuildOS%.%__BuildArch%.%__BuildType% +if not exist "%_CoreFXLogsDir%" (mkdir "%_CoreFXLogsDir%") + +set _CoreFXTestSetupUtilityName=CoreFX.TestUtils.TestFileSetup +set _CoreFXTestSetupUtility=%__ProjectFilesDir%\src\Common\CoreFX\TestFileSetup\%_CoreFXTestSetupUtilityName%.csproj + +call :ResolveDependencies +if errorlevel 1 exit /b 1 + +if defined __GenerateTestHostOnly ( + exit /b 0 +) + +if not exist %_CoreFXTestHost%\dotnet.exe echo CoreFX test host not found, please run runtest.cmd again && exit /b 1 + +set /p _CoreFXTestRemoteURL=< "%__ProjectFilesDir%\CoreFX\CoreFXTestListURL.txt" +if not defined __CoreFXTestList ( set __CoreFXTestList=%__ProjectFilesDir%\CoreFX\TopN.CoreFX.x64.Windows.issues.json ) + + +set _CoreFXTestExecutable=xunit.console.netcore.exe +set _CoreFXTestExecutableArgs= --notrait category=nonnetcoreapptests --notrait category=nonwindowstests --notrait category=failing --notrait category=IgnoreForCI --notrait category=OuterLoop --notrait Benchmark=true + +REM Set the log file name to something Jenkins can understand +set _CoreFX_TestLogFileName=testResults.xml +set _CoreFX_TestRunScriptName=CoreCLR_RunTest.cmd +if "%__CoreFXTestsRunAllAvailable%" == "true" ( + set _CoreFX_RunCommand=--runAllTests +) else ( + set _CoreFX_RunCommand=--runSpecifiedTests +) + + +echo Downloading and Running CoreFX Test Binaries +echo %__MsgPrefix%call "%_dotnet%" "%_CoreFXTestUtilitiesOutputPath%\%_CoreFXTestSetupUtilityName%.dll" --clean --outputDirectory "%_CoreFXTestBinariesPath%" --testListJsonPath "%__CoreFXTestList%" --testUrl "!_CoreFXTestRemoteURL!" %_CoreFX_RunCommand% --dotnetPath "%_CoreFXTestHost%\dotnet.exe" --executable %_CoreFXTestExecutable% --logPath %_CoreFXLogsDir% %_CoreFXTestExecutableArgs% +call "%_dotnet%" "%_CoreFXTestUtilitiesOutputPath%\%_CoreFXTestSetupUtilityName%.dll" --clean --outputDirectory "%_CoreFXTestBinariesPath%" --testListJsonPath "%__CoreFXTestList%" --testUrl "!_CoreFXTestRemoteURL!" %_CoreFX_RunCommand% --dotnetPath "%_CoreFXTestHost%\dotnet.exe" --executable %_CoreFXTestExecutable% --log %_CoreFXLogsDir% %_CoreFXTestExecutableArgs% +if errorlevel 1 ( + echo %__MsgPrefix%Running CoreFX tests finished with Failures + echo %__MsgPrefix%Check %_CoreFXLogsDir% for test run logs + exit /b 1 +) +) goto TestsDone REM ========================================================================================= @@ -473,6 +542,39 @@ if errorlevel 1 ( ) echo %__MsgPrefix%Created the runtime layout with all dependencies in %CORE_ROOT% +if "%__CoreFXTests%"=="true" goto ResolveCoreFXDependencies + +exit /b 0 + +REM ========================================================================================= +REM === +REM === Generate the "testhost" directory for running CoreFX tests; download dependencies. +REM === +REM ========================================================================================= + +:ResolveCoreFXDependencies +set __BuildLogRootName=Tests_GenerateTestHost +echo %__MsgPrefix%Building CoreFX Test Host +call :msbuild "%__ProjectFilesDir%\runtest.proj" /p:GenerateTestHost=true +if errorlevel 1 ( + echo %__MsgPrefix%Test Host Dependency Resolution Failed + exit /b 1 +) +echo %__MsgPrefix%Created the Test Host layout with all dependencies in %_CoreFXTestHost% + +REM Publish and call the CoreFX test helper projects - should this be integrated into runtest.proj? +REM Build Helper project +echo "%_dotnet%" msbuild /t:Restore "%_CoreFXTestSetupUtility%" +call "%_dotnet%" msbuild /t:Restore "%_CoreFXTestSetupUtility%" +if errorlevel 1 ( + exit /b 1 +) +echo "%_dotnet%" msbuild "/p:Configuration=%CoreRT_BuildType%" "/p:OSGroup=%CoreRT_BuildOS%" "/p:Platform=%CoreRT_BuildArch%" "/p:OutputPath=%_CoreFXTestUtilitiesOutputPath%" "%_CoreFXTestSetupUtility%" +call "%_dotnet%" msbuild "/p:Configuration=%CoreRT_BuildType%" "/p:OSGroup=%CoreRT_BuildOS%" "/p:Platform=%CoreRT_BuildArch%" "/p:OutputPath=%_CoreFXTestUtilitiesOutputPath%" "%_CoreFXTestSetupUtility%" +if errorlevel 1 ( + exit /b 1 +) + exit /b 0 REM ========================================================================================= diff --git a/src/coreclr/tests/runtest.proj b/src/coreclr/tests/runtest.proj index 3ab7b9f..aaff5b1 100644 --- a/src/coreclr/tests/runtest.proj +++ b/src/coreclr/tests/runtest.proj @@ -10,6 +10,14 @@ 1 + + + 9.9.9 + $(BinDir)testhost\ + $(TestHostRootPath)host\fxr\$(NETCoreAppTestSharedFxVersion)\ + $(TestHostRootPath)shared\Microsoft.NETCore.App\$(NETCoreAppTestSharedFxVersion)\ + + <_SkipTestDir Include="@(DisabledTestDir)" /> @@ -363,6 +371,91 @@ namespace $([System.String]::Copy($(Category)).Replace(".","_").Replace("\",""). Condition=" '$(BuildTestsAgainstPackages)'=='true' " /> + + + + + + + + + + + + + + + + + + + <_sharedFrameworkDepsJson Include="$(ToolsDir)dotnetcli\shared\Microsoft.NETCore.App\*\Microsoft.NETCore.App.deps.json" /> + + + + <_OriginalDepsJsonPath>%(_sharedFrameworkDepsJson.FullPath) + <_OutputTestSharedFrameworkDepsPath>$(NETCoreAppTestSharedFrameworkPath)\Microsoft.NETCore.App.deps.json + + + + + + + + + hostfxr + dotnet.exe + dll + + + + libhostfxr + so + dylib + dotnet + + + + + + + + + + + + + + + + + + + + + + + + @@ -396,6 +489,10 @@ namespace $([System.String]::Copy($(Category)).Replace(".","_").Replace("\",""). Targets="CreateTestOverlay" Condition=" '$(GenerateRuntimeLayout)'=='true' "/> + + diff --git a/src/coreclr/tests/src/Common/CoreFX/CoreFX.depproj b/src/coreclr/tests/src/Common/CoreFX/CoreFX.depproj new file mode 100644 index 0000000..5863b19 --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/CoreFX.depproj @@ -0,0 +1,173 @@ + + + + + $(SourceDir)Common\CoreFX\obj + C# + .NETCoreApp,Version=v2.0 + netcoreapp2.0 + true + SharedLibrary + false + + 1.3.0-preview3-26501-04 + 2.2.0-preview1-02902-01 + + 2.0.1 + 2.2.0-beta2-build3300 + + + + + win-x64 + linux-x64 + osx-x64 + $(RuntimeIdentifier) + + + + + $(CoreFXXUnitPackageVersion) + + + $(CoreFXXUnitPackageVersion) + + + $(CoreFXXUnitPackageVersion) + + + $(XUnitAbstractionsVersion) + + + $(CoreFXXUnitPackageVersion) + + + $(CoreFXXUnitPackageVersion) + + + $(CoreFXXUnitPackageVersion) + + + $(XUnitNetcoreExtensionsVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(SystemCompositionVersions) + + + $(SystemCompositionVersions) + + + $(SystemCompositionVersions) + + + $(SystemCompositionVersions) + + + $(SystemCompositionVersions) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + + + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + $(MicrosoftPrivateCoreFxNETCoreAppPackageVersion) + + + + netcoreapp2.2 + + + + + + + $(SourceDir)Common\CoreFX\obj\project.assets.json + + diff --git a/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/CoreFX.TestUtils.TestFileSetup.csproj b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/CoreFX.TestUtils.TestFileSetup.csproj new file mode 100644 index 0000000..90a5cf1 --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/CoreFX.TestUtils.TestFileSetup.csproj @@ -0,0 +1,35 @@ + + + + 0.1.0-e160909-1 + 11.0.2 + 3.0.10 + 2.3.0-beta1-build3642 + + 2.2.0-beta2-build3300 + 2.0.1 + 2.1.0-preview2-02516-02 + 4.5.0-preview2-26219-0 + $(RestoreSources);https://dotnet.myget.org/F/dotnet-corefxlab/api/v3/index.json + + + + netcoreapp2.0 + Exe + 99.9 + + + + + $(NewtonsoftJsonVersion) + + + $(NewtonsoftJsonSchemaVersion) + + + + $(SystemCommandLineVersion) + + + diff --git a/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Helpers/TestFileHelper.cs b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Helpers/TestFileHelper.cs new file mode 100644 index 0000000..c30b97d --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Helpers/TestFileHelper.cs @@ -0,0 +1,326 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Collections.Generic; +using System.CommandLine; +using System.Diagnostics; +using System.IO; +using System.IO.Compression; +using System.Net.Http; +using System.Text; +using System.Threading.Tasks; +using Newtonsoft.Json; +using Newtonsoft.Json.Schema; +using Newtonsoft.Json.Schema.Generation; + +namespace CoreFX.TestUtils.TestFileSetup.Helpers +{ + /// + /// Defines the set of flags that represent exit codes + /// + [Flags] + public enum ExitCode : int + { + Success = 0, + TestFailure = 1, + HttpError = 2, + IOError = 3, + JsonSchemaValidationError = 4, + UnknownError = 10 + + } + + /// + /// This helper class is used to fetch CoreFX tests from a specified URL, unarchive them and create a flat directory structure + /// through which to iterate. + /// + public class TestFileHelper + { + private HttpClient httpClient; + public HttpClient HttpClient + { + get + { + if (httpClient == null) + { + httpClient = new HttpClient(); + } + return httpClient; + } + set{ httpClient = value; } + } + + private HashSet disabledTests; + + /// + /// Default constructor - initialize list of disabled tests + /// + public TestFileHelper() { + disabledTests = new HashSet(); + } + + /// + /// Deserialize a list of JSON objects defining test assemblies + /// + /// The path on disk to the test list. The test list must conform to a schema generated from XUnitTestAssembly + /// + public Dictionary DeserializeTestJson(string testDefinitionFilePath) + { + JSchemaGenerator jsonGenerator = new JSchemaGenerator(); + + // Generate a JSON schema from the XUnitTestAssembly class against which to validate the test list + JSchema testDefinitionSchema = jsonGenerator.Generate(typeof(IList)); + IList testAssemblies = new List(); + + IList validationMessages = new List(); + + using (var sr = new StreamReader(testDefinitionFilePath)) + using (var jsonReader = new JsonTextReader(sr)) + using (var jsonValidationReader = new JSchemaValidatingReader(jsonReader)) + { + // Create schema validator + jsonValidationReader.Schema = testDefinitionSchema; + jsonValidationReader.ValidationEventHandler += (o, a) => validationMessages.Add(a.Message); + + // Deserialize json test assembly definitions + JsonSerializer serializer = new JsonSerializer(); + try + { + testAssemblies = serializer.Deserialize>(jsonValidationReader); + } + catch (JsonSerializationException ex) + { + // Invalid definition + throw new AggregateException(ex); + } + } + + if (validationMessages.Count != 0) + { + StringBuilder aggregateExceptionMessage = new StringBuilder(); + foreach (string validationMessage in validationMessages) + { + aggregateExceptionMessage.Append("JSON Validation Error: "); + aggregateExceptionMessage.Append(validationMessage); + aggregateExceptionMessage.AppendLine(); + } + + throw new AggregateException(new JSchemaValidationException(aggregateExceptionMessage.ToString())); + + } + // Generate a map of test assembly names to their object representations - this is used to download and match them to their on-disk representations + var nameToTestAssemblyDef = new Dictionary(); + + // Map test names to their definitions + foreach (XUnitTestAssembly assembly in testAssemblies) + { + // Filter disabled tests + if(assembly.IsEnabled) + nameToTestAssemblyDef.Add(assembly.Name, assembly); + else + disabledTests.Add(assembly.Name); + } + + return nameToTestAssemblyDef; + } + + /// + /// Layout tests on disk. This method sets up every downloaded test as it would appear after running build-test.[cmd/sh] in CoreFX + /// + /// URL to a test list - we expect a test list, which conforms to the Helix layout + /// Directory to which the tests are downloaded + /// The mapping of tests parsed from a test definition list to their names + /// Optional argument, which denotes whether all tests available in the test list downloaded from jsonUrl should be run + /// + public async Task SetupTests(string jsonUrl, string destinationDirectory, Dictionary testDefinitions = null, bool runAllTests = false) + { + Debug.Assert(Directory.Exists(destinationDirectory)); + // testDefinitions should not be empty unless we're running all tests with no exclusions + Debug.Assert(runAllTests || testDefinitions != null); + + // Download archives to a temporary directory + string tempDirPath = Path.Combine(destinationDirectory, "temp"); + if (!Directory.Exists(tempDirPath)) + { + Directory.CreateDirectory(tempDirPath); + } + // Map test names to their URLs, specified by the test list found at jsonUrl + Dictionary testPayloads = await GetTestUrls(jsonUrl, testDefinitions, runAllTests); + + // If none were found or the testList did not have the expected format - return + if (testPayloads == null) + { + return; + } + + // Download and unzip tests + await GetTestArchives(testPayloads, tempDirPath); + ExpandArchivesInDirectory(tempDirPath, destinationDirectory); + + // Generate response file for each tests + RSPGenerator rspGenerator = new RSPGenerator(); + foreach (XUnitTestAssembly assembly in testDefinitions.Values) + { + rspGenerator.GenerateRSPFile(assembly, Path.Combine(destinationDirectory, assembly.Name)); + } + + Directory.Delete(tempDirPath); + } + + /// + /// Maps test names to their respective URLs as found in the test list found at the specified URL + /// + /// URL to a test list - we expect a test list, which conforms to the Helix layout + /// The mapping of tests parsed from a test definition list to their names + /// Optional argument, which denotes whether all tests available in the test list downloaded from jsonUrl should be run + /// + public async Task> GetTestUrls(string jsonUrl, Dictionary testDefinitions = null, bool runAllTests = false) + { + // testDefinitions should not be empty unless we're running all tests with no exclusions + Debug.Assert(runAllTests || testDefinitions != null); + // Set up the json stream reader + using (var responseStream = await HttpClient.GetStreamAsync(jsonUrl)) + using (var streamReader = new StreamReader(responseStream)) + using (var jsonReader = new JsonTextReader(streamReader)) + { + // Manual parsing - we only need to key-value pairs from each object and this avoids deserializing all of the work items into objects + string markedTestName = string.Empty; + string currentPropertyName = string.Empty; + + // The expected layout is produced by regular Helix runs - this allows us to parse and run tests from any Helix test list without special considerations + // The expected fields are + // { "WorkItemId": "" , "PayloadUri":"" } + + while (jsonReader.Read()) + { + if (jsonReader.Value != null) + { + switch (jsonReader.TokenType) + { + case JsonToken.PropertyName: + currentPropertyName = jsonReader.Value.ToString(); + break; + case JsonToken.String: + // Test Name Value + if (currentPropertyName.Equals("WorkItemId")) + { + string currentTestName = jsonReader.Value.ToString(); + + // If the test has been marked as disabled in the test list - ignore it + if ((runAllTests || testDefinitions.ContainsKey(currentTestName)) && !disabledTests.Contains(currentTestName)) + { + markedTestName = currentTestName; + } + } + // Test URL value + else if (currentPropertyName.Equals("PayloadUri") && markedTestName != string.Empty) + { + if (!testDefinitions.ContainsKey(markedTestName)) + { + testDefinitions[markedTestName] = new XUnitTestAssembly() { Name = markedTestName }; + } + testDefinitions[markedTestName].Url = jsonReader.Value.ToString(); + markedTestName = string.Empty; + } + break; + } + } + } + + } + return testDefinitions; + } + + /// + /// Download each test from its specified URL + /// + /// The mapping of tests parsed from a test definition list to their names. The test definitions are populated with test URLs + /// Directory to which to download tests + /// + public async Task GetTestArchives(Dictionary testPayloads, string downloadDir) + { + foreach (string testName in testPayloads.Keys) + { + string payloadUri = testPayloads[testName].Url; + + // Check URL for validity + if (!Uri.IsWellFormedUriString(payloadUri, UriKind.Absolute)) + continue; + + // Download tests from specified URL + using (var response = await HttpClient.GetStreamAsync(payloadUri)) + { + if (response.CanRead) + { + // Create the test setup directory if it doesn't exist + if (!Directory.Exists(downloadDir)) + { + Directory.CreateDirectory(downloadDir); + } + + // CoreFX test archives are output as .zip regardless of platform + string archivePath = Path.Combine(downloadDir, testName + ".zip"); + + // Copy to a temp folder + using (FileStream file = new FileStream(archivePath, FileMode.Create)) + { + await response.CopyToAsync(file); + } + + } + } + } + } + + /// + /// Expand Archives + /// + /// Directory containing archives + /// Directory to which to unpack archives + /// Optional parameter stating, whether archives should be deleted once downloaded + public void ExpandArchivesInDirectory(string archiveDirectory, string destinationDirectory, bool cleanup = true) + { + Debug.Assert(Directory.Exists(archiveDirectory)); + Debug.Assert(Directory.Exists(destinationDirectory)); + + // Get all archives in the directory + string[] archives = Directory.GetFiles(archiveDirectory, "*.zip", SearchOption.TopDirectoryOnly); + + foreach (string archivePath in archives) + { + string destinationDirName = Path.Combine(destinationDirectory, Path.GetFileNameWithoutExtension(archivePath)); + + ZipFile.ExtractToDirectory(archivePath, destinationDirName); + + // Delete archives if cleanup was + if (cleanup) + { + File.Delete(archivePath); + } + } + } + + /// + /// Cleans build directory + /// + /// Directory the contents of which to delete. + public void CleanBuild(string directoryToClean) + { + Debug.Assert(Directory.Exists(directoryToClean)); + DirectoryInfo dirInfo = new DirectoryInfo(directoryToClean); + + foreach (FileInfo file in dirInfo.EnumerateFiles()) + { + file.Delete(); + } + + foreach (DirectoryInfo dir in dirInfo.EnumerateDirectories()) + { + dir.Delete(true); + } + } + + } +} diff --git a/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Helpers/TestRunHelper.cs b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Helpers/TestRunHelper.cs new file mode 100644 index 0000000..a8a9791 --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Helpers/TestRunHelper.cs @@ -0,0 +1,158 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Runtime.InteropServices; +using System.Text; +using System.Threading.Tasks; + +namespace CoreFX.TestUtils.TestFileSetup.Helpers +{ + /// + /// A class which runs all tests conforming to the current format of CoreFX tests - + /// Each folder: + /// is named for the test it contains - e.g. System.Collections.Tests + /// contains a test assembly named for the library it tests - e.g. System.Collections.Tests.dll + /// contains a test executable with the specified name - e.g. xunit.console.netcore.exe + /// + public class NetCoreTestRunHelper + { + + public string DotnetExecutablePath { get; set; } + + public string logRootOutputPath { get; set; } + + /// + /// Default Constructor + /// + /// Path to the dotnet executable, which is used to run the test executable In CoreFX tests this will be the built test host + /// Path to which to output test run logs + public NetCoreTestRunHelper(string DotnetExecutablePath, string logRootOutputPath) + { + this.DotnetExecutablePath = DotnetExecutablePath; + this.logRootOutputPath = logRootOutputPath; + } + + /// + /// Run a single test executabke + /// + /// Directory from which to start the test executable + /// Name of the test executable + /// Test trait exclusions to pass to the runner. + /// Path to which to output the single test run's results + /// 0 if single test run is succesful; 1 if not + public int RunExecutable(string workingDirectory, string executableName, IReadOnlyList xunitTestTraits, string logOutputPath) + { + // Calculate and create the path to the test log + string logPath = Path.Combine(logOutputPath, Path.GetFileName(workingDirectory)); + if (!Directory.Exists(logPath)) + Directory.CreateDirectory(logPath); + + // Calculate the arguments to pass to the test runner + string arguments = CalculateCommandLineArguments(workingDirectory, executableName, xunitTestTraits, Path.Combine(logPath,"testResults.xml")); + + // Create and initialize the test executable process + ProcessStartInfo startInfo = new ProcessStartInfo(DotnetExecutablePath, arguments) + { + Arguments = arguments, + WorkingDirectory = workingDirectory + }; + + + Process executableProcess = new Process(); + executableProcess.StartInfo = startInfo; + executableProcess.EnableRaisingEvents = true; + executableProcess.Start(); + executableProcess.WaitForExit(); + + return executableProcess.ExitCode; + } + /// + /// Run all test executables conforming to the specified pattern in a directory + /// + /// Directory containing tests to run + /// Name of the test executable contained in folders + /// Test trait exclusions to pass to the runner. + /// Maximum number of tests to run in parallel + /// Root path to which to output the all test runs' results + /// 0 if entire test run is succesful; 1 if not + public int RunAllExecutablesInDirectory(string rootDirectory, string executableName, IReadOnlyList xunitTestTraits, int processLimit, string logRootOutputPath = null) + { + int result = 0; + // Do a Depth-First Search to find and run executables with the same name + Stack directories = new Stack(); + List testDirectories = new List(); + // Push rootdir + directories.Push(rootDirectory); + + while (directories.Count > 0) + { + string currentDirectory = directories.Pop(); + + // If a directory contains an executable with the specified name - add it + if (File.Exists(Path.Combine(currentDirectory, executableName))) + testDirectories.Add(currentDirectory); + + foreach (string subDir in Directory.GetDirectories(currentDirectory)) + directories.Push(subDir); + } + + // Initialize max degree of parallelism + ParallelOptions parallelOptions = new ParallelOptions(); + parallelOptions.MaxDegreeOfParallelism = processLimit; + + Parallel.ForEach(testDirectories, parallelOptions, + (testDirectory) => + { + if (RunExecutable(testDirectory, executableName, xunitTestTraits, logRootOutputPath) != 0) + { + // If any tests fail mark the whole run as failed + Console.WriteLine("Test Run Failed " + testDirectory); + result = 1; + } + } + ); + return result; + } + + /// + /// Calculate the commandline arguments to pass to the test executable + /// + /// Current test directory name - assumed to have the same name as the test + /// >Name of the test executable contained in the folder + /// Test trait exclusions to pass to the runner. + /// Path to which to output the single test run's results + /// A string representing command line arguments to be passed to the console test runner + private string CalculateCommandLineArguments(string testDirectory, string executableName, IReadOnlyList xunitTestTraits, string logPath) + { + StringBuilder arguments = new StringBuilder(); + + // Append test executable name + arguments.Append($"\"{Path.Combine(testDirectory, Path.GetFileName(executableName))}\" "); + + // Append test name dll\ + arguments.Append($"\"{Path.Combine(testDirectory, Path.GetFileName(testDirectory))}.dll\" "); + + // Append RSP file + arguments.Append($"@\"{Path.Combine(testDirectory, Path.GetFileName(testDirectory))}.rsp\" "); + + if (!String.IsNullOrEmpty(logPath)) + { + // Add logging information + arguments.Append($"-xml {logPath} "); + } + + // Append all additional arguments + foreach (string traitToExclude in xunitTestTraits) + { + arguments.Append($"-notrait {traitToExclude} "); + } + + return arguments.ToString(); + } + } +} diff --git a/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Program.cs b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Program.cs new file mode 100644 index 0000000..107f242 --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/Program.cs @@ -0,0 +1,176 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Collections.Generic; +using System.CommandLine; +using System.IO; +using System.Net.Http; +using System.Text; +using CoreFX.TestUtils.TestFileSetup.Helpers; +using Newtonsoft.Json; +using Newtonsoft.Json.Schema; + +namespace CoreFX.TestUtils.TestFileSetup +{ + /// + /// This is a driver class, which downloads archived CoreFX test assemblies from a specified URL, lays out their contents + /// and subsequently runs them in parallel. + /// This is invoked from .tests\runtests.cmd and depends on a test host (CoreCLR components with a dotnet executable) being present + /// + public class Program + { + // Helper class to lay out files on disk + private static TestFileHelper testFileHelper; + // Helper class to run tests in parallel + private static NetCoreTestRunHelper testRunHelper; + + // Test Set-up Options + private static string outputDir; + private static string testUrl; + private static string testListPath; + private static bool cleanTestBuild; + + // Test Run Options + private static string dotnetPath; + private static bool runSpecifiedTests; + private static bool runAllTests; + private static int maximumDegreeOfParalellization; + private static string logRootOutputPath; + + private static ExitCode exitCode; + private static string executableName; + private static IReadOnlyList traitExclusions = Array.Empty(); + + public static void Main(string[] args) + { + // Initialize default options + exitCode = ExitCode.Success; + maximumDegreeOfParalellization = Environment.ProcessorCount; + runSpecifiedTests = false; + runAllTests = false; + cleanTestBuild = false; + + ArgumentSyntax argSyntax = ParseCommandLine(args); + try + { + // Download and lay out files on disk + SetupTests(runAllTests); + + // Only run tests if the relevant commandline switch is passed + if (runSpecifiedTests || runAllTests) + { + exitCode = RunTests(); + + } + } + catch (AggregateException e) + { + // Handle failure cases and exit gracefully + e.Handle(innerExc => + { + + if (innerExc is HttpRequestException) + { + exitCode = ExitCode.HttpError; + Console.WriteLine("Error downloading tests from: " + testUrl); + Console.WriteLine(innerExc.Message); + return true; + } + else if (innerExc is IOException) + { + exitCode = ExitCode.IOError; + Console.WriteLine(innerExc.Message); + return true; + } + else if (innerExc is JSchemaValidationException || innerExc is JsonSerializationException) + { + exitCode = ExitCode.JsonSchemaValidationError; + Console.WriteLine("Error validating test list: "); + Console.WriteLine(innerExc.Message); + return true; + } + else + { + exitCode = ExitCode.UnknownError; + } + return false; + }); + } + + Environment.Exit((int)exitCode); + } + + /// + /// Parse passed Command Line arguments + /// + /// + /// + private static ArgumentSyntax ParseCommandLine(string[] args) + { + ArgumentSyntax argSyntax = ArgumentSyntax.Parse(args, syntax => + { + syntax.DefineOption("out|outDir|outputDirectory", ref outputDir, "Directory where tests are downloaded"); + syntax.DefineOption("testUrl", ref testUrl, "URL, pointing to the list of tests"); + syntax.DefineOption("testListJsonPath", ref testListPath, "JSON-formatted list of test assembly names to download"); + syntax.DefineOption("clean|cleanOutputDir", ref cleanTestBuild, "Clean test assembly output directory"); + syntax.DefineOption("runSpecified|runSpecifiedTests", ref runSpecifiedTests, "Run specified Tests after setup"); + syntax.DefineOption("runAll|runAllTests", ref runAllTests, "Run All available Tests in the specified TestList"); + syntax.DefineOption("dotnet|dotnetPath", ref dotnetPath, "Path to dotnet executable used to run tests."); + syntax.DefineOption("executable|executableName", ref executableName, "Name of the test executable to start"); + syntax.DefineOption("log|logPath|logRootOutputPath", ref logRootOutputPath, "Run Tests after setup"); + syntax.DefineOption("maxProcessCount|numberOfParallelTests|maximumDegreeOfParalellization", ref maximumDegreeOfParalellization, "Maximum number of concurrently executing processes"); + syntax.DefineOptionList("notrait", ref traitExclusions, "Traits to be excluded from test runs"); + + }); + + if (runSpecifiedTests || runAllTests) + { + if (String.IsNullOrEmpty(dotnetPath)) + throw new ArgumentException("Please supply a test host location to run tests."); + + if (!File.Exists(dotnetPath)) + throw new ArgumentException("Invalid testhost path. Please supply a test host location to run tests."); + } + + return argSyntax; + } + + /// + /// Method, which calls into the Test File Setup helper class to download and layout test assemblies on disk + /// + /// Specifies whether all tests available in the test list should be run + private static void SetupTests(bool runAll = false) + { + testFileHelper = new TestFileHelper(); + + if (!Directory.Exists(outputDir)) + Directory.CreateDirectory(outputDir); + + // If the --clean switch has been specified delete existing assemblies + if (cleanTestBuild) + { + testFileHelper.CleanBuild(outputDir); + } + + // Map test names to their definitions + Dictionary testAssemblyDefinitions = testFileHelper.DeserializeTestJson(testListPath); + + testFileHelper.SetupTests(testUrl, outputDir, testAssemblyDefinitions, runAll).Wait(); + } + + /// + /// Runs all tests in a directory + /// Only tests, the executable driver of which has the same name as the passed argument are executed - e.g. xunit.console.netcore.exe + /// + /// + private static ExitCode RunTests() + { + testRunHelper = new NetCoreTestRunHelper(dotnetPath, logRootOutputPath); + int result = testRunHelper.RunAllExecutablesInDirectory(outputDir, executableName, traitExclusions, maximumDegreeOfParalellization, logRootOutputPath); + + return result == 0 ? ExitCode.Success : ExitCode.TestFailure; + } + } +} diff --git a/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/RSPGenerator.cs b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/RSPGenerator.cs new file mode 100644 index 0000000..ae32302 --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/RSPGenerator.cs @@ -0,0 +1,83 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Collections.Generic; +using System.IO; +using System.Text; + +namespace CoreFX.TestUtils.TestFileSetup +{ + /// + /// A class which generates .rsp files to be passed to the test executable + /// The file contains a list of methods, classes and namespaces to be excluded from running. + /// + public class RSPGenerator + { + /// + /// Generate an rsp file from an XUnitTestAssembly class + /// + /// The XUnitTestAssembly object parsed from a specified test list + /// Path to which to output a .rsp file + public void GenerateRSPFile(XUnitTestAssembly testDefinition, string outputPath) + { + if (!Directory.Exists(outputPath)) + { + Directory.CreateDirectory(outputPath); + } + string rspFilePath = Path.Combine(outputPath, testDefinition.Name + ".rsp"); + + if (File.Exists(rspFilePath)) + File.Delete(rspFilePath); + + // Write RSP file to disk + using (StreamWriter sr = File.CreateText(rspFilePath)) + { + // If no exclusions are defined, we don't need to generate an .rsp file + if (testDefinition.Exclusions == null) + return; + + // Method exclusions + if (testDefinition.Exclusions.Methods != null) + { + foreach (Exclusion exclusion in testDefinition.Exclusions.Methods) + { + if (String.IsNullOrWhiteSpace(exclusion.Name)) + continue; + sr.Write("-skipmethod "); + sr.Write(exclusion.Name); + sr.WriteLine(); + } + } + + // Class exclusions + if (testDefinition.Exclusions.Classes != null) + { + foreach (Exclusion exclusion in testDefinition.Exclusions.Classes) + { + if (String.IsNullOrWhiteSpace(exclusion.Name)) + continue; + sr.Write("-skipclass "); + sr.Write(exclusion.Name); + sr.WriteLine(); + } + + } + + // Namespace exclusions + if (testDefinition.Exclusions.Namespaces != null) + { + foreach (Exclusion exclusion in testDefinition.Exclusions.Namespaces) + { + if (String.IsNullOrWhiteSpace(exclusion.Name)) + continue; + sr.Write("-skipnamespace "); + sr.Write(exclusion.Name); + sr.WriteLine(); + } + } + } + } + } +} diff --git a/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/XUnit/XUnitTestAssembly.cs b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/XUnit/XUnitTestAssembly.cs new file mode 100644 index 0000000..efe7885 --- /dev/null +++ b/src/coreclr/tests/src/Common/CoreFX/TestFileSetup/XUnit/XUnitTestAssembly.cs @@ -0,0 +1,63 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Collections.Generic; +using System.Text; +using Newtonsoft.Json; + +namespace CoreFX.TestUtils.TestFileSetup +{ + /// + /// A class representing a CoreFX test assembly to be run + /// + public class XUnitTestAssembly + { + [JsonRequired] + [JsonProperty("name")] + public string Name; + + [JsonRequired] + [JsonProperty("enabled")] + public bool IsEnabled; + + [JsonRequired] + [JsonProperty("exclusions")] + public Exclusions Exclusions; + + // Used to assign a test url or to override it via the json file definition - mark it as optional in the test definition + [JsonIgnore] + [JsonProperty(Required = Required.Default)] + public string Url; + + } + /// + /// Class representing a collection of test exclusions + /// + public class Exclusions + { + [JsonProperty("namespaces")] + public Exclusion[] Namespaces; + + [JsonProperty("classes")] + public Exclusion[] Classes; + + [JsonProperty("methods")] + public Exclusion[] Methods; + } + + /// + /// Class representing a single test exclusion + /// + public class Exclusion + { + [JsonRequired] + [JsonProperty("name", Required = Required.DisallowNull)] + public string Name; + + [JsonRequired] + [JsonProperty("reason", Required = Required.DisallowNull)] + public string Reason; + } +} diff --git a/src/coreclr/tests/src/dirs.proj b/src/coreclr/tests/src/dirs.proj index 6009e2e..58c3559 100644 --- a/src/coreclr/tests/src/dirs.proj +++ b/src/coreclr/tests/src/dirs.proj @@ -20,6 +20,7 @@ +