[run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) will clone dotnet/corefx and run steps 2-4 above automatically. It is primarily intended to be run by the dotnet/coreclr CI system, but it might provide a useful reference or shortcut for individuals running the tests locally.
## Using the built CoreCLR testhost
-**These instructions are currently Windows only.**
Instead of copying CoreCLR binaries you can also test your changes with an existing CoreFX build or CoreCLR's CI assemblies
### Locally-built CoreFX
+
Once you have finished steps 1, 2. and 4. above execute the following instructions to test your local CLR changes with the built-CoreFX changes.
-1. From `<coreclr_root>` run `build-test.cmd <arch> <build_type> skipmanaged` to generate the test host.
+1. From `<coreclr_root>` run
+`build-test.cmd <arch> <build_type> buildtesthostonly`
+
+-or-
+
+`build-test.sh <arch> <build_type> generatetesthostonly`
+
+to generate the test host.
2. Navigate to `<corefx_root>\bin\tests\` and then the test you would like to run
3. Run
```cmd
<coreclr_root>\bin\<os>.<arch>.<build_type>\testhost\dotnet.exe <corefx_root>\bin\tests\<testname>\xunit.console.netcore.exe <testname>.dll
```
+-or-
+
+```sh
+<coreclr_root>/bin/<os>.<arch>.<build_type>/testhost/dotnet <corefx_root>/bin/tests/<testname>/xunit.console.netcore.exe <testname>.dll
+```
+
followed by any extra command-line arguments.
For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR.
```
-pushd C:\corefx\bin\tests\System.Collections.Tests
+cd C:\corefx\bin\tests\System.Collections.Tests
C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonwindowstests
```
+-or-
+
+```
+cd ~/corefx/bin/tests/System.Collections.Tests
+~/coreclr/bin/tests/Linux.x64.Release/testhost/dotnet .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonlinuxtests
+```
+
### CI Script
-CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs. To run tests against pre-built binaries you can execute the following from the CoreCLR repo root:
+CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs.
-1. `.\build.cmd <arch> <build_type>`
-2. `.\build-test.cmd <arch> <build_type> skipmanaged` - generates the test host
+To run tests against pre-built binaries you can execute the following from the CoreCLR repo root:
+
+#### Windows
+1. `.\build.cmd <arch> <build_type> skiptests`
+2. `.\build-test.cmd <arch> <build_type> buildtesthostonly` - generates the test host
3. `.\tests\runtest.cmd <arch> <build_type> corefxtests|corefxtestsall` - runs CoreFX tests
+#### Linux and OSX
+1. `./build.sh <arch> <build_type> skiptests`
+2. `./build-test.sh <arch> <build_type> generatetesthostonly`
+3. `./tests/runtest.sh --corefxtests|--corefxtestsall --testHostDir=<path_to_testhost> --coreclr-src<path_to_coreclr>`
+
CoreFXTests - runs all tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList`
CoreFXTestsAll - runs all tests available in the test list found at the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt`.
+#### Linux - specific
+<path_to_testhost> - path to the coreclr test host built in step 2.
+<path_to_coreclr> - path to coreclr source
### Helix Testing
To use Helix-built binaries, substitute the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt` with one acquired from a Helix test run and run the commands above.
-### Test List Format
+#### Workflow
+The CoreFX tests CI jobs run against cached test binaries in blob storage. This means that tests might need to be disabled until the test binaries are refreshed as breaking changes are merged in both CoreCLR and CoreFX. If you suspect a test is not failing because of a functional regression, but rather because it's stale you can add it to either the [Windows](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/TopN.CoreFX.x64.Windows.issues.json) or [Unix](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/TopN.CoreFX.x64.Unix.issues.json) test exclusion lists.
+
+#### Test List Format
The tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList` should conform to the following format -
```json
{
if [ ! -f "${__BuildToolsDir}/Microsoft.CSharp.targets" ]; then
ln -s "${__BuildToolsDir}/Microsoft.CSharp.Targets" "${__BuildToolsDir}/Microsoft.CSharp.targets"
fi
+
}
+generate_testhost()
+{
+ export TEST_HOST=$xUnitTestBinBase/testhost
+
+ if [ -d "${TEST_HOST}" ]; then
+ rm -rf $TEST_HOST
+ fi
+
+ echo "${__MsgPrefix}Creating test overlay..."
+ mkdir -p $TEST_HOST
+
+ build_Tests_internal "Tests_Generate_TestHost" "${__ProjectDir}/tests/runtest.proj" "-testHost" "Creating test host"
+}
+
+
build_Tests()
{
__TestDir=$__ProjectDir/tests
__RebuildTests=0
__BuildTestWrappers=0
__GenerateLayoutOnly=
+__GenerateTestHostOnly=
__priority1=
CORE_ROOT=
generatelayoutonly)
__GenerateLayoutOnly=1
;;
-
generatetesthostonly)
- exit 0
+ __GenerateTestHostOnly=1
;;
-
buildagainstpackages)
__BuildAgainstPackagesArg=1
;;
# Specify path to be set for CMAKE_INSTALL_PREFIX.
# This is where all built CoreClr libraries will copied to.
export __CMakeBinDir="$__BinDir"
+if [ [ ! -d "$__BinDir" ] || [ ! -d "$__BinDir/bin" ] ]; then
+ if [ [ -z "$__GenerateLayoutOnly" ] && [ -z "$__GenerateTestHostOnly" ] ]; then
-if [ ! -d "$__BinDir" ] || [ ! -d "$__BinDir/bin" ]; then
-
- echo "Cannot find build directory for the CoreCLR Product or native tests."
- echo "Please make sure CoreCLR and native tests are built before building managed tests."
- echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
+ echo "Cannot find build directory for the CoreCLR native tests."
+ echo "Please make sure native tests are built before building managed tests."
+ echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
+ else
+ echo "Cannot find build directory for the CoreCLR Product."
+ echo "Please make sure CoreCLR and native tests are built before building managed tests."
+ echo "Example use: './build.sh $__BuildArch $__BuildType' "
+ fi
exit 1
fi
+
+
# Configure environment if we are doing a cross compile.
if [ $__CrossBuild == 1 ]; then
export CROSSCOMPILE=1
__CoreClrVersion=1.1.0
__sharedFxDir=$__BuildToolsDir/dotnetcli/shared/Microsoft.NETCore.App/$__CoreClrVersion/
-echo "Building Tests..."
-if [ -z "$__GenerateLayoutOnly" ]; then
+if [[ (-z "$__GenerateLayoutOnly") && (-z "$__GenerateTestHostOnly") ]]; then
+ echo "Building Tests..."
build_Tests
else
+ echo "Generating test layout..."
generate_layout
+ if [ ! -z "$__GenerateTestHostOnly" ]; then
+ echo "Generating test host..."
+ generate_testhost
+ fi
fi
if [ $? -ne 0 ]; then
-https://cloudcijobs.blob.core.windows.net/corertci/TestList_Linux.json
\ No newline at end of file
+https://cloudcijobs.blob.core.windows.net/coreclrci/CoreFXArchives_Linux/TestList.json
\ No newline at end of file
-https://cloudcijobs.blob.core.windows.net/corertci/TestList_OSX.json
\ No newline at end of file
+https://cloudcijobs.blob.core.windows.net/coreclrci/CoreFXArchives_OSX/TestList.json
\ No newline at end of file
-// The expected format of this file can be found at https://github.com/dotnet/coreclr/blob/master/Documentation/building/testing-with-corefx.md
[
{
- "name": "System.Collections.Tests",
+ "name": "System.Diagnostics.DiagnosticSource.Tests",
+ "enabled": true,
"exclusions": {
"namespaces": null,
"classes": null,
"methods": [
{
- "name": "System.Collections.Tests.SortedList_Generic_Tests_string_string.ICollection_Generic_Remove_DefaultValueContainedInCollection",
- "reason": "Exclusion Sample"
+ "name": "System.Diagnostics.Tests.DiagnosticSourceEventSourceBridgeTests.TestShortcutKeywords",
+ "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\n ↓ (pos 0)\\nExpected: Activity1Start\\nActual: Event\\n ↑ (pos 0)"
}
]
}
},
{
- "name": "System.Runtime.Extensions.Tests",
+ "name": "System.Diagnostics.Tracing.Tests",
+ "enabled": true,
"exclusions": {
"namespaces": null,
"classes": null,
- "methods": []
+ "methods": [
+ {
+ "name": "BasicEventSourceTests.TestsUserErrors.Test_BadEventSource_MismatchedIds",
+ "reason": "Xunit.Sdk.EqualException Assert.Equal() Failure\\nExpected: 2\\nActual: 1"
+ }
+ ]
+ }
+ },
+ {
+ "name": "System.Drawing.Common.Tests",
+ "enabled": true,
+ "exclusions": {
+ "namespaces": null,
+ "classes": null,
+ "methods": [
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.InterpolationMode_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'InterpolationMode'.\\nParameter name: value"
+ },
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.CompositingQuality_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'CompositingQuality'.\\nParameter name: value"
+ },
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.PixelOffsetMode_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'PixelOffsetMode'.\\nParameter name: value"
+ },
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.PageUnit_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-1) is invalid for Enum type 'GraphicsUnit'.\\nParameter name: value"
+ },
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.SmoothingMode_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-2) is invalid for Enum type 'SmoothingMode'.\\nParameter name: value"
+ },
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.CompositingMode_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-1) is invalid for Enum type 'CompositingMode'.\\nParameter name: value"
+ },
+ {
+ "name": "System.Drawing.Tests.GraphicsTests.TextRenderingHint_SetInvalid_ThrowsInvalidEnumArgumentException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ComponentModel.InvalidEnumArgumentException)\\nActual: typeof(System.ComponentModel.InvalidEnumArgumentException): The value of argument 'value' (-1) is invalid for Enum type 'TextRenderingHint'.\\nParameter name: value"
+ }
+ ]
+ }
+ },
+ {
+ "name": "System.IO.Pipes.Tests",
+ "enabled": true,
+ "exclusions": {
+ "namespaces": null,
+ "classes": null,
+ "methods": [
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOutRead_ClientInOutWrite.Server_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOutWrite_ClientInOutRead.Server_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOut_ClientIn.Server_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOut_ClientOut.Server_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerOut_ClientIn.Server_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerIn_ClientOut.Server_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOutRead_ClientInOutWrite.Client_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOutWrite_ClientInOutRead.Client_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOut_ClientIn.Client_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerInOut_ClientOut.Client_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerOut_ClientIn.Client_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ },
+ {
+ "name": "System.IO.Pipes.Tests.NamedPipeTest_Simple_ServerIn_ClientOut.Client_ReadWriteCancelledToken_Throws_OperationCanceledException",
+ "reason": "https://github.com/dotnet/corefx/issues/16934"
+ }
+ ]
}
},
{
- "name": "System.Runtime.Tests",
+ "name": "System.Net.Http.Functional.Tests",
+ "enabled": true,
"exclusions": {
"namespaces": null,
"classes": null,
- "methods": []
+ "methods": [
+ {
+ "name": "System.Net.Http.Functional.Tests.PlatformHandler_DiagnosticsTest.SendAsync_ExpectedDiagnosticSourceActivityLogging",
+ "reason": "System.Diagnostics.RemoteExecutorTestBase+RemoteInvokeHandle+RemoteExecutionException System.Diagnostics.RemoteExecutorTestBase+RemoteInvokeHandle+RemoteExecutionException : Remote process failed with an unhandled exception."
+ },
+ {
+ "name": "System.Net.Http.Functional.Tests.SocketsHttpHandler_DiagnosticsTest.SendAsync_ExpectedDiagnosticSourceActivityLogging",
+ "reason": "System.Diagnostics.RemoteExecutorTestBase+RemoteInvokeHandle+RemoteExecutionException System.Diagnostics.RemoteExecutorTestBase+RemoteInvokeHandle+RemoteExecutionException : Remote process failed with an unhandled exception."
+ }
+ ]
+ }
+ },
+ {
+ "name": "System.Net.HttpListener.Tests",
+ "enabled": true,
+ "exclusions": {
+ "namespaces": null,
+ "classes": null,
+ "methods": [
+ {
+ "name": "System.Net.Tests.HttpRequestStreamTests.Read_NullBuffer_ThrowsArgumentNullException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ArgumentNullException)\\nActual: typeof(System.NullReferenceException): Object reference not set to an instance of an object."
+ },
+ {
+ "name": "System.Net.Tests.HttpResponseStreamTests.Write_NullBuffer_ThrowsArgumentNullException",
+ "reason": "Xunit.Sdk.ThrowsException Assert.Throws() Failure\\nExpected: typeof(System.ArgumentNullException)\\nActual: typeof(System.NullReferenceException): Object reference not set to an instance of an object."
+ }
+ ]
}
},
{
- "name": "System.Threading.Tasks.Tests",
+ "name": "System.Security.Cryptography.X509Certificates.Tests",
+ "enabled": true,
"exclusions": {
"namespaces": null,
"classes": null,
- "methods": []
+ "methods": [
+ {
+ "name": "System.Security.Cryptography.X509Certificates.Tests.X509StoreTests.Constructor_DefaultStoreName",
+ "reason": "Assert.Equal() Failure ↓ (pos 1) Expected: My Actual: MY ↑ (pos 1)"
+ }
+ ]
}
},
{
"name": "System.Threading.Tests",
+ "enabled": true,
"exclusions": {
"namespaces": null,
"classes": null,
- "methods": []
+ "methods": [
+ {
+ "name": "System.Threading.Tests.EventWaitHandleTests.Ctor_InvalidMode",
+ "reason": "Assert.Equal() Failure Expected: (null) Actual: mode"
+ }
+ ]
+ }
+ },
+ {
+ "name": "System.Xml.Xsl.XslTransformApi.Tests",
+ "enabled": true,
+ "exclusions": {
+ "namespaces": null,
+ "classes": null,
+ "methods": [
+ {
+ "name": "System.Xml.Tests.CXmlResolverTest.TC_AbsolutePath_Transform",
+ "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\nExpected: True\\nActual: False"
+ },
+ {
+ "name": "System.Xml.Tests.CTransformResolverTest.TC_AbsolutePath_Transform",
+ "reason": "Xunit.Sdk.TrueException Assert.True() Failure\\nExpected: True\\nActual: False"
+ }
+ ]
}
}
]
\ No newline at end of file
<PropertyGroup Condition="'$(OSGroup)'=='Windows_NT'">
<HostFxrFileName>hostfxr</HostFxrFileName>
- <DotnetExecutableName>dotnet.exe</DotnetExecutableName>
<HostFxrFileExtension>dll</HostFxrFileExtension>
+ <DotnetExecutableName>dotnet.exe</DotnetExecutableName>
+ <HostPolicyFileName>hostpolicy</HostPolicyFileName>
+ <HostPolicyExtension>dll</HostPolicyExtension>
</PropertyGroup>
<PropertyGroup Condition="'$(OSGroup)'!='Windows_NT'">
<HostFxrFileName>libhostfxr</HostFxrFileName>
<HostFxrFileExtension Condition="'$(OSGroup)' == 'Linux' Or '$(OSGroup)' == 'FreeBSD'">so</HostFxrFileExtension>
<HostFxrFileExtension Condition="$(OSGroup) =='OSX'">dylib</HostFxrFileExtension>
+ <HostPolicyFileName>libhostpolicy</HostPolicyFileName>
+ <HostPolicyExtension>$(HostFxrFileExtension)</HostPolicyExtension>
<DotnetExecutableName>dotnet</DotnetExecutableName>
</PropertyGroup>
<CoreCLRBinaries Include="$(CORE_ROOT)\**\*.*" Exclude="$(CORE_ROOT)\**\@(NetCoreAppPackagedAssemblies -> '%(Identity)' )" />
<HostFxFile Include="$(ToolsDir)\dotnetcli\**\$(HostFxrFileName).$(HostFxrFileExtension)" />
<DotnetExe Include="$(ToolsDir)\dotnetcli\$(DotnetExecutableName)" />
- <HostPolicyFile Include="$(ToolsDir)\dotnetcli\**\hostpolicy.dll" />
+ <HostPolicyFile Include="$(ToolsDir)\dotnetcli\**\$(HostPolicyFileName).$(HostPolicyExtension)" />
</ItemGroup>
<Copy SourceFiles="@(HostFxFile)"
echo 'CoreFX Test Options '
echo ' --corefxtests : Runs CoreFX tests'
echo ' --corefxtestsall : Runs all available CoreFX tests'
+ echo ' --corefxtestlist=<path> : Runs the CoreFX tests specified in the passed list'
+ echo ' --testHostDir=<path> : Directory containing a built test host including core binaries, test dependencies'
+ echo ' and a dotnet executable'
echo ''
echo 'Runtime Code Coverage options:'
echo ' --coreclr-coverage : Optional argument to get coreclr code coverage reports'
if [ ! -d "$coreClrBinDir" ]; then
exit_with_error "$errorSource" "Directory specified by --coreClrBinDir does not exist: $coreClrBinDir"
fi
- if [ -z "$coreFxBinDir" ]; then
- exit_with_error "$errorSource" "One of --coreOverlayDir or --coreFxBinDir must be specified." "$printUsage"
- fi
# Create the overlay
coreOverlayDir=$testRootDir/Tests/coreoverlay
copy_test_native_bin_to_test_root $coreOverlayDir
}
+function create_testhost
+{
+ if [ ! -d "$testHostDir" ]; then
+ exit_with_error "$errorSource" "Did not find the test host directory: $testHostDir"
+ fi
+
+ # Initialize test variables
+ local buildToolsDir=$coreClrSrc/Tools
+ local dotnetExe=$buildToolsDir/dotnetcli/dotnet
+ local coreClrSrcTestDir=$coreClrSrc/tests
+
+ if [ -z $coreClrBinDir ]; then
+ local coreClrBinDir=${coreClrSrc}/bin
+ export __CoreFXTestDir=${coreClrSrc}/bin/tests/CoreFX
+ else
+ export __CoreFXTestDir=${coreClrBinDir}/tests/CoreFX
+ fi
+
+ local coreFXTestSetupUtilityName=CoreFX.TestUtils.TestFileSetup
+ local coreFXTestSetupUtility="${coreClrSrcTestDir}/src/Common/CoreFX/TestFileSetup/${coreFXTestSetupUtilityName}.csproj"
+ local coreFXTestSetupUtilityOutputPath=${__CoreFXTestDir}/TestUtilities
+ local coreFXTestBinariesOutputPath=${__CoreFXTestDir}/tests_downloaded
+
+ if [ -z $CoreFXTestList]; then
+ local CoreFXTestList="${coreClrSrcTestDir}/CoreFX/TopN.CoreFX.x64.Unix.issues.json"
+ fi
+
+ case "${OSName}" in
+ # Check if we're running under OSX
+ Darwin)
+ local coreFXTestRemoteURL=$(<${coreClrSrcTestDir}/CoreFX/CoreFXTestListURL_OSX.txt)
+ local coreFXTestExclusionDef=nonosxtests
+ ;;
+ # Default to Linux
+ *)
+ local coreFXTestRemoteURL=$(<${coreClrSrcTestDir}/CoreFX/CoreFXTestListURL_Linux.txt)
+ local coreFXTestExclusionDef=nonlinuxtests
+ ;;
+ esac
+
+ local coreFXTestExecutable=xunit.console.netcore.exe
+ local coreFXLogDir=${coreClrBinDir}/Logs/CoreFX/
+ local coreFXTestExecutableArgs="--notrait category=nonnetcoreapptests --notrait category=${coreFXTestExclusionDef} --notrait category=failing --notrait category=IgnoreForCI --notrait category=OuterLoop --notrait Benchmark=true"
+
+ chmod +x ${dotnetExe}
+ resetCommandArgs=("msbuild /t:Restore ${coreFXTestSetupUtility}")
+ echo "${dotnetExe} $resetCommandArgs"
+ "${dotnetExe}" $resetCommandArgs
+
+ buildCommandArgs=("msbuild ${coreFXTestSetupUtility} /p:OutputPath=${coreFXTestSetupUtilityOutputPath} /p:Platform=${_arch} /p:Configuration=Release")
+ echo "${dotnetExe} $buildCommandArgs"
+ "${dotnetExe}" $buildCommandArgs
+
+ if [ "${RunCoreFXTestsAll}" == "1" ]; then
+ local coreFXRunCommand=--runAllTests
+ else
+ local coreFXRunCommand=--runSpecifiedTests
+ fi
+
+ local buildTestSetupUtilArgs=("${coreFXTestSetupUtilityOutputPath}/${coreFXTestSetupUtilityName}.dll --clean --outputDirectory ${coreFXTestBinariesOutputPath} --testListJsonPath ${CoreFXTestList} ${coreFXRunCommand} --dotnetPath ${testHostDir}/dotnet --testUrl ${coreFXTestRemoteURL} --executable ${coreFXTestExecutable} --log ${coreFXLogDir} ${coreFXTestExecutableArgs}")
+ echo "${dotnetExe} $buildTestSetupUtilArgs"
+ "${dotnetExe}" $buildTestSetupUtilArgs
+
+ local exitCode=$?
+ if [ $exitCode != 0 ]; then
+ echo Running CoreFX tests finished with failures
+ else
+ echo Running CoreFX tests finished successfully
+ fi
+
+ echo Check ${coreFXLogDir} for test run logs
+
+ exit ${exitCode}
+}
+
declare -a skipCrossGenFiles
function is_skip_crossgen_test {
export RunCrossGen=1
;;
--corefxtests)
- exit 0
+ export RunCoreFXTests=1
;;
--corefxtestsall)
- exit 0
+ export RunCoreFXTests=1
+ export RunCoreFXTestsAll=1
+ ;;
+ --corefxtestlist)
+ export RunCoreFXTests=1
+ export CoreFXTestList=${i#*=}
+ ;;
+ --testHostDir=*)
+ export testHostDir=${i#*=}
;;
--sequential)
((maxProcesses = 1))
export COMPlus_gcServer="$serverGC"
+if [ "$RunCoreFXTests" == 1 ];
+then
+ if [ -z "$coreClrSrc" ]
+ then
+ echo "Coreclr src files are required to run CoreFX tests"
+ echo "Coreclr src files root path can be passed using '--coreclr-src' argument"
+ print_usage
+ exit $EXIT_CODE_EXCEPTION
+ fi
+
+ if [ -z "$testHostDir" ]; then
+ echo "--testHostDir is required to run CoreFX tests"
+ print_usage
+ exit $EXIT_CODE_EXCEPTION
+ fi
+
+ if [ ! -f "$testHostDir/dotnet" ]; then
+ echo "Executable dotnet not found in $testHostDir"
+ exit $EXIT_CODE_EXCEPTION
+ fi
+
+ if [ ! -d "$testHostDir" ]; then
+ echo "Directory specified by --testHostDir does not exist: $testRootDir"
+ exit $EXIT_CODE_EXCEPTION
+ fi
+
+ create_testhost
+ exit 0
+fi
+
if [ -z "$testRootDir" ]; then
echo "--testRootDir is required."
print_usage
<XUnitAbstractionsVersion>2.0.1</XUnitAbstractionsVersion>
<CoreFXXUnitPackageVersion>2.2.0-beta2-build3300</CoreFXXUnitPackageVersion>
<MicrosoftDiagnosticsTracingTraceVentVersion>2.0.19</MicrosoftDiagnosticsTracingTraceVentVersion>
+ <MicrosoftDotnetPlatformAbstractionsVersion>2.1.0</MicrosoftDotnetPlatformAbstractionsVersion>
</PropertyGroup>
<!-- Switch RuntimeIdentifier according to currently running OSGroup -->
<Version>$(MicrosoftDiagnosticsTracingTraceVentVersion)</Version>
</PackageReference>
</ItemGroup>
+
+ <ItemGroup Condition="'$(OSGroup)' != 'Windows_NT'">
+ <PackageReference Include="Microsoft.DotNet.PlatformAbstractions">
+ <Version>$(MicrosoftDotnetPlatformAbstractionsVersion)</Version>
+ </PackageReference>
+ </ItemGroup>
+
<ItemGroup>
<PackageToInclude Include="@(PackageReference -> '%(Identity)' )"/>
</ItemGroup>
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="..\..\..\..\..\dependencies.props" />
<PropertyGroup>
+ <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
<SystemCommandLineVersion>0.1.0-e160909-1</SystemCommandLineVersion>
<NewtonsoftJsonVersion>11.0.2</NewtonsoftJsonVersion>
<NewtonsoftJsonSchemaVersion>3.0.10</NewtonsoftJsonSchemaVersion>
// Check URL for validity
if (!Uri.IsWellFormedUriString(payloadUri, UriKind.Absolute))
continue;
-
+ Console.WriteLine("Downloading " + testName + " from " + payloadUri);
// Download tests from specified URL
using (var response = await HttpClient.GetStreamAsync(payloadUri))
{
/// <summary>
/// This is a driver class, which downloads archived CoreFX test assemblies from a specified URL, lays out their contents
/// and subsequently runs them in parallel.
- /// This is invoked from .tests\runtests.cmd and depends on a test host (CoreCLR components with a dotnet executable) being present
+ /// This is invoked from .tests/runtests.[cmd|sh] and depends on a test host (CoreCLR components with a dotnet executable) being present
/// </summary>
public class Program
{
// Only run tests if the relevant commandline switch is passed
if (runSpecifiedTests || runAllTests)
- {
exitCode = RunTests();
-
- }
}
catch (AggregateException e)
{
{
exitCode = ExitCode.UnknownError;
}
+
return false;
});
}
projects / platform / upstream / coreclr.git / commitdiff