From 96609bc07cc823c9336061713926556fff2c1bc0 Mon Sep 17 00:00:00 2001 From: Bruce Forstall Date: Sat, 29 Jun 2019 09:50:02 -0700 Subject: [PATCH] Update documentation for corefx testing in coreclr repo (#25493) * Update documentation for corefx testing in coreclr repo * Apply suggestions from code review Co-Authored-By: Jan Kotas * Update a command line; improve clarity --- Documentation/building/testing-with-corefx.md | 263 +++++++++++++------------- 1 file changed, 132 insertions(+), 131 deletions(-) diff --git a/Documentation/building/testing-with-corefx.md b/Documentation/building/testing-with-corefx.md index 798b39d..34df235 100644 --- a/Documentation/building/testing-with-corefx.md +++ b/Documentation/building/testing-with-corefx.md @@ -1,138 +1,188 @@ Testing with CoreFX =================== -You can use CoreFX tests to validate your changes to CoreCLR. There are two basic options: +You can use CoreFX tests to validate your changes to CoreCLR. +The coreclr repo Azure DevOps CI system runs CoreFX tests against +every pull request, and regularly runs the CoreFX tests under +many different configurations (e.g., platforms, stress modes). -1. Build the CoreFX product and tests against a build of CoreCLR, or -2. Use a snapshot of the CoreFX test build with a build of CoreCLR. +There are two basic options: -Both mechanisms are exposed to certain types of breaking changes which can cause test failures. -However, we have a test exclusion mechanism for option #2, with exclusions specified in the -CoreCLR tree, not the CoreFX tree. This can make it possible to exclude tests that fail for -transient breaking change reasons, as well as for more long-lasting reasons. +1. Build the CoreFX product and tests with a build of CoreCLR, or +2. Use a published snapshot of the CoreFX test build with a build of CoreCLR. -Mechanism #2 is used to run CoreFX tests in the CI against every CoreCLR pull request. +Mechanism #1 is the easiest. Mechanism #2 is how the CI system runs tests, +as it is optimized for our distributed Helix test running system. # Building CoreFX against CoreCLR -In general, refer to the -[CoreFX Developer Guide](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md) -for information about CoreFX build scripts. +In all cases, first build the version of CoreCLR that you wish to test. You do not need to build the coreclr tests. Typically, this will be a Checked build (faster than Debug, but with asserts that Release builds do not have). In the examples here, coreclr is assumed to be cloned to `f:\git\coreclr`. -Normally when you build CoreFX it is built against a "last known good" version of CoreCLR. -To run CoreFX tests against a current, "live", version of CoreCLR (for example, a CoreCLR -you have built yourself), including with an updated System.Private.CoreLib.dll, -[use these instructions](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits). +For example: +``` +f:\git\coreclr> build.cmd x64 checked skiptests +``` -## Replace runtime between building CoreFX product and tests +Next, build CoreFX from a clone of the [CoreFX repo](https://github.com/dotnet/corefx). +For these examples, CoreFX is assumed to be cloned to `f:\git\corefx`. For general +information about how to build CoreFX, refer to the +[CoreFX Developer Guide](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md). -A variation on the above is to build CoreFX normally, then overwrite the "last known good" CoreCLR -it used with your build of CoreCLR. +Normally when you build CoreFX it is built against a "last known good" Release version of CoreCLR. -Do the following: +There are two options here: +1. Build CoreFX against your just-built CoreCLR. +2. Build CoreFX normally, against the "last known good" version of CoreCLR, and then overwrite the "last known good" CoreCLR. -1. Build the CoreCLR you wish to test under ``. -2. Build the CoreFX repo (using `build.[cmd|sh]`) under ``, but don't build tests yet. -3. Copy the contents of the CoreCLR binary root you wish to test into the CoreFX runtime -folder created in step #2. +Option #1 might fail to build if CoreCLR has breaking changes that have not propagated to CoreFX yet. +Option #2 should always succeed the build, since the CoreFX CI has verified this build already. +Option #2 is generally recommended. -For example: +## Building CoreFX against just-built CoreCLR -`copy \bin\Product\Windows_NT..\* \artifacts\bin\testhost\netcoreapp-Windows_NT--\shared\Microsoft.NETCore.App\9.9.9` +To build CoreFX tests against a current, "live", version of CoreCLR, including with an updated System.Private.CoreLib.dll, +[use these instructions](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits). --or- +For example: +``` +f:\git\corefx> build.cmd -configuration Release -arch x64 -restore -build -buildtests -test /p:CoreCLROverridePath=f:\git\coreclr\bin\Product\Windows_NT.x64.Checked +``` -`cp /bin/Product/../* /artifacts/bin/testhost/netcoreapp---/shared/Microsoft.NETCore.App/9.9.9` - -4. Build and run the CoreFX tests using `build.[cmd|sh] -test` as described in the Developer Guide. +Note that this will replace the coreclr used in the build, and because `-test` is passed, will also run the tests. -### CI Script +## Replace CoreCLR after building CoreFX normally -[run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) -will clone dotnet/corefx and run steps 2-4 above automatically. It is primarily intended -to be run by the dotnet/coreclr CI system, but it might provide a useful reference or -shortcut for individuals running the tests locally. +Do the following: -# Using the built CoreCLR test host +1. Build the CoreFX repo, but don't build tests yet. -Here is an alternative method to the one described above. You can test your changes with -an existing CoreFX build or CoreCLR's cached CoreFX test build assemblies. +``` +f:\git\corefx> build.cmd -configuration Release -arch x64 -restore -build +``` -The "test host" is a dotnet CLI layout that includes both the CoreCLR and the CoreFX you want to test. +This creates a "testhost" directory with a subdirectory that includes the coreclr bits, e.g., `f:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\3.0.0`. -## Locally-built CoreFX +2. Copy the contents of the CoreCLR build you wish to test into the CoreFX runtime +folder created in step #1. -First, build CoreCLR (building the tests is not required) and CoreFX (including the tests), -as described above: +``` +f:\git\corefx> copy f:\git\coreclr\bin\Product\Windows_NT.x64.Checked\* f:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\3.0.0 +``` -1. Build the CoreCLR you wish to test under ``. -2. Build the CoreFX repo under ``. -3. Build the CoreFX tests using `build.[cmd|sh] -test`. +3. Optionally, create a script that contains any environment variables you want to set when running each CoreFX test. Disabling TieredCompilation or setting a JIT stress mode is a common case. E.g., -Once these are built, execute the following commands to test your local CoreCLR changes -with the built CoreFX changes. +``` +f:\git\corefx> echo set COMPlus_TieredCompilation=0>f:\git\corefx\SetStressModes.bat +``` -1. From `` run: +4. Build and run the CoreFX tests. Optionally, pass in a file that will be passed to xunit to provide extra xunit arguments. Typically, this is used to exclude known failing tests. -For Windows: ``` -build-test.cmd buildtesthostonly +f:\git\corefx> build.cmd -configuration Release -arch x64 -buildtests -test /p:WithoutCategories=IgnoreForCI /p:PreExecutionTestScript=f:\git\corefx\SetStressModes.bat /p:TestRspFile=f:\git\coreclr\tests\CoreFX\CoreFX.issues.rsp ``` -For Linux: +## Automating the CoreFX build and test run process + +The script [tests\scripts\run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) clones the corefx repo, and then implements Option 2, described above. This simplifies the whole process to a single script invocation. + +First, build CoreCLR as usual. Then, invoke the script. Specify the build architecture, the build type, where you want corefx to be put, optionally a script of environment variables to set before running the tests, and optionally a test exclusion file (as above). For example: + ``` -build-test.sh generatetesthostonly +f:\git\coreclr> echo set COMPlus_TieredCompilation=0>f:\git\coreclr\SetStressModes.bat +f:\git\coreclr> python -u f:\git\coreclr\tests\scripts\run-corefx-tests.py -arch x64 -build_type Checked -fx_root f:\git\coreclr\_fx -env_script f:\git\coreclr\SetStressModes.bat -exclusion_rsp_file f:\git\coreclr\tests\CoreFX\CoreFX.issues.rsp ``` -to generate the test host. +## Handling cross-compilation testing -2. Navigate to `\bin\tests\` and then into the directory for the test -you would like to run. +The above instructions work fine if you are building and testing on the same machine, +but what if you are building on one machine and testing on another? This happens, +for example, when building for Windows arm32 on a Windows x64 machine, +or building for Linux arm64 on a Linux x64 machine (possibly in Docker). +In these cases, build all the tests, copy them to the target machine, and run tests +there. -3. Run: +To do that, remove `-test` from the command-line used to build CoreFX tests. Without `-test`, +the tests will be built but not run. -For Windows: -```cmd -\bin\..\testhost\dotnet.exe .\xunit.console.netcore.exe .dll +If using `run-corefx-tests.py`, pass the argument `-no_run_tests`. + +After the tests are copied to the remote machine, you want to run them. Use one of the scripts +[tests\scripts\run-corefx-tests.bat](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.bat) or +[tests\scripts\run-corefx-tests.sh](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.sh) +to run all the tests (consult the scripts for proper usage). Or, run a single test as described below. + +## Other corefx build considerations + +To build for Linux arm32, you need to make sure to build using clang 5 (the default is clang 3.9). +You might need to pass `/p:BuildNativeCompiler=--clang5.0` to the corefx build sjcripts. + +## Running a single CoreFX test assembly + +Once you've built the CoreFX tests (possibly with replaced CoreCLR bits), you can also run just a single test. E.g., + +``` +f:\git\corefx> cd f:\git\corefx\artifacts\bin\System.Buffers.Tests\netcoreapp-Release +f:\git\corefx\artifacts\bin\System.Buffers.Tests\netcoreapp-Release> RunTests.cmd -r f:\git\corefx\artifacts\bin\testhost\netcoreapp-Windows_NT-Release-x64 ``` -For Linux: -```sh -/bin/../testhost/dotnet ./xunit.console.netcore.exe .dll +Alternatively, you can run the tests from from the test source directory, as follows: + +``` +f:\git\corefx> cd f:\git\corefx\src\System.Buffers\tests +f:\git\corefx\src\System.Buffers\tests> dotnet msbuild /t:Test /p:ForceRunTests=true;ConfigurationGroup=Release ``` -followed by any extra command-line arguments for xunit. +# Using a published snapshot of CoreFX tests + +The corefx official build system publishes a set of corefx test packages for consumption +by the coreclr CI. You can use this set of published files, but it is complicated, especially +if you wish to run more than one or a few tests. -For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR: +The process builds a "test host", which is a directory layout like the dotnet CLI, and uses that +when invoking the tests. CoreFX product packages, and packages needed to run CoreFX tests, +are restored, and the CoreCLR to test is copied in. For Windows: + +1. `.\build.cmd skiptests` -- build the CoreCLR you want to test +2. `.\build-test.cmd buildtesthostonly` -- this generates the test host + +For Linux and macOS: + +1. `./build.sh skiptests` +2. `./build-test.sh generatetesthostonly` + +The published tests are summarized in a `corefx-test-assets.xml` file that lives here: + ``` -cd C:\corefx\artifacts\bin\tests\System.Collections.Tests -C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonwindowstests +https://dotnetfeed.blob.core.windows.net/dotnet-core/corefx-tests/$(MicrosoftPrivateCoreFxNETCoreAppVersion)/$(__BuildOS).$(__BuildArch)/$(_TargetGroup)/corefx-test-assets.xml ``` -For Linux: +where `MicrosoftPrivateCoreFxNETCoreAppVersion` is defined in `eng\Versions.props`. For example: + ``` -cd ~/corefx/bin/tests/System.Collections.Tests -~/coreclr/artifacts/bin/tests/Linux.x64.Release/testhost/dotnet ./xunit.console.netcore.exe ./System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonlinuxtests +https://dotnetfeed.blob.core.windows.net/dotnet-core/corefx-tests/4.6.0-preview8.19326.15/Linux.arm64/netcoreapp/corefx-test-assets.xml ``` -## Running against a cached copy of the CoreFX tests +This file lists all the published test assets. You can download each one, unpack it, and +then use the generated test host to run the test. + +Here is an example test file: +``` +https://dotnetfeed.blob.core.windows.net/dotnet-core/corefx-tests/4.6.0-preview8.19326.15/Linux.arm64/netcoreapp/tests/AnyOS.AnyCPU.Release/CoreFx.Private.TestUtilities.Tests.zip +``` -CoreCLR has an alternative way to run CoreFX tests, built for CI jobs. +========================= -To run tests against pre-built binaries you can execute the following from the CoreCLR repo root: +TBD: The following describes some automation for running CoreFX tests from a similar, but older system. +These instructions currently do not work (but perhaps should be revived to work). For Windows: -1. `.\build.cmd skiptests` -2. `.\build-test.cmd buildtesthostonly` -- this generates the test host 3. `.\tests\runtest.cmd corefxtests|corefxtestsall` -- this runs the CoreFX tests For Linux and macOS: -1. `./build.sh skiptests` -2. `./build-test.sh generatetesthostonly` 3. `./tests/runtest.sh --corefxtests|--corefxtestsall --testHostDir= --coreclr-src=` where: @@ -143,59 +193,10 @@ The set of tests run are based on the `corefxtests` or `corefxtestsall` argument + CoreFXTests - runs all tests defined in the dotnet/coreclr repo in `tests\CoreFX\CoreFX.issues.json`, or the test list specified with the optional argument `CoreFXTestList`. + CoreFXTestsAll - runs all tests available, ignoring exclusions. The full list of tests is found at the URL in the dotnet/coreclr repo at `.\tests\CoreFX`: one of `CoreFXTestListURL.txt`, `CoreFXTestListURL_Linux.txt`, or `CoreFXTestListURL_OSX.txt`, based on platform. -## Helix Testing - -To use Helix-built binaries, substitute the URL in `.\tests\CoreFX\CoreFXTestListURL.txt` -with one acquired from a Helix test run and run the commands above. - -## Workflow - -The CoreFX tests CI jobs run against cached test binaries in blob storage. This means that -tests might need to be disabled until the test binaries are refreshed as breaking changes -are merged in both CoreCLR and CoreFX. If you suspect a test is not failing because of a -functional regression, but rather because it's stale you can add it to the -[test exclusion file](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/CoreFX.issues.json). - -## Test List Format - -The tests defined in CoreFX.issues.json or the test list specified with the argument -`CoreFXTestList` should conform to the following format. -```js -[ // array of assemblies - { // one per assembly - "name": "", //e.g. System.Collections.Concurrent.Tests - "enabled": true|false, // Defines whether a test assembly should be run. If set to false any tests with the same name will not be run even if corefxtestsall is specified - "exclusions": { - "namespaces": // Can be null - [ - { - "name": "System.Collections.Concurrent.Tests", // All test methods under this namespace will be skipped - "reason": "" // This should be a link to the GitHub issue describing the problem - } - ] - "classes": // Can be null - [ - { - "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests", // All test methods in this class will be skipped - "reason": "" - } - ] - "methods": // Can be null - [ - { - "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ReferenceType_null", - "reason": "" - }, - { - "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ValueType_null_add", - "reason": "" - } - ] - } - }, - { // next assembly - ... - } -] -``` - \ No newline at end of file +========================= + +# CoreFX test exclusions + +The CoreCLR CI system runs CoreFX tests against a just-built CoreCLR. If tests need to be +disabled due to transitory breaking change, for instance, update the +[test exclusion file](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/CoreFX.issues.rsp). -- 2.7.4