4 You can use CoreFX tests to validate your changes to CoreCLR. There are two basic options:
6 1. Build the CoreFX product and tests against a build of CoreCLR, or
7 2. Use a snapshot of the CoreFX test build with a build of CoreCLR.
9 Both mechanisms are exposed to certain types of breaking changes which can cause test failures.
10 However, we have a test exclusion mechanism for option #2, with exclusions specified in the
11 CoreCLR tree, not the CoreFX tree. This can make it possible to exclude tests that fail for
12 transient breaking change reasons, as well as for more long-lasting reasons.
14 Mechanism #2 is used to run CoreFX tests in the CI against every CoreCLR pull request.
16 # Building CoreFX against CoreCLR
18 In general, refer to the
19 [CoreFX Developer Guide](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md)
20 for information about CoreFX build scripts.
22 Normally when you build CoreFX it is built against a "last known good" version of CoreCLR.
23 To run CoreFX tests against a current, "live", version of CoreCLR (for example, a CoreCLR
24 you have built yourself), including with an updated System.Private.CoreLib.dll,
25 [use these instructions](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits).
27 ## Replace runtime between building CoreFX product and tests
29 A variation on the above is to build CoreFX normally, then overwrite the "last known good" CoreCLR
30 it used with your build of CoreCLR.
34 1. Build the CoreCLR you wish to test under `<coreclr_root>`.
35 2. Build the CoreFX repo (using `build.[cmd|sh]`) under `<corefx_root>`, but don't build tests yet.
36 3. Copy the contents of the CoreCLR binary root you wish to test into the CoreFX runtime
37 folder created in step #2.
41 `copy <coreclr_root>\bin\Product\Windows_NT.<arch>.<build_type>\* <corefx_root>\artifacts\bin\testhost\netcoreapp-Windows_NT-<build_type>-<arch>\shared\Microsoft.NETCore.App\9.9.9`
45 `cp <coreclr_root>/bin/Product/<os>.<arch>.<build_type>/* <corefx_root>/artifacts/bin/testhost/netcoreapp-<os>-<build_type>-<arch>/shared/Microsoft.NETCore.App/9.9.9`
47 4. Build and run the CoreFX tests using `build.[cmd|sh] -test` as described in the Developer Guide.
51 [run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py)
52 will clone dotnet/corefx and run steps 2-4 above automatically. It is primarily intended
53 to be run by the dotnet/coreclr CI system, but it might provide a useful reference or
54 shortcut for individuals running the tests locally.
56 # Using the built CoreCLR test host
58 Here is an alternative method to the one described above. You can test your changes with
59 an existing CoreFX build or CoreCLR's cached CoreFX test build assemblies.
61 The "test host" is a dotnet CLI layout that includes both the CoreCLR and the CoreFX you want to test.
63 ## Locally-built CoreFX
65 First, build CoreCLR (building the tests is not required) and CoreFX (including the tests),
68 1. Build the CoreCLR you wish to test under `<coreclr_root>`.
69 2. Build the CoreFX repo under `<corefx_root>`.
70 3. Build the CoreFX tests using `build.[cmd|sh] -test`.
72 Once these are built, execute the following commands to test your local CoreCLR changes
73 with the built CoreFX changes.
75 1. From `<coreclr_root>` run:
79 build-test.cmd <arch> <build_type> buildtesthostonly
84 build-test.sh <arch> <build_type> generatetesthostonly
87 to generate the test host.
89 2. Navigate to `<corefx_root>\bin\tests\` and then into the directory for the test
90 you would like to run.
96 <coreclr_root>\bin\<os>.<arch>.<build_type>\testhost\dotnet.exe .\xunit.console.netcore.exe <testname>.dll
101 <coreclr_root>/bin/<os>.<arch>.<build_type>/testhost/dotnet ./xunit.console.netcore.exe <testname>.dll
104 followed by any extra command-line arguments for xunit.
106 For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR:
110 cd C:\corefx\artifacts\bin\tests\System.Collections.Tests
111 C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonwindowstests
116 cd ~/corefx/bin/tests/System.Collections.Tests
117 ~/coreclr/artifacts/bin/tests/Linux.x64.Release/testhost/dotnet ./xunit.console.netcore.exe ./System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonlinuxtests
120 ## Running against a cached copy of the CoreFX tests
122 CoreCLR has an alternative way to run CoreFX tests, built for CI jobs.
124 To run tests against pre-built binaries you can execute the following from the CoreCLR repo root:
128 1. `.\build.cmd <arch> <build_type> skiptests`
129 2. `.\build-test.cmd <arch> <build_type> buildtesthostonly` -- this generates the test host
130 3. `.\tests\runtest.cmd <arch> <build_type> corefxtests|corefxtestsall` -- this runs the CoreFX tests
134 1. `./build.sh <arch> <build_type> skiptests`
135 2. `./build-test.sh <arch> <build_type> generatetesthostonly`
136 3. `./tests/runtest.sh --corefxtests|--corefxtestsall --testHostDir=<path_to_testhost> --coreclr-src=<path_to_coreclr_root>`
139 + `<path_to_testhost>` - path to the CoreCLR test host built in step 2.
140 + `<path_to_coreclr_root>` - path to root of CoreCLR clone. Required to build the TestFileSetup tool for CoreFX testing.
142 The set of tests run are based on the `corefxtests` or `corefxtestsall` arguments, as follows:
143 + CoreFXTests - runs all tests defined in the dotnet/coreclr repo in `tests\CoreFX\CoreFX.issues.json`, or the test list specified with the optional argument `CoreFXTestList`.
144 + CoreFXTestsAll - runs all tests available, ignoring exclusions. The full list of tests is found at the URL in the dotnet/coreclr repo at `.\tests\CoreFX`: one of `CoreFXTestListURL.txt`, `CoreFXTestListURL_Linux.txt`, or `CoreFXTestListURL_OSX.txt`, based on platform.
148 To use Helix-built binaries, substitute the URL in `.\tests\CoreFX\CoreFXTestListURL.txt`
149 with one acquired from a Helix test run and run the commands above.
153 The CoreFX tests CI jobs run against cached test binaries in blob storage. This means that
154 tests might need to be disabled until the test binaries are refreshed as breaking changes
155 are merged in both CoreCLR and CoreFX. If you suspect a test is not failing because of a
156 functional regression, but rather because it's stale you can add it to the
157 [test exclusion file](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/CoreFX.issues.json).
161 The tests defined in CoreFX.issues.json or the test list specified with the argument
162 `CoreFXTestList` should conform to the following format.
164 [ // array of assemblies
165 { // one per assembly
166 "name": "<Fully Qualified Assembly Name>", //e.g. System.Collections.Concurrent.Tests
167 "enabled": true|false, // Defines whether a test assembly should be run. If set to false any tests with the same name will not be run even if corefxtestsall is specified
169 "namespaces": // Can be null
172 "name": "System.Collections.Concurrent.Tests", // All test methods under this namespace will be skipped
173 "reason": "<Reason for exclusion>" // This should be a link to the GitHub issue describing the problem
176 "classes": // Can be null
179 "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests", // All test methods in this class will be skipped
180 "reason": "<Reason for exclusion>"
183 "methods": // Can be null
186 "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ReferenceType_null",
187 "reason": "<Reason for exclusion>"
190 "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ValueType_null_add",
191 "reason": "<Reason for exclusion>"