[wasm] Add support for a random test case orderer, for xunit tests (#65628)
authorAnkit Jain <radical@gmail.com>
Mon, 21 Feb 2022 21:58:09 +0000 (16:58 -0500)
committerGitHub <noreply@github.com>
Mon, 21 Feb 2022 21:58:09 +0000 (16:58 -0500)
commitcacb3bf4f73e25b20a2032691ff71214f1d16c6b
tree140220aa90c3a901a92229db465935bf8ff0da4c
parenta01fe0b33d3c8107b2f798daca6fefe50e5a71a7
[wasm] Add support for a random test case orderer, for xunit tests (#65628)

* [wasm] Add support for a random test case orderer, for xunit tests

This is enabled by default for wasm with
`$(XUnitUseRandomizedTestOrderer)=true`.

When the library tests run, they print two messages like:

```
  info: Using random seed for test cases: 700149826
  info: Using random seed for collections: 700149826
  info: Starting:    System.Collections.Immutable.Tests.dll
```

These seeds are picked randomly every time the tests are run. To run the
tests with a specific seed, use environment variable
`XUNIT_RANDOM_ORDER_SEED`.

When running with tests, this can be used as:

    `WasmXHarnessMonoArgs="--setenv=XUNIT_RANDOM_ORDER_SEED=<seed>`

* Enable test orderer only for libraries tests

* [wasm] Automatically pass XUNIT_RANDOM_ORDER_SEED envvar to wasm apps

* Disable random test ordering for `System.Xml.RW.XmlWriterApi.Tests`

`System.Xml.Tests.TCCloseOutput.*` tests seem to depend on the order of
execution.

```
[06:35:14] fail: [FAIL] System.Xml.Tests.TCCloseOutput.CloseOutput_4(utils: XmlWriterUtils { Async = False, WriterType = UTF8Writer }, outputType: "Stream")
[06:35:14] info: System.IO.FileNotFoundException : File Not Found: writer.out
[06:35:14] info:    at XmlCoreTest.Common.FilePathUtil.getStream(String filename)
[06:35:14] info:    at System.Xml.Tests.TCCloseOutput.CloseOutput_4(XmlWriterUtils utils, String outputType)
[06:35:14] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)
```

* [wasm] Disable random tests ordering for `System.Runtime.Loader`

It seems to cause failures like, reproducible with
`XUNIT_RANDOM_ORDER_SEED=2106784294`:

```
[06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en")
[06:25:43] info: Assert.Same() Failure
[06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0
[06:25:43] info: Actual:   "Empty" System.Runtime.Loader.AssemblyLoadContext #4
[06:25:43] info:    at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture)
[06:25:43] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)
```

* [wasm] Disable random test ordering for `System.Runtime.Numerics`

Randomized runs seem to fail with:

```
[03:29:54] info: Starting:    System.Runtime.Numerics.Tests.dll
[03:29:58] fail: [FAIL] System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests
[03:29:58] info: Assert.Equal() Failure
[03:29:58] info:           ↓ (pos 0)
[03:29:58] info: Expected: -0
[03:29:58] info: Actual:   0
[03:29:58] info:           ↑ (pos 0)
[03:29:58] info:    at System.Numerics.Tests.cast_toTest.VerifyDoubleExplicitCastToBigInteger(Double value)
[03:29:58] info:    at System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests()
[03:29:58] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)
```

Reproducible with `XUNIT_RANDOM_SEED_ORDER=1883302047`.

* Add issues for the failing tests
eng/testing/WasmRunnerTemplate.cmd
eng/testing/WasmRunnerTemplate.sh
eng/testing/tests.mobile.targets
eng/testing/tests.wasm.targets
src/libraries/Common/tests/TestUtilities/RandomTestCaseOrderer.cs [new file with mode: 0644]
src/libraries/Common/tests/TestUtilities/RandomTestCollectionOrderer.cs [new file with mode: 0644]
src/libraries/Common/tests/TestUtilities/TestUtilities.csproj
src/libraries/Common/tests/Tests/RandomizedTestOrderAssemblyInfo.cs [new file with mode: 0644]
src/libraries/System.Private.Xml/tests/Writers/XmlWriterApi/System.Xml.RW.XmlWriterApi.Tests.csproj
src/libraries/System.Runtime.Loader/tests/System.Runtime.Loader.Tests.csproj
src/libraries/System.Runtime.Numerics/tests/System.Runtime.Numerics.Tests.csproj