We need to ensure that the CoreCLR repo contains a suitably
hookable test script. Core testing is driven by xunit but there's typically a
-wrapper around this (runtest.cmd today) to facilitate test execution.
+wrapper around this (run.cmd today) to facilitate test execution.
-The proposal is to implement platform-neutral variant of runtest.cmd that
+The proposal is to implement platform-neutral variant of run.cmd that
contains all the existing functionality plus some additional capabilities for
JIT testing. Initially this will mean:
the libraries subset (`-subset libs`) before this command can be run.
```
-./src/coreclr/build-test.sh generatelayoutonly
+./src/tests/build.sh generatelayoutonly
```
After the build is complete you will find the output in the `artifacts/tests/coreclr/Linux.x64.Debug/Tests/Core_Root` folder.
Running a single test
===================
-After `src/coreclr/build-test.sh` is run, `corerun` from the Core_Root folder is ready to be run. This can be done by using the full absolute path to `corerun`, or by setting an environment variable to the Core_Root folder.
+After `src/tests/build.sh` is run, `corerun` from the Core_Root folder is ready to be run. This can be done by using the full absolute path to `corerun`, or by setting an environment variable to the Core_Root folder.
```sh
export CORE_ROOT=/runtime/artifacts/tests/coreclr/Linux.x64.Debug/Tests/Core_Root
The Core_Root folder will have the built binaries, from `src/coreclr/build.sh` and it will also include the CoreFX packages required to run tests.
```
-./src/coreclr/build-test.sh generatelayoutonly
+./src/tests/build.sh generatelayoutonly
```
After the build is complete you will be able to find the output in the `artifacts/tests/coreclr/OSX.x64.Debug/Tests/Core_Root` folder.
Running a single test
===================
-After `src/coreclr/build-test.sh` is run, corerun from the Core_Root folder is ready to be run. This can be done by using the full absolute path to corerun, or by setting
+After `src/tests/build.sh` is run, corerun from the Core_Root folder is ready to be run. This can be done by using the full absolute path to corerun, or by setting
an environment variable to the Core_Root folder.
```sh
```powershell
.\build.cmd -subset clr+libs -c release
-cd src\coreclr
-.\build-test.cmd Release generatelayoutonly
+.\src\tests\build.cmd Release generatelayoutonly
```
For Linux:
```bash
./build.sh -subset clr+libs -c release
-cd src/coreclr
-./build-test.sh -release -generatelayoutonly
+./src/tests/build.sh -release -generatelayoutonly
```
### Generate a Configuration File for ASP.NET Benchmarking Runs
* [macOS](../../building/coreclr/osx-instructions.md)
* [Windows](../../building/coreclr/README.md)
2) [Build the libraries](../../building/libraries/README.md) in Release configuration. Pass the configuration of CoreCLR you just built to the build script (e.g. `-runtimeconfiguration debug`).
-3) From the `src/coreclr` directory run the following command:
- * Non-Windows - `./build-test.sh`
- * Windows - `build-test.cmd`
+3) From the root directory run the following command:
+ * Non-Windows - `src/tests/build.sh`
+ * Windows - `src\tests\build.cmd`
* Supply `-h` for usage flags
### Test priority
### Examples
* Build all tests priority 1 and higher
- * `build-test.cmd -priority=1`
- * `build-test.sh -priority1`
+ * `build.cmd -priority=1`
+ * `build.sh -priority1`
## Build Individual Test
Note: CoreCLR must be built prior to building an individual test. See the first step, above, for building all tests.
* Native Test: Build the generated CMake projects
- * Projects are auto-generated when the `build-test.sh`/`build-test.cmd` script is run
- * It is possible to explicitly run only the native test build with `build-test.sh/cmd skipmanaged`
+ * Projects are auto-generated when the `build.sh`/`build.cmd` script is run
+ * It is possible to explicitly run only the native test build with `build.sh/cmd skipmanaged`
* Managed Test: Invoke `dotnet build` on the project directly. `dotnet` can be the `dotnet.sh` or `dotnet.cmd` script in the repo root.
```
<runtime-repo-root>/dotnet.sh build <runtime-repo-root>/src/coreclr/tests/src/JIT/CodegenBringupTests/Array1_d.csproj /p:Configuration=Release
To build the tests on Unix:
```sh
-./src/coreclr/build-test.sh
+./src/tests/build.sh
```
Please note that this builds the Priority 0 tests. To build priority 1:
```sh
-./src/coreclr/build-test.sh -priority1
+./src/tests/build.sh -priority1
```
## Building Individual Tests
The following instructions assume that on the Unix machine:
- The CoreCLR repo is cloned at `/mnt/coreclr`
-`src/coreclr/build-test.sh` will have set up the `Core_Root` directory correctly after the test build.
+`src/tests/build.sh` will have set up the `Core_Root` directory correctly after the test build.
```sh
-./src/coreclr/tests/runtest.sh x64 checked
+./src/tests/run.sh x64 checked
```
Please use the following command for help.
```sh
-./src/coreclr/tests/runtest.sh -h
+./src/tests/run.sh -h
```
### Unsupported and temporarily disabled tests
This property disables building of a test in a default build. It also
disables running a test in the bash/batch wrapper scripts. It allows the
test to be built on any target in CI when the `allTargets` option is
-passed to the `build-test.*` scripts.
+passed to the `build.*` scripts.
Tests which never should be built or run are marked
Building coreclr tests must be done using a specific script as follows:
```
-> src\coreclr\build-test.cmd
+> src\tests\build.cmd
```
## Building Precompiled Tests
```
-> src\coreclr\build-test.cmd crossgen
+> src\tests\build.cmd crossgen
```
This will use `crossgen.exe` to precompile test executables before they are executed.
## Building Specific Priority Tests
```
-> src\coreclr\build-test.cmd -priority=1
+> src\tests\build.cmd -priority=1
```
The above is an example of requesting that priority '1' and below be built. The default priority value is '0'. If '1' is specified, all tests with `CLRTestPriorty` `0` **and** `1` will be built and run.
To run a priority '0' and '1' and crossgen'd test pass:
```
-> src\coreclr\build-test.cmd crossgen -priority=1
+> src\tests\build.cmd crossgen -priority=1
```
For additional supported parameters use the following:
```
-> src\coreclr\build-test.cmd -?
+> src\tests\build.cmd -?
```
### Building Individual Tests
-**Note:** `build-test.cmd skipmanaged [Any additional flags]` needs to be run at least once if the individual test has native assets.
+**Note:** `build.cmd skipmanaged [Any additional flags]` needs to be run at least once if the individual test has native assets.
* Native Test: Build the generated Visual Studio solution or makefile corresponding to test cmake file.
Will list supported parameters.
```
-> src\coreclr\tests\runtest.cmd /?
+> src\tests\run.cmd /?
```
In order to run all of the tests using your checked build:
```
-> src\coreclr\tests\runtest.cmd checked
+> src\tests\run.cmd checked
```
This will generate a report named `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT_x64_Checked.html`) in the subdirectory `<repo_root>\artifacts\log`. Any tests that failed will be listed in `TestRunResults_Windows_NT_x64_Checked.err`.
If you wish to re-run a failed test, follow the following steps:
-1) Set an environment variable, `CORE_ROOT`, pointing to the path to product binaries that was passed to runtest.cmd.
+1) Set an environment variable, `CORE_ROOT`, pointing to the path to product binaries that was passed to run.cmd.
For example using a checked build the location would be: `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Tests\Core_Root`
2) Run the failed test, the command to which is also present in the test report for a failed test. It will be something like `<repo_root>\artifacts\tests\coreclr\Windows_NT.x64.Checked\Exceptions\Finalization\Finalizer.cmd`.
## How CoreCLR Tests use corerun.exe
-When you execute 'runtime/src/coreclr/build-test.cmd' one of the things that it does is set up a directory where it
+When you execute 'runtime/src/tests/build.cmd' one of the things that it does is set up a directory where it
gathers the CoreCLR that has just been built with the pieces of the class library that tests need.
It places this runtime in the directory
```bat
off the CoreCLR Repository. The way the tests are expected to work is that you set the environment
variable CORE_ROOT to this directory
(you don't have to set CORE_LIBRARIES) and you can run any tests. For example after building the tests
-(running build-test at the repository base) and running 'test\runtest') you can do the following
+(running src\tests\build from the repository base) and running 'src\tests\run') you can do the following
```bat
set PATH=%PATH%;%CoreCLR%\artifacts\Product\Windows_NT.x64.Debug
echo ''
echo 'Typical command line:'
echo ''
- echo 'coreclr/tests/runtest.sh'
+ echo 'src/tests/run.sh'
echo ' --testRootDir="temp/Windows_NT.x64.Debug"'
echo ' --testNativeBinDir="runtime/artifacts/obj/Linux.x64.Debug/tests"'
echo ' --coreOverlayDir="runtime/artifacts/tests/Linux.x64.Debug/Tests/Core_Root"'
xunitOutputPath=
xunitTestOutputPath=
-# Variables for text file output. These can be passed back to runtest.sh using the "--playlist" argument
+# Variables for text file output. These can be passed back to run.sh using the "--playlist" argument
# to rerun specific tests.
testsPassOutputPath=
testsFailOutputPath=
Notes:
Using the env passed, create a temporary file to use as the
- test_env to be passed for runtest.cmd. Note that this only happens
+ test_env to be passed for run.cmd. Note that this only happens
on windows, until xunit is used on unix there is no managed code run
- in runtest.sh.
+ in run.sh.
"""
global gc_stress
<!-- Run coreclr tests using runtest.py -->
<Target Name="RunCoreClrTests" DependsOnTargets="ValidateLocalDotnet;PatchCoreClrCoreRoot">
- <Exec Condition="'$(OS)' == 'Windows_NT'" Command="$(MonoProjectRoot)..\coreclr\tests\runtest.cmd $(CoreClrTestConfig)" ContinueOnError="ErrorAndContinue" />
- <Exec Condition="'$(OS)' != 'Windows_NT'" Command="$(MonoProjectRoot)../coreclr/tests/./runtest.sh $(CoreClrTestConfig)" ContinueOnError="ErrorAndContinue" />
+ <Exec Condition="'$(OS)' == 'Windows_NT'" Command="$(MonoProjectRoot)..\tests\run.cmd $(CoreClrTestConfig)" ContinueOnError="ErrorAndContinue" />
+ <Exec Condition="'$(OS)' != 'Windows_NT'" Command="$(MonoProjectRoot)../tests/run.sh $(CoreClrTestConfig)" ContinueOnError="ErrorAndContinue" />
</Target>
<!-- Run Libraries tests using testhost -->
# run specific coreclr test, e.g.:
# make run-tests-coreclr CoreClrTest="bash ../../artifacts/tests/coreclr/OSX.x64.Release/JIT/opt/InstructionCombining/DivToMul/DivToMul.sh"
-# NOTE: make sure you've built them (cd src/coreclr && ./build-test.sh release)
+# NOTE: make sure you've built them (cd src/tests && ./build.sh release)
run-tests-coreclr:
$(DOTNET) msbuild /t:RunCoreClrTest /p:CoreClrTest="$(CoreClrTest)" $(MONO_PROJ)
echo "Run Tests..."
- nextCommand="$__TestDir/runtest.sh --testRootDir=$__TestBinDir --coreClrBinDir=$__BinDir --coreFxBinDir=$CORE_ROOT --testNativeBinDir=$__testNativeBinDir"
+ nextCommand="$__TestDir/run.sh --testRootDir=$__TestBinDir --coreClrBinDir=$__BinDir --coreFxBinDir=$CORE_ROOT --testNativeBinDir=$__testNativeBinDir"
echo "$nextCommand"
eval $nextCommand
echo " coreFxBinDir = $CORE_ROOT"
echo " testNativeBinDir = $__testNativeBinDir"
echo " -------------------------------------------------- "
- echo " Example runtest.sh command"
+ echo " Example run.sh command"
echo ""
- echo " ./tests/runtest.sh --coreOverlayDir=$CORE_ROOT --testNativeBinDir=$__testNativeBinDir --testRootDir=$__TestBinDir --copyNativeTestBin $__BuildType"
+ echo " src/tests/run.sh --coreOverlayDir=$CORE_ROOT --testNativeBinDir=$__testNativeBinDir --testRootDir=$__TestBinDir --copyNativeTestBin $__BuildType"
echo " -------------------------------------------------- "
echo "To run single test use the following command:"
echo " bash ${__TestBinDir}/__TEST_PATH__/__TEST_NAME__.sh -coreroot=${CORE_ROOT}"
</ExcludeList>
</ItemGroup>
- <!-- runtest.proj finds all the *.cmd/*.sh scripts in a test folder and creates corresponding test methods.
+ <!-- run.proj finds all the *.cmd/*.sh scripts in a test folder and creates corresponding test methods.
Exclude these scripts to avoid creating such methods for the superpmicollect dependent test projects
and running them separately from superpmicollect test. These should be excluded regardless of RuntimeFlavor/os/arch-->
echo ''
echo 'Typical command line:'
echo ''
- echo '
coreclr/tests/runtest.sh <arch> <configurations>'
+ echo '
src/tests/run.sh <arch> <configurations>'
echo ''
echo 'Optional arguments:'
echo ' --testRootDir=<path> : Root directory of the test build (e.g. runtime/artifacts/tests/Windows_NT.x64.Debug).'
echo ''
echo 'CoreCLR test runner wrapper script.'
echo ''
- echo 'Run tests using runtest.sh, then rerun the failures, if any,'
+ echo 'Run tests using run.sh, then rerun the failures, if any,'
echo 'until the number of failures stabilizes. Thus, when running'
echo 'flaky tests, or running tests on a flaky platform, only the'
echo 'repeatable, "real", failures are reported.'
echo ''
- echo 'Tests are rerun in sequential mode (passing --sequential to runtest.sh).'
+ echo 'Tests are rerun in sequential mode (passing --sequential to run.sh).'
echo 'This hopefully avoids resource exhaustion and other parallel run problems.'
echo ''
echo 'A maximum number of iterations can be specified.'
echo ''
echo 'Command line:'
echo ''
- echo 'runtesttilstable.sh [options] [arguments for runtest.sh]'
+ echo 'runtesttilstable.sh [options] [arguments for run.sh]'
echo ''
- echo 'Any unknown argument is passed directly to runtest.sh.'
+ echo 'Any unknown argument is passed directly to run.sh.'
echo ''
echo 'Optional arguments:'
echo ' -h|--help : Show usage information.'
;;
--testRootDir=*)
testRootDir=${i#*=}
- # Also pass it on to runtest.sh
+ # Also pass it on to run.sh
__UnprocessedBuildArgs="$__UnprocessedBuildArgs $i"
;;
*)
esac
done
-# Check testRootDir; this check is also done by runtest.sh.
+# Check testRootDir; this check is also done by run.sh.
if [ -z "$testRootDir" ]; then
echo "--testRootDir is required."
# Now start running the tests.
-nextcmd="${scriptPath}/runtest.sh ${playlistArgument} ${__UnprocessedBuildArgs}"
+nextcmd="${scriptPath}/run.sh ${playlistArgument} ${__UnprocessedBuildArgs}"
echo "Running: $nextcmd"
$nextcmd
exitCode=$?
fi
mv "$testRootDir/coreclrtests.fail.txt" "$retryFile"
- nextcmd="${scriptPath}/runtest.sh --sequential --playlist=${retryFile} ${__UnprocessedBuildArgs}"
+ nextcmd="${scriptPath}/run.sh --sequential --playlist=${retryFile} ${__UnprocessedBuildArgs}"
echo "Running: $nextcmd"
$nextcmd
exitCode=$?
projects / platform / upstream / dotnet / runtime.git / commitdiff