From: Ankit Jain Date: Fri, 18 Feb 2022 16:16:31 +0000 (-0500) Subject: [wasm] Rationalize wasm jobs run on CI (#65035) X-Git-Tag: accepted/tizen/unified/riscv/20231226.055536~10738 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=463bd7585479a8bf454df9d65d0aa5199d82e1fd;p=platform%2Fupstream%2Fdotnet%2Fruntime.git [wasm] Rationalize wasm jobs run on CI (#65035) * Rationalize wasm jobs run on CI Essentially: - For rolling builds, we want to run all the wasm jobs, unconditionally - And `runtime-extra-platforms` runs the tests that were not in `runtime`, and `runtime-staging` - For PRs, we use the various path checks Changes: - split wasm jobs into various templates - combine, and simplify some jobs * `runtime-extra-platforms`, and `runtime-wasm` run only when manually triggered with a comment - `/azp run ` * `runtime`, and `runtime-staging`, run jobs only when relevant paths change. And for `EAT`, and `AOT`, only smoke tests are run. * And when `runtime-wasm` is triggered manually, it runs *all* the wasm jobs completely | . | runtime | runtime-staging | runtime-extra-platforms(manual only) | runtime-wasm (manual only) | | ----------------- | -------------------- | --------------- | ------------------------------------ | ------- | | libtests | linux: all, only-pc | windows: all, only-pc | linux+windows: all, only-pc | linux+windows: all, always | | libtests eat | linux: smoke, only-pc | - | linux: all, only-pc | linux: all, always | | libtests aot | linux: smoke, only-pc | windows: smoke, only-pc | linux+windows: all, only-pc | linux+windows: all, always | | | | | | | | Wasm.Build.Tests | linux: only-pc | windows: only-pc | linux+windows: only-pc | linux+windows | | Debugger tests | - | linux+windows: only-pc | linux+windows: only-pc | linux+windows | | Runtime tests | linux: only-pc | - | linux: only-pc | linux | * `runtime`, and `runtime-staging`, run all the wasm jobs unconditionally, but `EAT`, and `AOT` still run only smoke tests. * `runtime-extra-platforms` also runs by default. And it runs only the cases not covered by the above two pipelines. * jobs w/o `only-pc` are always run | . | runtime | runtime-staging | runtime-extra-platforms (always run) | runtime-wasm (manual only) | | ----------------- | ------------- | --------------- | ------------------------------------ | ------ | | libtests | linux: all(v8/chr/node) | windows: all | linux+windows: all with node | N/A | | libtests eat | linux: smoke | - | linux: all | | | libtests aot | linux: smoke | windows: smoke | linux+windows: all | | | | | | | | | Wasm.Build.Tests | linux: always | windows: always | none | | | Debugger tests | - | linux+windows: always | none | | | Runtime tests | linux: always | - | none | | * [wasm] Run tests with node only on runtime-wasm --- diff --git a/eng/pipelines/common/templates/wasm-build-tests.yml b/eng/pipelines/common/templates/wasm-build-tests.yml new file mode 100644 index 0000000..cd142b5 --- /dev/null +++ b/eng/pipelines/common/templates/wasm-build-tests.yml @@ -0,0 +1,41 @@ +parameters: + alwaysRun: false + isExtraPlatformsBuild: false + platforms: [] + +jobs: + +# +# Build the whole product using Mono and run libraries tests, for Wasm.Build.Tests +# +- template: /eng/pipelines/common/platform-matrix.yml + parameters: + jobTemplate: /eng/pipelines/common/global-build-job.yml + helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml + buildConfig: Release + runtimeFlavor: mono + platforms: ${{ parameters.platforms }} + variables: + # map dependencies variables to local variables + - name: wasmbuildtestsContainsChange + value: $[ dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'] ] + - name: alwaysRunVar + value: ${{ parameters.alwaysRun }} + jobParameters: + isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} + testGroup: innerloop + nameSuffix: WasmBuildTests + buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmBuildTests=true /p:TestAssemblies=false /p:BrowserHost=$(_hostedOs) + timeoutInMinutes: 180 + condition: >- + or( + eq(variables['alwaysRunVar'], true), + eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'], true)) + # extra steps, run tests + extraStepsTemplate: /eng/pipelines/libraries/helix.yml + extraStepsParameters: + creator: dotnet-bot + testRunNamePrefixSuffix: Mono_$(_BuildConfig)_$(_hostedOs) + extraHelixArguments: /p:BrowserHost=$(_hostedOs) + scenarios: + - buildwasmapps diff --git a/eng/pipelines/common/templates/wasm-debugger-tests.yml b/eng/pipelines/common/templates/wasm-debugger-tests.yml new file mode 100644 index 0000000..9d5bb43 --- /dev/null +++ b/eng/pipelines/common/templates/wasm-debugger-tests.yml @@ -0,0 +1,39 @@ +parameters: + alwaysRun: false + isExtraPlatformsBuild: false + platforms: [] + +jobs: + +# Wasm debugger tests - windows +- template: /eng/pipelines/common/platform-matrix.yml + parameters: + jobTemplate: /eng/pipelines/common/global-build-job.yml + helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml + buildConfig: Release + runtimeFlavor: mono + platforms: ${{ parameters.platforms }} + variables: + # map dependencies variables to local variables + - name: wasmdebuggertestsContainsChange + value: $[ dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'] ] + - name: alwaysRunVar + value: ${{ parameters.alwaysRun }} + jobParameters: + testGroup: innerloop + isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} + nameSuffix: Mono_DebuggerTests + buildArgs: -s mono+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmDebuggerTests=true /p:TestAssemblies=false /p:BrowserHost=$(_hostedOs) + timeoutInMinutes: 180 + condition: >- + or( + eq(variables['alwaysRunVar'], true), + eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'], true)) + # extra steps, run tests + extraStepsTemplate: /eng/pipelines/libraries/helix.yml + extraStepsParameters: + creator: dotnet-bot + testRunNamePrefixSuffix: Mono_$(_BuildConfig) + extraHelixArguments: /p:BrowserHost=$(_hostedOs) + scenarios: + - wasmdebuggertests diff --git a/eng/pipelines/common/templates/wasm-library-aot-tests.yml b/eng/pipelines/common/templates/wasm-library-aot-tests.yml new file mode 100644 index 0000000..395d845 --- /dev/null +++ b/eng/pipelines/common/templates/wasm-library-aot-tests.yml @@ -0,0 +1,24 @@ +parameters: + alwaysRun: false + extraBuildArgs: '' + extraHelixArgs: '' + isExtraPlatformsBuild: false + nameSuffix: '' + platforms: [] + runAOT: false + runSmokeOnlyArg: '' + +jobs: + +# +# Build for Browser/wasm, with EnableAggressiveTrimming=true +# +- template: /eng/pipelines/common/templates/wasm-library-tests.yml + parameters: + platforms: ${{ parameters.platforms }} + nameSuffix: ${{ parameters.nameSuffix }} + isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} + extraBuildArgs: /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=${{ parameters.runAOT }} ${{ parameters.extraBuildArgs }} + extraHelixArgs: /p:NeedsToBuildWasmAppsOnHelix=true ${{ parameters.extraHelixArgs }} + alwaysRun: ${{ parameters.alwaysRun }} + runSmokeOnlyArg: $(_runSmokeTestsOnlyArg) diff --git a/eng/pipelines/common/templates/wasm-library-tests.yml b/eng/pipelines/common/templates/wasm-library-tests.yml new file mode 100644 index 0000000..56bf074 --- /dev/null +++ b/eng/pipelines/common/templates/wasm-library-tests.yml @@ -0,0 +1,51 @@ +parameters: + alwaysRun: false + extraBuildArgs: '' + extraHelixArgs: '' + isExtraPlatformsBuild: false + nameSuffix: '' + platforms: [] + runSmokeOnlyArg: '' + scenarios: ['normal'] + +jobs: + +# +# Build for Browser/wasm and test it +# +- template: /eng/pipelines/common/platform-matrix.yml + parameters: + jobTemplate: /eng/pipelines/common/global-build-job.yml + helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml + buildConfig: Release + runtimeFlavor: mono + platforms: ${{ parameters.platforms }} + variables: + # map dependencies variables to local variables + - name: librariesContainsChange + value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] + - name: monoContainsChange + value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] + - name: alwaysRunVar + value: ${{ parameters.alwaysRun }} + jobParameters: + isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} + testGroup: innerloop + nameSuffix: LibraryTests${{ parameters.nameSuffix }} + buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:BrowserHost=$(_hostedOs) ${{ parameters.runSmokeOnlyArg }} ${{ parameters.extraBuildArgs }} + timeoutInMinutes: 180 + # always run for runtime-wasm builds (triggered manually) + # Always run for rolling builds + # Else run on path changes + condition: >- + or( + eq(variables['alwaysRunVar'], true), + eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), + eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true)) + # extra steps, run tests + extraStepsTemplate: /eng/pipelines/libraries/helix.yml + extraStepsParameters: + creator: dotnet-bot + testRunNamePrefixSuffix: Mono_$(_BuildConfig) + extraHelixArguments: /p:BrowserHost=$(_hostedOs) ${{ parameters.runSmokeOnlyArg }} ${{ parameters.extraHelixArgs }} + scenarios: ${{ parameters.scenarios }} diff --git a/eng/pipelines/common/templates/wasm-runtime-tests.yml b/eng/pipelines/common/templates/wasm-runtime-tests.yml new file mode 100644 index 0000000..58ace91 --- /dev/null +++ b/eng/pipelines/common/templates/wasm-runtime-tests.yml @@ -0,0 +1,39 @@ +parameters: + alwaysRun: false + isExtraPlatformsBuild: false + platforms: [] + +jobs: + +# +# Build the whole product using Mono and run runtime tests +# +- template: /eng/pipelines/common/platform-matrix.yml + parameters: + jobTemplate: /eng/pipelines/common/global-build-job.yml + helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml + buildConfig: Release + runtimeFlavor: mono + platforms: ${{ parameters.platforms }} + variables: + - name: alwaysRunVar + value: ${{ parameters.alwaysRun }} + - name: timeoutPerTestInMinutes + value: 10 + - name: timeoutPerTestCollectionInMinutes + value: 200 + jobParameters: + testGroup: innerloop + isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} + nameSuffix: AllSubsets_Mono_RuntimeTests + buildArgs: -s mono+libs -c $(_BuildConfig) + timeoutInMinutes: 180 + condition: >- + or( + eq(variables['alwaysRunVar'], true), + eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), + eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true)) + extraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.yml + extraStepsParameters: + creator: dotnet-bot + testRunNamePrefixSuffix: Mono_$(_BuildConfig) diff --git a/eng/pipelines/runtime-extra-platforms-wasm.yml b/eng/pipelines/runtime-extra-platforms-wasm.yml index af51523..9dc8b84 100644 --- a/eng/pipelines/runtime-extra-platforms-wasm.yml +++ b/eng/pipelines/runtime-extra-platforms-wasm.yml @@ -4,230 +4,92 @@ # /azp run runtime-wasm parameters: - isExtraPlatformsBuild: '' + isWasmOnlyBuild: false + isRollingBuild: false jobs: # -# Build for Browser/wasm and test it on v8, browser, and nodejs +# ************ Rolling builds: ************* +# - only run eat, and aot tests +# - rest are covered by runtime, and runtime-staging # -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - - Browser_wasm_win - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: LibraryTests - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:BrowserHost=$(_hostedOs) - timeoutInMinutes: 180 - # always run for runtime-wasm builds (triggered manually) - # Always run for rolling builds - # Else run on path changes - condition: >- - or( - eq(variables['isWasmOnlyBuild'], true), - eq(variables['isRollingBuild'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - extraHelixArguments: /p:BrowserHost=$(_hostedOs) - scenarios: - - normal - - WasmTestOnBrowser - - WasmTestOnNodeJs +- ${{ if eq(parameters.isRollingBuild, true) }}: + # EAT Library tests - only run on linux + - template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm + nameSuffix: _EAT + runAOT: false + alwaysRun: true -# -# Build for Browser/wasm, with EnableAggressiveTrimming=true -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} - testGroup: innerloop - nameSuffix: LibraryTests_EAT - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=false - timeoutInMinutes: 180 - # always run for runtime-wasm builds (triggered manually) - # Always run for rolling builds - # Else run on path changes - condition: >- - or( - eq(variables['isWasmOnlyBuild'], true), - eq(variables['isRollingBuild'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true $(_runSmokeTestsOnlyArg) - scenarios: - - normal + # AOT Library tests + - template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm + - Browser_wasm_win + nameSuffix: _AOT + runAOT: true + alwaysRun: true # -# Build for Browser/wasm with RunAOTCompilation=true +# ********** For !rolling builds, IOW - PR builds ************* +# - run everything, if relevant paths changed +# - For runtime-wasm, force run all the jobs # -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - - Browser_wasm_win - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - isExtraPlatforms: ${{ parameters.isExtraPlatformsBuild }} - testGroup: innerloop - nameSuffix: LibraryTests_AOT - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true /p:BrowserHost=$(_hostedOs) - timeoutInMinutes: 180 - # always run for runtime-wasm builds (triggered manually) - # Always run for rolling builds - # Else run on path changes - condition: >- - or( - eq(variables['isWasmOnlyBuild'], true), - eq(variables['isRollingBuild'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_AOT_$(_BuildConfig)_$(_hostedOs) - extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true $(_runSmokeTestsOnlyArg) /p:BrowserHost=$(_hostedOs) - scenarios: +- ${{ if ne(parameters.isRollingBuild, true) }}: + # Library tests + - template: /eng/pipelines/common/templates/wasm-library-tests.yml + parameters: + platforms: + - Browser_wasm + - Browser_wasm_win + # Don't run for rolling builds, as this is covered + alwaysRun: ${{ parameters.isWasmOnlyBuild }} + scenarios: - normal + - WasmTestOnBrowser + - WasmTestOnNodeJs -## WebAssembly legs, run only when runtime-wasm is triggered + # EAT Library tests - only run on linux + - template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm + nameSuffix: _EAT + runAOT: false + alwaysRun: ${{ parameters.isWasmOnlyBuild }} -# -# runtime-wasm: Build the whole product using Mono and run libraries tests, for Wasm.Build.Tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - - Browser_wasm_win - jobParameters: - testGroup: innerloop - nameSuffix: WasmBuildTests - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmBuildTests=true /p:TestAssemblies=false /p:BrowserHost=$(_hostedOs) - timeoutInMinutes: 180 - # always run for runtime-wasm builds (triggered manually) - condition: >- - eq(variables['isWasmOnlyBuild'], true) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig)_$(_hostedOs) - extraHelixArguments: /p:BrowserHost=$(_hostedOs) - scenarios: - - buildwasmapps + # AOT Library tests + - template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm + - Browser_wasm_win + nameSuffix: _AOT + runAOT: true + alwaysRun: ${{ parameters.isWasmOnlyBuild }} -# -# runtime-wasm: Build the whole product using Mono and run libraries tests, for wasm debugger tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - - Browser_wasm_win - jobParameters: - testGroup: innerloop - nameSuffix: Mono_DebuggerTests - buildArgs: -s mono+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmDebuggerTests=true /p:TestAssemblies=false /p:BrowserHost=$(_hostedOs) - timeoutInMinutes: 180 - # always run for runtime-wasm builds (triggered manually) - condition: >- - eq(variables['isWasmOnlyBuild'], true) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig)_$(_hostedOs) - extraHelixArguments: /p:BrowserHost=$(_hostedOs) - scenarios: - - wasmdebuggertests + # Wasm.Build.Tests + - template: /eng/pipelines/common/templates/wasm-build-tests.yml + parameters: + platforms: + - Browser_wasm + - Browser_wasm_win + alwaysRun: ${{ parameters.isWasmOnlyBuild }} -# -# runtime-wasm: Build the whole product using Mono and run runtime tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - - ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}: - - name: _HelixSource - value: pr/dotnet/runtime/$(Build.SourceBranch) - - ${{ if and(eq(variables['System.TeamProject'], 'public'), ne(variables['Build.Reason'], 'PullRequest')) }}: - - name: _HelixSource - value: ci/dotnet/runtime/$(Build.SourceBranch) - - name: timeoutPerTestInMinutes - value: 10 - - name: timeoutPerTestCollectionInMinutes - value: 200 - jobParameters: - testGroup: innerloop - nameSuffix: AllSubsets_Mono_RuntimeTests - buildArgs: -s mono+libs -c $(_BuildConfig) - timeoutInMinutes: 180 - # always run for runtime-wasm builds (triggered manually) - condition: >- - eq(variables['isWasmOnlyBuild'], true) - extraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) + # Debugger tests + - template: /eng/pipelines/common/templates/wasm-debugger-tests.yml + parameters: + platforms: + - Browser_wasm + - Browser_wasm_win + alwaysRun: ${{ parameters.isWasmOnlyBuild }} + - template: /eng/pipelines/common/templates/wasm-runtime-tests.yml + parameters: + platforms: + - Browser_wasm + alwaysRun: ${{ parameters.isWasmOnlyBuild }} diff --git a/eng/pipelines/runtime-staging.yml b/eng/pipelines/runtime-staging.yml index 5ff5ac6..fab86e3 100644 --- a/eng/pipelines/runtime-staging.yml +++ b/eng/pipelines/runtime-staging.yml @@ -55,6 +55,7 @@ variables: - template: /eng/pipelines/common/variables.yml jobs: + # # Evaluate paths # @@ -62,6 +63,42 @@ jobs: - template: /eng/pipelines/common/evaluate-default-paths.yml # +# WebAssembly legs - run windows builds. Also, debugger tests which are currently +# unstable +# +- template: /eng/pipelines/common/templates/wasm-library-tests.yml + parameters: + platforms: + - Browser_wasm_win + alwaysRun: ${{ variables.isRollingBuild }} + scenarios: + - normal + - WasmTestOnBrowser + +# AOT Library tests +- template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm_win + nameSuffix: _AOT + runSmokeOnlyArg: $(_runSmokeTestsOnlyArg) + runAOT: true + alwaysRun: ${{ variables.isRollingBuild }} + +- template: /eng/pipelines/common/templates/wasm-build-tests.yml + parameters: + platforms: + - Browser_wasm_win + alwaysRun: ${{ variables.isRollingBuild }} + +- template: /eng/pipelines/common/templates/wasm-debugger-tests.yml + parameters: + platforms: + - Browser_wasm + - Browser_wasm_win + alwaysRun: ${{ variables.isRollingBuild }} + +# # iOS/tvOS interp - requires AOT Compilation and Interp flags # Build the whole product using Mono and run libraries tests # @@ -221,184 +258,6 @@ jobs: testRunNamePrefixSuffix: Mono_$(_BuildConfig) # -# Build the whole product using Mono and run libraries tests, for Wasm.Build.Tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm_win - variables: - # map dependencies variables to local variables - - name: wasmbuildtestsContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'] ] - jobParameters: - isExtraPlatforms: ${{ variables.isExtraPlatformsBuild }} - testGroup: innerloop - nameSuffix: WasmBuildTests - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmBuildTests=true /p:TestAssemblies=false /p:BrowserHost=$(_hostedOs) - timeoutInMinutes: 180 - condition: >- - eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'], true) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig)_$(_hostedOs) - extraHelixArguments: /p:BrowserHost=$(_hostedOs) - scenarios: - - buildwasmapps - condition: >- - eq(variables['wasmbuildtestsContainsChange'], true) - -# -# Build Browser_wasm, on windows, run console and browser tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: release - runtimeFlavor: mono - platforms: - - Browser_wasm_win - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: LibraryTests - buildArgs: -subset mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:BrowserHost=windows - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Windows_wasm_$(_BuildConfig) - extraHelixArguments: /p:BrowserHost=windows - scenarios: - - normal - - wasmtestonbrowser - condition: >- - or( - eq(variables['librariesContainsChange'], true), - eq(variables['monoContainsChange'], true), - eq(variables['isRollingBuild'], true)) - -# -# Build for Browser/wasm with RunAOTCompilation=true -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm_win - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - isExtraPlatforms: ${{ variables.isExtraPlatformsBuild }} - testGroup: innerloop - nameSuffix: LibraryTests_AOT - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true /p:BrowserHost=$(_hostedOs) - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_AOT_$(_BuildConfig)_$(_hostedOs) - extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true $(_runSmokeTestsOnlyArg) /p:BrowserHost=$(_hostedOs) - scenarios: - - normal - condition: >- - or( - eq(variables['librariesContainsChange'], true), - eq(variables['monoContainsChange'], true)) - -# Wasm debugger tests - windows -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm_win - variables: - # map dependencies variables to local variables - - name: wasmdebuggertestsContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: Mono_DebuggerTests - buildArgs: -s mono+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmDebuggerTests=true /p:TestAssemblies=false /p:BrowserHost=windows - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'], true), - eq(variables['isRollingBuild'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - extraHelixArguments: /p:BrowserHost=windows - scenarios: - - wasmdebuggertests - -# Wasm debugger tests - linux -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - # map dependencies variables to local variables - - name: wasmdebuggertestsContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: Mono_DebuggerTests - buildArgs: -s mono+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmDebuggerTests=true /p:TestAssemblies=false - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'], true), - eq(variables['isRollingBuild'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - scenarios: - - wasmdebuggertests - -# # Build the whole product using Mono for Android and run runtime tests with Android devices # - template: /eng/pipelines/common/platform-matrix.yml diff --git a/eng/pipelines/runtime-wasm.yml b/eng/pipelines/runtime-wasm.yml index da464d4..62de930 100644 --- a/eng/pipelines/runtime-wasm.yml +++ b/eng/pipelines/runtime-wasm.yml @@ -17,4 +17,5 @@ jobs: - template: /eng/pipelines/runtime-extra-platforms-wasm.yml parameters: - isExtraPlatformsBuild: false + isWasmOnlyBuild: ${{ variables.isWasmOnlyBuild }} + isRollingBuild: ${{ variables.isRollingBuild }} diff --git a/eng/pipelines/runtime.yml b/eng/pipelines/runtime.yml index 5f6b973..d6ab57e 100644 --- a/eng/pipelines/runtime.yml +++ b/eng/pipelines/runtime.yml @@ -355,46 +355,48 @@ jobs: eq(variables['isRollingBuild'], true)) # -# Build the whole product using Mono and run libraries tests, multi-scenario +# WebAssembly legs # -- template: /eng/pipelines/common/platform-matrix.yml +- template: /eng/pipelines/common/templates/wasm-library-tests.yml parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono platforms: - - Browser_wasm - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: LibraryTests - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true), - eq(variables['isRollingBuild'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - scenarios: - - normal - - WasmTestOnBrowser - condition: >- - or( - eq(variables['librariesContainsChange'], true), - eq(variables['monoContainsChange'], true), - eq(variables['isRollingBuild'], true)) + - Browser_wasm + alwaysRun: ${{ variables.isRollingBuild }} + scenarios: + - normal + - WasmTestOnBrowser + +# EAT Library tests - only run on linux +- template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm + nameSuffix: _EAT + runAOT: false + runSmokeOnlyArg: $(_runSmokeTestsOnlyArg) + alwaysRun: ${{ variables.isRollingBuild }} + +# AOT Library tests +- template: /eng/pipelines/common/templates/wasm-library-aot-tests.yml + parameters: + platforms: + - Browser_wasm + nameSuffix: _AOT + runAOT: true + runSmokeOnlyArg: $(_runSmokeTestsOnlyArg) + alwaysRun: ${{ variables.isRollingBuild }} + +- template: /eng/pipelines/common/templates/wasm-build-tests.yml + parameters: + platforms: + - Browser_wasm + alwaysRun: ${{ variables.isRollingBuild }} + +- template: /eng/pipelines/common/templates/wasm-runtime-tests.yml + parameters: + platforms: + - Browser_wasm + alwaysRun: ${{ variables.isRollingBuild }} # # iOS/tvOS devices - Full AOT + AggressiveTrimming to reduce size @@ -481,163 +483,6 @@ jobs: eq(variables['isRollingBuild'], true)) # -# Build the whole product using Mono and run libraries tests, for Wasm.Build.Tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - # map dependencies variables to local variables - - name: wasmbuildtestsContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: WasmBuildTests - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:TestWasmBuildTests=true /p:TestAssemblies=false - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'], true), - eq(variables['isRollingBuild'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - scenarios: - - buildwasmapps - condition: >- - or( - eq(variables['wasmbuildtestsContainsChange'], true), - eq(variables['isRollingBuild'], true)) - -# -# Build for Browser/wasm, with EnableAggressiveTrimming=true -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: LibraryTests_EAT - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=false - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true), - eq(variables['isRollingBuild'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true $(_runSmokeTestsOnlyArg) - scenarios: - - normal - condition: >- - or( - eq(variables['librariesContainsChange'], true), - eq(variables['monoContainsChange'], true), - eq(variables['isRollingBuild'], true)) - -# -# Build for Browser/wasm with RunAOTCompilation=true -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - # map dependencies variables to local variables - - name: librariesContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ] - - name: monoContainsChange - value: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ] - jobParameters: - testGroup: innerloop - nameSuffix: LibraryTests_AOT - buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true $(_runSmokeTestsOnlyArg) /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_installer.containsChange'], true), - eq(variables['isRollingBuild'], true)) - # extra steps, run tests - extraStepsTemplate: /eng/pipelines/libraries/helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true $(_runSmokeTestsOnlyArg) - scenarios: - - normal - condition: >- - or( - eq(variables['librariesContainsChange'], true), - eq(variables['monoContainsChange'], true), - eq(variables['isRollingBuild'], true)) - -# -# Build the whole product using Mono and run runtime tests -# -- template: /eng/pipelines/common/platform-matrix.yml - parameters: - jobTemplate: /eng/pipelines/common/global-build-job.yml - helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml - buildConfig: Release - runtimeFlavor: mono - platforms: - - Browser_wasm - variables: - - ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}: - - name: _HelixSource - value: pr/dotnet/runtime/$(Build.SourceBranch) - - ${{ if and(eq(variables['System.TeamProject'], 'public'), ne(variables['Build.Reason'], 'PullRequest')) }}: - - name: _HelixSource - value: ci/dotnet/runtime/$(Build.SourceBranch) - - name: timeoutPerTestInMinutes - value: 10 - - name: timeoutPerTestCollectionInMinutes - value: 200 - jobParameters: - testGroup: innerloop - nameSuffix: AllSubsets_Mono_RuntimeTests - buildArgs: -s mono+libs -c $(_BuildConfig) - timeoutInMinutes: 180 - condition: >- - or( - eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true), - eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true), - eq(variables['isRollingBuild'], true)) - extraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.yml - extraStepsParameters: - creator: dotnet-bot - testRunNamePrefixSuffix: Mono_$(_BuildConfig) - -# # Build Mono and Installer on LLVMJIT mode # - template: /eng/pipelines/common/platform-matrix.yml diff --git a/src/mono/wasm/README.md b/src/mono/wasm/README.md index a016b9a..86e7b56 100644 --- a/src/mono/wasm/README.md +++ b/src/mono/wasm/README.md @@ -194,3 +194,42 @@ Bumping Emscripten version involves these steps: * Is enforced via [eslint](https://eslint.org/) and rules are in `./.eslintrc.js` * You could check the style by running `npm run lint` in `src/mono/wasm/runtime` directory * You can install [plugin into your VS Code](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) to show you the errors as you type + + +## Builds on CI + +* For PRs, tests are generally triggered based on path changes. But if you have a change which would not trigger the relevant builds, then you can run `runtime-wasm` pipeline manually to run all of them. Comment `/azp run runtime-wasm` on the PR. + +### How do I know which jobs run on CI, and when? + +## PR: +* `runtime-extra-platforms`, and `runtime-wasm` run only when manually triggered with a comment - `/azp run ` +* `runtime`, and `runtime-staging`, run jobs only when relevant paths change. And for `EAT`, and `AOT`, only smoke tests are run. +* And when `runtime-wasm` is triggered manually, it runs *all* the wasm jobs completely + +| . | runtime | runtime-staging | runtime-extra-platforms(manual only) | runtime-wasm (manual only) | +| ----------------- | -------------------- | --------------- | ------------------------------------ | ------- | +| libtests | linux: all, only-pc | windows: all, only-pc | linux+windows: all, only-pc | linux+windows: all, always | +| libtests eat | linux: smoke, only-pc | - | linux: all, only-pc | linux: all, always | +| libtests aot | linux: smoke, only-pc | windows: smoke, only-pc | linux+windows: all, only-pc | linux+windows: all, always | +| | | | | | +| Wasm.Build.Tests | linux: only-pc | windows: only-pc | linux+windows: only-pc | linux+windows | +| Debugger tests | - | linux+windows: only-pc | linux+windows: only-pc | linux+windows | +| Runtime tests | linux: only-pc | - | linux: only-pc | linux | + +## Rolling build (twice a day): + +* `runtime`, and `runtime-staging`, run all the wasm jobs unconditionally, but `EAT`, and `AOT` still run only smoke tests. +* `runtime-extra-platforms` also runs by default. And it runs only the cases not covered by the above two pipelines. + +* jobs w/o `only-pc` are always run + +| . | runtime | runtime-staging | runtime-extra-platforms (always run) | runtime-wasm (manual only) | +| ----------------- | ------------- | --------------- | ------------------------------------ | ------ | +| libtests | linux: all(v8/chr) | windows: all | none | N/A | +| libtests eat | linux: smoke | - | linux: all | | +| libtests aot | linux: smoke | windows: smoke | linux+windows: all | | +| | | | | | +| Wasm.Build.Tests | linux: always | windows: always | none | | +| Debugger tests | - | linux+windows: always | none | | +| Runtime tests | linux: always | - | none | |