Enable WASM AOT microbenchmark run (#55353)

* bring up WASM AOT microbenchmarks run

* bring up WASM AOT microbenchmarks run

* adding diagnostice log for testing

* commend out some jobs while testing to save lab run resource

* commend out some jobs while testing to save lab run resource

* Add WASM AOT in not scheduled run

* Only run selected jobs

* add "ls -l" to print out permission before calling performance-setup.sh

* add "ls -l" to print out permission before calling performance-setup.sh

* Add "--buildTimeout 3600" to benchmarkdotnet to extend default timeout time. Adjust -aotcompilermode format to "--aotcompilermode=wasm".

* Add "--buildTimeout 3600" to benchmarkdotnet to extend default timeout time. Adjust -aotcompilermode format to "--aotcompilermode=wasm".

* Correct runtimeSrcDir value

* Correct --runtimeSrcDir location to $HELIX_CORRELATION_PAYLOAD

* Correct --runtimeSrcDir location to $HELIX_CORRELATION_PAYLOAD

* Remove not used parameters in WASM AOT mode and  add temperary diagnostic info

* Remove not used parameters in WASM AOT mode and  add temperary diagnostic info

* make \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm to be the root directory of runtime repo source

* Only copy runtime repo source directory to helix payload when it's wasm aot

* Add diagnosic log for testing only

* omite unneeded runtime repo file from helix payload

* omite unneeded runtime repo file from helix payload

* use rsync instead of cp exclusion command since cp doesn't work with sub directory as expected.

* Add temp diagnostic logging, should be removed before PR.

* remove artifacts/obj from helix payload

* remove artifacts/obj from helix payload

* copy back helix job for diagnosis and silent rsync

* For testing only, clone performance repor alicial/hikeMicro-benchmark private branch, which contains copying back helix job folder for diagnosis

* enable rsync verbose for diagnosis

* add --keepfiles for wasm intepreter as well for diagnosis

* Remove temporary diagnostic code

* recover all the jobs in the pipeline

* recover all the jobs in the pipeline

* copy wasm build drop to the location that aot build expects

* using private perf branch to enable binlog for benchmarkdotnet for diagnosis

* comment out none wasm runs temporarily for testing only

* comment out none wasm runs temporarily for testing only

* add "--keepfiles" to keep diagnostic files

* fixed $wasm_dotnet_path/artifacts/BrowserWasm/artifacts path

* instead of copy, move $wasm_dotnet_path/artifacts/BrowserWasm/artifacts/* to save space

* benchmarkdotnet using --cli \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm/dotnet.sh instead

* benchmarkdotnet using --cli \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm/dotnet.sh instead

* add executable permission

* Enable other non run to compare

* Enable other non run to compare

* remove "--cli \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm/dotnet.sh"

* fixes per CR

* copy EMSDK_PATH to expected location in helix payload

* install emsdk

* fix emsdk path

* minor fix

* removed more not used files to reduce helix payload size

* remove more jobs for testing purpose

* Fix the download path

* Add BenchmarkDotNet sources to the payload

* Fix __download__ path

* set BenchmarkDotNetSources to use benchmarkdotnet built from source

* use benchmarkdotnet private branch:alicial/tempwasmaot for BenchmarkDotNetSources

* use benchmarkdonet private branch with BenchmarkDotNetSources option

* Debug prints

* Clone B.NET to different location

Fix rsync exclusion
Add more debug prints

* Move B.NET back, clone upstream B.NET

To avoid:

    [INFO] ERROR(S):
    [INFO]   Option 'BenchmarkDotNetSources' is unknown.

* Use my branch for performance repo

* Revert "set BenchmarkDotNetSources to use benchmarkdotnet built from source"

This reverts commit bccfadde684922c0440d928224c8e7ec7c220fe9.

To avoid:

    [INFO] ERROR(S):
    [INFO]   Option 'BenchmarkDotNetSources' is unknown.

* Disable one more job I missed in merge

* Switch back to nuget packages for BenchmarkDotNet

* tempararily use pr-alicias-branch to test new fixes

* [mono] Fix invalid memory write

Fixes https://github.com/dotnet/runtime/issues/56526 and probably also
https://github.com/dotnet/runtime/issues/53546

Allocate enough memory when `SIZEOF_REGISTER == 4`, so that code like

    defs [ins->dreg + 1] = NULL;
    defs [ins->dreg + 2] = NULL;

doesn't write after allocated range.

* [mono] Fix an uninitialized memory access in the ABCREM pass.

Detected by valgrind.

* clean up PR

* add missing param

* fix perf.yml and make wasm interpreter to be consistent with wasm AOT

* comment out more job for fast testing

* remove extra job

* exclude copying __download__ dir instead of delete it.

* add parameters.codeGenType

* fix aot javascriptengine

* correct livelibrariesbuildconfig from release to Release

* install emsdk before copy $source_directory

* add back all the temporarily commented out jobs in perf.yml

* clean up work around code

* minor clean up

* remove performance submodule

* switch back to use performance repo main branch

* add --quiet to git clone

Co-authored-by: Radek Doulik <radekdoulik@gmail.com>
Co-authored-by: Zoltan Varga <vargaz@gmail.com>

diff --git a/eng/pipelines/coreclr/perf.yml b/eng/pipelines/coreclr/perf.yml
index 6fc0b85..5726e5a 100644 (file)
--- a/eng/pipelines/coreclr/perf.yml
+++ b/eng/pipelines/coreclr/perf.yml
@@ -93,6 +93,25 @@ jobs:
         logicalmachine: 'perftiger'
         javascriptEngine: 'javascriptcore'
 
+  #run mono wasm aot microbenchmarks perf job
+  - template: /eng/pipelines/common/platform-matrix.yml
+    parameters:
+      jobtemplate: /eng/pipelines/coreclr/templates/perf-job.yml # note: should we move this file out of coreclr tempelates because it contains mono jobs?
+      buildconfig: release
+      runtimeflavor: wasm
+      platforms:
+      - linux_x64
+      jobparameters:
+        testgroup: perf
+        livelibrariesbuildconfig: Release
+        runtimetype: wasm
+        codegentype: 'aot'
+        projectfile: microbenchmarks.proj
+        runkind: micro
+        runjobtemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
+        logicalmachine: 'perftiger'
+        javascriptengine: 'javascriptcore'
+
 - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'Schedule')) }}:
 
   # build coreclr and libraries
@@ -288,7 +307,7 @@ jobs:
         runJobTemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
         logicalmachine: 'perftiger'
 
-  # run mono wasm microbenchmarks perf job
+  # run mono wasm interpreter (default) microbenchmarks perf job
   - template: /eng/pipelines/common/platform-matrix.yml
     parameters:
       jobTemplate: /eng/pipelines/coreclr/templates/perf-job.yml # NOTE: should we move this file out of coreclr tempelates because it contains mono jobs?
@@ -307,6 +326,25 @@ jobs:
         logicalmachine: 'perftiger'
         javascriptEngine: 'v8'
 
+  #run mono wasm aot microbenchmarks perf job
+  - template: /eng/pipelines/common/platform-matrix.yml
+    parameters:
+      jobtemplate: /eng/pipelines/coreclr/templates/perf-job.yml # note: should we move this file out of coreclr tempelates because it contains mono jobs?
+      buildconfig: release
+      runtimeflavor: wasm
+      platforms:
+      - linux_x64
+      jobparameters:
+        testgroup: perf
+        livelibrariesbuildconfig: Release
+        runtimetype: wasm
+        codegentype: 'aot'
+        projectfile: microbenchmarks.proj
+        runkind: micro
+        runjobtemplate: /eng/pipelines/coreclr/templates/run-performance-job.yml
+        logicalmachine: 'perftiger'
+        javascriptEngine: 'v8'
+
   # run mono aot microbenchmarks perf job
   - template: /eng/pipelines/common/platform-matrix.yml
     parameters:

diff --git a/eng/pipelines/coreclr/templates/perf-job.yml b/eng/pipelines/coreclr/templates/perf-job.yml
index 59b7843..b3f66a2 100644 (file)
--- a/eng/pipelines/coreclr/templates/perf-job.yml
+++ b/eng/pipelines/coreclr/templates/perf-job.yml
@@ -56,9 +56,9 @@ jobs:
       - ${{ format('libraries_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.liveLibrariesBuildConfig) }}
     - ${{ if and(eq(parameters.runtimeType, 'mono'), ne(parameters.codeGenType, 'AOT')) }}:
       - ${{ format('mono_{0}_product_build_{1}{2}_{3}_{4}', parameters.runtimeVariant, parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }}
-    - ${{ if eq(parameters.runtimeType, 'wasm') }}:
+    - ${{ if eq(parameters.runtimeType, 'wasm')}}:
       - ${{ format('build_{0}{1}_{2}_{3}_{4}', 'Browser', '', 'wasm', parameters.buildConfig, parameters.runtimeType) }}
-    - ${{ if eq(parameters.codeGenType, 'AOT')}}:
+    - ${{ if and(eq(parameters.codeGenType, 'AOT'), ne(parameters.runtimeType, 'wasm'))}}:
       - ${{ format('build_{0}{1}_{2}_{3}_{4}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.codeGenType) }}
     - ${{ if eq(parameters.runtimeType, 'AndroidMono')}}:
       - ${{ 'build_Android_arm64_release_AndroidMono' }}
@@ -73,9 +73,11 @@ jobs:
     ${{ if and(ne(parameters.osGroup, 'windows'), not(in(parameters.runtimeType, 'AndroidMono', 'iOSMono'))) }}:
       ${{ if and(eq(parameters.runtimeType, 'mono'), ne(parameters.codeGenType, 'AOT')) }}:
         extraSetupParameters: --architecture ${{ parameters.archType }} --monodotnet $(Build.SourcesDirectory)/.dotnet-mono
-      ${{ if eq(parameters.runtimeType, 'wasm') }}:
+      ${{ if and(eq(parameters.runtimeType, 'wasm'), ne(parameters.codeGenType, 'AOT')) }}:
         extraSetupParameters: --architecture ${{ parameters.archType }} --wasm $(librariesDownloadDir)/bin/wasm --javascriptengine ${{ parameters.javascriptEngine }}
-      ${{ if eq(parameters.codeGenType, 'AOT') }}:
+      ${{ if and(eq(parameters.runtimeType, 'wasm'), eq(parameters.codeGenType, 'AOT')) }}:
+        extraSetupParameters: --architecture ${{ parameters.archType }} --wasm $(librariesDownloadDir)/bin/wasm --wasmaot --javascriptengine ${{ parameters.javascriptEngine }}
+      ${{ if and(eq(parameters.codeGenType, 'AOT'), ne(parameters.runtimeType, 'wasm')) }}:
         extraSetupParameters: --architecture ${{ parameters.archType }} --monoaot $(librariesDownloadDir)/bin/aot
       ${{ if and(eq(parameters.runtimeType, 'coreclr'), ne(parameters.osSubGroup, '_musl')) }}:
         extraSetupParameters: --corerootdirectory $(Build.SourcesDirectory)/artifacts/tests/coreclr/${{ parameters.osGroup }}.${{ parameters.archType }}.Release/Tests/Core_Root --architecture ${{ parameters.archType }}
@@ -134,8 +136,8 @@ jobs:
       - script: "mkdir $(librariesDownloadDir)/bin/wasm;unzip -o $(librariesDownloadDir)/BrowserWasm/artifacts/packages/Release/Shipping/Microsoft.NETCore.App.Runtime.Mono.browser-wasm.6.0.0-ci.nupkg data/* runtimes/* -d $(librariesDownloadDir)/bin/wasm;cp src/mono/wasm/runtime-test.js $(librariesDownloadDir)/bin/wasm/runtime-test.js;find $(librariesDownloadDir)/bin/wasm -type f -exec chmod 664 {} \\;"
         displayName: "Create wasm directory (Linux)"
 
-    # Download AOT
-    - ${{ if eq(parameters.codeGenType, 'AOT') }}:
+    # Download mono AOT
+    - ${{ if and(eq(parameters.codeGenType, 'AOT'), ne(parameters.runtimeType, 'wasm')) }}:
       - template: /eng/pipelines/common/download-artifact-step.yml
         parameters:
           unpackFolder: $(librariesDownloadDir)/LinuxMonoAOT

diff --git a/eng/testing/performance/performance-setup.sh b/eng/testing/performance/performance-setup.sh
index 671c627..c99ad1f 100755 (executable)
--- a/eng/testing/performance/performance-setup.sh
+++ b/eng/testing/performance/performance-setup.sh
@@ -124,6 +124,10 @@ while (($# > 0)); do
       wasm_runtime_loc=$2
       shift 2
       ;;
+    --wasmaot)
+      wasmaot=true
+      shift 1
+      ;;
     --compare)
       compare=true
       shift 1
@@ -158,6 +162,7 @@ while (($# > 0)); do
       echo "  --internal                     If the benchmarks are running as an official job."
       echo "  --monodotnet                   Pass the path to the mono dotnet for mono performance testing."
       echo "  --wasm                         Path to the unpacked wasm runtime pack."
+      echo "  --wasmaot                      Indicate wasm aot"  
       echo "  --latestdotnet                 --dotnet-versions will not be specified. --dotnet-versions defaults to LKG version in global.json "
       echo "  --alpine                       Set for runs on Alpine"
       echo ""
@@ -184,6 +189,7 @@ fi
 
 payload_directory=$source_directory/Payload
 performance_directory=$payload_directory/performance
+benchmark_directory=$payload_directory/BenchmarkDotNet
 workitem_directory=$source_directory/workitem
 extra_benchmark_dotnet_arguments="--iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --stopOnFirstError true"
 perflab_arguments=
@@ -228,7 +234,11 @@ if [[ "$mono_dotnet" != "" ]] && [[ "$monointerpreter" == "false" ]]; then
 fi
 
 if [[ "$wasm_runtime_loc" != "" ]]; then
-    configurations="CompilationMode=wasm RunKind=$kind"
+    if [[ "$wasmaot" == "true" ]]; then
+        configurations="CompilationMode=wasm AOT=true RunKind=$kind"
+    else
+        configurations="CompilationMode=wasm RunKind=$kind"
+    fi
     if [[ "$javascript_engine" == "javascriptcore" ]]; then
       configurations="$configurations JSEngine=javascriptcore"
     fi
@@ -254,8 +264,10 @@ if [[ "$run_from_perf_repo" = true ]]; then
     performance_directory=$workitem_directory
     setup_arguments="--perf-hash $commit_sha $common_setup_arguments"
 else
-    git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance $performance_directory
-    
+    git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $performance_directory
+    # uncomment to use BenchmarkDotNet sources instead of nuget packages
+    # git clone https://github.com/dotnet/BenchmarkDotNet.git $benchmark_directory
+
     docs_directory=$performance_directory/docs
     mv $docs_directory $workitem_directory
 fi
@@ -264,7 +276,21 @@ if [[ "$wasm_runtime_loc" != "" ]]; then
     using_wasm=true
     wasm_dotnet_path=$payload_directory/dotnet-wasm
     mv $wasm_runtime_loc $wasm_dotnet_path
-    extra_benchmark_dotnet_arguments="$extra_benchmark_dotnet_arguments --wasmMainJS \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm/runtime-test.js --wasmEngine /home/helixbot/.jsvu/$javascript_engine --customRuntimePack \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm"
+    # install emsdk, $source_directory/src/mono/wasm/ has the nuget.config with require feed. EMSDK may be available in the payload in a different directory, should visit this install to avoid deplicated payload.
+    pushd $source_directory/src/mono/wasm/
+    make provision-wasm
+    EMSDK_PATH = $source_directory/src/mono/wasm/emsdk
+    popd
+    # wasm aot and interpreter need some source code from dotnet\runtime repo
+    rsync -aq --progress $source_directory/* $wasm_dotnet_path --exclude Payload --exclude docs --exclude src/coreclr --exclude src/tests --exclude artifacts/obj --exclude artifacts/log --exclude artifacts/tests --exclude __download__
+    # copy wasm build drop to the location that aot and interpreter build expects
+    rsync -a --progress $wasm_dotnet_path/artifacts/BrowserWasm/artifacts/* $wasm_dotnet_path/artifacts
+    rm -r $wasm_dotnet_path/artifacts/BrowserWasm/artifacts
+    if [[ "$wasmaot" == "true" ]]; then
+        extra_benchmark_dotnet_arguments="$extra_benchmark_dotnet_arguments --wasmEngine /home/helixbot/.jsvu/$javascript_engine --runtimeSrcDir \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm --aotcompilermode wasm --buildTimeout 3600" 
+    else
+        extra_benchmark_dotnet_arguments="$extra_benchmark_dotnet_arguments --wasmEngine /home/helixbot/.jsvu/$javascript_engine --runtimeSrcDir \$HELIX_CORRELATION_PAYLOAD/dotnet-wasm"
+    fi
 fi
 
 if [[ "$mono_dotnet" != "" ]] && [[ "$monoaot" == "false" ]]; then