[wasm] Add incremental build support (#57113)
authorAnkit Jain <radical@gmail.com>
Tue, 17 Aug 2021 16:58:50 +0000 (12:58 -0400)
committerGitHub <noreply@github.com>
Tue, 17 Aug 2021 16:58:50 +0000 (11:58 -0500)
commit3e3b00c53b127ec20e1d4d74bb75992fb650e08a
treed6341ffcdd2c9b47b01c7594f106d37e33a80246
parentb838bca7605b0935705dc2af4cbccd0e99e2d343
[wasm] Add incremental build support (#57113)

* Update tasks to support incremental build

MonoAOTCompiler:
- Compiles assemblies to .bc files.
- Hashes for the .bc files are stored in a json cache file.
- And uses .depfile generated by mono-aot-cross, to figure out the
  if any of the dependencies have changed
- Writes out the actual .bc file only if the assembly, or it's
  dependencies changed

EmccCompile.cs: Support a `%(Dependencies)` metadata on the source
files, to compile the files only when needed.

* Update wasm targets

* don't pass unncessary args to RunTests

* Add tests for incremental builds

* Disable non-wasm builds, for testing

* Add miggins LogAsErrorException.cs

* Bump sdk for workload testing to 6.0.100-rc.1.21410.3

* fix build

* MonoAOTCompiler: use the full path to copy the final .bc file

* Make the method used with `MemberData`, static

otherwise xunit just shows a cryptic:
```
    Wasm.Build.Tests.RebuildTests.NoOpRebuild [STARTING]
    Wasm.Build.Tests.RebuildTests.NoOpRebuild [FAIL]
      System.NotSupportedException : Specified method is not supported.
```

* add back builds

* Split Wasm.Build.Tests into multiple helix jobs

* Revert "add back builds"

This reverts commit 1d031c04e13780ec73180ba6f06a37ee42c24203.

* Split up native rebuild tests

* remove non-test classes

* add back builds

This reverts commit b008130a7886c2e2b9f16c83641c1b8c936082f6.

* MonoAOTCompiler: make cache optional

* MonoAOTCompiler: handle the case where we have a cache entry, but the file on disk doesn't exist

* Fix aot compiler task output

* MonoAOTCompiler: Use hashes of .bc files instead of assemblies

`--depfile` isn't supported on aot config used by android, and fails
with:

```
* Assertion at /__w/1/s/src/mono/mono/mini/aot-compiler.c:14216, condition `acfg->aot_opts.llvm_only && acfg->aot_opts.asm_only && acfg->aot_opts.llvm_outfile' not met
```

Instead, use hashes of the .bc.tmp files generated, with the existing
.bc files.

* MonoAOTCompiler: Support more than one output file

The earlier implementation assumed that there would be only one output
file. But in some cases (eg. android), there are more than one, like
`.s`, `.dll-llvm.o`.

* -bump sdk for workload testing

* MonoAOTCompiler: don't use tmp files at all, when cache isn't being

.. used.

Co-authored-by: Larry Ewing <lewing@microsoft.com>
25 files changed:
eng/Versions.props
eng/testing/scenarios/BuildWasmAppsJobsList.txt [new file with mode: 0644]
eng/testing/tests.wasm.targets
src/libraries/sendtohelixhelp.proj
src/mono/wasm/build/WasmApp.Native.targets
src/mono/wasm/build/WasmApp.targets
src/tasks/AotCompilerTask/MonoAOTCompiler.cs
src/tasks/AotCompilerTask/MonoAOTCompiler.csproj
src/tasks/Common/LogAsErrorException.cs [new file with mode: 0644]
src/tasks/Common/Utils.cs
src/tasks/WasmAppBuilder/EmccCompile.cs
src/tasks/WasmAppBuilder/IcallTableGenerator.cs
src/tasks/WasmAppBuilder/PInvokeTableGenerator.cs
src/tasks/WasmAppBuilder/WasmAppBuilder.cs
src/tests/BuildWasmApps/Wasm.Build.Tests/BuildAndRunAttribute.cs
src/tests/BuildWasmApps/Wasm.Build.Tests/BuildTestBase.cs
src/tests/BuildWasmApps/Wasm.Build.Tests/HelperExtensions.cs
src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/FlagsChangeRebuildTest.cs [new file with mode: 0644]
src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/NativeRebuildTestsBase.cs [new file with mode: 0644]
src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/NoopNativeRebuildTest.cs [new file with mode: 0644]
src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/ReferenceNewAssemblyRebuildTest.cs [new file with mode: 0644]
src/tests/BuildWasmApps/Wasm.Build.Tests/NativeRebuildTests/SimpleSourceChangeRebuildTest.cs [new file with mode: 0644]
src/tests/BuildWasmApps/Wasm.Build.Tests/RebuildTests.cs
src/tests/BuildWasmApps/Wasm.Build.Tests/SharedBuildPerTestClassFixture.cs
src/tests/BuildWasmApps/Wasm.Build.Tests/Wasm.Build.Tests.csproj