Single-File: MemoryMap the bundle file (dotnet/core-setup#6576)

* Single-File: MemoryMap the bundle file

This changes makes a few changes in preparation for
loading IL files from the single-file bundle.

**Memory Mapping**

At startup, the entire bundle-file is memory mapped.
The individual embedded files are then extracted out to actual files on disk on startup.
However, this change will facilitate later checkins to:
* Loading files directly from bundle (ex: IL files using PELoader, config files using memory-streams, etc ).
* Extracting out files to disk lazily
Further details are available in [this document](https://github.com/dotnet/designs/blob/master/accepted/single-file/design.md#the-host)

**Refactoring**

* Static bundle_runner: The bundle_runner is made a static class that holds a global pointer
  to the memory-mapped application bundle, and meta-data about the contents.
  This facilitates the implementation of call-backs for accessing content directly from the bundle,
  as explained in [this document](https://github.com/dotnet/designs/blob/master/accepted/single-file/direct.md)

* A few new classes are created by factoring out functionality from bundle_runner
   * Extractor: Functionality for extracting contents of a bundle to files on disk.
   * Reader: This is a helper class that implements sequentional reading on memory-mapped bundle file.
   * dir-Util: Extraction directory handling utilities that assist in processing the bundle.

* Naming: bundle::bundle_runner is renamed as bundle::runner to remove the redundancy.
  Similarly, the newly refactored classes are named bundle::extractor and bundle::reader.

Commit migrated from https://github.com/dotnet/core-setup/commit/9bb119b2a4b82d54e73669928a3b831cc07a1291

Increase bundle's button size: avoid loc clipping (dotnet/core-setup#7757)

Commit migrated from https://github.com/dotnet/core-setup/commit/6762e5be2e74bcf35a38a1c1cd842d1ac61e0357

adding ci legs to release 3.1 (dotnet/core-setup#7756)

Commit migrated from https://github.com/dotnet/core-setup/commit/b5212c7d4d0a0c6a8b2cdb310f377653e509f9ac

Add single quotes to warnAsError:false (dotnet/core-setup#7765)

Commit migrated from https://github.com/dotnet/core-setup/commit/8a5f710c474a955ae82ce449fb9185a781782558

Use existing Runtime loc for WD Runtime (dotnet/core-setup#7758)

Copy over branding from .NET Core Runtime bundle installer to Windows Desktop Runtime bundle installer, then remove the "WelcomeHeaderMessage" and "WelcomeDescription" strings that could be confusing in the WD context. The result is a WindowsDesktop bundle installer with no useful description, but with UI localization.

Commit migrated from https://github.com/dotnet/core-setup/commit/00dfc821b3ba4a56add8161ad6549e8f70ae5add

SharedHost always installs to 'ProgramFiles'\dotnet (dotnet/core-setup#7658)

* When installing the SharedHost; copy all files to the traditional default install location: 'ProgramFiles'\dotnet

Commit migrated from https://github.com/dotnet/core-setup/commit/36894a82f9c0a24d4142b48597e764fd722a9c20

HostModel:Enable TestWithAdditionalContentAfterBundleMetadata (dotnet/core-setup#6968)

The above test was disabled in PR dotnet/core-setup#6885 because the lab runs
showed failures because of "not enough disk space"

This PR now enables the test, since the test job no longer uses the hosted pool.

Commit migrated from https://github.com/dotnet/core-setup/commit/b44e5a25b376cfa6e0581cd69af05ce655d72f40

 Switch to Arcade yaml stage-based publishing with custom steps (dotnet/core-setup#7725)

* Switch to stage-based publishing with custom steps

* Workaround: allow known symbol publisher warnings

Use Arcade-supported symbol publish workaround

* Workaround: run gather drop dry to avoid flakiness

* Use hosted pool for custom publish

Commit migrated from https://github.com/dotnet/core-setup/commit/002bc9ef5d0d6d433321df325b5928aaf80282a4

Enable hosted builds on arm64 (dotnet/core-setup#7696)

To enable building core-setup on arm64 (hosted, not cross compiled), we
need to do a few things:

- Set the right TargetArchitecture

  Use the currently running architecture to decide the HostArch. Use that
  as the default value of TargetArchitecture, unless another value was
  provided by the build configuration.

- Use the right coreclr JIT

  If we are cross-compiling, we need to use the x86_arm64 libclrjit.so.
  But if we are building on an arm64 host, we need to pick the normal-RID
  (eg, linux-arm64) libclrjit.so from the two:

    ./.packages/transport.runtime.linux-arm64.microsoft.netcore.jit/###/runtimes/linux-arm64/native/libclrjit.so
    ./.packages/transport.runtime.linux-arm64.microsoft.netcore.jit/###/runtimes/x64_arm64/native/libclrjit.so

- Use a version of SourceLink that supports arm64

  We need to upgrade SourceLink to a version that contains
  https://github.com/dotnet/sourcelink/pull/288. This commit just
  updates it to the latest version.

Fixes dotnet/core-setup#7653

Commit migrated from https://github.com/dotnet/core-setup/commit/4b82b1edad5fb7b1f9e865a8429c2da33c921854

Add HostModel AppHost tests (dotnet/core-setup#7586)

This change ports the Import [AppHost updater tests](https://github.com/dotnet/sdk/blob/dotnet/core-setup@a1cc8ec5a9741a5a2191fe022e12dd8a7f7348c7/src/Tasks/Microsoft.NET.Build.Tasks.UnitTests/GivenAnAppHost.cs) previously in SDK branch to work with HostModel library.

Commit migrated from https://github.com/dotnet/core-setup/commit/aae3e31d0011b09c8277d2f06f00b07e1fba8cb5

HostModel: Retry ResourceUpdate on Win32 File-Lock error (dotnet/core-setup#7665)

* HostModel: Retry ResourceUpdate on Win32 File-Lock error

On Win32 PE files, the SDK copies resources on the AppHost binary
from the intermediate assembly.

This update is performed using native Win32 operations.
This change retries the resource update if the operation fails
because the file is locked (say because of AntiVirus scan).

This change is similar to: https://github.com/dotnet/core-setup/pull/7617
which handles failures in managed code via IOException.

The Retry logic is factored out to the RetryUtil class. Currently the
HostWriter is the only client for the class. But it is made public
because other components (like Bundler) may use it in future.

Commit migrated from https://github.com/dotnet/core-setup/commit/6f3ead36128f5ef3287a847fe9f4cf0d1177a057

Minor fix in Native Hosting doc (dotnet/core-setup#7647)

Commit migrated from https://github.com/dotnet/core-setup/commit/0bd21a0ed332e425a708237d93d0e115acbd3885

Adds PresentationUI.dll to WPF Profile (dotnet/core-setup#7661)

* Fixes https://github.com/dotnet/wpf/issues/1423

Add PresentationUI.dll to WPF profile

* Update WPF Dependencies

Commit migrated from https://github.com/dotnet/core-setup/commit/0588e55d5e1401e8b4e43ebcdef4d992c9bd451b

HostWriter: Update Retry-count (dotnet/core-setup#7654)

Update Retry counters in HostWriter to match the
retry numbers in AppHost.

Commit migrated from https://github.com/dotnet/core-setup/commit/98b76e058ed209f98e56627da08afb032d22a0a4

HostModel: Retry updates on IO error (dotnet/core-setup#7617)

* HostModel: Retry updates on IO error

The HostModel implements services for updating the AppHost DLL
- Write App-dll name
- Set GUI/CUI bit
- Read/Write BundleHeader marker

These updates involve multiple file open/close operations.
An Antivirus scanner may intercept in-between and cause these operations
to fail with IO-Error.

So, this change adds a few retries to the IO operations, similar to
https://github.com/dotnet/core-setup/pull/7135

Fix for dotnet/core-setup#7597

Commit migrated from https://github.com/dotnet/core-setup/commit/f19be99463d48d2fae98030d26c5f394a18d7df6

Skip framework validation for secondary components loaded into a self-contained app (dotnet/core-setup#7618)

* Skip framework validation for secondary components loaded into a
self-contained app

* Add tests for loading a component in a self-contained app

Commit migrated from https://github.com/dotnet/core-setup/commit/4789e65dc121d81431d88ef2a7377be735d51a10

Reference WPF theme assemblies by default (dotnet/core-setup#7601)

* Reference WPF theme assemblies by default

* Fix test expecting >0 ReferencedByDefault attrs

Commit migrated from https://github.com/dotnet/core-setup/commit/6f12a57e3c1a5e0bc592745b2648b041819ae4ed

Add root "shared" dir to WindowsDesktop zip (dotnet/core-setup#7581)

Also fix a small bug where both the new and old infrastructure were creating a zip for the NETCoreApp sfx. This wasn't causing a problem in parallel because they are ordered, but it made it unclear to a reader which infra was responsible for the zip.

Commit migrated from https://github.com/dotnet/core-setup/commit/732ef42a49e82b1bfbe1363fa6799b34d7e8ef34

Add support for ProgID registration (dotnet/core-setup#7551)

* Add support for ProgID registration

* Update COM activation documentation

Commit migrated from https://github.com/dotnet/core-setup/commit/490c40397e4645cf0c47e528466b6d4a301cc04c

Remove WindowsDesktop bundle WelcomeDescription (dotnet/core-setup#7569)

Commit migrated from https://github.com/dotnet/core-setup/commit/edd4598118afda2032bb58b4be5fcfff5fcfc335

Use Prod pool provider (dotnet/core-setup#7571)

Commit migrated from https://github.com/dotnet/core-setup/commit/362cc2a93cfceaeb5e135503123eb7f09586bd03

Explicitly set DOTNET_MULTILEVEL_LOOKUP in tests (dotnet/core-setup#7541)

Commit migrated from https://github.com/dotnet/core-setup/commit/9b9814cc595dce09a126186d5926b1b19e8a09f3

Create WindowsDesktop sharedfx bundle installer (dotnet/core-setup#7540)

* Copy ExecWithRetries from BuildTools

Adds working dir, log importance to ExecWithRetries. Also Inline attemptMessage and add log importance comment.

* Add WindowsDesktop sfx bundle installer, refactor

Adds bundle creation support to the packaging-tools, and factors the framework pack and runtime WiX files into a single, more flexible, "product" set of WiX files.

The old MSI generation directory is removed entirely: with the new "product" WiX files, the host and hostfxr MSI creation can move to the new system. The VS insertion nupkg generation code is also ported from PowerShell to MSBuild to fit the new system.

The old system is still used for some macOS pkgs and the macOS bundle installer, some Debian/RPM packages, and some tarballs/zips.

Commit migrated from https://github.com/dotnet/core-setup/commit/ccf0024ba099103a5b759613c48b8cd46af8754a

Move DisableVSPublish out of template (dotnet/core-setup#7542)

Commit migrated from https://github.com/dotnet/core-setup/commit/9e5e0e555fe309c3ad749615969b1ae7b65afeb1

Fix how we determine line/column from a JsonReaderState (dotnet/core-setup#7510)

Previously these fields were public and so simple `.GetField("name")` worked. Now they are internal, so we need to use the overload which takes `BindingFlags` and ask for private reflection.

Commit migrated from https://github.com/dotnet/core-setup/commit/907f46239dd63de667797504fdb7fadbf8424a37

Merge pull request dotnet/core-setup#7398 from tlakollo/visualstudiopreview

Support Visual Studio Preview Install Path discovery at Build

Commit migrated from https://github.com/dotnet/core-setup/commit/be08a953bf365d9c52dddbcaa38745dbc39e198c

Tested that non preview Visual Studio versions work with the -prerelease
flag. Changing the code to match corefx implementation.

Commit migrated from https://github.com/dotnet/core-setup/commit/e0bd8f304182d4e25bd96252468063c2ecffb790

Make apphost.exe and comhost.dll MSI id stable (dotnet/core-setup#7387)

* Make apphost.exe and comhost.dll MSI id stable

Commit migrated from https://github.com/dotnet/core-setup/commit/5ce778beaf29a76d48c09713d951440ef610af4c

Enabling and conditioning the 'NuGetCommand' task. (dotnet/core-setup#7397)

* Enabling and conditioning the 'NuGetCommand' task.

* Update eng/jobs/windows-build.yml

Co-Authored-By: Davis Goodin <dagood@users.noreply.github.com>
* Update eng/jobs/windows-build.yml

Co-Authored-By: Davis Goodin <dagood@users.noreply.github.com>
* Updates per code review.

Commit migrated from https://github.com/dotnet/core-setup/commit/38fa929b45d17e98120f31643e79598c13592f40

New code snippet to support Visual Studio Install Path discovery at
build

Commit migrated from https://github.com/dotnet/core-setup/commit/98c47d473b4f751fd84019713c252e939b2c4415

Merge pull request dotnet/core-setup#7375 from wli3/port-fixes2

Simply aggregate exception happens in failed apphost deletion

Commit migrated from https://github.com/dotnet/core-setup/commit/0e8f2ed755a65bb52edbd6390ea8a35845999239

Simply aggregate exception happens in failed apphost deletion

Commit migrated from https://github.com/dotnet/core-setup/commit/bad4dd11c3fcb1fd685008da33841b82aa195ef2

Merge pull request dotnet/core-setup#7373 from wli3/port-fixes

port changes in SDK after Microsoft.NET.HostModel.AppHost is created

Commit migrated from https://github.com/dotnet/core-setup/commit/f5776f8b584d91b647306376bab5c9e8a75c6f92

Add targeting pack data/PackageOverrides.txt files (dotnet/core-setup#7369)

Commit migrated from https://github.com/dotnet/core-setup/commit/c2aa08390bbfe2b3dc8bee2210d9379b4bea4372

Delete output apphost upon failure of the CreateAppHost task.

Port https://github.com/dotnet/sdk/pull/3427 by @peterhuene with my edit

This commit deletes the output apphost when the CreateAppHost task fails from
an exception.

Partially fixes dotnet/core-setup#2989

-------------
* deletes the output apphost when the `CreateAppHost` task fails from
an exception.

* aggreates the Exception for future logging from deleting the failed apphost

Commit migrated from https://github.com/dotnet/core-setup/commit/7f0eb2869f2f37a899cdd02b7bbc4bd78e4e7038

Fix P/Invoke signatures for the resource updater.

Port https://github.com/dotnet/sdk/pull/3310 by @peterhuene

The resource updater was not specifying a CharSet for the Windows API
P/Invoke signatures and therefore the ANSI versions of the APIs were
being used.

This caused an unhandled exception when customizing the apphost on
Windows when the intermediate apphost path contained Unicode characters
because the ANSI resource updating API couldn't find the file.

Fixes ABdotnet/core-setup#900429.

Commit migrated from https://github.com/dotnet/core-setup/commit/0059721f0532ae386341fa15eccb7c0d37e5a6f3

Always build full platform manifest (dotnet/core-setup#7341)

Unless intentionally disabled or building from source, always create the full platform manifest that lists assets from all platforms. With the platform manifest included in runtime packs, it needs to be generated the same across all official build jobs. Producing it locally during a default build also reduces the diff from dev to official builds.

Commit migrated from https://github.com/dotnet/core-setup/commit/1bac3a54fe65834258718fbbc6c62369f0cbebf7

Fix platform manifest package id column (dotnet/core-setup#7322)

Fixes "__PackageId__" placeholder appearing in outputs. Added test to ensure this doesn't happen.

Also adds platform manifest to runtime packs.

Commit migrated from https://github.com/dotnet/core-setup/commit/7e3b0d0caf21b89608a3373e040041cc458722ba

Disable overriding CentOS as RHEL. (dotnet/core-setup#7318)

Commit migrated from https://github.com/dotnet/core-setup/commit/daeb68797d196f14368d8ec23aafb545e1c0f94e

Remove DiaSymReader prebuilt in source-build (dotnet/core-setup#7319)

Commit migrated from https://github.com/dotnet/core-setup/commit/22326dadefcc7115c0edbd41450dca65b8d163f8

Add ReferencedByDefault=false to WD theme DLLs (dotnet/core-setup#7320)

Commit migrated from https://github.com/dotnet/core-setup/commit/0366d3f06d9dadbf1c1253e8b975ce6cb3a317d5

Merge pull request dotnet/core-setup#7277 from AaronRobinsonMSFT/comhost_errorinfo_test_5113

Comhost IErrorInfo test

Commit migrated from https://github.com/dotnet/core-setup/commit/e3135aa808ffbbd9316d3e660b03b407a678a2eb

Add test for IErrorInfo propagation during COM activation failure.

Commit migrated from https://github.com/dotnet/core-setup/commit/a480b039445763a942746f32aa173fdfc40d2276

Propagate error writer to hostfxr during delegate acquisition.

Commit migrated from https://github.com/dotnet/core-setup/commit/92fb649965eb488cf5918d371cc1d73d323af2ff

Merge pull request dotnet/core-setup#7293 from dagood/arcade-merge/master

# Conflicts:
# eng/Version.Details.xml
# eng/Versions.props

Commit migrated from https://github.com/dotnet/core-setup/commit/970b3a5143dfda81556f2b5efc4bfc9c053465eb

Use access(2) to determine if a file/directory exists

access(2) does not need to validate and fill in the buffer passed to the
kernel, so it's slightly faster.

Commit migrated from https://github.com/dotnet/core-setup/commit/7d70bbe81b4b86d491e30f957551441464c85375

Construct directory entry in place while reading directories

Commit migrated from https://github.com/dotnet/core-setup/commit/44650f0cdc1307f7d0a451a227a1b8d75f42a3fe

Use fstatat() in pal::readdir() instead of building a std::string

fstatat(2) is equivalent to stat(2), but it takes a file descriptor for
a directory, so that lookups are always relative to that file
descriptor.  There's no need to construct a temporary std::string.

Commit migrated from https://github.com/dotnet/core-setup/commit/6a6266a1ab4ba99b3c42b16cc8dc822db35fb45f

Merge branch 'master' into arcade-merge/master

# Conflicts:
# eng/Version.Details.xml
# eng/Versions.props

Commit migrated from https://github.com/dotnet/core-setup/commit/5e8221cfd2da93462ba35dc0c4bc23b0d6d3fd32

Fix up tests for 5.0

Commit migrated from https://github.com/dotnet/core-setup/commit/3b8cebc9528342972f0229314aa375f9b09d0038

Fix up 5.0 alpha 1 versioning w/ SemVer v2

Commit migrated from https://github.com/dotnet/core-setup/commit/cafcf54a8caccf388715618943cb641fd49f912f

Use tracing infrastructure to log realpath() and getcwd() errors

Add some more context to them (whenever possible, too).

Commit migrated from https://github.com/dotnet/core-setup/commit/053fa0473d74176fd4aca2e4ab12d47fbd787791

Merge branch 'master' into dev/dagood/into-master

# Conflicts:
# BranchInfo.props
# dir.props
# src/managed/CommonManaged.props
# src/test/HostActivation.Tests/DependencyResolution/PortableAppRidAssetResolution.cs
# src/test/HostActivation.Tests/DependencyResolution/ResolveComponentDependencies.cs
# tools-local/setuptools/dotnet-deb-tool/dotnet-deb-tool.csproj

Commit migrated from https://github.com/dotnet/core-setup/commit/76796fa18c0d23279f0d92205eaf3b8b62cd1a73

[dev/arcade-migration] Fix Linux package versions (dotnet/core-setup#7261)

* Fix dev deb/rpm build script

* Fix deb/rpm versions: use --ci, fix version props

Commit migrated from https://github.com/dotnet/core-setup/commit/8f90d9b792b3da42df518da28468ef1677107113

yaml updates for building snaps (dotnet/core-setup#7272)

Commit migrated from https://github.com/dotnet/core-setup/commit/5057bc7688b810721b13ae8e0fc1df54cd79c809

[dev/arcade-migration] Fix package and shared framework diffs (dotnet/core-setup#7259)

* Always include CoreCLR in NCA versions txt file

This makes it appear where it's expected. It shouldn't be a problem that it now appears in a package that doesn't truly use CoreCLR. If anything, it makes it easier to find, which could be useful.

* Fix M.NETCore.App.Internal build file differences

Fix GetDependencyDataFileInclusionDefaults to actually affect the condition.

Use a template for the props file and have each pkgproj fill it, so that each pkgproj can configure it. Use non-".Internal" name to refer to platform manifest to keep that filename consistent.

* Fix latest commit usage

* Fix names of deps.json and runtimeconfig.json

* Remove Packaging tools package from sfx deps json

* Add comment about props file placeholders

Commit migrated from https://github.com/dotnet/core-setup/commit/32d0950119e1537f136efc5ee5bb13b800f4bda0

[dev/arcade-migration] Fix regressions found in MSI diff (dotnet/core-setup#7238)

* Fix VersionComparisonDate: copy from buildtools

* Adjust ShortFrameworkName placement for sfxproj

Fixes MSIs missing the framework name. This commit also adds a check to prevent regression.

* Move ProductVersion usage from static to target

Fixes the netstandard package version and the version used for the layout, which fixes its install location in the MSI. Adds regression test.

Commit migrated from https://github.com/dotnet/core-setup/commit/11c7fb051ae5a3a417990008965168dc6acd887b

Specify /Zi and -g compile options to enable debugging information (dotnet/core-setup#7248)

Commit migrated from https://github.com/dotnet/core-setup/commit/27f0bb094c4ed99fdb44fa81ea8248ff4b21bc14

Fixes couple of places in the bundle runner where we might leak memory. (dotnet/core-setup#7171)

Note that in reality this is a non-issue: The leaks are only present if something fails and in case of failure the process will immediately exit anyway.

I did this mostly for my own education on how to write this correctly. And since I had the changes read, we might as well merge it.

The structures are simple enough that we can rely on RVO alone. In one case there will be a copy (when inserting the files into the list), but that should be lot faster than explicit allocation.

Also remove any default ctors defined as they don't add any value.

Commit migrated from https://github.com/dotnet/core-setup/commit/e02bb0a4fa18355495e6af653de92d75d5a26ad3

Use a relaxed JavaScriptEncoder for writing JSON to disk. (dotnet/core-setup#7241)

Commit migrated from https://github.com/dotnet/core-setup/commit/43118387f9edb83f0abe12120a071c988b8b9e04

Test component dependency resolution via native host tests (dotnet/core-setup#6919)

Change component dependency resolution tests to not use managed test app which requires full framework to run (and is slow). Instead implement component dependency resolution into native host test app and change all tests to use that.

This makes the tests much faster and it also allows for more flexible tests (it's easy to mock shared frameworks now).

Notable refactorings:

Implement error writer redirector in the native tests - this is necessary to correctly handle multi-threaded error redirection. Using C++ templates to get multiple distinct function pointers (for each thread separately)
Split hostfxr exports into its own class which can be reused by multiple tests
Introduce hostpolicy exports
Implement case class for component dependency resolution tests with helpers and such
Added test which validates per-assembly version resolution behavior

Commit migrated from https://github.com/dotnet/core-setup/commit/2870b217650af9f49ffa1b835b998d58e25638c8

Merge pull request dotnet/core-setup#7221 from wtgodbe/50

Update branding to 5.0

Commit migrated from https://github.com/dotnet/core-setup/commit/6bc158d4eb36a742a7926f85f12a0d3b8b3e9c0a

[dev/arcade-migration] Improve dev scenarios (dotnet/core-setup#7233)

* Add '/p:Subset=' support for dev scenarios

* Fix readme table generation

* Improve VS build: update helper projects, doc

* Document Subsets better; add polish and safety

* Update windows-instructions.md, add Subsets hint

Commit migrated from https://github.com/dotnet/core-setup/commit/c9c64e72f18b6bf16a07dffa0a60fcee8477ea98

Update Prerelease to alpha1

Commit migrated from https://github.com/dotnet/core-setup/commit/a77adb256164ad9028f8cd5d49427fd770539f77

Remove AppWithSubDirs hard-coded netcoreapp3.0

Commit migrated from https://github.com/dotnet/core-setup/commit/5b613dcbf66aeb7f5a694b9fad0ec740d95ea62f

Use SemVer v2 versioning

Commit migrated from https://github.com/dotnet/core-setup/commit/cf79d6fce1b6f2f9557caf2ddb4a10d1583da585

Update branding to 5.0

Commit migrated from https://github.com/dotnet/core-setup/commit/21217b1f4078188f9c2f1ac8390ee774db243ab1

Don't always skip sourcelink: use a targeted fix (dotnet/core-setup#7219)

Commit migrated from https://github.com/dotnet/core-setup/commit/ff7422d7346979b44e3cc3a9ec1235c62a57ac9e

Enable BAR publish (dotnet/core-setup#7217)

Commit migrated from https://github.com/dotnet/core-setup/commit/3fbed74b39cd63761b9981dea4f983b678a62fce

Merge pull request dotnet/core-setup#7216 from dagood/dev-merge-from-master

Merge branch 'master' => 'dev/arcade-migration'

Commit migrated from https://github.com/dotnet/core-setup/commit/d7b62c4b66063d6ec7e77460d1d7587e71c48dab

Merge branch 'master' into dev/arcade-migration

# Conflicts:
# BuildToolsVersion.txt
# NuGet.config
# dependencies.props
# eng/Versions.props
# eng/jobs/bash-build.yml
# eng/jobs/windows-build.yml

Commit migrated from https://github.com/dotnet/core-setup/commit/a0c4047ed17874f72af66cb26085821b1649d8f8

Merge pull request dotnet/core-setup#7210 from dagood/fix-arcade-platformmanifest

[dev/arcade-migration] Add simple sharedfx package test project, fix PlatformManifest creation

Commit migrated from https://github.com/dotnet/core-setup/commit/99d2a27673278823e11c39d57ac25ce925fd89a5

Avoid building WD on non-supported platforms

Commit migrated from https://github.com/dotnet/core-setup/commit/c823497b00d73f33e2d55626ebda090c8db42a4e

Add platform manifest, simpler set of RID props

Commit migrated from https://github.com/dotnet/core-setup/commit/a61c69df207317e6de0c18cc4e66835b71ccf06b

Use file, not env, for test context variables

This is the recommended Arcade way to transfer this information. The Arcade SDK doens't support passing env vars while running tests.

Commit migrated from https://github.com/dotnet/core-setup/commit/fb43a6a3abab1b25356b7ee146586d14bb20b195

Add simple sfx packaging tests

Commit migrated from https://github.com/dotnet/core-setup/commit/e611ece7a0313320b96eeea376a97219f901a8e1

Minor fixes to the native hosting doc (dotnet/core-setup#7122)

The main part is to remove the `.a` file reference since we don't actually need or produce it.

Commit migrated from https://github.com/dotnet/core-setup/commit/622202134ce0103c953714879c0bb1746aaac434

Merge pull request dotnet/core-setup#7153 from jcagme/restoreSources

Copy sources from Versions.props to NuGet.config

Commit migrated from https://github.com/dotnet/core-setup/commit/99c7780cb9e40835278152d23909c3079367b39d

Implement roll-forward policies for .NET Core SDK resolution. (dotnet/core-setup#6953)

Implements additional roll-forward policies for resolving a .NET
Core SDK based upon new SDK settings in `global.json`.

Two new values are now supported in `global.json` in the `sdk` settings:

* `rollForward` - the roll-forward policy to use (see below).
* `allowPrerelease` - controls whether or not preview SDKS may be resolved
(defaults to `true` to allow resolving prerelease SDKs).

The new roll-forward policies are:

* `patch` - If the requested SDK is installed, use it. Otherwise, use the
  latest installed patch level that matches the requested major, minor, and
  feature band.
* `feature` - If the requested major, minor, and feature band is installed, use
the latest patch level for the specified feature band.  Otherwise, roll-forward
to the next available feature band and use the latest patch level for that
feature band.
* `minor` - If the requested major, minor, and feature band is installed, use
the latest patch level for the specified feature band.  Otherwise, roll-forward
to the next available feature band available for the same major version and use
the latest patch level for that feature band.
* `major` - If the requested major, minor, and feature band is installed, use
the latest patch level for the specified feature band.  Otherwise, roll-forward
to the next available feature band available without restriction and use the
latest patch level for that feature band.
* `latestPatch` - Use the latest installed patch level that matches the
requested major, minor, and feature band.
* `latestFeature` - Use the latest installed patch level for the latest
installed feature band that matches the requested major and minor version.
* `latestMinor` - Use the latest installed patch level for the latest installed
feature band for the latest installed minor version that matches the requested
major version.
* `latestMajor` - Use the absolutely latest .NET Core SDK available.

For backwards compatibility, the policy when a `version` is not specified in
`global.json` will be `latestMajor`.  The default policy when a `version` is
specified but `rollForward` is not specified will be `patch`.

Commit migrated from https://github.com/dotnet/core-setup/commit/5cd0245860335e0b307f6a46b99c6f9244887318

Retry extraction of bundle to avoid issues with locked files by AV (dotnet/core-setup#7135)

Retry extraction of bundle to avoid issues with locked files by AV software
Address some memory leaks

Commit migrated from https://github.com/dotnet/core-setup/commit/552dba193c0f21a03687d2f14861c7a4dcf8dd0f

Skip pal::realpath() error logging while probing directories

Both paths calling pal::realpath() and check its return value to see if
the probed paths actually exist in the filesystem; they do have logic
to handle those paths not existing, and printing error messages while
doing so isn't helpful for the user.

This happens when an application is published with one user but is
executed with another user -- for instance, when using containers (see
issue dotnet/core-setup#4038 for details).

Fixes dotnet/core-setup#4038.

Commit migrated from https://github.com/dotnet/core-setup/commit/024d2c0150c1ea356cfb33b2bcec91b39509084c

Merge pull request dotnet/core-setup#7084 from dagood/arcade-sign-publish

[dev/arcade-migration] Sign and publish using Arcade SDK

Commit migrated from https://github.com/dotnet/core-setup/commit/8d445c071754cbb95b28b7b84d066adf92689660

Add publish to official build

This also modernizes publish flow. Each job stores bits in build artifacts, rather than using the dotnetcli blob storage account as an intermediate location.

Some leftover code from ProdCon v1 that was difficult to follow is also removed.

Dev "mock" official builds are also made less risky by accepting override variables and adding a safety that skips official variable groups when the build pipeline name is changed.

Commit migrated from https://github.com/dotnet/core-setup/commit/30f9f19d905ccd63f178af5a326629ad98c73159

Update issue-guide.md (dotnet/core-setup#7117)

Commit migrated from https://github.com/dotnet/core-setup/commit/c4105af1c2255a2eb390d08aac4bf8272c3554da

Add BuildTools publish step tasks and dependencies

Copies the closure of C# code from the BuildTools repo that's necessary for the Core-Setup publish tasks. There are no changes except automatically removing and sorting the using statements.

Source: https://github.com/dotnet/buildtools/tree/dotnet/core-setup@55d43483866c7caeeace96355add3a9b12fa5795

Using the existing BuildTools code reduces the risk of behavior differences vs. trying to find equivalents in Arcade. The upcoming new Arcade-powered publish functionality makes short-term effort to deduplicate these tasks throwaway work.

Commit migrated from https://github.com/dotnet/core-setup/commit/ea269ba71cc53b8bf217f20953fcbad85647033b

Update BuildTools to preview4-04104-01 (dotnet/core-setup#6671)

Commit migrated from https://github.com/dotnet/core-setup/commit/3a2d0293bc7717a6fa038f2a12a9a91391205b06

Add signing using Arcade SDK infrastructure

Arcade SDK doesn't support signing the containers that this repo produces, so this commit adds a structure that allows projects to depend on signed assets. This involves adding signing/*.proj projects for other projects to depend on, and implementing signing in a way that allows multiple passes.

Commit migrated from https://github.com/dotnet/core-setup/commit/29414edc1132925e64e703f377b308695c5b64ba

Add parameters struct to nethost get_hostfxr_path (dotnet/core-setup#7020)

Allow specifying dotnet_root to find the appropriate hostfxr under the
specified root

Commit migrated from https://github.com/dotnet/core-setup/commit/4ecb63fde6338bb8659c76ec21239bb165f8c2a4

Add FileUtils.CopyIntoDirectory for CoreFX change

Commit migrated from https://github.com/dotnet/core-setup/commit/228c1b004d846767b640bacb58431abe687ad973

Leave any cross-targeting file off RuntimeList.xml (dotnet/core-setup#7041)

Commit migrated from https://github.com/dotnet/core-setup/commit/32cdcac88710b83a83c25dd1e2be946e3c180931

Update timeouts to 120 minutes for all jobs (dotnet/core-setup#7040)

* Update timeouts to 120 minutes for all jobs

* Use default value of 120 in Windows & Linux jobs

* Revert OSX change

Commit migrated from https://github.com/dotnet/core-setup/commit/97997f0c2429898dbf46124d39da872c3665ef18

Make Microsoft.NET.HostModel.AppHost.ResourceUpdater public (dotnet/core-setup#6989)

Make Microsoft.NET.HostModel.AppHost.ResourceUpdater class public,
so that it can be consumed from the SDK repo.

Commit migrated from https://github.com/dotnet/core-setup/commit/1d808bfb7cacb5ad6234a2f658d3fcd7d1ec5229

Fix test project restore for Preview 6 SDK

Preview 5 automatically added some daily build restore sources, but Preview 6 doesn't. This commit adds one back manually.

Commit migrated from https://github.com/dotnet/core-setup/commit/8cba83adbff69b28d4474b33156776699955061e

Merge 'master' into dev/arcade-migration

# Conflicts:
# build.proj
# eng/Versions.props

Commit migrated from https://github.com/dotnet/core-setup/commit/457e30a2ddde823477f245f3bde4e38a765b8332

Merge pull request dotnet/core-setup#6967 from dagood/arcade-reviewable

[dev/arcade-migration] Fix tests for CI platforms, add PR validation

Commit migrated from https://github.com/dotnet/core-setup/commit/ed69566cc944a319eb23b8281456e658d7ba822f

Fix official build MSI signing (dotnet/core-setup#7032)

Removes GenerateVSNugetPackages target dependencies causing MSIs to be built again after they were signed, overwriting the signed copies.

Commit migrated from https://github.com/dotnet/core-setup/commit/a2d5b0faba873d09b0c00918366799c5e2b102f3

Add dev/arcade-migration PR trigger

Commit migrated from https://github.com/dotnet/core-setup/commit/8d78a214ae6687ce1a8927ccf42952b7745f77c8

Fix Deb/RPM package build

This includes simplifying dotnet-deb-tool from a tool package to an ordinary project. This avoids two complications: tool packages used by DotNetCliToolReference can only target .NET Core 2.2 and below, and the Pack requirements are more involved with the new SDK.

Adds some new '<project> -> <artifact>' log lines. It was previously hard to tell what the build was producing; this output clarifies in the usual MSBuild project style.

Commit migrated from https://github.com/dotnet/core-setup/commit/58efb317558e34b805ec8e8650a3b3dcede21fcd

Fix build on macOS

Commit migrated from https://github.com/dotnet/core-setup/commit/09d46b89a25abfcbd20a425ce018d27950fb082a

Fix yaml CI definitions

Commit migrated from https://github.com/dotnet/core-setup/commit/4fc3d0f65b69c418daf663dfc350c02dfb84382e