* Initial change.
* Filename update.
* WBT for WASM.
* Updated docs.
* Run Hybrid wbt on CI.
* Feedback + hierarchy compatible with #86255.
* Feedback. Test program code to assets.
# Hybrid Globalization
-Description, purpose and instruction how to use.
+Originally, internalization data is loaded from ICU data files. In `HybridGlobalization` mode we are leveraging the platform-native internationalization APIs, where it is possible, to allow for loading smaller ICU data files. We still need to rely on ICU files because for a bunch of globalization data no API equivalent is available. For some existing equivalents, the behavior does not fully match the original. The differences you can expect after switching on the mode are listed in this document. Expected size savings can be found under each platform section below.
+
+Hybrid has lower priority than Invariant. To switch on the mode set the property in the build file:
+```
+<HybridGlobalization>true</HybridGlobalization>
+```
## Behavioral differences
### WASM
-For WebAssembly in Browser we are using Web API instead of some ICU data. Ideally, we would use `System.Runtime.InteropServices.JavaScript` to call JS code from inside of C# but we cannot reference any assemblies from inside of `System.Private.CoreLib`. That is why we are using iCalls instead.
+For WebAssembly in Browser we are using Web API instead of some ICU data. Ideally, we would use `System.Runtime.InteropServices.JavaScript` to call JS code from inside of C# but we cannot reference any assemblies from inside of `System.Private.CoreLib`. That is why we are using iCalls instead. The host support depends on used Web API functions support - see **dependencies** in each section.
+
+Hybrid has higher priority than sharding or custom modes, described in globalization-icu-wasm.md.
**SortKey**
ICU-based case change does not respect final-sigma rule, but hybrid does, so "ΒΌΛΟΣ" -> "βόλος", not "βόλοσ".
+Dependencies:
+- [String.prototype.toUpperCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toUpperCase)
+- [String.prototype.toLoweCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLowerCase)
+- [String.prototype.toLocaleUpperCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleUpperCase)
+- [String.prototype.toLocaleLoweCase()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/toLocaleLowerCase)
+
**String comparison**
Affected public APIs:
- String.Compare,
- String.Equals.
+Dependencies:
+- [String.prototype.localeCompare()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare)
+
The number of `CompareOptions` and `StringComparison` combinations is limited. Originally supported combinations can be found [here for CompareOptions](https://learn.microsoft.com/dotnet/api/system.globalization.compareoptions) and [here for StringComparison](https://learn.microsoft.com/dotnet/api/system.stringcomparison).
+
- `IgnoreWidth` is not supported because there is no equivalent in Web API. Throws `PlatformNotSupportedException`.
``` JS
let high = String.fromCharCode(65281) // %uff83 = ﾃ
- String.StartsWith
- String.EndsWith
+Dependencies:
+- [String.prototype.normalize()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize)
+- [String.prototype.localeCompare()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare)
+
Web API does not expose locale-sensitive endsWith/startsWith function. As a workaround, both strings get normalized and weightless characters are removed. Resulting strings are cut to the same length and comparison is performed. This approach, beyond having the same compare option limitations as described under **String comparison**, has additional limitations connected with the workaround used. Because we are normalizing strings to be able to cut them, we cannot calculate the match length on the original strings. Methods that calculate this information throw PlatformNotSupported exception:
- [CompareInfo.IsPrefix](https://learn.microsoft.com/en-us/dotnet/api/system.globalization.compareinfo.isprefix?view=net-8.0#system-globalization-compareinfo-isprefix(system-readonlyspan((system-char))-system-readonlyspan((system-char))-system-globalization-compareoptions-system-int32@))
Wasm.Build.Tests.CleanTests
Wasm.Build.Tests.ConfigSrcTests
Wasm.Build.Tests.IcuShardingTests
+Wasm.Build.Tests.HybridGlobalizationTests
Wasm.Build.Tests.InvariantGlobalizationTests
Wasm.Build.Tests.MainWithArgsTests
Wasm.Build.Tests.NativeBuildTests
<WasmAssembliesToBundle Include="@(ReferenceCopyLocalPaths)" Condition="'%(ReferenceCopyLocalPaths.Extension)' == '.dll'" />
</ItemGroup>
</Target>
-
+
<Target Name="_ResolveGlobalizationConfiguration">
<Error Condition="'$(BlazorIcuDataFileName)' != '' AND !$([System.IO.Path]::GetFileName('$(BlazorIcuDataFileName)').StartsWith('icudt'))" Text="File name in %24(BlazorIcuDataFileName) has to start with 'icudt'." />
<Warning Condition="'$(InvariantGlobalization)' == 'true' AND '$(BlazorWebAssemblyLoadAllGlobalizationData)' == 'true'" Text="%24(BlazorWebAssemblyLoadAllGlobalizationData) has no effect when %24(InvariantGlobalization) is set to true." />
<Warning Condition="'$(InvariantGlobalization)' == 'true' AND '$(BlazorIcuDataFileName)' != ''" Text="%24(BlazorIcuDataFileName) has no effect when %24(InvariantGlobalization) is set to true." />
<Warning Condition="'$(BlazorWebAssemblyLoadAllGlobalizationData)' == 'true' AND '$(BlazorIcuDataFileName)' != ''" Text="%24(BlazorIcuDataFileName) has no effect when %24(BlazorWebAssemblyLoadAllGlobalizationData) is set to true." />
+ <Warning Condition="'$(InvariantGlobalization)' == 'true' AND '$(HybridGlobalization)' == 'true'" Text="%24(HybridGlobalization) has no effect when %24(InvariantGlobalization) is set to true." />
+ <Warning Condition="'$(BlazorIcuDataFileName)' != '' AND '$(HybridGlobalization)' == 'true'" Text="%24(HybridGlobalization) has no effect when %24(BlazorIcuDataFileName) is set." />
<PropertyGroup>
+ <HybridGlobalization Condition="'$(BlazorIcuDataFileName)' != ''">false</HybridGlobalization>
<_BlazorWebAssemblyLoadAllGlobalizationData Condition="'$(InvariantGlobalization)' != 'true'">$(BlazorWebAssemblyLoadAllGlobalizationData)</_BlazorWebAssemblyLoadAllGlobalizationData>
<_BlazorWebAssemblyLoadAllGlobalizationData Condition="'$(_BlazorWebAssemblyLoadAllGlobalizationData)' == ''">false</_BlazorWebAssemblyLoadAllGlobalizationData>
- <_BlazorIcuDataFileName Condition="'$(InvariantGlobalization)' != 'true' AND '$(BlazorWebAssemblyLoadAllGlobalizationData)' != 'true'">$(BlazorIcuDataFileName)</_BlazorIcuDataFileName>
+ <_IsHybridGlobalization Condition="'$(InvariantGlobalization)' != 'true' AND '$(HybridGlobalization)' == 'true'"></_IsHybridGlobalization>
+ <_BlazorIcuDataFileName Condition="'$(InvariantGlobalization)' != 'true' AND '$(BlazorWebAssemblyLoadAllGlobalizationData)' != 'true' AND '$(HybridGlobalization)' != 'true'">$(BlazorIcuDataFileName)</_BlazorIcuDataFileName>
<_LoadCustomIcuData>false</_LoadCustomIcuData>
<_LoadCustomIcuData Condition="'$(_BlazorIcuDataFileName)' != ''">true</_LoadCustomIcuData>
</PropertyGroup>
<_WasmCopyOutputSymbolsToOutputDirectory Condition="'$(_WasmCopyOutputSymbolsToOutputDirectory)'==''">true</_WasmCopyOutputSymbolsToOutputDirectory>
<_WasmEnableThreads>$(WasmEnableThreads)</_WasmEnableThreads>
<_WasmEnableThreads Condition="'$(_WasmEnableThreads)' == ''">false</_WasmEnableThreads>
-
+
<_WasmEnableWebcil>$(WasmEnableWebcil)</_WasmEnableWebcil>
<_WasmEnableWebcil Condition="'$(TargetFrameworkIdentifier)' != '.NETCoreApp' or '$(_TargetingNET80OrLater)' != 'true'">false</_WasmEnableWebcil>
<_WasmEnableWebcil Condition="'$(_WasmEnableWebcil)' == ''">true</_WasmEnableWebcil>
LazyLoadedAssemblies="@(BlazorWebAssemblyLazyLoad)"
InvariantGlobalization="$(InvariantGlobalization)"
LoadCustomIcuData="$(_LoadCustomIcuData)"
+ IsHybridGlobalization="$(_IsHybridGlobalization)"
LoadAllICUData="$(_BlazorWebAssemblyLoadAllGlobalizationData)"
StartupMemoryCache="$(_BlazorWebAssemblyStartupMemoryCache)"
Jiterpreter="$(_BlazorWebAssemblyJiterpreter)"
<StaticWebAsset Include="@(_NewWebCilPublishStaticWebAssets)" />
<!-- TODO: Probably doesn't do anything as of now, original https://github.com/dotnet/aspnetcore/pull/34798 -->
- <PublishBlazorBootStaticWebAsset
+ <PublishBlazorBootStaticWebAsset
Include="@(StaticWebAsset)"
- Condition="'%(AssetKind)' != 'Build' and
+ Condition="'%(AssetKind)' != 'Build' and
(('%(StaticWebAsset.AssetTraitName)' == 'WasmResource' and '%(StaticWebAsset.AssetTraitValue)' != 'manifest' and '%(StaticWebAsset.AssetTraitValue)' != 'boot') or
'%(StaticWebAsset.AssetTraitName)' == 'Culture')" />
</ItemGroup>
</Target>
- <Target
+ <Target
Name="ComputeWasmExtensions"
AfterTargets="ProcessPublishFilesForWasm"
DependsOnTargets="$(ComputeBlazorExtensionsDependsOn)" >
LazyLoadedAssemblies="@(BlazorWebAssemblyLazyLoad)"
InvariantGlobalization="$(InvariantGlobalization)"
LoadCustomIcuData="$(_LoadCustomIcuData)"
+ IsHybridGlobalization="$(_IsHybridGlobalization)"
LoadAllICUData="$(_BlazorWebAssemblyLoadAllGlobalizationData)"
StartupMemoryCache="$(_BlazorWebAssemblyStartupMemoryCache)"
Jiterpreter="$(_BlazorWebAssemblyJiterpreter)"
bool expectCJK = false;
bool expectNOCJK = false;
bool expectFULL = false;
+ bool expectHYBRID = false;
switch (globalizationMode)
{
case GlobalizationMode.Invariant:
case GlobalizationMode.FullIcu:
expectFULL = true;
break;
+ case GlobalizationMode.Hybrid:
+ expectHYBRID = true;
+ break;
case GlobalizationMode.PredefinedIcu:
if (string.IsNullOrEmpty(predefinedIcudt))
throw new ArgumentException("WasmBuildTest is invalid, value for predefinedIcudt is required when GlobalizationMode=PredefinedIcu.");
AssertFilesExist(bundleDir, new[] { "icudt_EFIGS.dat" }, expectToExist: expectEFIGS);
AssertFilesExist(bundleDir, new[] { "icudt_CJK.dat" }, expectToExist: expectCJK);
AssertFilesExist(bundleDir, new[] { "icudt_no_CJK.dat" }, expectToExist: expectNOCJK);
+ AssertFilesExist(bundleDir, new[] { "icudt_hybrid.dat" }, expectToExist: expectHYBRID);
}
}
{
Invariant, // no icu
FullIcu, // full icu data: icudt.dat is loaded
- PredefinedIcu // user set WasmIcuDataFileName value and we are loading that file
+ PredefinedIcu, // user set WasmIcuDataFileName value and we are loading that file
+ Hybrid // reduced icu, missing data is provided by platform-native functions (web api for wasm)
};
public enum NativeFilesType { FromRuntimePack, Relinked, AOT };
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System.Collections.Generic;
+using System.IO;
+using Xunit;
+using Xunit.Abstractions;
+
+#nullable enable
+
+namespace Wasm.Build.Tests
+{
+ public class HybridGlobalizationTests : BuildTestBase
+ {
+ public HybridGlobalizationTests(ITestOutputHelper output, SharedBuildPerTestClassFixture buildContext)
+ : base(output, buildContext)
+ {
+ }
+
+ public static IEnumerable<object?[]> HybridGlobalizationTestData(bool aot, RunHost host)
+ => ConfigWithAOTData(aot)
+ .WithRunHosts(host)
+ .UnwrapItemsAsArrays();
+
+ [Theory]
+ [MemberData(nameof(HybridGlobalizationTestData), parameters: new object[] { /*aot*/ false, RunHost.All })]
+ [MemberData(nameof(HybridGlobalizationTestData), parameters: new object[] { /*aot*/ true, RunHost.All })]
+ public void AOT_HybridGlobalizationTests(BuildArgs buildArgs, RunHost host, string id)
+ => TestHybridGlobalizationTests(buildArgs, host, id);
+
+ [Theory]
+ [MemberData(nameof(HybridGlobalizationTestData), parameters: new object[] { /*aot*/ false, RunHost.All })]
+ public void RelinkingWithoutAOT(BuildArgs buildArgs, RunHost host, string id)
+ => TestHybridGlobalizationTests(buildArgs, host, id,
+ extraProperties: "<WasmBuildNative>true</WasmBuildNative>",
+ dotnetWasmFromRuntimePack: false);
+
+ private void TestHybridGlobalizationTests(BuildArgs buildArgs, RunHost host, string id, string extraProperties="", bool? dotnetWasmFromRuntimePack=null)
+ {
+ string projectName = $"hybrid";
+ extraProperties = $"{extraProperties}<HybridGlobalization>true</HybridGlobalization>";
+
+ buildArgs = buildArgs with { ProjectName = projectName };
+ buildArgs = ExpandBuildArgs(buildArgs, extraProperties);
+
+ if (dotnetWasmFromRuntimePack == null)
+ dotnetWasmFromRuntimePack = !(buildArgs.AOT || buildArgs.Config == "Release");
+
+ string programText = File.ReadAllText(Path.Combine(BuildEnvironment.TestAssetsPath, "Wasm.Buid.Tests.Programs", "HybridGlobalization.cs"));
+
+ BuildProject(buildArgs,
+ id: id,
+ new BuildProjectOptions(
+ InitProject: () => File.WriteAllText(Path.Combine(_projectDir!, "Program.cs"), programText),
+ DotnetWasmFromRuntimePack: dotnetWasmFromRuntimePack,
+ GlobalizationMode: GlobalizationMode.Hybrid));
+
+ string output = RunAndTestWasmApp(buildArgs, expectedExitCode: 42, host: host, id: id);
+ Assert.Contains("HybridGlobalization works, thrown exception as expected", output);
+ }
+ }
+}
<Target Name="_GetWasmGenerateAppBundleDependencies">
<Warning Condition="'$(InvariantGlobalization)' == 'true' and '$(HybridGlobalization)' == 'true'" Text="%24(HybridGlobalization) has no effect when %24(InvariantGlobalization) is set to true." />
+ <Warning Condition="'$(WasmIcuDataFileName)' != '' and '$(HybridGlobalization)' == 'true'" Text="%24(WasmIcuDataFileName) has no effect when %24(HybridGlobalization) is set to true." />
<PropertyGroup>
- <HybridGlobalization Condition="'$(InvariantGlobalization)' == 'true'">false</HybridGlobalization>
<_HasDotnetWasm Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.native.wasm'">true</_HasDotnetWasm>
<_HasDotnetJsWorker Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.native.worker.js'">true</_HasDotnetJsWorker>
<_HasDotnetJsSymbols Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.native.js.symbols'">true</_HasDotnetJsSymbols>
<_HasDotnetNativeJs Condition="'%(WasmNativeAsset.FileName)%(WasmNativeAsset.Extension)' == 'dotnet.native.js'">true</_HasDotnetNativeJs>
- <_WasmIcuDataFileName Condition="'$(WasmIcuDataFileName)' != '' and Exists('$(WasmIcuDataFileName)')">$(WasmIcuDataFileName)</_WasmIcuDataFileName>
- <_WasmIcuDataFileName Condition="'$(WasmIcuDataFileName)' != '' and !Exists('$(WasmIcuDataFileName)')">$(MicrosoftNetCoreAppRuntimePackRidNativeDir)$(WasmIcuDataFileName)</_WasmIcuDataFileName>
- </PropertyGroup>
-
- <PropertyGroup Condition="'$(HybridGlobalization)' == 'true' and '$(WasmIcuDataFileName)' == ''">
- <!-- to be renamed to icudt_wasm.dat when the contents of the file get defined and it gets added to repo -->
- <_WasmIcuDataFileName>$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt.dat</_WasmIcuDataFileName>
+ <HybridGlobalization Condition="'$(InvariantGlobalization)' == 'true'">false</HybridGlobalization>
+ <_WasmIcuDataFileName Condition="'$(HybridGlobalization)' != 'true' and '$(WasmIcuDataFileName)' != '' and Exists('$(WasmIcuDataFileName)')">$(WasmIcuDataFileName)</_WasmIcuDataFileName>
+ <_WasmIcuDataFileName Condition="'$(HybridGlobalization)' != 'true' and '$(WasmIcuDataFileName)' != '' and !Exists('$(WasmIcuDataFileName)')">$(MicrosoftNetCoreAppRuntimePackRidNativeDir)$(WasmIcuDataFileName)</_WasmIcuDataFileName>
</PropertyGroup>
<ItemGroup>
</ItemGroup>
<ItemGroup Condition="'$(InvariantGlobalization)' != 'true'">
- <_IcuAvailableDataFiles Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt_*" />
- <WasmIcuDataFileNames Condition="'$(WasmIncludeFullIcuData)' == 'true'" Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt.dat"/>
- <WasmIcuDataFileNames Condition="'$(WasmIncludeFullIcuData)' != 'true' and '$(_WasmIcuDataFileName)' == ''" Include="@(_IcuAvailableDataFiles)"/>
- <WasmIcuDataFileNames Condition="'$(WasmIncludeFullIcuData)' != 'true' and '$(_WasmIcuDataFileName)' != ''" Include="$(_WasmIcuDataFileName)"/>
+ <_HybridGlobalizationDataFiles Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt_hybrid.dat"/>
+ <_IcuAvailableDataFiles Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt_*" Exclude="@(_HybridGlobalizationDataFiles)"/>
+ <WasmIcuDataFileNames Condition="'$(HybridGlobalization)' == 'true'" Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt_hybrid.dat"/>
+ <WasmIcuDataFileNames Condition="'$(HybridGlobalization)' != 'true' and '$(WasmIncludeFullIcuData)' == 'true'" Include="$(MicrosoftNetCoreAppRuntimePackRidNativeDir)icudt.dat"/>
+ <WasmIcuDataFileNames Condition="'$(HybridGlobalization)' != 'true' and '$(WasmIncludeFullIcuData)' != 'true' and '$(_WasmIcuDataFileName)' == ''" Include="@(_IcuAvailableDataFiles)"/>
+ <WasmIcuDataFileNames Condition="'$(HybridGlobalization)' != 'true' and '$(WasmIncludeFullIcuData)' != 'true' and '$(_WasmIcuDataFileName)' != ''" Include="$(_WasmIcuDataFileName)"/>
<WasmNativeAsset Include="@(WasmIcuDataFileNames)"/>
</ItemGroup>
+
</Target>
<Target Name="_WasmGenerateAppBundle"
//!
//! This is generated file, see src/mono/wasm/runtime/rollup.config.js
-//! This is not considered public API with backward compatibility guarantees.
+//! This is not considered public API with backward compatibility guarantees.
declare interface NativePointer {
__brandNativePointer: "NativePointer";
Sharded = 0,
All = 1,
Invariant = 2,
- Custom = 3
+ Custom = 3,
+ Hybrid = 4
}
declare global {
}
}
- const combinedICUResourceName = "icudt.dat";
+ if (bootConfig.icuDataMode === ICUDataMode.Hybrid)
+ {
+ const reducedICUResourceName = "icudt_hybrid.dat";
+ return reducedICUResourceName;
+ }
+
if (!culture || bootConfig.icuDataMode === ICUDataMode.All) {
+ const combinedICUResourceName = "icudt.dat";
return combinedICUResourceName;
}
Sharded = 0,
All = 1,
Invariant = 2,
- Custom = 3
-}
\ No newline at end of file
+ Custom = 3,
+ Hybrid = 4
+}
--- /dev/null
+using System;
+using System.Globalization;
+
+try
+{
+ CompareInfo compareInfo = new CultureInfo("es-ES").CompareInfo;
+ int shouldBeEqual = compareInfo.Compare("A\u0300", "\u00C0", CompareOptions.None);
+ if (shouldBeEqual != 0)
+ {
+ return 1;
+ }
+ int shouldThrow = compareInfo.Compare("A\u0300", "\u00C0", CompareOptions.IgnoreNonSpace);
+ Console.WriteLine($"Did not throw as expected but returned {shouldThrow} as a result. Using CompareOptions.IgnoreNonSpace option alone should be unavailable in HybridGlobalization mode.");
+}
+catch (PlatformNotSupportedException pnse)
+{
+ Console.WriteLine($"HybridGlobalization works, thrown exception as expected: {pnse}.");
+ return 42;
+}
+catch (Exception ex)
+{
+ Console.WriteLine($"HybridGlobalization failed, unexpected exception was thrown: {ex}.");
+ return 2;
+}
+return 3;
/// Load custom icu file provided by the developer.
/// </summary>
Custom = 3,
+
+ /// <summary>
+ /// Use the reduced icudt_hybrid.dat file
+ /// </summary>
+ Hybrid = 4,
}
[DataContract]
public bool LoadAllICUData { get; set; }
+ public bool IsHybridGlobalization { get; set; }
+
public bool LoadCustomIcuData { get; set; }
public string InvariantGlobalization { get; set; }
{
icuDataMode = ICUDataMode.Invariant;
}
+ else if (IsHybridGlobalization)
+ {
+ icuDataMode = ICUDataMode.Hybrid;
+ }
else if (LoadAllICUData)
{
icuDataMode = ICUDataMode.All;
projects / platform / upstream / dotnet / runtime.git / commitdiff