`System.Globalization.Nls.Tests.ToUpper/ToLower_netcore` test failures in local environment

[Gnbrkm41]

Description

Run tests in System.Globalization.Nls.Tests test project. The characters after the ToLower/ToUpper call do not change even though we expect it to change.

Test	Duration	Traits	Error Message
System.Globalization.Tests.TextInfoMiscTests.ToLower_Netcore Failed	24 ms		Assert.Equal() Failure            ↓ (pos 1) Expected: 𐐨 Actual:   𐐀            ↑ (pos 1)
System.Globalization.Tests.TextInfoMiscTests.ToUpper_netcore Failed	< 1 ms		Assert.Equal() Failure            ↓ (pos 1) Expected: 𐐀 Actual:   𐐨            ↑ (pos 1)

Please note that this only seems to happen when no culture name is specified (new CultureInfo("")); the exact same test but with the culture name en-US / fr / fr-FR passes fine.

runtime/src/libraries/System.Globalization/tests/System/Globalization/TextInfoTests.cs

Lines 174 to 184 in f4e9146

	private static readonly string [] s_cultureNames = new string[] { "", "en-US", "fr", "fr-FR" };
	// ToLower_TestData_netcore has the data which is specific to netcore framework
	public static IEnumerable<object[]> ToLower_TestData_netcore()
	{
	foreach (string cultureName in s_cultureNames)
	{
	// DESERT CAPITAL LETTER LONG I has a lower case variant (but not on Windows 7).
	yield return new object[] { cultureName, "\U00010400", PlatformDetection.IsWindows7 ? "\U00010400" : "\U00010428" };
	}
	}

Configuration

• Windows 10 Pro Insider Preview, Build 20170 (Also noticed in a couple previous flights), x64
• Locale ko-KR
• Ran with VS2019 16.7 P4, with ./build.cmd -vs System.Globalization -runtimeConfiguration Release -librariesConfiguration Debug

Other information

Possibly a Windows NLS issue, rather than the libraries implementation being wrong? Using builds from the dev channel probably doesn't help a lot 😅

Tagging subscribers to this area: @tarekgh, @safern, @krwq
Notify danmosemsft if you want to be subscribed.

[tarekgh]

@Gnbrkm41 I am running on Windows 10 v19577 and I am not seeing this behavior. So this most likely a new change coming from Windows. I am trying to confirm that now. Thanks for reporting that.

[Gnbrkm41]

System.Text.Tests.RuneTests.Casing_Invariant under System.Runtime.Nls.Tests appears to be failing with the same test data.

[tarekgh]

do you see any other failures than with this data "\U00010400"?

[Gnbrkm41]

No, case conversion between \U00010400 and \U00010428 are the only failures I'm noticing with existing test data.

[MattGal]

Also observed on ARM64 Windows 10.0.19619

[GrabYourPitchforks]

@tarekgh Do you know if Windows ever removes case conversion mappings between versions? Win7 didn't support case conversion between characters from the Deseret alphabet, Win8 did, and now it looks like recent versions of Win10 may have backed it out?

[tarekgh]

@GrabYourPitchforks

Do you know if Windows ever removes case conversion mappings between versions? Win7 didn't support case conversion between characters from the Deseret alphabet, Win8 did, and now it looks like recent versions of Win10 may have backed it out?

I am trying to investigate the issue. I doubt Windows removed it intentionally but any way I am accessing some machine with same reported version and seeing the casing not changed here. I'll dig more to understand what could be the problem. I'll report back when having more info.

[tarekgh]

@Gnbrkm41 I am not able to repro it so far. I have tried on 2 machines with different architecture. could you please create a stand alone dotnet app and run the following code:

using System;
using System.Runtime.InteropServices;
using System.Reflection;
using System.Globalization;

namespace apps
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(RuntimeInformation.OSDescription);
            Console.WriteLine(RuntimeInformation.FrameworkDescription);
            Console.WriteLine($"UseNls: {typeof(object).Assembly.GetType("System.Globalization.GlobalizationMode").GetProperty("UseNls", BindingFlags.NonPublic | BindingFlags.Static).GetValue(null)}");
            Console.WriteLine($"Invariant: {typeof(object).Assembly.GetType("System.Globalization.GlobalizationMode").GetProperty("Invariant", BindingFlags.NonPublic | BindingFlags.Static).GetValue(null)}");

            CultureInfo ci = new CultureInfo("ko-KR");
            foreach (char c in ci.TextInfo.ToLower("\U00010400"))
            {
                Console.Write($"\\u{(int)c:x4}, ");
            }
            Console.WriteLine();
        }
    }
}

and the project:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <RuntimeHostConfigurationOption Include="System.Globalization.UseNls" Value="true" />
  </ItemGroup>
</Project>

on my machine I am getting the output:

Microsoft Windows 10.0.20170
.NET 5.0.0-preview.8.20361.2
UseNls: True
Invariant: False
\ud801, \udc28,