A (possible) bug in handling non-ASCII characters in the Location header

[aliostad]

This snippet prints 404 in .NET Core while 200 in .NET Framework:

var url = "https://www.ilna.ir/fa/tiny/news-724227";
var client = new HttpClient();
var result = client.GetAsync(url).Result;
Console.WriteLine(result.StatusCode);

Basically there is a 301 response which has non-ASCII characters in the location header and gets garbled in .NET Core.

All browsers, cURL and .NET Framework and python happily follow the redirect.

---

To provide more context, the Location header gets url-encoded as

https://www.ilna.ir/%C3%98%C2%A8%C3%98%C2%AE%C3%98%C2%B4-%D8%A7%D9%82%D8%AA%D8%B5%D8%A7%D8%AF%DB%8C-4/724227-%DA%A9%D9%86%D8%AA%D8%B1%D9%84-%D9%86%D9%88%D8%B3%D8%A7%D9%86%D8%A7%D8%AA-%D8%A7%D8%B1%D8%B2%DB%8C-%D9%85%D9%87%D9%85-%D8%AA%D8%B1-%D8%A7%D8%B2-%DA%A9%D8%A7%D9%87%D8%B4-%D9%86%D8%B1%D8%AE-%D8%A7%D8%B1%D8%B2-%D8%A7%D8%B3%D8%AA-%D8%A8%D8%B1%D8%A7%DB%8C-%DA%A9%D8%A7%D8%B1%D8%A2%D9%81%D8%B1%DB%8C%D9%86%D8%A7%D9%86-%D9%81%D8%B1%D8%B4-%D9%82%D8%B1%D9%85%D8%B2-%D9%BE%D9%87%D9%86-%DA%A9%D8%B1%D8%AF%D9%87-%D8%A7%DB%8C%D9%85

Instead of

https://www.ilna.ir/%D8%A8%D8%AE%D8%B4-%D8%A7%D9%82%D8%AA%D8%B5%D8%A7%D8%AF%DB%8C-4/724227-%DA%A9%D9%86%D8%AA%D8%B1%D9%84-%D9%86%D9%88%D8%B3%D8%A7%D9%86%D8%A7%D8%AA-%D8%A7%D8%B1%D8%B2%DB%8C-%D9%85%D9%87%D9%85-%D8%AA%D8%B1-%D8%A7%D8%B2-%DA%A9%D8%A7%D9%87%D8%B4-%D9%86%D8%B1%D8%AE-%D8%A7%D8%B1%D8%B2-%D8%A7%D8%B3%D8%AA-%D8%A8%D8%B1%D8%A7%DB%8C-%DA%A9%D8%A7%D8%B1%D8%A2%D9%81%D8%B1%DB%8C%D9%86%D8%A7%D9%86-%D9%81%D8%B1%D8%B4-%D9%82%D8%B1%D9%85%D8%B2-%D9%BE%D9%87%D9%86-%DA%A9%D8%B1%D8%AF%D9%87-%D8%A7%DB%8C%D9%85

Note the difference of %C3%98%C2%A8%C3%98%C2%AE%C3%98%C2%B4 (wrong) vs %D8%A8%D8%AE%D8%B4 (correct).

[JoeStead]

I can confirm I also get a 404 running on Linux Mint. Here's my dotnet --info output:

.NET Core SDK (reflecting any global.json):
 Version:   2.2.103
 Commit:    8edbc2570a

Runtime Environment:
 OS Name:     linuxmint
 OS Version:  19.1
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/2.2.103/

Host (useful for support):
  Version: 2.2.1
  Commit:  878dd11e62

.NET Core SDKs installed:
  2.2.103 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Running curl -IL https://www.ilna.ir/fa/tiny/news-724227 shows a 301 followed by 200.

In summary: I can reproduce this on my machine.

[davidsh]

@geoffkizer

[geoffkizer]

Possibly related to #24679

[wfurt]

BTW I did test and CurlHandler works e.g. gives 200 back.

[aliostad]

Is this coming in 3.0?

[wfurt]

This is because Location contains unescaped UTF8 characters:

ASCII:
https://www.ilna.ir/??????-%D8%A7%D9%82%D8%AA%D8%B5%D8%A7%D8%AF%DB%8C-4/724227-%DA%A9%D9%86%D8%AA%D8%B1%D9%84-%D9%86%D9%88%D8%B3%D8%A7%D9%86%D8%A7%D8%AA-%D8%A7%D8%B1%D8%B2%DB%8C-%D9%85%D9%87%D9%85-%D8%AA%D8%B1-%D8%A7%D8%B2-%DA%A9%D8%A7%D9%87%D8%B4-%D9%86%D8%B1%D8%AE-%D8%A7%D8%B1%D8%B2-%D8%A7%D8%B3%D8%AA-%D8%A8%D8%B1%D8%A7%DB%8C-%DA%A9%D8%A7%D8%B1%D8%A2%D9%81%D8%B1%DB%8C%D9%86%D8%A7%D9%86-%D9%81%D8%B1%D8%B4-%D9%82%D8%B1%D9%85%D8%B2-%D9%BE%D9%87%D9%86-%DA%A9%D8%B1%D8%AF%D9%87-%D8%A7%DB%8C%D9%85
UTF8:
https://www.ilna.ir/بخش-%D8%A7%D9%82%D8%AA%D8%B5%D8%A7%D8%AF%DB%8C-4/724227-%DA%A9%D9%86%D8%AA%D8%B1%D9%84-%D9%86%D9%88%D8%B3%D8%A7%D9%86%D8%A7%D8%AA-%D8%A7%D8%B1%D8%B2%DB%8C-%D9%85%D9%87%D9%85-%D8%AA%D8%B1-%D8%A7%D8%B2-%DA%A9%D8%A7%D9%87%D8%B4-%D9%86%D8%B1%D8%AE-%D8%A7%D8%B1%D8%B2-%D8%A7%D8%B3%D8%AA-%D8%A8%D8%B1%D8%A7%DB%8C-%DA%A9%D8%A7%D8%B1%D8%A2%D9%81%D8%B1%DB%8C%D9%86%D8%A7%D9%86-%D9%81%D8%B1%D8%B4-%D9%82%D8%B1%D9%85%D8%B2-%D9%BE%D9%87%D9%86-%DA%A9%D8%B1%D8%AF%D9%87-%D8%A7%DB%8C%D9%85

I don't think that is legal. cc: @wtgodbe

I guess curl works because it does not care about encoding, normalization and escaping.

[aliostad]

This is because Location contains unescaped UTF8 characters

That is the title of the issue (non-ascii)

I guess curl works because it does not care about encoding, normalization and escaping.

Or actually cares enough? Bear in mind it works in .NET Framework (and pretty much every other tool/lang I have tested) - unless there is a good reason, the behaviour should be the same.

[davidsh]

All browsers, cURL and .NET Framework and python happily follow the redirect.

I agree this is a regression from .NET Framework behavior. Despite the RFCs indicating that Unicode characters should be percent-encoded, we do find raw UTF-8 octets in header values like this. In fact, Windows native HTTP stacks like WinInet and WinHTTP support raw UTF-8 octets as well.

In DNS resolution, Windows, supports both punnycode encoding of Unicode hostnames as well as UTF-8 encoding. Although the latter is usually percent-encoded.

We will investigate to see how we can match .NET Framework behavior here.

[davidsh]

@karelz