Skip to content

Use ANSI encoding (GetACP) for tool stdout/stderr instead of OEM - #13873

Open
huulinhnguyen-dev wants to merge 9 commits into
dotnet:mainfrom
huulinhnguyen-dev:dev/huulinhnguyen/fix-native-tool-stdout-ansi-encoding
Open

Use ANSI encoding (GetACP) for tool stdout/stderr instead of OEM#13873
huulinhnguyen-dev wants to merge 9 commits into
dotnet:mainfrom
huulinhnguyen-dev:dev/huulinhnguyen/fix-native-tool-stdout-ansi-encoding

Conversation

@huulinhnguyen-dev

@huulinhnguyen-dev huulinhnguyen-dev commented May 27, 2026

Copy link
Copy Markdown
Contributor

Fixes #12290

Context

On Windows with Western European locales (e.g., French), the system ANSI code page (GetACP = CP1252) and OEM code page (GetOEMCP = CP850) differ. Native Windows tools such as MSVC link.exe and cl.exe write their output using the ANSI code page, but MSBuild reads captured tool output using the OEM code page, causing non-ASCII characters to be garbled (e.g., éÚ, àÓ).

Simply switching the default to ANSI would be a breaking change for tools that rely on the current OEM decoding. Instead, this PR keeps the existing OEM default and adds an opt-in knob so the targets that invoke link.exe/cl.exe can request ANSI decoding explicitly.

Changes Made

  • src/Framework/EncodingUtilities.cs
    • Added CurrentSystemAnsiEncoding (ANSI code page via PInvoke.GetACP(), Encoding.Default on .NET Framework).
    • Refactored the OEM/ANSI resolution into a shared GetCurrentSystemEncoding(bool useOemCodePage) helper (removes duplication with CurrentSystemOemEncoding).
    • Added the "ansi" special value (UseAnsiEncoding) and GetEncodingFromName(string), which resolves "ansi" (case-insensitive) to the ANSI code page and any other value via Encoding.GetEncoding.
  • src/Framework/NativeMethods.txt — added GetACP P/Invoke declaration.
  • src/Utilities/ToolTask.cs — exposed new public virtual StdOutEncoding/StdErrEncoding properties (previously only on Exec). An explicitly-set value is honored first in StandardOutputEncoding/StandardErrorEncoding; when unset, the default is unchanged (OEM code page). Setting the value to "ansi" selects the ANSI code page.
  • src/Tasks/Exec.csStdOutEncoding/StdErrEncoding now override the ToolTask properties and resolve values through EncodingUtilities.GetEncodingFromName, so "ansi" is supported there too. The default remains OEM.

Testing

  • src/Tasks.UnitTests/Exec_Tests.cs:
    • ExecTask_DefaultStdEncodingIsOem — default decoding stays OEM.
    • ExecTask_AnsiStdEncodingKnobSelectsAnsiCodePage (ansi/ANSI/Ansi) — the knob selects the ANSI code page.
    • ExecTask_AnsiStdEncodingKnobExecutesSuccessfully — the "ansi" value keeps parameters valid and the task runs.
  • src/Utilities.UnitTests/ToolTask_Tests.cs:
    • StdEncodingDefaultsToOem — default decoding stays OEM.
    • StdEncodingAnsiKnobSelectsAnsiCodePage (ansi/ANSI/Ansi) — the knob selects the ANSI code page.
    • StdEncodingHonorsExplicitNamedEncoding — an explicit named encoding (e.g. utf-8) is honored over the default.

Notes

  • This is only half of the fix. It exposes an opt-in ANSI-encoding knob but keeps the default unchanged (OEM), so it does not by itself resolve Build tools' output encoding broken building a C++ project #12290. The garbled C++ tool output is only fixed once the Visual Studio C++ targets (a separate repo) are updated to set StdOutEncoding="ansi"/StdErrEncoding="ansi" when invoking link.exe/cl.exe. The issue should therefore not be auto-closed by this PR alone.

@huulinhnguyen-dev
huulinhnguyen-dev marked this pull request as ready for review May 28, 2026 06:44
Copilot AI review requested due to automatic review settings May 28, 2026 06:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes #12290, where MSBuild garbles non-ASCII characters in native Windows tool output (e.g., MSVC link.exe/cl.exe) on locales where the ANSI and OEM code pages differ (e.g., French Windows: ANSI=CP1252, OEM=CP850). The fix changes the default decoding of child tool stdout/stderr from the system OEM code page (GetOEMCP) to the system ANSI code page (GetACP) in both ToolTask and the Exec task.

Changes:

  • Add EncodingUtilities.CurrentSystemAnsiEncoding (uses Encoding.Default on .NET Framework, PInvoke.GetACP() on .NET Core) and the GetACP P/Invoke entry.
  • ToolTask.StandardOutputEncoding/StandardErrorEncoding and Exec's constructor now default to ANSI instead of OEM.
  • Update three existing Exec encoding tests and add ExecTask_DefaultStdEncodingIsAnsi / ExecTask_UseUtf8AlwaysOverridesAnsiDefault.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Framework/EncodingUtilities.cs Adds the new CurrentSystemAnsiEncoding property with caching, mirroring the OEM variant.
src/Framework/NativeMethods.txt Adds GetACP to the CsWin32 P/Invoke generator input.
src/Utilities/ToolTask.cs Replaces OEM default with ANSI via new GetDefaultToolEncoding helper; updates remarks.
src/Tasks/Exec.cs Constructor now seeds the shadowed stdout/stderr encoding fields with ANSI instead of OEM.
src/Tasks.UnitTests/Exec_Tests.cs Updates existing assertions from OEM to ANSI and adds new coverage for the default and the UTF‑8 override.

Comment thread src/Utilities/ToolTask.cs Outdated
Comment thread src/Tasks/Exec.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comment thread src/Tasks.UnitTests/Exec_Tests.cs
Comment thread src/Tasks.UnitTests/Exec_Tests.cs Outdated
Comment thread documentation/wiki/ChangeWaves.md Outdated
huulinhnguyen-dev and others added 3 commits May 28, 2026 14:42
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@AlesProkop AlesProkop left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented. Also, I think we are missing tests for the ToolTask changes. If so, could you maybe please add some?

Comment thread documentation/wiki/ChangeWaves.md Outdated
Comment thread src/Tasks.UnitTests/Exec_Tests.cs Outdated
Comment thread src/Tasks/Exec.cs Outdated
Comment thread src/Utilities/ToolTask.cs Outdated
Comment thread src/Framework/EncodingUtilities.cs Outdated

@JanProvaznik JanProvaznik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a breaking change that some tools rely on the current code page.

We should instead expose a knob with the default staying the same but the targets that invoke link.exe and cl.exe will parametrize exec to use this.

Keep the OEM default and add settable StdOutEncoding/StdErrEncoding parameters on ToolTask so ToolTask-derived tasks (CL/Link) can opt into the system ANSI code page via the special value ansi. Share the encoding-name parser via EncodingUtilities and make Exec override the base properties.
@AlesProkop
AlesProkop requested a review from JanProvaznik July 7, 2026 13:57

@JanProvaznik JanProvaznik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see this in a coordinated change with the VS targets that define the cl and link exe using tasks/execs so that it proves that the shape of the API is easily usable and is validated to fix the reported problem rather than just adding an API.

Comment thread src/Utilities/ToolTask.cs
/// Set to the special value "ansi" to use the current system ANSI code page (GetACP), which some native
/// tools (e.g., MSVC link.exe/cl.exe) use for their output. When not set, the current system OEM code page is used.
/// </summary>
public virtual string StdOutEncoding

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider improving the naming
having in one class StdOutEncoding and StandardOutputEncoding is not acceptable.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this is adding a widely used public api surface area I'd like a review from @rainersigwald as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build tools' output encoding broken building a C++ project

4 participants