Proposal: deterministic encoding for child-process I/O

[kaluchi]

Summary

Fix Windows shell mojibake for non-ASCII output. Two changes:

1. Decode child-process pipe output via TextDecoder('utf-8') in
   shellExecutionService (cpSpawn path). Remove the
   systemEncoding.ts module and chardet npm dependency.
2. Inject chcp 65001 prefix into commands run on Windows through
   the PTY path so the new ConPTY session uses UTF-8 codepage.

Fixes #20968. Companion side-effect fix: #27241.

Goal

Deterministic encoding for child-process I/O. UTF-8 for the byte
stream regardless of host locale or active console codepage. Remove
heuristic lookups (chcp / system locale / chardet sniffing).

For legacy programs, the user and the agent can pipe through
iconv, recode, or platform equivalents. Transcoding depends on
the source (what the user runs in shell), so gemini-cli leaves it to
the caller.

Why (part 1: decoder)

The console's display settings (chcp, LANG) describe how text is
rendered. The byte encoding inside a redirected stdout pipe is
independent — modern toolchains (node, python 3 with PEP 540/686,
git, go, rust) write UTF-8 to redirected stdout regardless.

shellExecutionService previously queried getCachedEncodingForBuffer
which mixed the system locale (chcp / LANG) with chardet on the
first chunk. On hosts with a non-UTF-8 system codepage, the system
signal won and the decoder mis-interpreted UTF-8 output from modern
children as windows-1252 / shift_jis / iso-8859-1.

Repro on Windows with conhost CP = 1252, before fix:
node -e \"console.log('你好世界')\" → ä½ å¥½ä¸–ç•Œ.
After fix: 你好世界.

Why (part 2: PTY-path chcp injection)

On Windows the PTY path goes through ConPTY (via @lydell/node-pty).
Each new ConPTY session inherits its codepage from the system OEMCP:

• microsoft/terminal/src/host/srvinit.cpp:44 reads GetOEMCP() into
  Globals.uiOEMCP.
• src/host/settings.cpp:41 defaults _uCodePage to uiOEMCP.
• src/host/_stream.cpp:341 converts each byte from the child via
  MultiByteToWideChar(gci.OutputCP, ...).

On locales without Beta: Use Unicode UTF-8 for worldwide language
support the OEMCP is a legacy codepage (850 866
936 zh-CN, 932 ja-JP, ...). UTF-8 output from child processes
(perl, python, node, ...) inside the PTY is then misinterpreted as
that legacy codepage, double-encoded back to UTF-8 for the VT
stream, and reaches the decoder as valid UTF-8 mojibake.

CreatePseudoConsole accepts no codepage argument
(microsoft/terminal#9174, open since 2021). The only way to change
the ConPTY codepage is from inside the session via
SetConsoleOutputCP, intercepted by conhost
src/host/getset.cpp:1144. So the prefix chcp 65001 >\$null; is
injected into the command before ptyProcess.spawn for Windows +
pwsh/cmd + PTY path. The cpSpawn fallback (no ConPTY) is left
unchanged.

History

The chardet + system-locale path was introduced in #1949 (closes
#1945) to fix garbled Chinese output on Windows hosts with chcp 950
(Big5) / 936 (GBK) running native programs that emitted bytes in
the system codepage. Modern toolchains write UTF-8 to redirected
stdout regardless of chcp / LANG, making UTF-8 the right primary
signal.

The systemEncoding.ts module added by that PR had no other internal
consumers in the repo and is removed along with the chardet dependency.

Prior art

UTF-8 as default for transport / output is the industry consensus:

• openai/codex — bytes_to_string_smart in
  codex-rs/protocol/src/exec_output.rs
  validates UTF-8 first via `std::str::from_utf8`. Tracking issue:
  Windows: shell wrapper via powershell.exe -Command mojibakes UTF-8 files on non-English locales (PowerShell 5.1 Get-Content defaults to ANSI codepage) openai/codex#23044.
• Java 18 (JEP 400, 2022) — UTF-8 default charset globally.
• Python 3.15+ (PEP 686, 2024) — UTF-8 mode default; opt-in
  shipped earlier in 3.7 (PEP 540).
• PowerShell 7 — `$OutputEncoding` UTF-8 by default.

Tests

`shellExecutionService.test.ts` (66 tests) passes. Platform-specific
PTY-path expectation updated to assert the `chcp 65001 >$null;`
prefix; cpSpawn-fallback expectation unchanged (no injection there).

Context

Found while investigating Windows encoding issues tracked in #20968.
The companion side-effect fix in `process-utils.ts` (un-NoProfile
diagnostics spawn) is in #27241.

[kaluchi]

@scidomino — could you take a look at this as a follow-up to the discussion on the recently-merged #25900? This PR fully fixes all the Windows encoding issues I found in that repo (without breaking anything for others, and UTF-8 is now the default on all platforms).

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves long-standing encoding issues (mojibake) on Windows by enforcing UTF-8 as the standard for child-process I/O. By removing complex heuristic lookups and explicitly setting the console codepage for PTY sessions, the system now provides predictable text handling regardless of the host's locale or console settings.

Highlights

• Standardized Encoding: Replaced heuristic-based encoding detection with a deterministic UTF-8 decoder for all child-process I/O.
• Dependency Cleanup: Removed the 'chardet' npm dependency and the 'systemEncoding.ts' module as they are no longer required.
• Windows PTY Support: Added automatic injection of 'chcp 65001' into commands run on Windows PTY sessions to ensure consistent UTF-8 output.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request removes the chardet dependency and the systemEncoding utility, standardizing output decoding to UTF-8 across the core package. It also introduces a mechanism to ensure the Windows ConPTY session uses a UTF-8 codepage by prefixing commands with chcp 65001. Feedback from the review highlights a potential syntax error when prefixing empty or whitespace-only commands in cmd shells and identifies a ReferenceError caused by the use of an undefined isWindows variable in the prepareCommand method.