fix: character encoding issues in shell command processor by boylin0 · Pull Request #1949 · google-gemini/gemini-cli

boylin0 · 2025-06-26T14:46:04Z

Fixes #1945

Add system encoding detection for Windows (chcp) and Unix-like systems (locale)
Replace StringDecoder with TextDecoder for better multi-byte character support
Add comprehensive Windows code page to encoding mappings
Remove deprecated decoder.end() calls that are not needed with TextDecoder

This resolves issues with garbled output when shell commands produce non-UTF-8 encoded text, particularly on Windows systems with different code pages.

TLDR

This PR improves the handling of shell command outputs with non-UTF-8 encodings by dynamically detecting system encoding on Windows and Unix-like OSes. It replaces StringDecoder with TextDecoder to properly decode multi-byte characters such as Chinese, fixing the garbled text problem.

Dive Deeper

Previously, shell command outputs containing Chinese or other non-UTF-8 characters appeared corrupted due to improper decoding. The original implementation used StringDecoder, which does not support Big5 or other non-UTF-8 encodings well. On Windows, code pages vary and can be detected via chcp, while on Unix-like systems, locale provides the encoding info. By integrating these detection methods and switching to TextDecoder (which supports multi-byte decoding natively), this change ensures the correct interpretation of command outputs across different environments. Deprecated calls to decoder.end() were removed because TextDecoder does not require them.

Reviewer Test Plan

Run shell commands via gemini on Windows with a code page like 950 (Big5) or 936 (GBK) that outputs Chinese characters, and verify the output is displayed correctly without garbled text.
Repeat similar tests on Unix/Linux/macOS with locales that produce Chinese or other multi-byte output.
Confirm that no regressions occur with UTF-8 encoded outputs.
Review the code for correct usage of system encoding detection and proper replacement of decoding logic.

Testing Matrix

	🍏	🪟	🐧
npm run	✅	✅	✅
npx	❓	❓	❓
Docker	❓	❓	❓
Podman	❓	-	-
Seatbelt	❓	-	-

Linked issues / bugs

#1945: Garbled output from shell commands containing Chinese characters due to encoding issues

Fixes google-gemini#1945 - Add system encoding detection for Windows (chcp) and Unix-like systems (locale) - Replace StringDecoder with TextDecoder for better multi-byte character support - Add comprehensive Windows code page to encoding mappings - Remove deprecated decoder.end() calls that are not needed with TextDecoder This resolves issues with garbled output when shell commands produce non-UTF-8 encoded text, particularly on Windows systems with different code pages.

chaizp · 2025-06-28T05:07:20Z

Today I found that when entering Chinese, only the last character will be displayed in the terminal, and the way of copying text will also only show the last character....

umairidris · 2025-06-30T02:56:28Z

/gemini review

gemini-code-assist

Code Review

The pull request addresses character encoding issues in the shell command processor by adding system encoding detection and replacing StringDecoder with TextDecoder. However, there are potential issues with the fallback mechanisms for encoding detection and handling unsupported code pages. Additionally, the use of stripAnsi might remove valuable formatting information. Addressing these issues will improve the robustness and user experience of the shell command processor.

- Add NaN check for parsed code page value in getSystemEncoding() - Rename unused catch variable to indicate intentional non-use - Prevent potential issues from invalid code page values Resolves PR feedback in google-gemini#1949

- Implement locale charmap fallback for Unix-like systems when env vars are missing (Addresses code review feedback from PR google-gemini#1949)

Fall back to utf-8 instead of `cp${cp}` for unknown code pages and log warning to prevent TextDecoder errors.

jacob314

Thank you for the pull request! This is a very important polish item.

- Add cachedSystemEncoding variable to store detected encoding - Add getCachedSystemEncoding() function to return cached value or detect once - Update executeShellCommand to use cached encoding instead of calling getSystemEncoding() repeatedly

…ream handling - Add { stream: true } option to decoder.decode() calls for proper streaming - Handle final bytes by calling decode() at end-of-stream - Based on WHATWG Encoding specification: https://encoding.spec.whatwg.org/#interface-textdecoder

- Add chardet dependency for robust encoding detection - Enhance getCachedSystemEncoding to use chardet as fallback - Improve system encoding detection with better error handling - Initialize TextDecoders lazily after encoding detection - Return null instead of fallback encoding when detection fails - Add detectEncodingFromBuffer function for buffer-based detection - Update Windows code page mapping with better error messages - Improve comment formatting and documentation

…eShellCommand function

…for testing Includes function comments and types for clarity and a cache reset utility for test support.

- Windows code page mapping tests - Buffer encoding detection tests - Cross-platform system encoding tests - Edge cases and error handling tests

…initialized state for fixing the tests

boylin0 · 2025-07-19T13:45:32Z

approved after these comments are addressed and some tests are added. If tests mock the output of the commands to enable them to run on platforms other than windows that is fine.

Thanks again for this! This will really help users who aren't using utf8.

@jacob314 Thanks so much for the thorough code review and your patience! All comments have been addressed, and I’ve added tests with mocked command output to ensure cross-platform compatibility. Please feel free to let me know if you spot any other issues or have further suggestions.

…tent test behavior

jacob314

Great work! Thank you for sticking with this pull request.

…ini#1949)

Co-authored-by: Jacob Richman <jacob314@gmail.com> Co-authored-by: Sandy Tao <sandytao520@icloud.com>

…ini#1949) Co-authored-by: Jacob Richman <jacob314@gmail.com> Co-authored-by: Sandy Tao <sandytao520@icloud.com>

boylin0 requested a review from a team as a code owner June 26, 2025 14:46

Merge branch 'main' into main

07e8ded

boylin0 changed the title ~~Fix character encoding issues in shell command processor~~ fix: character encoding issues in shell command processor Jun 26, 2025

Merge branch 'main' into main

923b04c

boylin0 mentioned this pull request Jun 30, 2025

Shell Command Output with Chinese Characters Displays as Garbage #1945

Closed

gemini-code-assist Bot reviewed Jun 30, 2025

View reviewed changes

boylin0 added 3 commits June 30, 2025 18:35

fix: improve system encoding detection with robust fallbacks

0cb8755

- Implement locale charmap fallback for Unix-like systems when env vars are missing (Addresses code review feedback from PR google-gemini#1949)

fix: handle unsupported Windows code pages safely

55fdc7c

Fall back to utf-8 instead of `cp${cp}` for unknown code pages and log warning to prevent TextDecoder errors.

boylin0 closed this Jun 30, 2025

boylin0 reopened this Jun 30, 2025

gemini-cli Bot added kind/bug priority/p1 Important and should be addressed in the near term. labels Jul 3, 2025

jacob314 reviewed Jul 10, 2025

View reviewed changes

Comment thread packages/cli/src/ui/hooks/shellCommandProcessor.ts

jacob314 reviewed Jul 10, 2025

View reviewed changes

Comment thread packages/cli/src/ui/hooks/shellCommandProcessor.ts Outdated

jacob314 reviewed Jul 10, 2025

View reviewed changes

Comment thread packages/cli/src/ui/hooks/shellCommandProcessor.ts Outdated

jacob314 reviewed Jul 10, 2025

View reviewed changes

boylin0 and others added 8 commits July 12, 2025 13:01

Merge branch 'main' into main

5f1ef8f

fix(package): remove unnecessary dependency on @google/gemini-cli

3e6d8a9

style(shellCommandProcessor): remove unnecessary blank line in execut…

a6cb358

…eShellCommand function

fix(package): add missing chardet dependency

41b524a

Merge branch 'main' into main

7588ae3

jacob314 reviewed Jul 12, 2025

View reviewed changes

Comment thread packages/cli/src/ui/hooks/shellCommandProcessor.ts Outdated

boylin0 force-pushed the main branch from d25e52f to f650be2 Compare July 19, 2025 11:08

Merge branch 'fix/shell-command-processor-encoding'

dd63380

boylin0 reopened this Jul 19, 2025

boylin0 added 5 commits July 19, 2025 20:29

refactor(cli): integrate core encoding utilities; add reset function …

b871ab7

…for testing Includes function comments and types for clarity and a cache reset utility for test support.

test(core): add systemEncoding utility test suite

c629c8f

- Windows code page mapping tests - Buffer encoding detection tests - Cross-platform system encoding tests - Edge cases and error handling tests

refactor(deps): move chardet dependency into packages\core

3583693

style(shellCommandProcessor): format code

30a1f7e

fix(systemEncoding): update cachedSystemEncoding use undefined for un…

1aa9491

…initialized state for fixing the tests

boylin0 requested a review from jacob314 July 19, 2025 13:51

boylin0 added 2 commits July 19, 2025 22:22

style(systemEncoding): format code

b14869e

fix(shellCommandProcessor): ensure homedir mock is defined for consis…

b474dd5

…tent test behavior

gemini-cli Bot added priority/p2 Important but can be addressed in a future release. and removed priority/p1 Important and should be addressed in the near term. labels Jul 21, 2025

Merge branch 'main' into main

909d170

jacob314 approved these changes Jul 21, 2025

View reviewed changes

jacob314 added this pull request to the merge queue Jul 21, 2025

Merged via the queue into google-gemini:main with commit 12765eb Jul 21, 2025
10 checks passed

thacio added a commit to thacio/auditaria that referenced this pull request Jul 22, 2025

fix: character encoding issues in shell command processor (google-gem…

d93a2a7

…ini#1949)

galdawave pushed a commit that referenced this pull request Jul 22, 2025

fix: character encoding issues in shell command processor (#1949)

dad9f00

Co-authored-by: Jacob Richman <jacob314@gmail.com> Co-authored-by: Sandy Tao <sandytao520@icloud.com>

jkcinouye pushed a commit that referenced this pull request Jul 25, 2025

fix: character encoding issues in shell command processor (#1949)

3391077

Co-authored-by: Jacob Richman <jacob314@gmail.com> Co-authored-by: Sandy Tao <sandytao520@icloud.com>

frikyfriky11 mentioned this pull request Aug 13, 2025

Inconsistent Release Changelog Generation: 'Full Changelog' always starts from v0.1.12 #5663

Closed

involvex pushed a commit to involvex/gemini-cli that referenced this pull request Sep 11, 2025

fix: character encoding issues in shell command processor (google-gem…

ee40db9

…ini#1949) Co-authored-by: Jacob Richman <jacob314@gmail.com> Co-authored-by: Sandy Tao <sandytao520@icloud.com>

kaluchi mentioned this pull request May 19, 2026

Proposal: deterministic encoding for child-process I/O #27247

Merged

sripasg added the size/l A large sized PR label Jun 2, 2026

Navigation Menu

Uh oh!

fix: character encoding issues in shell command processor - #1949

fix: character encoding issues in shell command processor#1949
jacob314 merged 27 commits into
google-gemini:mainfrom
boylin0:main

boylin0 commented Jun 26, 2025 •

edited

Loading

Uh oh!

chaizp commented Jun 28, 2025

Uh oh!

umairidris commented Jun 30, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jacob314 left a comment

Uh oh!

Uh oh!

boylin0 commented Jul 19, 2025 •

edited

Loading

Uh oh!

jacob314 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

boylin0 commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Dive Deeper

Reviewer Test Plan

Testing Matrix

Linked issues / bugs

Uh oh!

chaizp commented Jun 28, 2025

Uh oh!

umairidris commented Jun 30, 2025

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jacob314 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

boylin0 commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacob314 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

boylin0 commented Jun 26, 2025 •

edited

Loading

boylin0 commented Jul 19, 2025 •

edited

Loading