Skip to content

gh-150771: Fix email serialization for shift_jis and euc-jp#151120

Merged
serhiy-storchaka merged 8 commits into
python:mainfrom
bhuvi27:gh-150771-fix-email-shift-jis-euc-jp
Jun 16, 2026
Merged

gh-150771: Fix email serialization for shift_jis and euc-jp#151120
serhiy-storchaka merged 8 commits into
python:mainfrom
bhuvi27:gh-150771-fix-email-shift-jis-euc-jp

Conversation

@bhuvi27

@bhuvi27 bhuvi27 commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes #150771

Creating a message with set_content(..., charset='shift_jis') or charset='euc-jp' raised UnicodeEncodeError on str(m) because the payload was encoded with the input charset while the Content-Type uses the output
charset (iso-2022-jp).

Use Charset.output_charset in set_text_content so the payload and Content-Type agree from the start.

@bhuvi27 bhuvi27 requested a review from a team as a code owner June 9, 2026 02:15
@bedevere-app bedevere-app Bot mentioned this pull request Jun 9, 2026
Closed
@bedevere-app

bedevere-app Bot commented Jun 9, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@bhuvi27 bhuvi27 force-pushed the gh-150771-fix-email-shift-jis-euc-jp branch from 5928e5b to f071b32 Compare June 9, 2026 02:48
@bhuvi27
pythongh-150771: Fix email serialization for shift_jis and euc-jp
6795f58 
Convert surrogate-escaped payloads through the input charset before
encoding to iso-2022-jp, fixing UnicodeEncodeError when printing
messages created with set_content().
@bhuvi27 bhuvi27 force-pushed the gh-150771-fix-email-shift-jis-euc-jp branch from f071b32 to 6795f58 Compare June 9, 2026 02:49
bhuvi27 added 5 commits June 10, 2026 08:44
@bhuvi27
Merge branch 'main' into pythongh-150771-fix-email-shift-jis-euc-jp
2f08540
@bhuvi27
Merge branch 'main' into pythongh-150771-fix-email-shift-jis-euc-jp
24f6067
@bhuvi27
pythongh-150771: Use output charset in set_text_content for shift_jis…
f6c53cb 
…/euc-jp

Encode the payload with the charset output mapping (iso-2022-jp) when
set_content is called with shift_jis or euc-jp, instead of patching
serialization in body_encode and set_payload. Reverts those changes.
@bhuvi27
Merge remote-tracking branch 'origin/pythongh-150771-fix-email-shift-…
52810e5 
…jis-euc-jp'
@bhuvi27
pythongh-150771: Fix NEWS entry doc references for docs CI
e2f3984 
Use plain backticks for set_content() instead of a broken :func: target.
serhiy-storchaka
serhiy-storchaka approved these changes Jun 14, 2026
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍

Do you think an assertion for bytes(m) would be useful?

Comment thread Lib/test/test_email/test_contentmanager.py
@bhuvi27 @cursoragent
pythongh-150771: Add bytes(m) assertions for shift_jis and euc-jp tests
8692456 
Mirror test_set_text_charset_cp949 per review feedback.

Co-authored-by: Cursor <cursoragent@cursor.com>
serhiy-storchaka
serhiy-storchaka reviewed Jun 16, 2026
View reviewed changes
Comment thread Lib/test/test_email/test_contentmanager.py Outdated
@bhuvi27
pythongh-150771: Use Unicode escapes in bytes(m) test expectations
5736d59 
Match cp949 test style per review feedback.
@serhiy-storchaka serhiy-storchaka merged commit 0777a58 into python:main Jun 16, 2026
54 checks passed
@serhiy-storchaka serhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Jun 16, 2026
@miss-islington-app

miss-islington-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

Thanks @bhuvi27 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

@miss-islington-app

miss-islington-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

Thanks @bhuvi27 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

@miss-islington-app

miss-islington-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

Thanks @bhuvi27 for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

@miss-islington-app

miss-islington-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

Sorry, @bhuvi27 and @serhiy-storchaka, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 0777a58d8012bbdd0d72654b56f9112686ae6ff0 3.14

@miss-islington-app

miss-islington-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

Sorry, @bhuvi27 and @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 0777a58d8012bbdd0d72654b56f9112686ae6ff0 3.13

@bedevere-app

bedevere-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

GH-151541 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jun 16, 2026
serhiy-storchaka pushed a commit that referenced this pull request Jun 16, 2026
@miss-islington @bhuvi27
[3.15] gh-150771: Fix email serialization for shift_jis and euc-jp (G…
ab61101 
…H-151120) (GH-151541)

Encode the payload with output_charset instead of input_charset.
(cherry picked from commit 0777a58)

Co-authored-by: dev <b.chouksey27@gmail.com>
philthompson10 pushed a commit to philthompson10/cpython that referenced this pull request Jun 17, 2026
@bhuvi27 @philthompson10
pythongh-150771: Fix email serialization for shift_jis and euc-jp (py…
6f4f0e3 
…thonGH-151120)

Encode the payload with output_charset instead of input_charset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes

Assignees

@serhiy-storchaka serhiy-storchaka

Labels

needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Issues with Shift_JIS and EUC-JP in email

2 participants

@bhuvi27 @serhiy-storchaka