email package content-transfer-encoding behaviour changed

[ThomasAH]

BPO	1525919
Nosy	@warsaw, @bitdancer, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2006-07-20.14:22:21.000>
labels = ['type-bug', 'expert-email', '3.10', '3.11', '3.9', 'library', 'docs']
title = 'email package content-transfer-encoding behaviour changed'
updated_at = <Date 2021-12-13.09:55:31.268>
user = 'https://bugs.python.org/ThomasAH'

bugs.python.org fields:

activity = <Date 2021-12-13.09:55:31.268>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)', 'email']
creation = <Date 2006-07-20.14:22:21.000>
creator = 'ThomasAH'
dependencies = []
files = []
hgrepos = []
issue_num = 1525919
keywords = []
message_count = 11.0
messages = ['29229', '29230', '58248', '73949', '105045', '184663', '184685', '184692', '184698', '408387', '408433']
nosy_count = 9.0
nosy_names = ['barry', 'ThomasAH', 'rdemetrescu', 'paulproteus', 'r.david.murray', 'docs@python', 'python-dev', 'littleq0903', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1525919'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[ThomasAH]

from email.Message import Message
from email.Charset import Charset, QP
text = "="
msg = Message()
charset = Charset("utf-8")
charset.header_encoding = QP
charset.body_encoding = QP
msg.set_charset(charset)
msg.set_payload(text)
print msg.as_string()

Gives

MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

=3D

With the email package from python2.4.3 and 2.4.4c0 the
last '=3D' becomes just '=', so an extra
msg.body_encode(text) is needed.

[ThomasAH]

Logged In: YES
user_id=839582

One program which got hit by this is MoinMoin, see
http://moinmoin.wikiwikiweb.de/MoinMoinBugs/ResetPasswordEmailImproperlyEncoded

[rdemetrescu]

I am not sure if it is related, but anyway...

MIMEText behaviour has changed from python 2.4 to 2.5.

# Python 2.4

>>> from email.MIMEText import MIMEText
>>> m = MIMEText(None, 'html', 'iso-8859-1')
>>> m.set_payload('abc ' * 50)
>>> print m
From nobody Thu Dec  6 12:52:40 2007
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc=
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc ab=
c abc abc abc abc abc abc abc abc abc abc abc abc=20

# Python 2.5

>>> from email.MIMEText import MIMEText
>>> m = MIMEText(None, 'html', 'iso-8859-1')
>>> m.set_payload('abc ' * 50)
>>> print m
From nobody Thu Dec  6 14:46:07 2007
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc
abc abc abc abc abc abc abc abc abc abc abc abc abc abc

However, if we initialize MIMEText with the text, we get the correct output:

# python 2.5

>>> from email.MIMEText import MIMEText
>>> m = MIMEText('abc ' * 50, 'html', 'iso-8859-1')
>>> print m
From nobody Thu Dec  6 13:01:17 2007
Content-Type: text/html; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc=
abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc abc ab=
c abc abc abc abc abc abc abc abc abc abc abc abc=20

If I want to set payload after MIMEText is already created, I need to
use this workaround::

#python 2.5
from email.MIMEText import MIMEText
m = MIMEText(None, 'html', 'iso-8859-1')
m.set_payload(m._charset.body_encode('abc' * 50))

PS: The issue's versions field is filled with "Python 2.4". Shouldn't it
be "Python 2.5" ?

[paulproteus]

Another way to see this issue is that the email module double-encodes
when one attempts to use quoted-printable encoding. This has to be
worked around by e.g. MoinMoin.

It's easy to get proper base64-encoded output of email.mime.text:

 	>>> mt = email.mime.text.MIMEText('Ta mère', 'plain', 'utf-8')
 	>>> 'Content-Transfer-Encoding: base64' in mt.as_string()
 	True
 	>>> mt.as_string().split('\n')[-2]
 	'VGEgbcOocmU='

There we go, all nice and base64'd.

I can *not* figure out how to get quoted-printable-encoding. I found
http://docs.python.org/lib/module-email.encoders.html , so I thought
great - I'll just encode my MIMEText object:

 	>>> email.encoders.encode_quopri(mt)
 	>>> 'Content-Transfer-Encoding: quoted-printable' in mt.as_string()
 	True

Great! Except it's actually double-encoded, and the headers admit to as
much. You see here that, in addition to the quoted-printable header
just discovered, there is also a base64-related header, and the result
is not strictly QP encoding but QP(base64(payload)).

 	>>> 'Content-Transfer-Encoding: base64' in mt.as_string()
 	True
 	>>> mt.as_string().split('\n')[-2]
 	'VGEgbcOocmU=3D'

It should look like:

 	>>> quopri.encodestring('Ta mère')
 	'Ta m=C3=A8re'

I raised this issue on the Baypiggies list
<http://mail.python.org/pipermail/baypiggies/2008-September/003983.html\>, but
luckily I found this here bug. This is with Python 2.5.2-0ubuntu1 from
Ubuntu 8.04.

paulproteus@alchemy:~ $ python --version
Python 2.5.2

If we can come to a decision as to how this *should* work, I could
contribute a patch and/or tests to fix it. I could even perhaps write a
new section of the Python documentation of the email module explaining this.

[ThomasAH]

Roger Demetrescu, I filed the issue with "Python 2.4", because the behavior changed somewhere between 2.4.2 and 2.4.3

The updated link to the MoinMoin bug entry is:
http://moinmo.in/MoinMoinBugs/ResetPasswordEmailImproperlyEncoded

The workaround I use to be compatible with <= 2.4.2 and >= 2.4.3 is:

    msg.set_payload('=')
    if msg.as_string().endswith('='):
        text = charset.body_encode(text)
    msg.set_payload(text)