SimpleHTTPServer reports wrong content-length for text files

[irmen]

BPO	839496
Nosy	@birkenfeld, @irmen, @amauryfa, @bluebird75
Files	diff.txt: SimpleHTTPServer patch httptest.zip: test case that shows the problem on Windows.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-07-20.11:26:37.960>
created_at = <Date 2003-11-10.20:42:43.000>
labels = ['easy', 'library']
title = 'SimpleHTTPServer reports wrong content-length for text files'
updated_at = <Date 2008-07-20.11:26:37.958>
user = 'https://github.com/irmen'

bugs.python.org fields:

activity = <Date 2008-07-20.11:26:37.958>
actor = 'georg.brandl'
assignee = 'jlgijsbers'
closed = True
closed_date = <Date 2008-07-20.11:26:37.960>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2003-11-10.20:42:43.000>
creator = 'irmen'
dependencies = []
files = ['5678', '5679']
hgrepos = []
issue_num = 839496
keywords = ['patch', 'easy']
message_count = 11.0
messages = ['44870', '44871', '44872', '44873', '44874', '44875', '44876', '68140', '69332', '69355', '70081']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'jlgijsbers', 'irmen', 'amaury.forgeotdarc', 'ggenellina', 'bluebird']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue839496'
versions = ['Python 2.6', 'Python 2.5']

[irmen]

(Python 2.3.2 on Windows)

SimpleHTTPServer reports the size of the file on disk
as Content-Length. This works except for text files.
If the content type starts with "text/" it is opening the
file in 'text' mode rather than 'binary' mode. At least on
Windows this causes newline translations, thereby making
the actual size of the content transmitted *less* than
the content-length!

I don't know why SimpleHTTPServer is reading text files
with text mode. The included patch removes this distinction
so all files are opened in binary mode (and, also on
windows,
the actual size transmitted is the same as the reported
content-length).

--Irmen de Jong

[irmen]

Logged In: YES
user_id=129426

This bug is also still present in the SimpleHTTPServer.py
from Python 2.3.3 (and in the current CVS version, too).

Is there a reason why it treats text files differently? If
so, then at least the reported content-length must be fixed.

[irmen]

Logged In: YES
user_id=129426

The attached trivial patch removes the special case for text
files.

[irmen]

Logged In: YES
user_id=129426

The attached httptest.zip contains a test scenario. When run
on windows, it will show the problem.
First start 'startserver.py' and then from the same
directory run test.py.
I get this:

[E:\temp\httptest]python test.py
The reported content-length is: 1047 bytes
The real filesize is: 1047 bytes
The data I actually received from the httpserver is: 1028 bytes

[irmen]

Logged In: YES
user_id=129426

Hm, perhaps the easy way out (see my patch) is not the best
solution:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1

It seems it's best to convert text responses to CR LF
format? If we should do this, we must somehow 're-calculate'
the content-length after the CR LF conversion.

[irmen]

Logged In: YES
user_id=129426

Upon re-reading the w3 spec, it seems that we're safe. As
long as the use of CR or LF or CR+LF is consistent in the
whole text file.
The spec says: "HTTP relaxes this requirement [=the
requirement of being in canonical form] and allows the
transport of text media with plain CR or LF alone
representing a line break when it is done consistently for
an entire entity-body. HTTP applications MUST accept CRLF,
bare CR, and bare LF as being representative of a line break
in text media received via HTTP."

So my patch is safe, I think.

[jlgijsbers]

Logged In: YES
user_id=469548

Okay, I've checked in the fix on maint24 and HEAD.

[ggenellina]

As noted by Leo Jay in this message
<http://mail.python.org/pipermail/python-list/2008-June/495718.html\>
this bug was supposedly fixed but it is still present.

Looks like the patch was only applied to release24-maint, not to the
trunk. Both the 2.5 and the 2.6 sources don't have the patch applied.
3.0 doesn't have this problem but probably it was fixed independently.

[bluebird75]

I confirm that the problem is present on python2.5 on windows, and that
the attached patch fixes it.

[amauryfa]

Committed as r64762.
Maybe a 2.5 backport candidate, but I fear that it may break existing code.