Pillow 3.0 raises errors on open when using tempfile.TemporaryFile

[kmike]

Here is a Travis job which starts to fail because of Pillow update: https://travis-ci.org/scrapy/scrapy/jobs/83354354. Sorry, I haven't checked what happens and why, so this bug report is very vague. Maybe someone with more detailed knowledge can spot an issue immediately. If not, I'll take a deeper look later.

This line fails: https://github.com/scrapy/scrapy/blob/fe15f93e533be36e81e0385691fe5571c88b0b31/tests/test_pipeline_images.py#L81. This is _create_image function: https://github.com/scrapy/scrapy/blob/fe15f93e533be36e81e0385691fe5571c88b0b31/tests/test_pipeline_images.py#L208.

There are also some errors printed to stdout in this job; they can be a different Pillow issue.

[wiredfool]

How are you installing pillow?

[kmike]

It is in tox.ini deps; we're running tox on Travis. I believe Travis uses Ubuntu 12.04 and tox just runs 'pip install'.

[hugovk]

Yes, Travis is Ubuntu 12.04. If you pin Pillow to 2.9.0, what do you get?

[kmike]

Just tried it on OS X locally: Scrapy testing suite fails with Pillow 3.0, but passes if I pin Pillow to 2.9.0. So with Pillow 2.9.0 there are no exceptions. I haven't seen these error messages in stdout before either.

The command to get a test failure is tox -r -e py27 tests/test_pipeline_images.py executed from https://github.com/scrapy/scrapy checkout.

[wiredfool]

I wonder if Travis has libjpeg-dev and libz-dev packages installed.

[wiredfool]

So, on OS X it should be installing a binary wheel, so that shouldn't have the jpeg or zlib build issue.

[kmike]

These packages should be installed: Travis tests used to work before the upgrade, they started to fail when Pilow 3.0 was released. Or are there new dependencies in Pillow 3.0?

[kmike]

right, and it also fails in OS X, so this is not OS-specific.

[hugovk]

Pillow 3.0.0 now requires libjpeg and zlib, before they were optional: #1412

[wiredfool]

They were formerly optional, but commonly used and if not used, indicative of installing pillow without a working tool chain. Now they are required unless disabled.

[kmike]

But it should fail during installation, not at runtime, right? Travis tox log shows that Pillow==3.0.0 is installed.

[wiredfool]

Yes.. It should fail on install. So, then not that problem. Also ruled out by the OS X failure.

What image is failing in this test? It's looking like it's just not getting recognized, and it's falling through to the we can't identify this image error.

Could be bad exif info borking a jpeg, could be an mpo change. That's what I remember touching in JpegImagePlugin this time through.

[kmike]

I think stdout log errors are not related to the test failure, they are different.

Ok, a short example to reproduce the issue. This was previously supported, but not now:

from PIL import Image
from tempfile import TemporaryFile

buf = TemporaryFile()
im = Image.new('RGB', (100, 100), (0, 127, 255))
im.save(buf, 'JPEG')

buf.seek(0)
im = Image.open(buf)

I'm not sure why is the code written this way though.

[kmike]

Previously we used BytesIO instead of TemporaryFile, but as I recall BytesIO buffers were not supported by Pillow.

[wiredfool]

So, I just ran this:

from helper import PillowTestCase
from PIL import Image

class test_scrapy(PillowTestCase):
    def _create_image(self, format, *a, **kw):
        buf = open(self.tempfile('temp.jpg'), 'w+b')
        Image.new(*a, **kw).save(buf, format)
        buf.seek(0)
        return Image.open(buf)

    def test_create_image(self):
        SIZE = (100, 100)
        # straigh forward case: RGB and JPEG
        COLOUR = (0, 127, 255)
        im = self._create_image('JPEG', 'RGB', SIZE, COLOUR)

        self.assert_image_equal(Image.new('RGB', (100,100), COLOUR), im)

and it passes.