fix integer overflow in _imaging.c

[nedwill]

Fixes issue #1710. I added the canonical integer overflow check used across the CPython codebase.

Perhaps it's better to raise an OverflowError but I'm trying to keep some consistency with the surrounding code.

[wiredfool]

Thanks for reporting this. I think we're going to include this in 3.1.1 today.

[wiredfool]

I'm trying to add the test from the issue in a way that fails pre-patch, but I'm getting a failed malloc:

142     kk = malloc(xsize * kmax * sizeof(float));
(gdb) n
143     if ( ! kk)
(gdb) p kk
$1 = (float *) 0x0
(gdb) n
144         return (Imaging) ImagingError_MemoryError();

I've tried this on a small and large memory machine where it should have enough memory to allocate several Gb, but each time it winds up giving me a MemoryError. Unfortunately, this is indistinguishable from the post-patch case.

edit -- the test I'm using:

from PIL import Image
im = Image.open('Tests/images/hopper.png').convert('L')
xsize = 0x100000008 / 4
ysize = 1000 
im.im.resize((xsize, ysize), Image.LINEAR) 

[wiredfool]

closed in favor of #1714

[nedwill]

@wiredfool it looks like you've already merged in the fix, but just to follow up: Because malloc takes a size_t as its argument, I think the overflow only occurs on 32-bit systems, where sizeof(size_t) == sizeof(int). You could raise OverflowError instead of MemoryError if you want to distinguish these cases.

[wiredfool]

Makes sense, I'll check it out on a 32 bit vm