Improve large array allocation speed

[mrocklin]

I would like to be able to allocate NumPy arrays quickly. Currently calling numpy.ones runs at about 2-3GB/s on consumer laptops (tested a 2018 Macbook pro and ThinkPad Carbon 4th gen with an i7)

My general intuition is that I should be able to allocate memory more quickly than this. Is that true? Is there anything that can be done here?

Reproducing code example:

import numpy

a = numpy.ones(shape=(1000000000), dtype='u1')  # 1 GB

# CPU times: user 293 ms, sys: 286 ms, total: 579 ms
# Wall time: 580 ms

>>> numpy.__version__
'1.17.0'

Numpy/Python version information:

1.17.0 3.7.1 (default, Oct 23 2018, 14:07:42)
[Clang 4.0.1 (tags/RELEASE_401/final)]

[charris]

There are probably three things involved here: the OS, memory bandwidth (initialize to 1), and the numpy/python allocation implementation. How does the speed of empty/zeros compare? @juliantaylor would know more about this.

[mrocklin]

How does the speed of empty/zeros compare?

They're free to start, but we'll suffer the allocation costs eventually when we write to those pages

In [1]: import numpy

In [2]: %%time
   ...: a = numpy.zeros(shape=(1000000000), dtype='u1')  # 1 GB
   ...:
   ...:
CPU times: user 16 µs, sys: 9 µs, total: 25 µs
Wall time: 29.8 µs

In [3]: %%time
   ...: a = numpy.empty(shape=(1000000000), dtype='u1')  # 1 GB
   ...:
   ...:
CPU times: user 17 µs, sys: 11 µs, total: 28 µs
Wall time: 33.1 µs

[charris]

I get slightly faster times for the uninitialized allocations, however, np.ones is much faster

In [7]: %%time 
   ...: a = numpy.ones(shape=(1000000000), dtype='u1') 
   ...:  
   ...:                                                                         
CPU times: user 26 ms, sys: 55.5 ms, total: 81.4 ms
Wall time: 81.7 ms

This is on a six year old desktop Intel(R) Core(TM) i5-4670K CPU @ 3.40GHz with 1866MHz 2x8GB (16GB) memory. It seems that memory bandwidth is the determining factor here.

[charris]

And perhaps compiler also, this is on fedora 30, gcc 9.1.1.

[mrocklin]

I would love it if I could get that performance :)

I'm now very curious what the determining factors are here.

[charris]

I'd have to check in the BIOS to see how the memory is actually set up, I don't think DDR4 is supported, so am not sure any more what is actually in there. It may be overclocked a bit.

EDIT: G.SKILL · DDR3 · 8 GB · 1,866 MHz · 240-pin

[charris]

My guess it that it is a combination of compiler and laptop. IIRC, laptops tend to have slow memory. When I bought my current setup I think common laptops were running about 1/4 the speed. Power saving settings can also have a big impact.

[pentschev]

I don't think that the memory frequency is the determining factor here. I'm on a desktop with i7-7800X @3.50 GHz, 1x16GB DDR4 @2400 MHz, CPU governor set to performance and I am only seeing a bit faster results than @mrocklin:

In [1]: import numpy as np

In [2]: np.__version__
Out[2]: '1.17.0'

In [3]: %%time
   ...: a = np.ones(int(1e9), dtype='u1')
   ...:
   ...:
CPU times: user 128 ms, sys: 208 ms, total: 336 ms
Wall time: 335 ms

I think it may have something to do with the compiler or some Linux configuration.

@charris if you watch top/htop/free do you see 1GB more of resident memory being allocated after np.ones get executed?

[charris]

@pentschev Yes.

[charris]

I wonder if it may have something to do with disk backup of the files in memory. I'm running two SSDs in RAID 0 (yes, I live dangerously :)

[seberg]

(If it is slow) could there also be things about MADV_HUGEPAGE not kicking in, possibly even because of OS settings?

EDIT: There is also a note about hugepages in the 1.16 release notes, for linux /sys/kernel/mm/transparent_hugepage/enabled should be madvise or always.

[pentschev]

@seberg I had checked that before and madvise was. Just for peace-of-mind I tried setting it to always now, and I do see an improvement after that:

In [3]: %%time
   ...: a = np.ones(int(1e9), dtype='u1')
   ...:
   ...:
CPU times: user 48.1 ms, sys: 93.5 ms, total: 142 ms
Wall time: 141 ms

So it looks like madvise may not be enough for all cases. Also, in NumPy if MADV_HUGEPAGE is available, it's supposed to be default for any buffer sizes >= 4MB:

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/alloc.c#L77-L84

[pentschev]

Not really sure why I had to have always set for this to be faster, from the kernel docs:

"madvise" will enter direct reclaim like "always" but only for regions
that are have used madvise(MADV_HUGEPAGE). This is the default behaviour.

[charris]

I should mention that my memory has dual channel access (two slots out of four used), which might also make a difference.

EDIT: Dual channel memory in two slots, memory is overclocked. Specs for the Asus Z87I-DELUXE motherboard.