Problem statement
Halide's memcpy performance is quite poor: performance_memcpy allows a 3x throughput gap that still sometimes trips.
I wrote an experimental branch (alexreinking/non-temporal-hack) that asks LLVM to attach non-temporal load/store hints globally when HL_EXPERIMENTAL_NONTEMPORAL_STORES=1 and/or HL_EXPERIMENTAL_NONTEMPORAL_LOADS=1 are set.
On ARM, I see the following:
areinking@Mac non-temporal-hack % HL_EXPERIMENTAL_NONTEMPORAL_LOADS=1 ./build/macOS/test/performance/performance_memcpy
system memcpy: 70.61 GB/s
halide memcpy: 69.44 GB/s
Success!
areinking@Mac non-temporal-hack % HL_EXPERIMENTAL_NONTEMPORAL_LOADS=0 ./build/macOS/test/performance/performance_memcpy
system memcpy: 78.58 GB/s
halide memcpy: 55.63 GB/s
Success!
The benchmarks are pretty noisy, but I can't get a comparable results without the non-temporal loads. On my machine (Apple M3), enabling NT stores does not improve performance, likely because it's writing a whole cache-line anyway, and the cache can skip RFO.
Proposed solution
Add two new scheduling directives:
Func::stream_loads() (also applies to ImageParam) that ensures that loads from this Func are done non-temporally. (e.g. ldnp or movntdqa).Func::stream_stores() that ensures that stores to the buffer backing this Func are done non-temporally (e.g. stnp or movntiq)
On ARM, there are no coherency issues. On x86, non-temporal stores must be followed by sfence before they can be safely read; this will require additional analysis. I haven't looked into other backends (GPUs, RISC-V, Hexagon, etc.).
The proposed stream_loads directive interacts especially nicely with the .in() directive: in a residual-network-like computation, you re-load an old buffer, streaming it once just for computation. Loads in the network would be temporal; loads during the residual application would be non-temporal ("streaming").
Non-temporal loads are also meaningful on some platforms for matrix-vector operations, like ARM. They are meaningless on others: x86 ignores non-temporal hints on write-back memory (i.e. everything Halide has cared about to date).
Additional context
This issue was brought to my attention by @mcourteaux
Problem statement
Halide's memcpy performance is quite poor:
performance_memcpyallows a 3x throughput gap that still sometimes trips.I wrote an experimental branch (
alexreinking/non-temporal-hack) that asks LLVM to attach non-temporal load/store hints globally whenHL_EXPERIMENTAL_NONTEMPORAL_STORES=1and/orHL_EXPERIMENTAL_NONTEMPORAL_LOADS=1are set.On ARM, I see the following:
The benchmarks are pretty noisy, but I can't get a comparable results without the non-temporal loads. On my machine (Apple M3), enabling NT stores does not improve performance, likely because it's writing a whole cache-line anyway, and the cache can skip RFO.
Proposed solution
Add two new scheduling directives:
Func::stream_loads()(also applies toImageParam) that ensures that loads from this Func are done non-temporally. (e.g.ldnpormovntdqa).Func::stream_stores()that ensures that stores to the buffer backing this Func are done non-temporally (e.g.stnpormovntiq)On ARM, there are no coherency issues. On x86, non-temporal stores must be followed by
sfencebefore they can be safely read; this will require additional analysis. I haven't looked into other backends (GPUs, RISC-V, Hexagon, etc.).The proposed
stream_loadsdirective interacts especially nicely with the.in()directive: in a residual-network-like computation, you re-load an old buffer, streaming it once just for computation. Loads in the network would be temporal; loads during the residual application would be non-temporal ("streaming").Non-temporal loads are also meaningful on some platforms for matrix-vector operations, like ARM. They are meaningless on others: x86 ignores non-temporal hints on write-back memory (i.e. everything Halide has cared about to date).
Additional context
This issue was brought to my attention by @mcourteaux