Singificant performance regression on large compression level (18) when switching to API v2

[kostja]

Hello,

We've recently updated ZSTD from version 1.5.1 to version 1.5.7 and observed a 4x drop in compression speed at level 18 when using the new API. Let me clarify: 1.5.7 itself is fine and doesn't regress against 1.5.1, when the old, deprecaed API is in use. It's the switch to the new API that drops the speed 4x.

Unfortunately I don't have an isolated test case, but the code can be found at:

https://git.picodata.io/core/tarantool/-/commit/392485ed92973ab836b72714a9489b86f68ab9c5

The test is a database benchmark that fills 128KB pages of LSM tree with random data and compresses them with ZSTD. The drop in compression speed when simply switching to the new API is from 40MB/s to 10MB/s on my build box.

[Bulat-Ziganshin]

Kostja published results of his benchmark at https://git.picodata.io/core/tarantool/-/commit/6add6bcb8eee7d6fa1eaa85fddd1d4064009a756

Here they are:

ZSTD	level 3	level 9	level 18
1.5.7, API v2	240 MB/s	107 MB/s	10.4 MB/s
1.5.7, API v1	223 MB/s	125 MB/s	35 MB/s
1.5.1, API v1	169 MB/s	124 MB/s	34 MB/s

API v1 here means sequence of ZSTD_compressBegin + ZSTD_compressContinue + ... + ZSTD_compressEnd, while API v2 means sequence of calls to ZSTD_compressStream2. The resulting compressed data are anyway concatenated and later are decompressed together by a sequence of calls to ZSTD_decompressStream.

Why ZSTD_compressStream2 calls become so much slower than ZSTD_compressContinue on level 18?

[Cyan4973]

Thanks for the feedback,

it does not seem a reasonable outcome.
API v2 shouldn't be much slower than APIv1.

I'll have to reproduce the issue locally in order to study it.

The first thing that surprises me is that 35 MB/s seems really fast for level 18,
my local M1 Pro only achieves ~4.5 MB/s on silesia,
I'm concerned that such a result might be data dependent (hence difficult to reproduce locally).

[Cyan4973]

So, we have this benchmark tool, tests/fullbench/,
which is designed to test multiple scenarios and entry points into the API.
It's not necessarily complete, but it already features a bunch of scenarios.

And it happens that the comparison presented here can be imitated with:
./tests/fullbench -l18 -b4 <sample> => compressContinue
./tests/fullbench -l18 -b7 <sample> => compressStream
./tests/fullbench -l18 -b13 <sample> => compressStream2, continue

I tested them with a bunch of samples, and could not observe the behavior described in this issue: all these variants produce approximately the same compression ratio and speed (though, in my cases, much slower than reported above, likely as a consequence of using samples with different characteristics).

Now, it could be that devil is in the details,
and we would need to look directly at the source code to understand what is going on,
maybe there is some little detail that makes a lot of difference.

[Cyan4973]

fills 128KB pages of LSM tree with random data

Wait, that's the test?

[Cyan4973]

OK so I tested ./tests/fullbench -l18 -b{4,7,13} with some random data,
and immediately the speed jumped to ~45 MB/s, becoming comparable to your measurements.

But I still could not observe any noticeable difference between the 3 APIs, they all scored ~45 MB/s.
This suggest there may be other differences at play.

Also, I can't stress it enough, compressing random data is not a good test.
It's an edge case, not representative of real performance on real compressible data.
I would recommend using anything else, like some lorem ipsum generator, or whatever sample is more representative of your data. Compression of random data (or rather lack of) is essentially a specialized code path.