TPC-H q8 hangs with xxhash64 enabled

[andygrove]

Describe the bug

I can run TPC-H queries @ 100GB locally with commit 8f8a0d9, but not with any commit after that because the job hangs and eventually fails with out of memory errors.

I will investigate more and post findings on this issue.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

The issue starts with the PR that added xxhash64 (#424).

I wonder if this is an issue with hashing itself or just that more queries are running natively now and I am hitting some other issue 🤔

[viirya]

Is the failed one with OOM a query with xxhash64 so it is not natively run before?

[andygrove]

The issue is happening with query 8. The behavior I see is that all/most cores are at 100% utilization, and memory is not actually high, but the cluster becomes unresponsive before an OutOfMemory error is shown in the driver.

[advancedxy]

all/most cores are at 100% utilization

So it might be more about cpu usage rather than memory consumption? If it’s indeed related to xxhash64, one possible reason might be that every hash call it creates a XXHash64 struct on the stack.

Thanks for filing this, it would be great that we can have a minimum reproduction example with smaller size. I can help with this issue.

[parthchandra]

Was this running on a K8s cluster? It should not result in a OutOfMemory if cpu limits are reached.

[andygrove]

This is running locally. I ran this again comparing xxhash64 enabled vs disabled and it is very clear that early on in processing q8 all the cores are busy and Spark logging stops and then there are heartbeat timeouts and eventually the driver fails with OOM but I do not see system memory usage being very high so I think the cluster timeouts lead to the OOM issue in the driver somehow.

[parthchandra]

@andygrove how are you enabling/disabling xxhash64? I just ran tpch8 locally in the profiler and I see no signs of xxhash64 in the output.

[parthchandra]

JFR output from async_profiler for tpch q8 (sf1) running locally on mac. (unzip, then open in intellij or tool of your choice).
CometTPCHQueryBenchmark_2024_06_06_102425.jfr.gz

[parthchandra]

hashAgg_doAggregateWithKeys_0$ stands out as an expensive call. I suppose it makes sense to profile the xxhash code independently, profiling an entire tpch query produces a lot of noise.

[andygrove]

@andygrove how are you enabling/disabling xxhash64?

I have been doing this manually by commenting out the case XxHash64(children, seed) => section in QueryPlanSerde

[andygrove]

I just ran tpch8 locally in the profiler and I see no signs of xxhash64 in the output.

Also, I am using the configs from https://datafusion.apache.org/comet/contributor-guide/benchmarking.html#running-benchmarks-against-apache-spark-with-apache-datafusion-comet-enabled

[parthchandra]

Got it. I think the calls to xxhash64 native code are not being profiled successfully by async_profiler. Looks like the profiler loses track of the native call stack in generated code. I'll experiment and see if I can find some way to get more info.
In the run above while most of the executor threads were in hash aggregation (when cpu usage was at its peak), some threads were in the LZ4 decompressor and some in the parquet reader both of which were probably adding to the CPU load.
Also, the peak (CPU: 95%) existed for a very short while compared to the overall run.