[GLUTEN-6650] Remove bloomfilter from partition filter

[WangGuangxin]

What changes were proposed in this pull request?

If BloomFilterMightContains is pushed down to partition filter, it's executed in Spark driver using codegen. But since the bloom filter contructed by native are different with Spark, a NPE will throw in this case

(Fixes: #6650)

How was this patch tested?

UT

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[github-actions]

#6650

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

Do bloom filters still take effect in Gluten if we do this?

[WangGuangxin]

It doesn't task effect in this case, and partition filter will not re-evalute in following FilterExec or other operator.

[zhztheplayer]

So the PR completely disables runtime bloom-filter in Gluten? Am I missing something?

[WangGuangxin]

completely

@zhztheplayer Only disable the case when bloomfilter is applied on hive table's partition column ( join key is hive table's partition column).

In most case, bloom filter is applied on data column, which is not affected by this PR.

[zhztheplayer]

But since the bloom filter contructed by native are different with Spark, a NPE will throw in this case

I remember we had made Velox bloom-filter functions runnable in vanilla Spark operators so there shouldn't have compatibility issues. #5435. Am I missing something? Or there is a bug?

[WangGuangxin]

But since the bloom filter contructed by native are different with Spark, a NPE will throw in this case

I remember we had made Velox bloom-filter functions runnable in vanilla Spark operators so there shouldn't have compatibility issues. #5435. Am I missing something? Or there is a bug?

@zhztheplayer Got it. Let me check.

[WangGuangxin]

But since the bloom filter contructed by native are different with Spark, a NPE will throw in this case

I remember we had made Velox bloom-filter functions runnable in vanilla Spark operators so there shouldn't have compatibility issues. #5435. Am I missing something? Or there is a bug?

@zhztheplayer Got it. Let me check.

@zhztheplayer I digged into it, the main reason is that in this case, its a partition filter and the bloom filter is evaluted in driver side, so we need to change VeloxBloomFilterMightContain's codegen from

val bf = ctx.addMutableState(className, "bloomFilter")
    ctx.addPartitionInitializationStatement(s"$bf = $className.readFrom($bfData);")

to

val bf = ctx.addMutableState(className, "bloomFilter", bf => s"$bf = $className.readFrom($bfData);")

Otherwise, the bf object is not initialized when executed in driver and a NPE is thrown.

But after this change, the VeloxBloomFilter still cannot evaluted on driver side, since most of JNI and Resource Management will check whether we are in a spark task or not.
such as

Runtimes.contextInstance

and

Spillable reservation listener must be used in a Spark task.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

[zhztheplayer]

Sorry for missing the latest message and I am now revisiting. Thanks! @WangGuangxin