feat: add MemWAL sharding evaluator

[jackye1995]

Adds an Arrow-native MemWAL sharding evaluator and exposes it through the Java API/JNI.

• Evaluates MemWAL sharding specs against Arrow RecordBatch values for bucket, identity, and unsharded fields.
• Resolves sharding source IDs through a Java-provided source-id-to-column map.
• Adds Java-facing ShardingEvaluator returning an Arrow reader for the evaluated sharding key batch.

This is needed by lance-spark to route writes using Lance's sharding semantics instead of duplicating Spark-side bucket logic.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 78.84131% with 84 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/mem_wal/sharding.rs	79.16%	68 Missing and 12 partials ⚠️
rust/lance/src/dataset/mem_wal/api.rs	50.00%	3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

I think the Python sharding spec round-trip needs one fix before this lands.

[Xuanwo]

This makes dict specs returned by Dataset.mem_wal_index_details() unusable with the new evaluator.

mem_wal_index_details() currently serializes each sharding field with field_id, source_ids, transform, result_type, and parameters, but it does not include expression. Since this parser now requires expression to be present, the natural flow below fails with Missing sharding spec field 'expression':

spec = ds.mem_wal_index_details()["sharding_specs"][0]
evaluate_sharding_spec(batch, spec, LanceSchema.from_pyarrow(batch.schema))

Could we either include expression in the dict returned by mem_wal_index_details() or treat a missing expression key as None here? I think adding it to mem_wal_index_details() is cleaner because that keeps the exported spec shape complete and round-trippable.

[jackye1995]

Addressed in 4f2ddc5 by including expression in the Python Dataset.mem_wal_index_details() sharding field dict. I also extended test_initialize_mem_wal_bucket_sharding to pass the returned dict spec directly into evaluate_sharding_spec(...), covering the round-trip path from the comment.

Submitted as changes requested by mistake; intended as a non-blocking review comment.

[hamersaw]

Looks great to me! It feels a little off to be burying the sharding (partitioning / clustering) metadata into the memwal index implementation IMO. I think there are a lot of use-cases where we use this metadata (ex. writing dataset, Spark SPJ, etc) that is not tied to mem_wal in any way.

[jackye1995]

Looks great to me! It feels a little off to be burying the sharding (partitioning / clustering) metadata into the memwal index implementation IMO. I think there are a lot of use-cases where we use this metadata (ex. writing dataset, Spark SPJ, etc) that is not tied to mem_wal in any way.

I agree, I think one related proposal is to lift this into some independent module like lance-sharding.

But on the other side, the good thing about this is user who sets this can directly write with WAL. It's just we don't handle the read path.