[Proposal] Native, vectorised, zero-copy execution path for Druid

[Shekharrajak]

Motivation

Native, vectorised, zero-copy execution path for Druid via FFM and Arrow — staged from segment reader → filter/project → DataFusion (follow-up to #19039 item 1)

Druid today does the entire query hot path on the JVM. Each row passes through a ColumnSelector interface, materialises into Java objects, gets compared/aggregated by JIT-compiled byte code - means :

• No SIMD on filter / project / aggregate hot loops.
• GC pressure from per-block wrapper objects on cold-cache scans.
• 2 GiB ceiling on off-heap buffers (see Speed up node boot times by parallelizing buffer acquisition #19025).
• Bottleneck on MSQ joins / aggregates

Proposed changes

Phase 0 — Spike: native segment column reader via FFM
Phase 1 — Layer0 production-shape: native column reader for all common types
Phase 2 — L1: vectorised native filter + project over Arrow batches
Phase 3 — L2: DataFusion as candidate aggregate/join executor in MSQ stages

Rationale

Operational impact

Performance 2–3× speedup on

• speedup on cold-cache scan + filter for typical Druid datasources.
• speedup on MSQ fact-fact joins.
• speedup on external Parquet ingestion.
• lower p99 for high-cardinality GroupBy.
• faster execution under memory pressure (DataFusion graceful spill).

Operational

• 2 GiB ByteBuffer ceiling lifted on the native path (aligns with Speed up node boot times by parallelizing buffer acquisition #19025).
• Reduced GC pressure → fewer p99 latency spikes.

Test plan (optional)

Future work (optional)

[jtuglu1]

Thanks for raising this. I was waiting until Druid v38 was cut (so compiler can be switched to JDK 25). This way, we can use latest version of FFM.

[jtuglu1]

I think using Datafusion would be a good idea (much better than the current query planner/execution engine set up), but one thing to call out here is that Datafusion always materializes rows into ArrowBatch between operators, which is a bit different in some cases compared to Druid. This materialization cost can be expensive.

[jtuglu1]

To be clear, I would much rather rely on Datafusion to implement the core physical operators (since they do it much better, and we get future performance improvements for free), the only caveat being we will need to ensure we can get the same or better read amplification than the current native JDK Druid code does, e.g. where we can rely on column stats, row count, bitmap/dictionary indices to pushdown filters/projections and avoid decompressing/scanning more than we need to.

[Shekharrajak]

Thanks @jtuglu1 for checking.

Yes, I understand the complexity and work for this initiative due to JNI/FFM and prerequisite if we want vectorization support.

We can actually evaluate the Java bindings for Apache DataFusion: https://github.com/apache/datafusion-java and see if those work can be minimised.