Epic: duckdb performance/feature parity with Parquet

[myrrc]

We would like to support most/all the features the duckdb parquet extension enables

In other words, where and why are we not better than Parquet on all benchmarks we check:

clickbench: q23[1][2], q35, q40, q41, q42.
statpopgen: q2[5][6], q3[6], q4[6], q5[5][6], q6[5], q9[6], q10[6].
tpch S3 sf=1: q6, q17, q21.
tpch NVME sf=10: q4, q12 (vortex-compact).
tpch NVME sf=100: q20, q21.
tpch S3 sf=100: q6, q17, q21.
tpcds NVME sf=1: many, notable ones are q2, q3 (big regression), q5, q6, q31

clickbench q14-q18: measuring duckdb performance, scan takes 5% CPU time; q32-q34: scan takes 2.2% CPU time: q40-42: scan takes 2% CPU time.

Features that will solve this:

• [1] Lazy materialization: Late materialization support for duckdb #7721
• [2] Post-filter cardinality estimates: Mimic duckdb's post-filter cardinality estimates #7895
• [3] Partition stats Add file-wise partition stats for duckdb #7615
• [5] duckdb: flatten runend arrays on export if requested #7951
• [6] Slice list/list_view elements in duckdb exporter #8020

Things proven to have none/marginal impact:

• Multi-file stats multi-file stats for duckdb #7907
• Substr expression substr expression #7898 (also checked pushing down more expressions).
• Expression pushdown Expression pushdown for duckdb #7727

Related:

• Clickbench 27: Scalar functions pushdown to Duckdb #8303

[myrrc]

Statpopgen 3, 9, 10: the issue is likely in mismatching list types.

samply record ./target/release/duckdb-bench statpopgen --formats vortex --queries 3 --opt=scale-factor=100

Flamegraphs show that actual work spent in duckdb::PipelineExecutor::Execute in Vortex is ~2.8%.
Hypothesis is that it's due to upcasting list types from U8 to INTEGER, Parquet stores it already as INTEGER, so no conversion happens.

• vortex: https://share.firefox.dev/4ugl4UW
• parquet: https://share.firefox.dev/3PjjfYa

[myrrc]

Further work on statpopgen requires duckdb support of type pushdown for casts duckdb/duckdb#22738 as well as our support for type pushdown https://github.com/vortex-data/vortex/tree/myrrc/duckdb-type-pushdown

[myrrc]

duckdb/duckdb#22788 Type pushdown PR for duckdb
#8033 Type pushdown PR for Vortex

[myrrc]

Sometimes flamegrahs show that scheduling for random-access and some other queries spends first 400ms doing actual work, and then just stalled dequeuing. #8068 aims to solve this but doesn't have significant impact