Use R2 bucket for duckdb libraries by myrrc · Pull Request #8486 · vortex-data/vortex

myrrc · 2026-06-18T09:30:55Z

Use ci-builds.vortex.dev R2 bucket as source for duckdb release and commit builds.
Mirror release builds from duckdb github releases page. Build commits from source.
For commit builds also try to download from R2 (useful for testing pre-release in CI)
Gate test_geometry for duckdb under release builds only. Running it for commits means we need to bundle "spatial" extension which is hard on macos due to openssl-dev symbols

codspeed-hq · 2026-06-18T09:38:11Z

Merging this PR will not alter performance

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 4 improved benchmarks
❌ 7 regressed benchmarks
✅ 1570 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`decompress_rd[f64, (10000, 0.01)]`	108.7 µs	139.1 µs	-21.89%
❌	Simulation	`decompress_rd[f64, (10000, 0.1)]`	109 µs	139.5 µs	-21.85%
❌	Simulation	`decompress_rd[f64, (10000, 0.0)]`	108.7 µs	139.1 µs	-21.83%
❌	Simulation	`decompress_rd[f32, (100000, 0.0)]`	496 µs	583.8 µs	-15.05%
❌	Simulation	`decompress_rd[f32, (10000, 0.1)]`	78.1 µs	91.2 µs	-14.43%
❌	Simulation	`decompress_rd[f32, (10000, 0.01)]`	78.1 µs	91 µs	-14.2%
❌	Simulation	`decompress_rd[f32, (10000, 0.0)]`	78.5 µs	91.2 µs	-13.91%
⚡	Simulation	`chunked_varbinview_opt_canonical_into[(1000, 10)]`	206.8 µs	170.2 µs	+21.46%
⚡	Simulation	`bitwise_not_vortex_buffer_mut[128]`	215.3 ns	186.1 ns	+15.67%
⚡	Simulation	`chunked_varbinview_into_canonical[(100, 100)]`	307.1 µs	272.8 µs	+12.59%
⚡	Simulation	`bitwise_not_vortex_buffer_mut[1024]`	275.6 ns	246.4 ns	+11.84%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing myrrc/duckdb-r2-infra (3c6ac96) with develop (575db9c)}

myrrc · 2026-06-18T13:43:43Z

Macos deployment error:

error:  cannot create /Users/runner/work/vortex/vortex/duckdb-08e34c447bae34eaee3723cac61f2878b6bdf787/data/csv/issue2628_������.csv
        Illegal byte sequence

DuckdbFS implementation for Vortex was introduced in #6198 as opt-out, but changed to opt-in in #6564 due to performance regressions. There were multiple issues (#6709, #6565 #6685) associated with it which differ from vortex's file system behaviour. It also requires additional dependencies CI which are a blocker for #8486 since MacOS runner doesn't bundle openssl for x86_64 on arm, and builds fail. As a long term goal, calling duckdb's blocking IO inside our event loop isn't the right abstraction. We want to allow duckdb to use its own IO outside vortex. Duckdb fs is also not maintaned actively so we're removing it Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>

myrrc · 2026-06-19T13:47:43Z

I've verified manually this works for commits, they are uploaded and used from R2 as well.

0ax1

Great to see this coming into shape! Couple of questions inline. In general, will this be able to handle diff build configs: debug, release, asan etc.? The build config should prob be part of the key with which we store builds.

0ax1 · 2026-06-22T15:13:11Z

+    needs: duckdb-mirror
+    if: ${{ !cancelled() }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 5


What's the rationale for the timeout duration here?

Chosen randomly

0ax1 · 2026-06-22T15:14:18Z

+    uses: ./.github/workflows/duckdb-r2.yml
+    secrets: inherit
+
+  duckdb-ready:


Do we really need this, or asked diff can we encode the dependency?

0ax1 · 2026-06-22T15:16:13Z

@@ -0,0 +1,197 @@
+name: DuckDB R2 mirror
+
+# Mirror DuckDB libraries referenced by vortex-duckdb/build.rs to R2 when they


What does mirror mean exactly? Should we extend the text here a bit on how the whole setup works with R2 and caching?

0ax1 · 2026-06-22T15:17:50Z

+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    outputs:
+      version: ${{ steps.resolve.outputs.version }}


Do we support individual commits? (by default DDB sets sth like version 0.0.0 or so right ?)

Yes, we support commits.

0ax1 · 2026-06-22T15:19:02Z

+      - name: Resolve version and check R2
+        id: resolve
+        run: |
+          set -Eeuo pipefail


Bit complex and long to inline a shell script into the GH action, wdyt?

0ax1 · 2026-06-22T15:19:45Z

+            matrix='{"include":[]}'
+            any_missing=false
+          else
+            include=$(printf '%s\n' "${entries[@]}" | jq -sc '.')


If the runners have python preinstalled, we could maybe consider that.

0ax1 · 2026-06-22T15:24:37Z

+
+            lib_dir="${src_dir}/build/release/src"
+            stage="stage"
+            rm -rf "$stage"


How come we need to clear the dir here? Isn't this empty for each new runner?

myrrc · 2026-06-22T15:32:19Z

In general, will this be able to handle diff build configs: debug, release, asan etc.?

Yes, but for the first version it won't to keep the diff small

0ax1 · 2026-06-22T15:44:27Z

    // extensions statically, otherwise DuckDB tries to load them from an http
    // endpoint with version 0.0.1 (all non-tagged builds) which doesn't exist.
    let static_extensions = match version {
-        DuckDBVersion::Release(_) => "parquet;jemalloc",


Why remove jemalloc?

It's not an extension anymore, the build prints this as a warning in the log.

0ax1 · 2026-06-22T16:00:06Z

+    timeout-minutes: 5
+    steps:
+      - name: Verify DuckDB mirror
+        if: ${{ needs.duckdb-mirror.result == 'failure' }}


There's more result types than success and failure. We could consider if: ${{ needs.duckdb-mirror.result != 'success' }} to signal a failure here.

Apparently there's also: needs.check.outputs.any_missing == 'true'`.

myrrc requested a review from a team June 18, 2026 09:30

myrrc added the changelog/ci label Jun 18, 2026

myrrc force-pushed the myrrc/duckdb-r2-infra branch from 8129660 to 8af34b4 Compare June 18, 2026 09:31

myrrc marked this pull request as draft June 18, 2026 09:35

myrrc force-pushed the myrrc/duckdb-r2-infra branch from c3d1f15 to d3fc1da Compare June 18, 2026 10:44

myrrc temporarily deployed to duckdb-build June 18, 2026 10:45 — with GitHub Actions Inactive

myrrc temporarily deployed to duckdb-build June 18, 2026 11:17 — with GitHub Actions Inactive

myrrc requested a review from joseph-isaacs June 18, 2026 11:19

myrrc marked this pull request as ready for review June 18, 2026 11:19

myrrc marked this pull request as draft June 18, 2026 11:19

myrrc temporarily deployed to duckdb-build June 18, 2026 13:39 — with GitHub Actions Inactive

myrrc had a problem deploying to duckdb-build June 18, 2026 13:39 — with GitHub Actions Failure

myrrc had a problem deploying to duckdb-build June 18, 2026 13:58 — with GitHub Actions Error

myrrc had a problem deploying to duckdb-build June 18, 2026 16:41 — with GitHub Actions Error

myrrc had a problem deploying to duckdb-build June 18, 2026 16:51 — with GitHub Actions Error

myrrc had a problem deploying to duckdb-build June 18, 2026 16:54 — with GitHub Actions Failure

myrrc had a problem deploying to duckdb-build June 18, 2026 17:13 — with GitHub Actions Error

myrrc had a problem deploying to duckdb-build June 19, 2026 09:48 — with GitHub Actions Failure

myrrc had a problem deploying to duckdb-build June 19, 2026 09:50 — with GitHub Actions Error

myrrc temporarily deployed to duckdb-build June 19, 2026 09:53 — with GitHub Actions Inactive

myrrc mentioned this pull request Jun 19, 2026

Remove duckdbfs #8468

Merged

myrrc requested a review from AdamGS June 19, 2026 10:16

myrrc force-pushed the myrrc/duckdb-r2-infra branch from e12de4f to 59d0cd7 Compare June 19, 2026 11:02

myrrc had a problem deploying to duckdb-build June 19, 2026 11:25 — with GitHub Actions Failure

myrrc temporarily deployed to duckdb-build June 19, 2026 13:06 — with GitHub Actions Inactive

initial

3c6ac96

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>

myrrc force-pushed the myrrc/duckdb-r2-infra branch from b32cb14 to 3c6ac96 Compare June 19, 2026 13:44

myrrc temporarily deployed to duckdb-build June 19, 2026 13:44 — with GitHub Actions Inactive

myrrc marked this pull request as ready for review June 19, 2026 13:46

myrrc requested review from robert3005 and removed request for joseph-isaacs June 19, 2026 13:47

myrrc added the ext/duckdb Relates to the DuckDB integration label Jun 22, 2026

myrrc enabled auto-merge (squash) June 22, 2026 13:22

robert3005 requested a review from 0ax1 June 22, 2026 13:25

0ax1 reviewed Jun 22, 2026

View reviewed changes

vortex-data deleted a comment from github-actions Bot Jun 22, 2026

vortex-data deleted a comment from 0ax1 Jun 22, 2026

0ax1 reviewed Jun 22, 2026

View reviewed changes

		@@ -0,0 +1,197 @@
		name: DuckDB R2 mirror

		# Mirror DuckDB libraries referenced by vortex-duckdb/build.rs to R2 when they

Conversation

myrrc commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Performance Changes

Uh oh!

myrrc commented Jun 18, 2026

Uh oh!

myrrc commented Jun 19, 2026

Uh oh!

0ax1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

myrrc commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

myrrc commented Jun 18, 2026 •

edited

Loading

codspeed-hq Bot commented Jun 18, 2026 •

edited

Loading