Skip to content

Register frozen Parquet files for DuckLake perf - #1027

Merged
bill-ph merged 2 commits into
mainfrom
agent/batch-ducklake-perf-backfill
Aug 4, 2026
Merged

Register frozen Parquet files for DuckLake perf#1027
bill-ph merged 2 commits into
mainfrom
agent/batch-ducklake-perf-backfill

Conversation

@bill-ph

@bill-ph bill-ph commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replace the memory/disk-heavy INSERT … SELECT backfill with ducklake_add_data_files registration of the frozen events and persons Parquet objects
  • create the DuckLake tables before registration, then apply the existing production partition specifications; the fixture keys are flat rather than Hive-partitioned
  • validate exact file-list registration and schema metadata without scanning the corpus before the perf step

Why

The first real mw-dev run failed during setup_posthog_tables: rewriting the production-sized fixture exhausted worker ephemeral storage. Registration reads Parquet footers and writes DuckLake catalog metadata only, so raw-view and DuckLake-table perf queries use the same immutable S3 objects.

Validation

  • go test ./tests/mw-dev/scenario ./tests/mw-dev/scenario/sql
  • local smoke test with the pinned DuckLake extension: register a Parquet file, apply partitioning afterward, query it successfully, and confirm no replacement Parquet files were written
  • git diff --check

Next: dispatch posthog_frozen_perf from this branch to validate mw-dev setup and performance.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 0 4 0
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +1 / -1
  • Assertions: +4 / -3
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

@bill-ph
bill-ph marked this pull request as ready for review August 3, 2026 22:58
@bill-ph bill-ph changed the title Stream DuckLake perf table backfill Register frozen Parquet files for DuckLake perf Aug 4, 2026
@bill-ph
bill-ph merged commit 61f4f32 into main Aug 4, 2026
33 of 34 checks passed
@bill-ph
bill-ph deleted the agent/batch-ducklake-perf-backfill branch August 4, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant