sources: Speed up MySQL ingestion by parallelizing#35569
Open
def- wants to merge 1 commit into
Open
Conversation
Contributor
|
Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone. PR title guidelines
Pre-merge checklist
|
9c63b2a to
f7effd1
Compare
2ee8d42 to
2e9f0da
Compare
db20263 to
f97084a
Compare
f97084a to
96687b4
Compare
792128d to
11d7b39
Compare
Partition each table's primary-key range across timely workers so they read disjoint PK ranges of the initial snapshot concurrently, reducing initial-load time for large tables with a single-column integer primary key. A snapshot leader establishes the consistent point and broadcasts SnapshotInfo to all workers over a timely feedback loop; each worker then reads its range under a CONSISTENT SNAPSHOT transaction. Tables without a suitable PK fall back to single-worker-per-table mode. Also adds a MySqlInitialLoadMultiWorker feature benchmark. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
11d7b39 to
a7eed48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a cluster with 8 workers, running on my 8 core / 16 thread dev server:
Co-written with Claude 🤖