Skip to content

PERF : modify SMJ shuffle file reader to skip validation #15948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
getChan wants to merge 11 commits into from
Closed

PERF : modify SMJ shuffle file reader to skip validation #15948

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
11bf46e
PERF : modify SMJ shuffle file reader to skip validation
getChan May 5, 2025
c869542
Merge branch 'main' into smj-skip-validation
getChan May 5, 2025
38255bd
Merge branch 'main' into smj-skip-validation
getChan May 6, 2025
4c19675
add bench
getChan May 6, 2025
ee2a6d6
cargo fmt
getChan May 6, 2025
a5bda29
cargo fmt
getChan May 6, 2025
b17f5fd
move bench
getChan May 6, 2025
28839e3
increase sample data size
getChan May 7, 2025
dab92e1
clippy
getChan May 7, 2025
0423d19
fmt
getChan May 8, 2025
dccea7f
Merge branch 'main' into smj-skip-validation
alamb Jul 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
PERF : modify SMJ shuffle file reader to skip validation
  • Loading branch information
@getChan
getChan committed May 5, 2025
commit 11bf46e48acf8d15415e00ace2fb79089e7b88a8
2 changes: 1 addition & 1 deletion datafusion/physical-plan/src/joins/sort_merge_join.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2313,7 +2313,7 @@ fn fetch_right_columns_from_batch_by_idxs(
Vec::with_capacity(buffered_indices.len());

let file = BufReader::new(File::open(spill_file.path())?);
let reader = StreamReader::try_new(file, None)?;
let reader = unsafe {StreamReader::try_new(file, None)?.with_skip_validation(true)};

for batch in reader {
batch?.columns().iter().for_each(|column| {
Expand Down
Loading