Skip to content

Add delete file index to pyiceberg and support equality delete reads #2255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
geruh wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Add delete file index to pyiceberg and support equality delete reads #2255

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
fd31e73
feat: add mor delete file index support
Jun 18, 2025
4f6b31e
code style and correctness
geruh Jul 29, 2025
d3dd58e
fix failing tests
geruh Jul 29, 2025
11f406e
_read_deletes and other changes
rutb327 Aug 14, 2025
e7853e2
Add dv path uniqueness check
rutb327 Aug 19, 2025
b3d9fa6
integration tests for delete_file_index
rutb327 Aug 19, 2025
f250985
fixture and test update
rutuja273 Aug 26, 2025
d77590e
task_to_record_batches function call update in test
rutuja273 Sep 10, 2025
13bb413
rebase delete file index
geruh Dec 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add dv path uniqueness check 
Co-authored-by: Sung Yun <107272191+sungwy@users.noreply.github.com>
  • Loading branch information
@rutb327 @sungwy @geruh
2 people authored and geruh committed Dec 7, 2025
commit e7853e2b9e4f04c3e7a17cb2551bb8eb4448e0a4
5 changes: 4 additions & 1 deletion pyiceberg/table/delete_file_index.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,10 @@ def add_delete_file(self, manifest_entry: ManifestEntry, partition_key: Optional
if delete_file.file_format == FileFormat.PUFFIN:
sequence_number = manifest_entry.sequence_number or 0
path = delete_file.file_path
self.dv[path] = (delete_file, sequence_number)
if path not in self.dv:
self.dv[path] = (delete_file, sequence_number)
else:
raise ValueError(f"Can't index multiple DVs for {path}: {self.dv[path]} and {(delete_file, sequence_number)}")
else:
pos_wrapper = PositionalDeleteFileWrapper(manifest_entry)

Expand Down