feat: add free-form metadata map to DataFragment#6920
Open
jackye1995 wants to merge 3 commits into
Open
Conversation
Add a `map<string, string> metadata` field to `DataFragment` for lightweight key-value metadata (e.g., creation time, process ID). Extend `UpdateConfig` transaction with `fragment_metadata_updates` to support updating fragment metadata with the same `UpdateMap` semantics used for table/schema/field metadata, including conflict detection for concurrent updates to the same fragment.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
- Add fragment_metadata_updates to has_new_fields check in proto deserialization to prevent silent data loss - Add conflict detection for UpdateConfig fragment metadata against Overwrite, Restore, Delete, and Update operations - Update spec doc with new conflict semantics
Old writers that do not understand fragment metadata must not overwrite a manifest containing it, as they would silently drop the metadata. Add FLAG_FRAGMENT_METADATA (bit 64) as a writer-only feature flag, set automatically when any fragment has non-empty metadata.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
map<string, string> metadatafield toDataFragmentprotobuf for lightweight per-fragment key-value metadata (e.g., creation time, process ID).UpdateConfigtransaction withfragment_metadata_updatesto support updating fragment metadata, using existingUpdateMapsemantics (merge or replace mode).UpdateConfigtransactions conflict if they modify metadata on the same fragment ID.Voting Discussion
This is a spec change. A corresponding [VOTE] discussion will be created separately.