Skip to content

feat(partitions): rework partitions using new message#3020

Open
numinnex wants to merge 12 commits intomasterfrom
partition_remaster
Open

feat(partitions): rework partitions using new message#3020
numinnex wants to merge 12 commits intomasterfrom
partition_remaster

Conversation

@numinnex
Copy link
Copy Markdown
Contributor

@numinnex numinnex commented Mar 24, 2026

Refactors partition messaging around Message<H, B>, where the backing storage is part of the type rather than
being encoded through the old TwoHalves prefix/tail layout.

Request messages now use a contiguous Owned backing, while response-style paths can use fragmented storage
directly. This removes the old prefix/tail copy-on-write design entirely, along with the aliasing and uniqueness
problems it introduced. In particular, storage reads no longer need to reconstruct a request-shaped message from
a split buffer, which was fragile once the tail could already be shared by cache-backed Frozen values. The new
design makes the ownership model explicit, keeps the request path simple, and avoids relying on prefix
detachment semantics to preserve correctness.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 24, 2026

Codecov Report

❌ Patch coverage is 12.77412% with 1591 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.80%. Comparing base (411a697) to head (cc2ddd6).

Files with missing lines Patch % Lines
core/common/src/types/send_messages2.rs 0.00% 408 Missing ⚠️
core/partitions/src/iggy_partitions.rs 0.00% 296 Missing ⚠️
core/iobuf/src/lib.rs 6.21% 181 Missing ⚠️
core/partitions/src/journal.rs 0.00% 169 Missing ⚠️
core/binary_protocol/src/consensus/message.rs 41.88% 110 Missing and 1 partial ⚠️
core/partitions/src/messages_writer.rs 0.00% 78 Missing ⚠️
core/partitions/src/iggy_index_writer.rs 0.00% 56 Missing ⚠️
core/partitions/src/iggy_partition.rs 0.00% 53 Missing ⚠️
core/partitions/src/log.rs 0.00% 30 Missing ⚠️
core/partitions/src/segment.rs 0.00% 29 Missing ⚠️
... and 18 more

❌ Your patch check has failed because the patch coverage (12.77%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3020      +/-   ##
============================================
- Coverage     71.74%   70.80%   -0.95%     
  Complexity      943      943              
============================================
  Files          1121     1126       +5     
  Lines         93800    94710     +910     
  Branches      71124    72047     +923     
============================================
- Hits          67301    67063     -238     
- Misses        23863    25002    +1139     
- Partials       2636     2645       +9     
Components Coverage Δ
Rust Core 71.08% <12.77%> (-1.28%) ⬇️
Java SDK 62.30% <ø> (ø)
C# SDK 67.47% <ø> (-0.17%) ⬇️
Python SDK 81.43% <ø> (ø)
Node SDK 91.53% <ø> (+0.21%) ⬆️
Go SDK 38.68% <ø> (ø)
Files with missing lines Coverage Δ
core/common/src/error/iggy_error.rs 100.00% <ø> (ø)
core/consensus/src/namespaced_pipeline.rs 97.32% <100.00%> (+0.12%) ⬆️
core/metadata/src/impls/recovery.rs 70.63% <100.00%> (+0.23%) ⬆️
core/shard/src/lib.rs 0.00% <ø> (ø)
core/partitions/src/lib.rs 0.00% <0.00%> (ø)
...ommon/src/types/segment_storage/messages_writer.rs 69.86% <0.00%> (-3.00%) ⬇️
core/metadata/src/stm/mod.rs 45.56% <0.00%> (-1.19%) ⬇️
core/simulator/src/client.rs 0.00% <0.00%> (ø)
core/consensus/src/impls.rs 42.17% <60.00%> (-0.31%) ⬇️
core/consensus/src/plane_mux.rs 0.00% <0.00%> (ø)
... and 22 more

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@numinnex numinnex force-pushed the partition_remaster branch from 3878d16 to b1ce9c3 Compare March 24, 2026 13:31
@tungtose
Copy link
Copy Markdown
Contributor

The simulator is currently hanging on the send messages to partition step

Copy link
Copy Markdown
Contributor

@hubcio hubcio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also outside of diff:
offset is Arc (iggy_partition.rs:43) but all stores use Ordering::Relaxed (iggy_partition.rs:108, iggy_partitions.rs:774). the Arc wrapper enables cross-thread sharing - if any other shard/thread reads this (metrics, consumer group coordination), Relaxed provides no ordering guarantees on ARM. should be Release on stores, Acquire on loads at cross-shard boundaries. on x86 this is harmless due to TSO, but iggy targets cross-platform.

@numinnex numinnex changed the title feat(partitions): rework partitions using prefix/tail message feat(partitions): rework partitions using new message Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants