Teach transmute_{ref,mut}! to handle slice DSTs by joshlf · Pull Request #2428 · google/zerocopy

joshlf · 2025-03-07T22:27:19Z

We generalize our existing support in util::macro_util, now relying on
PMEs to detect when transmutes cannot be supported. In order to continue
to support sized reference transmutation in a const context, we use
the autoref specialization trick to fall back to our old (const-safe)
implementation when both the source and destination type are Sized.

The main challenge in generalizing this support is making a trait
implementation available conditional on a PME. We do this using the
unsafe_with_size_eq! helper macro, which introduces
#[repr(transparent)] wrapper types, Src and Dst. Internal to this
macro, we implement SizeEq<Src<T>> for Dst<U>, with the implementation
of SizeEq::cast_from_raw containing the PME logic for size equality.
The macro's caller must promise to only use the Src and Dst types
to wrap the T and U types for which this PME logic has been applied
(or else the SizeEq impl could "leak" to types for which it is
unsound).

In addition, we generalize our prior support for transmuting between
unsized types. In particular, we previously used the SizeEq trait to
denote that two types have equal sizes in the face of a cast operation
(in particular, that *const T as *const U preserves referent size). In
this commit, we add support for metadata fix-up, which means that we
support casts for which *const T as *const U does not preserve
referent size. Instead, we compute an affine function at compile time
and apply it at runtime - computing the destination type's metadata as a
function of the source metadata, dst_meta = A + src_meta * B. A and
B are computed at compile time.

We generalize SizeEq to permit its cast_from_raw method to perform a
runtime metadata fix-up operation.

Makes progress on #1817

Co-authored-by: Jack Wrenn jswrenn@amazon.com

This PR is on branch transmute-ref-dst.

jswrenn · 2025-03-25T19:45:45Z

+/// Implementations of `cast_from_raw` must satisfy that method's safety
+/// post-condition.
+pub unsafe trait SizeCompat<Src: ?Sized> {


We don't need to do this now, but it'd be more idiomatic to name this CastFrom, after its method (and source of safety conditions). That name also conveys directionality, unlike SizeCompat.

codecov-commenter · 2025-03-25T20:16:03Z

Codecov Report

❌ Patch coverage is 73.20000% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.85%. Comparing base (f9c9703) to head (f5fcda2).
⚠️ Report is 349 commits behind head on main.

Files with missing lines	Patch %	Lines
src/layout.rs	33.33%	28 Missing ⚠️
src/macros.rs	80.64%	18 Missing ⚠️
src/impls.rs	0.00%	8 Missing ⚠️
src/doctests.rs	0.00%	4 Missing ⚠️
src/lib.rs	55.55%	4 Missing ⚠️
src/pointer/transmute.rs	40.00%	3 Missing ⚠️
src/util/macros.rs	88.23%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2428      +/-   ##
==========================================
- Coverage   89.25%   88.85%   -0.40%     
==========================================
  Files          19       20       +1     
  Lines        5126     5312     +186     
==========================================
+ Hits         4575     4720     +145     
- Misses        551      592      +41

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joshlf · 2025-03-25T20:35:58Z

    }
 }

+// TODO: Update all `TransmuteFrom` safety proofs.


jswrenn · 2025-06-03T16:47:22Z

+        // so permits interior mutation exactly when `T` does.
+        unsafe_impl!(T: ?Sized + Immutable => Immutable for $dst<T>);
+
+        $blk


Wonder if the outer unsafe required to currently call this macro leaks into cover $blk.

Tried offline; turned out to be very difficult, so we're going to punt for now.

jswrenn · 2025-06-03T16:49:46Z

+        if 1 == 0 {
+            let ptr = <$t as KnownLayout>::raw_dangling();
+            #[allow(unused_unsafe)]
+            // SAFETY: This call is never executed.
+            let ptr = unsafe { crate::pointer::PtrInner::new(ptr) };
+            #[allow(unused_unsafe)]
+            // SAFETY: This call is never executed.
+            let ptr = unsafe { cast!(ptr) };
+            let _ = <$dst<$u> as SizeEq<$src<$t>>>::cast_from_raw(ptr);
+        }


We should drop a comment as to why this block exists.

Is this okay? Can we rely on this PME occuring despite this code being dead?

Comment: Done.

Can we rely on the PME: Maybe not, but the follow-up PR will ensure that it's sound since SizeEq will no longer convey a safety post-condition that can be consumed by unsafe code - the only way to use it will be via SizeEq::cast_from_raw, which will guarantee that the PME is triggered.

We generalize our existing support in `util::macro_util`, now relying on PMEs to detect when transmutes cannot be supported. In order to continue to support sized reference transmutation in a `const` context, we use the autoref specialization trick to fall back to our old (`const`-safe) implementation when both the source and destination type are `Sized`. The main challenge in generalizing this support is making a trait implementation available conditional on a PME. We do this using the `unsafe_with_size_eq!` helper macro, which introduces `#[repr(transparent)]` wrapper types, `Src` and `Dst`. Internal to this macro, we implement `SizeEq<Src<T>> for Dst<U>`, with the implementation of `SizeEq::cast_from_raw` containing the PME logic for size equality. The macro's caller must promise to only use the `Src` and `Dst` types to wrap the `T` and `U` types for which this PME logic has been applied (or else the `SizeEq` impl could "leak" to types for which it is unsound). In addition, we generalize our prior support for transmuting between unsized types. In particular, we previously used the `SizeEq` trait to denote that two types have equal sizes in the face of a cast operation (in particular, that `*const T as *const U` preserves referent size). In this commit, we add support for metadata fix-up, which means that we support casts for which `*const T as *const U` does *not* preserve referent size. Instead, we compute an affine function at compile time and apply it at runtime - computing the destination type's metadata as a function of the source metadata, `dst_meta = A + src_meta * B`. `A` and `B` are computed at compile time. We generalize `SizeEq` to permit its `cast_from_raw` method to perform a runtime metadata fix-up operation. Makes progress on #1817 Co-authored-by: Jack Wrenn <jswrenn@amazon.com> gherrit-pr-id: Ib4bc62202e0b3b09d155333b525087f7aa8f02c2

This was referenced Mar 7, 2025

Remove obsolete safety proofs #2423

Merged

Implement traits for Cell #2421

Merged

[pointer] SizeEq supports raw pointer transmutes #2427

Merged

Teach transmute_{ref,mut}! to handle slice DSTs #1924

Closed

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 6 times, most recently from 829b199 to a7c40db Compare March 8, 2025 16:58

joshlf force-pushed the I70d5aa5ace6bd2e39e679eac7f00a66d4b843d57 branch from 4d09d7c to f2ec335 Compare March 8, 2025 16:58

Base automatically changed from I70d5aa5ace6bd2e39e679eac7f00a66d4b843d57 to main March 8, 2025 19:16

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 3 times, most recently from 98bbab7 to 5196fac Compare March 9, 2025 01:10

joshlf commented Mar 9, 2025

View reviewed changes

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 2 times, most recently from 398966d to 976b234 Compare March 9, 2025 03:03

joshlf changed the base branch from main to I691b42ce8c0c3c6e5990e7684fc66f8f5dd73d85 March 9, 2025 03:03

joshlf mentioned this pull request Mar 9, 2025

[layout] Move Kani proofs from crate root #2430

Merged

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 4 times, most recently from bd3b50a to e972a2c Compare March 9, 2025 06:47

joshlf commented Mar 9, 2025

View reviewed changes

Comment thread src/pointer/transmute.rs Outdated

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 2 times, most recently from e2b7f7d to 459de15 Compare March 11, 2025 18:51

Base automatically changed from I691b42ce8c0c3c6e5990e7684fc66f8f5dd73d85 to main March 11, 2025 19:21

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch from 459de15 to 424ddb4 Compare March 11, 2025 20:19

joshlf changed the base branch from main to Icdb256acfdb274f34312cf5b216a02ca426338a1 March 11, 2025 20:19

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 7 times, most recently from ac7fd3f to 85cbc73 Compare March 25, 2025 19:17

jswrenn reviewed Mar 25, 2025

View reviewed changes

Comment thread src/layout.rs Outdated

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 2 times, most recently from 5ebdc1a to 7d27689 Compare March 25, 2025 19:36

jswrenn requested changes Mar 25, 2025

View reviewed changes

Comment thread src/util/macro_util.rs

joshlf mentioned this pull request Mar 25, 2025

Support slice DST transmutes (including those requiring metadata fix-up) in transmute_{ref,mut}! #1817

Closed

11 tasks

jswrenn reviewed Mar 25, 2025

View reviewed changes

Comment thread src/util/macros.rs

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 2 times, most recently from c5ae106 to 2d585bc Compare March 25, 2025 19:48

jswrenn previously approved these changes Mar 25, 2025

View reviewed changes

joshlf force-pushed the Ib4bc62202e0b3b09d155333b525087f7aa8f02c2 branch 2 times, most recently from 8a04b30 to 3931fcf Compare March 25, 2025 19:58

joshlf commented Mar 25, 2025

View reviewed changes

Comment thread src/pointer/transmute.rs Outdated

}

}

// TODO: Update all `TransmuteFrom` safety proofs.

Copy link
Copy Markdown

Member Author

joshlf Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO

This was referenced Mar 28, 2025

[WIP] Update TransmuteFrom safety proofs #2469

Closed

Generalize SizeEq to SizeFrom, support shrinking #2472

Closed

joshlf mentioned this pull request Apr 7, 2025

[macros] Support shrinking reference transmutes #2487

Open

jswrenn reviewed Jun 3, 2025

View reviewed changes

Comment thread src/util/macros.rs

jswrenn reviewed Jun 3, 2025

View reviewed changes

jswrenn approved these changes Jun 3, 2025

View reviewed changes

joshlf mentioned this pull request Jun 4, 2025

Update SizeEq safety invariant #2564

Merged

Conversation

joshlf commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jswrenn Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joshlf Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jswrenn Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

joshlf Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

jswrenn Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

jswrenn Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

joshlf Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joshlf commented Mar 7, 2025 •

edited

Loading

codecov-commenter commented Mar 25, 2025 •

edited

Loading