Mesh: preserve empty tensors in memmap serialization - #1882
Draft
peterdsharpe wants to merge 1 commit into
Draft
Conversation
Two independent Mesh.save/load (memmap) defects. TensorDict records a zero-element tensor's shape and dtype in meta.json but writes no backing file, and its loader skips keys whose file is absent. Cells of shape (0, 3) loaded back as (0, 1) with a different dtype, zero-width point/cell/global data fields disappeared, and a mesh with no points, or with an Adjacency cached over a point cloud, failed to load at all. Mesh, DomainMesh, Adjacency, BVH, ClusterTree, DualInteractionPlan and SourceAggregates now rebuild those tensors from metadata on load. Separately, and independent of empty tensors, loading a mesh with a populated topology cache in a process that had not otherwise imported the adjacency API raised "Could not find name ...Adjacency" -- the offline-preprocess / dataloader-worker pattern. Importing Mesh now registers Adjacency with TensorDict. Both fixes are read-side only; the on-disk format is unchanged.
Contributor
CODEOWNERS review mapCurrent for commit ⏳ @peterdsharpe — 9 file(s)
No CODEOWNER
Comment |
Collaborator
Author
|
Update: probably going to fix this here; keeping this draft until we see if this merges: pytorch/tensordict#1750 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PhysicsNeMo Pull Request
Description
TensorDict records zero-element tensors in memmap metadata but writes no backing file, so loading silently drops them. This can change empty connectivity from
(0, 3)to the(0, 1)point-cloud sentinel, remove empty data fields, or crash while restoring empty adjacency and spatial structures. Loading a cachedAdjacencyin a fresh process can also fail because its tensorclass has not been registered.This PR:
__post_init__can replace or dereference them.Mesh,DomainMesh,Adjacency,BVH,ClusterTree, and interaction-plan tensorclasses.AdjacencywhenMeshis imported so fresh-process cache loading succeeds.Empty jagged tensors remain a documented, strict-xfail limitation because their shapes cannot be reconstructed from
meta.jsonalone.Verification
pytest test/mesh: 2357 passed, 704 skipped, 1 xfailed.Mesh,DomainMesh, andBVHcases.ruff check,ruff format --check, and docstring coverage checks pass.Checklist
Dependencies
None.
Review Process
All PRs are reviewed by the PhysicsNeMo team before merging.
Depending on which files are changed, GitHub may automatically assign a maintainer for review.
We are also testing AI-based code review tools (e.g., Greptile), which may add automated comments with a confidence score. This score reflects the AI's assessment of merge readiness and is not a qualitative judgment of the work or an indication that the PR will be accepted or rejected.
AI-generated feedback should be reviewed critically for usefulness. You are not required to respond to every AI comment, but they are intended to help both authors and reviewers. Please react to Greptile comments with 👍 or 👎 to provide feedback on their accuracy.