feat(#818 B2): category-aware mesh segmentation + #788 retrain verdict - #879
Conversation
…-category models
- MeshSegmenter::Part grows vegetation (trunk/branch/foliage/root/flower),
vehicle (vehicle_body/wheel/window/wing/rotor) and building
(wall/roof/window/door/chimney/foundation) labels; 'window' is one global
label shared by vehicle + building. Body keeps the original 7-channel
meshseg.onnx wire contract.
- Options::category {Auto, Body, Vegetation, Vehicle, Building}; per-category
ONNX files (meshseg_<cat>.onnx) resolved/downloaded via the same
QTMESH_SEGMENT_* overrides; local channel → global Part via
categoryChannelMap().
- Auto dispatch: tiny point-cloud category classifier
(meshseg_category.onnx) via resolveCategoryBlocking(); classifier
unavailable/offline → Body (pre-B2 behaviour). SmolVLM rejected as the
dispatcher (render + 500MB VLM vs ~0.1MB headless classifier); kept as a
follow-up GUI identify/name assist — decision in
docs/MESH_SEGMENTATION_STRATEGY.md.
- Geometric fallback is category-aware; bone-proximity hints apply to Body
only. CLI --category, MCP segment_mesh category arg, GUI Select-by-Part
resolves Auto transparently. 12 new unit tests.
- export-meshseg-onnx.py: --category selects body (unchanged recipe) /
vegetation / vehicle / building generators or the classifier; procedural
tree/vehicle/building surface samplers with exact by-construction labels.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dels
sync-hf-model-repos.sh gains segveg/segveh/segbld/segcls →
QtMeshEditor-mesh-segmentation-{vegetation,vehicle,building,category};
hosting note updated in THIRD_PARTY_AI_MODELS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #787 exterior-cap fix concentrated every attachment-end cap point on the junction-facing hemisphere — limb-labelled points exactly on the torso boundary — teaching the model to over-claim shoulders/hips (three retrain seeds scored 92.6/89.0/87.0 vs the 94.7 rig-truth bar, torso recall 0.80 → 0.50-0.68). A real character has no surface at the limb- torso junction, so capsule_surf gains cap0/cap1 toggles and every body plan suppresses the attachment-end cap (arm shoulder end, leg hip end, neck both ends). Same-label junctions (tail-torso, elbow, wrist) keep caps — they cannot bias a boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughMesh segmentation now supports Body, Vegetation, Vehicle, and Building categories through shared labels, specialized ONNX models, automatic classification, category-aware fallbacks, CLI/MCP/UI controls, updated training exports, and expanded tests and documentation. ChangesCategory-aware segmentation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef8e92f6cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const std::vector<int> chanMap = categoryChannelMap(cat); | ||
| const int C = static_cast<int>(chanMap.size()); | ||
| if (elems < static_cast<size_t>(N)) | ||
| return fallback(QStringLiteral("Segmentation output too small — used the geometric fallback.")); |
There was a problem hiding this comment.
When the cached/downloaded model does not match the selected category (for example a stale prototype or a QTMESH_SEGMENT_MODEL_BASE_URL that serves the classifier/body model for meshseg_vehicle.onnx), this now derives C from the category map but only checks elems >= N. If the tensor has fewer than C channels, channelsLast becomes false and the loop below reads d[c * N + i] past the output buffer instead of falling back, so an invalid or mismatched model can crash segmentation rather than reporting model failure.
Useful? React with 👍 / 👎.
…lect by Part Real-world failure (user report): a car with DETACHED wheels (separate nodes dropped below the hull — common in exports) classified as 'body'; the floating wheels stretch the normalised box so the hull sits in the upper half like a torso. Verified + fixed on two fronts: - Training: new detach_parts augmentation (minority-label clusters offset downward/outward) for the category segmenters and, stronger, for the classifier; classifier clouds are additionally spun by a FULL random yaw (category is a yaw-invariant question — it had inherited the segmenters' near-canonical facing augmentation). Retrained classifier probe: detached wheels 18→20/20, detached+45° 5→19/20, detached+90° 3→20/20; no category regression at random yaw. Published as meshseg_category v1.1. - GUI: Category dropdown (Auto/Body/Vegetation/Vehicle/Building) above Edit Mode's 'Select by Part (AI)' — the manual escape hatch, matching CLI --category / MCP category. An explicit non-body category also skips the rig-prior fast path (rig labels are body parts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Follow-up (real-world bug report during testing): detached-wheel cars classified as A real car mesh with wheels exported as detached nodes below the hull auto-classified as Fixes:
The vehicle segmenter is also retraining with the same augmentation (so detached wheels still label as |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/MESH_SEGMENTATION_STRATEGY.md`:
- Line 86: Update the paragraph beginning with “#788” so the issue reference is
escaped from Markdown heading parsing, preferably by wrapping “#788” in
backticks. Preserve the existing text and references to `#787`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e327b68f-2169-46d4-80fa-238242fa1bb2
📒 Files selected for processing (13)
CLAUDE.mdTHIRD_PARTY_AI_MODELS.mddocs/MESH_SEGMENTATION_STRATEGY.mdqml/PropertiesPanel.qmlscripts/export-meshseg-onnx.pyscripts/sync-hf-model-repos.shsrc/CLIPipeline.cppsrc/EditModeController.cppsrc/EditModeController.hsrc/MCPServer.cppsrc/MeshSegmenter.cppsrc/MeshSegmenter.hsrc/MeshSegmenter_test.cpp
|
|
||
| The current C++ contract is one 7-class body-centric label set | ||
| (`MeshSegmenter::Part`). That already covers **humanoids** and stretches to | ||
| #788 asked for a retrain with the #787 exterior-capsule-cap data fix, |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Escape the leading #788 to avoid the markdownlint MD018 warning.
The paragraph starts with #788, which markdownlint parses as a heading missing a space after #. Wrapping the reference in backticks (or rewording) clears the warning.
📝 Proposed fix
-#788 asked for a retrain with the `#787` exterior-capsule-cap data fix,
+Issue `#788` asked for a retrain with the `#787` exterior-capsule-cap data fix,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| #788 asked for a retrain with the #787 exterior-capsule-cap data fix, | |
| Issue `#788` asked for a retrain with the `#787` exterior-capsule-cap data fix, |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 86-86: No space after hash on atx style heading
(MD018, no-missing-space-atx)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/MESH_SEGMENTATION_STRATEGY.md` at line 86, Update the paragraph
beginning with “#788” so the issue reference is escaped from Markdown heading
parsing, preferably by wrapping “#788” in backticks. Preserve the existing text
and references to `#787`.
Source: Linters/SAST tools
|
1 participant
Summary
Implements #818 Track B2 (segmentation for any mesh category) and resolves #788 (meshseg retrain with the capsule-cap fix) with a data-backed verdict.
#818 B2 — one label vocabulary, per-category specialist models
MeshSegmenter::Partgrows vegetation (trunk/branch/foliage/root/flower), vehicle (vehicle_body/wheel/window/wing/rotor) and building (wall/roof/window/door/chimney/foundation) labels;windowis one global label shared by vehicle+building. The body model's 7-channelmeshseg.onnxwire contract is untouched.Options::category {Auto, Body, Vegetation, Vehicle, Building}; per-category ONNX files resolved/downloaded under the sameQTMESH_SEGMENT_*overrides; local channels map to global parts viacategoryChannelMap().meshseg_category.onnx, 99.1% 4-way val) throughresolveCategoryBlocking(); classifier unavailable/offline → Body (exact pre-B2 behaviour). Decision record (why several specialists + classifier, not one big softmax; why SmolVLM was rejected as the dispatcher and kept as a future GUI identify/name assist):docs/MESH_SEGMENTATION_STRATEGY.md.qtmesh segment --category …, MCPsegment_meshcategoryarg, GUI Select-by-Part resolves Auto transparently in its worker.export-meshseg-onnx.py --category vegetation|vehicle|building|classifierwith new procedural surface samplers (exact by-construction labels, 100% ours — no third-party data). Held-out synthetic val: vegetation 93.8%, vehicle 93.5%, building 86.9%, classifier 99.1%.QtMeshEditor-models/segment/+ dedicated reposQtMeshEditor-mesh-segmentation-{vegetation,vehicle,building,category}with model cards;sync-hf-model-repos.shgained the four mirrors. Verified end-to-end with real first-use downloads: bareqtmesh segment house.obj→ classifier download → building-model download → correct wall/roof/chimney split; Auto classified tree/car/house/human test meshes correctly.#788 — retrain verdict: published v2 weights stay
Six full-recipe retrains with the fixed script (3 seeds plain #787 fix, 3 with an additional junction-cap suppression fix) all missed the issue's ≥94.7% rig-truth bar (best 92.7%; distribution mean ≈89.5%), while ALL of them beat v2 on the 6-file held-out real set (97.2–97.9% vs 97.0%). Diagnosis + findings table in
docs/MESH_SEGMENTATION_STRATEGY.md: the exterior-cap fix concentrates limb attachment-end cap points exactly on the limb–torso boundary; the newcapsule_surf(cap0/cap1)suppression removes those junction caps entirely (a limb has no real surface there) and recovers torso recall, but v2's 94.7% on the tiny (effectively 2-character) OOD eval is partly a fortunate draw. Script↔weights correspondence is deliberately NOT restored; follow-up: widen the OOD eval set before any republish.Testing
MeshSegmenter*unit tests pass on an ONNX-enabled build (12 new category tests).Closes #788
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation