Mocap Slice E (#874): body capture — PoseCapPredictor + PoseIKSolver + canonical retarget - #883
Conversation
…l retarget (--body) Slice E of epic #869: video -> skeletal clip on the user's humanoid rig. - PoseCapPredictor (ONNX consumer #10): BlazePose detector (224, 2254 anchors) + landmark model (256 crop) -> 33 world landmarks (metres, hip-centred) + visibility + presence, with detector-skip tracking driven by the model's own auxiliary alignment landmarks (raw 33/34). Same model management as the face bundle (ai_models/mocap/pose/, shared env/QSettings overrides). - PoseIKSolver (pure data, headless-tested): world landmarks -> WORLD orientation quats for the 22 canonical CMU roles. Torso roles get a full hip/shoulder+spine basis (torso roll captured; abdomen blends the two), the head an ear-line+nose basis, and limb segments a primary-axis frame whose twist reference is TRUE parallel transport (previous secondary axis rotated by the shortest arc between successive segment directions — continuous and twist-free by construction, no candy-wrapping; torso-axis seeding on the first frame). Low-visibility landmarks invalidate only the roles they feed. - Retarget: MocapRecorder::recordBody feeds the [frame][22] world-quat stream straight into AnimationMerger::applyMotionClip(worldFrame=true) — the #411 delta-vs-frame-0 conjugation, standing-pose bind harvest, locked root, humanoid >=1/2-roles gate. No new retargeter. Frame 0 is the calibration frame. - RecordBodyClipCommand: one undo step per body take (skeletal-clip snapshot/restore). - CLI: 'qtmesh mocap --body' (combinable with --face in one decode pass), --algo sam3dbody|pose-ik + --no-model. sam3dbody is dispatched as the default quality path but its checkpoints are HF-gated (Slice A verdict) — until the export is hosted every request falls back to pose-ik with algorithmUsed/fallbackReason in the report (the SkinTokens->GVB pattern). --root-motion is rejected with a clear message (hip-centred world landmarks carry no root translation; lands with the SAM backend). - MCP capture_body_from_video (heavy): live-scene entity, single undoable clip, optional export, report JSON. - Sentry ai.assist.mocap_body (records algo); gamification noteOperation("mocap_body", {frames, tracks}) on CLI + MCP. Verified end-to-end on macOS with the Slice A models: 4-frame pose sequence onto a Quaternius rig — 13/22 roles resolved, 18 bone tracks, BodyCap clip exports to glb and reimports alongside the rig's original animations. 6 new PoseIKSolver tests (static-frame stability, 90-degree elbow bend recovery, torso twist isolation, sweep continuity, degenerate input, visibility gating); 44 mocap-suite tests green locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17f250a5b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| QJsonObject root; | ||
| if (face) | ||
| root.insert(QLatin1String("face"), reportToJson(report)); | ||
| if (body) | ||
| root.insert(QLatin1String("body"), bodyReportToJson(bodyReport)); | ||
| CLIPipeline::writeOutput(QString::fromUtf8( | ||
| QJsonDocument(reportToJson(report)).toJson(QJsonDocument::Indented))); | ||
| QJsonDocument(root).toJson(QJsonDocument::Indented))); |
There was a problem hiding this comment.
When an existing script runs qtmesh mocap ... --face --json without --body, this now emits the face report under a new face property instead of the previous top-level object produced by QJsonDocument(reportToJson(report)). Any caller reading fields such as clipName, framesProcessed, or error at the top level will break even though they did not opt into body capture; keep the old shape for face-only output and only nest when body is included.
Useful? React with 👍 / 👎.
| if (skel->hasAnimation(clip)) { | ||
| if (!options.replaceExisting) { | ||
| report.error = QStringLiteral( | ||
| "animation '%1' already exists (pass replace)").arg(options.clipName); | ||
| return report; | ||
| } | ||
| skel->removeAnimation(clip); |
There was a problem hiding this comment.
If the requested body clip name already exists and applyMotionClip then rejects the take, for example because the skeleton resolves fewer than half of the canonical humanoid roles, this removes the original animation before returning an error. The CLI/MCP path reports failure but the live scene has already lost the user's existing clip; applyMotionClip already performs its own replacement after its humanoid gate, so this pre-removal should be avoided or rolled back on failure.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in #909: dropped the premature removeAnimation() — applyMotionClip replaces the clip on success and leaves it untouched on a retarget failure, so a failed take no longer destroys the user's existing clip.
| if (!ent || ent->getMovableType() != "Entity") continue; | ||
| if (entityName.isEmpty() | ||
| || QString::fromStdString(ent->getName()) == entityName) { entity = ent; break; } |
There was a problem hiding this comment.
When entity_name is omitted, the tool schema says the target defaults to the first/selected entity, but this loop ignores SelectionSet and always picks the first entity returned by the manager. In a scene where the selected avatar is not first, or a static prop appears before the skinned character, capture_body_from_video will either fail the skeleton check or record onto the wrong mesh; resolve the current selection first as the other MCP tools do.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in #909: both capture_face/body_from_video now resolve an omitted entity_name via SelectionSet first (the schema's documented default), falling back to the first scene entity.
|
Consolidated into #909 (single epic PR targeting master). Review findings from this PR are addressed there — see the latest commit. |
1 participant
Part of epic #869. Closes #874. Stacked on #882 (Slice D) — merge order: #880 → #881 → #882 → this.
What this adds
qtmesh mocap <video> --body --mesh rigged.fbx -o out.glb: track a person in a video and retarget the full-body pose onto the mesh's humanoid rig as an ordinary skeletal clip.PoseCapPredictor(ONNX consumer Update and rename cmake.yml to deploy.yml #10) — BlazePose detector + landmark graphs from the Slice A conversion; 33 world landmarks (metres, hip-centred) + visibility + presence; detector-skip tracking from the model's own auxiliary alignment landmarks.PoseIKSolver(pure data, 6 headless tests) — world landmarks → WORLD orientations for the 22 canonical CMU roles: full torso basis (roll captured, abdomen blends hips↔chest), ear/nose head basis, and limb frames whose twist reference is true parallel transport (shortest-arc rotation of the previous twist axis — continuous and twist-free by construction; the naive transported-axis approach failed the 90°-swing test and was replaced).recordBodyfeeds the quat stream directly intoAnimationMerger::applyMotionClip(worldFrame=true): the existing AI: Text-to-motion via MDM (ONNX) — research/ambitious #411 machinery (delta vs frame 0 = calibration frame, locked root, ≥½-roles humanoid gate). No new retargeter.--algo sam3dbody|pose-ikwith sam3dbody as the declared quality path: its checkpoints are HF-gated (Slice A licensing verdict: PASS with conditions, access pending), so every request currently falls back to pose-ik withalgorithmUsed/fallbackReasonin the report — the SkinTokens→GeodesicVoxel house pattern; the SAM backend lights up in a follow-up once the export is hosted, with no interface change.RecordBodyClipCommand— one undo step per take; MCPcapture_body_from_video(heavy) with the same report.--root-motionis rejected with an explanatory message (pose-ik's world landmarks are hip-centred → no root translation; lands with the SAM backend).Verification
Male_Shirtrig — 13/22 canonical roles resolved, 18 bone tracks written,BodyCapexports to glb and reimports alongside the rig's original 11 animations.PoseIKSolvertests: static-frame stability, 90° elbow-bend recovery (upper arm untouched), torso-twist-on-chest-not-hips, 20-step sweep continuity, degenerate-input no-NaN, visibility gating. 44 mocap-suite tests green.🤖 Generated with Claude Code