Mocap Slice F (#875): live Performance Capture panel — MocapController + QML + MCP live tools - #884
Conversation
…MCP live tools Slice F of epic #869 — the headline feature: webcam preview drives the selected entity live; Record writes an ordinary undoable clip. - MocapController (QML_SINGLETON, registered under PropertiesPanel 1.0): Idle -> CameraStarting -> Previewing -> Recording state machine. Camera frames land in the latest-wins FrameMailbox; a dedicated worker thread drains it through FaceCapPredictor (+ One-Euro) and queues FaceSamples to the main thread, where ALL Ogre mutation happens (no BlockingQueuedConnection anywhere). - Live drive: mapped morph weights via MorphAnimationManager::setWeight and the Head bone via setManuallyControlled + orientation + _notifyManualBonesDirty, with the UvUnwrap-style snapshot/restore discipline — entering preview snapshots the mapped weights, the head bone's {manuallyControlled, orientation} and every enabled AnimationState (disabled during preview); leaving preview restores all of it exactly (covered by test). - Recording buffers samples in memory; stopRecording() commits the take as ONE RecordMocapClipCommand (Ctrl+Z discards); stopping preview mid-record commits rather than drops. Status line reports 'Recorded Ns -> clip (M keyframes) — Ctrl+Z to discard'; the clip appears in the Animations section/dope sheet for free (it's a normal clip). - calibrateNeutral() re-bases the head-pose zero on the next confident sample; auto-applied on the first confident frame of a preview. - qml/PropertiesPanel.qml: 'Performance Capture' CollapsibleSection in Animation-mode Mode Tools — device picker, preview thumbnail (data-URL image at reduced rate, the house preview pattern) with face-detected dot + live fps HUD, matched-channel summary, clip-name field, Neutral + Record/Stop buttons, status line; section hide stops the camera (the AutoRig onSectionVisibleChanged precedent); non-mocap builds show the rebuild hint. - MCP: list_capture_devices, start_live_capture, stop_live_capture — thin controller wrappers; start refuses headless --mcp (needs the GUI). - Sentry ai.assist.mocap_live breadcrumbs; gamification noteFeature('mocap', Gui) on preview start + noteOperation('mocap_face') on committed takes. Controller tests (CI lane, Ogre-gated): preview drives weights and restores the pre-preview value bit-exactly on stop, record -> single-undo clip, stop-during-recording commits, no-selection refusal. 48 runnable mocap-suite tests green locally; app + UnitTests build clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: addb0e04e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| d->camera->stop(); | ||
| if (d->workerThread.isRunning()) { | ||
| d->workerThread.quit(); | ||
| d->workerThread.wait(2000); |
There was a problem hiding this comment.
If the ONNX predictor is still inside processPending() when the user stops preview, wait(2000) can time out, but the code continues to drop d->worker and reset d->camera. The worker still holds mailbox = &d->camera->mailbox(), so a slow or hung inference on stop can leave a running thread with a dangling mailbox pointer and can crash or corrupt memory; handle a failed wait instead of freeing the camera while the thread may still be active.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in the consolidated PR #909: stopPreview() now loops workerThread.wait(2000) until the thread actually exits before releasing the camera/mailbox, so a slow ONNX step can no longer fall through into a use-after-free.
| } | ||
|
|
||
| // models first (blocking download with a visible status) | ||
| if (!FaceCapPredictor::modelsPresent()) { |
There was a problem hiding this comment.
On a fresh mocap build where the face models are not present, clicking Preview for a normal mesh with no ARKit morph targets and no Head bone will start the blocking model download path before the later mapping/head check rejects the selection as "nothing to drive." This wastes network/time and can also mask the real selection error when offline; build the mapping and refuse ineligible selections before calling ensureModelsBlocking().
Useful? React with 👍 / 👎.
|
Consolidated into #909 (single epic PR targeting master). Review findings from this PR are addressed there — see the latest commit. |
1 participant
Part of epic #869. Closes #875. Stacked on #883 (Slice E) — merge order: #880 → #881 → #882 → #883 → this.
What this adds
The headline feature: Animation Mode → Mode Tools → "Performance Capture" — webcam preview drives the selected entity's morph targets + Head bone in real time; Record writes an ordinary, undoable clip that plays on the existing timeline and exports through the existing exporters.
MocapController(QML_SINGLETON, MeshGenController threading pattern): camera frames → latest-wins mailbox → inference worker thread → queued samples → main-thread Ogre mutation. Live morphs viaMorphAnimationManager::setWeight; head via manual-bone control with exact snapshot/restore of weights, bone state and enabled AnimationStates (the UvUnwrap discipline — verified by test that a pre-preview weight comes back bit-exact).RecordMocapClipCommand(single Ctrl+Z discards the take; stop-mid-record commits rather than drops).Neutralcalibration,● Record/■ Stop, status line. Hiding the section (mode change/deselect) stops the camera. Non-mocap builds show the rebuild hint inside the section.list_capture_devices,start_live_capture,stop_live_capture(GUI-attached only; headless--mcpgets a clear refusal).Verification
ENABLE_MOCAP=ON; all 5 new MCP tool registrations verified in the binary.CameraFrameSourceand the same onSample pipeline the tests drive.Body-live intentionally stays offline-only in this slice (panel copy points at
qtmesh mocap --body), per the slice spec's fallback plan.🤖 Generated with Claude Code