feat(#869): Performance capture — video/webcam → face + body animation (consolidated epic) - #909
Merged
Conversation
…sing, MHR skeleton Slice A of epic #869 (performance capture). Offline dev tooling + decision records only; no app code. - scripts/export-facecap-onnx.py: converts the five MediaPipe TFLite models (face detector/landmarks/blendshapes, pose detector/landmarks) to ONNX and asserts numerical parity against the python mediapipe reference in the same run (landmarks <=0.59px, blendshapes <=0.0148, pose world <=1.02cm on the MediaPipe Apache-2.0 test images). Handles the fp16 block-sparse DENSIFY weights in the pose detector (interpreter densify pass) and the tf2onnx-optimizer-broken blendshapes graph (unoptimized fallback). - scripts/export-bodycap-onnx.py: extracts the 127-joint MHR skeleton (names/hierarchy/pre-rotations/rest world pose) from the Apache-2.0 mhr_model.pt into mhr_skeleton.json for the Slice E retarget; documents the SAM 3D Body export recipe (blocked on gated HF access). - docs/MOCAP_SPIKE.md: full pre/post-processing contracts (the Slice C/E implementation spec), conversion gotchas, parity numbers, latencies (face 11.2ms, pose 17.2ms per frame on M-series CPU), go/no-go. - THIRD_PARTY_AI_MODELS.md: MediaPipe (Apache-2.0) entry; SAM License due-diligence verdict (PASS with conditions) + rejected alternatives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice B of epic #869 (performance capture). - New ENABLE_MOCAP CMake option (default OFF; requires ENABLE_ONNX, errors otherwise) pulling in Qt6::Multimedia for the app + UnitTests. - src/Mocap/VideoFrameSource.{h,cpp}: one frame-source abstraction delivering timestamped RGB888 MocapFrames from (a) a video file (QMediaPlayer + QVideoSink, targetFps decimation, playback-driven — faster-than-realtime decode is a documented follow-up), (b) a live camera (QCamera + QMediaCaptureSession, device enumeration for the GUI picker / MCP, permission-denied mapped to a human-readable error, latest-wins FrameMailbox for a slower inference consumer), (c) an image sequence (the synchronous headless test double / CLI --frames-dir path). - FrameDecimator + FrameMailbox are pure data and headless-tested; 13 new tests pass (emission order/timestamps, 60->30 decimation, latest-wins drop semantics incl. cross-thread, open() failure paths, device enumeration). - test_main.cpp gains QTMESH_TESTS_SKIP_OGRE_PREFLIGHT so pure-data suites can run on machines with no GL/WindowServer at all; CI never sets it. - NSCameraUsageDescription added to Info.plist.in. - CI: Linux unit-test lane installs the qtmultimedia module and configures with -DENABLE_MOCAP=ON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice C of epic #869 (performance capture). Implements the contract proven in docs/MOCAP_SPIKE.md (Slice A) in C++. - FaceCapPredictor: three-session ONNX pipeline (BlazeFace detector -> Face Mesh V2 landmarks -> MLP-Mixer blendshapes) -> FaceSample {52 weights, head pose quat+translation, confidence}. Detector-skip tracking (next-frame ROI from the previous landmarks; detector re-runs only when presence drops). Models download on first use to AppData/ai_models/mocap/face/ (QTMESH_MOCAP_MODEL_BASE_URL / ai/mocapModelBaseUrl / QTMESH_MOCAP_NO_DOWNLOAD; UniRig multi-file pattern). Runtime I/O discovery, anchor-count sanity check, graceful degradation without ONNX/models. - FaceCapGeom (pure, QtGui-only): letterbox + inverse, SSD anchor gen (896 face / 2254 pose), TensorsToDetections decode + weighted NMS, face/pose ROI rects, rotated-crop tensor sampling (plain bilinear, BORDER_ZERO, the cv2-integer-index convention the spike parity pinned), landmark projection. - FaceCapPose (pure): weighted rigid fit via Horn's quaternion method with a self-contained 4x4 Jacobi eigensolver (no linear-algebra dependency); solveHeadPose fits the embedded canonical face model with MediaPipe's Procrustes basis weights. Convention documented in the header. - FaceCapMapper (pure, QtCore-only): canonical-52 -> mesh morph-target names via side-token expansion + normalization + alias table + JSON override sidecar; unmatched channels always reported. - OneEuroFilter (pure): scalar + quaternion (hemisphere-aligned slerp) variants. - FaceCapCanonicalData.h: generated constants (52 names in model order, the 146-landmark blendshape subset, 468 canonical vertices, Procrustes basis). 33 headless tests pass; the env-gated real-inference test (models + QTMESH_MOCAP_MODELS_DIR/QTMESH_MOCAP_TEST_IMAGE) verifies the full pipeline on a real photo: mouthSmileLeft 0.96 on the smiling MediaPipe portrait, identity head pose, confidence 1.0, and the detector-skip assertion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ace_from_video Slice D of epic #869 — the first user-visible milestone: video file -> animated mesh, end-to-end. - MocapRecorder (src/Mocap/): FaceSample stream -> morph weight keyframes on a named clip via the #519 pipeline (new MorphAnimationManager:: writeWeightKeyOn public static — the entity-explicit core of setMorphWeightKeyframe, which now delegates to it), with epsilon run-length suppression, first/last anchoring, jump pre-anchors and >0.5s face-lost gaps held at both edges. Head pose is calibrated on the take's first confident frame and keyed as rotation deltas on the Head bone (canonicalIndexForBone role 5) in '<clip>_Head', or as node-TRS deltas via NodeAnimationManager for static meshes. - RecordMocapClipCommand (src/commands/): one undo step per take — first redo snapshots the pre-existing weight/head clips keyframe-for-keyframe, undo restores them exactly (verified by test). - CLI 'qtmesh mocap' (src/Mocap/MocapCLI.cpp, the SceneLightsCLI pattern): import mesh -> mapping table (matched/unmatched printed, never silently dropped) -> FileFrameSource or --frames-dir image sequence -> FaceCapPredictor -> One-Euro -> recordFace -> optional re-export; --json emits the FaceRecordReport. Non-MOCAP builds print the standard rebuild hint. Registered in the dispatcher, subcommand list and usage text. - MCP capture_face_from_video: heavy tool, live-scene entity (selected or entity_name), single undoable clip, optional output_path export, report JSON; clean error on non-MOCAP builds. - Fixes a latent CLI bug: writeCliError text was silently lost at _exit() (unflushed static QTextStream) — every subcommand's stderr errors printed nothing; now flushed. - Sentry ai.assist.mocap_face breadcrumbs; gamification noteOperation("mocap_face", {frames, keyframes}) on CLI + MCP. Verified end-to-end on macOS with the Slice A models: a 6-frame image sequence (portrait -> rotated portrait) onto (a) a static OBJ — 5 head keys on the node path, glb exported — and (b) a minimal glTF with jawOpen/ mouthSmileLeft targets — 2 channels matched, 9 weight keys, exported glb carries the 'FaceCap' morph-weights animation (path:"weights"). Known upstream gap noted: the Assimp glTF exporter doesn't emit extras.targetNames, so reimported targets alias to Shape_N. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l retarget (--body) Slice E of epic #869: video -> skeletal clip on the user's humanoid rig. - PoseCapPredictor (ONNX consumer #10): BlazePose detector (224, 2254 anchors) + landmark model (256 crop) -> 33 world landmarks (metres, hip-centred) + visibility + presence, with detector-skip tracking driven by the model's own auxiliary alignment landmarks (raw 33/34). Same model management as the face bundle (ai_models/mocap/pose/, shared env/QSettings overrides). - PoseIKSolver (pure data, headless-tested): world landmarks -> WORLD orientation quats for the 22 canonical CMU roles. Torso roles get a full hip/shoulder+spine basis (torso roll captured; abdomen blends the two), the head an ear-line+nose basis, and limb segments a primary-axis frame whose twist reference is TRUE parallel transport (previous secondary axis rotated by the shortest arc between successive segment directions — continuous and twist-free by construction, no candy-wrapping; torso-axis seeding on the first frame). Low-visibility landmarks invalidate only the roles they feed. - Retarget: MocapRecorder::recordBody feeds the [frame][22] world-quat stream straight into AnimationMerger::applyMotionClip(worldFrame=true) — the #411 delta-vs-frame-0 conjugation, standing-pose bind harvest, locked root, humanoid >=1/2-roles gate. No new retargeter. Frame 0 is the calibration frame. - RecordBodyClipCommand: one undo step per body take (skeletal-clip snapshot/restore). - CLI: 'qtmesh mocap --body' (combinable with --face in one decode pass), --algo sam3dbody|pose-ik + --no-model. sam3dbody is dispatched as the default quality path but its checkpoints are HF-gated (Slice A verdict) — until the export is hosted every request falls back to pose-ik with algorithmUsed/fallbackReason in the report (the SkinTokens->GVB pattern). --root-motion is rejected with a clear message (hip-centred world landmarks carry no root translation; lands with the SAM backend). - MCP capture_body_from_video (heavy): live-scene entity, single undoable clip, optional export, report JSON. - Sentry ai.assist.mocap_body (records algo); gamification noteOperation("mocap_body", {frames, tracks}) on CLI + MCP. Verified end-to-end on macOS with the Slice A models: 4-frame pose sequence onto a Quaternius rig — 13/22 roles resolved, 18 bone tracks, BodyCap clip exports to glb and reimports alongside the rig's original animations. 6 new PoseIKSolver tests (static-frame stability, 90-degree elbow bend recovery, torso twist isolation, sweep continuity, degenerate input, visibility gating); 44 mocap-suite tests green locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…MCP live tools Slice F of epic #869 — the headline feature: webcam preview drives the selected entity live; Record writes an ordinary undoable clip. - MocapController (QML_SINGLETON, registered under PropertiesPanel 1.0): Idle -> CameraStarting -> Previewing -> Recording state machine. Camera frames land in the latest-wins FrameMailbox; a dedicated worker thread drains it through FaceCapPredictor (+ One-Euro) and queues FaceSamples to the main thread, where ALL Ogre mutation happens (no BlockingQueuedConnection anywhere). - Live drive: mapped morph weights via MorphAnimationManager::setWeight and the Head bone via setManuallyControlled + orientation + _notifyManualBonesDirty, with the UvUnwrap-style snapshot/restore discipline — entering preview snapshots the mapped weights, the head bone's {manuallyControlled, orientation} and every enabled AnimationState (disabled during preview); leaving preview restores all of it exactly (covered by test). - Recording buffers samples in memory; stopRecording() commits the take as ONE RecordMocapClipCommand (Ctrl+Z discards); stopping preview mid-record commits rather than drops. Status line reports 'Recorded Ns -> clip (M keyframes) — Ctrl+Z to discard'; the clip appears in the Animations section/dope sheet for free (it's a normal clip). - calibrateNeutral() re-bases the head-pose zero on the next confident sample; auto-applied on the first confident frame of a preview. - qml/PropertiesPanel.qml: 'Performance Capture' CollapsibleSection in Animation-mode Mode Tools — device picker, preview thumbnail (data-URL image at reduced rate, the house preview pattern) with face-detected dot + live fps HUD, matched-channel summary, clip-name field, Neutral + Record/Stop buttons, status line; section hide stops the camera (the AutoRig onSectionVisibleChanged precedent); non-mocap builds show the rebuild hint. - MCP: list_capture_devices, start_live_capture, stop_live_capture — thin controller wrappers; start refuses headless --mcp (needs the GUI). - Sentry ai.assist.mocap_live breadcrumbs; gamification noteFeature('mocap', Gui) on preview start + noteOperation('mocap_face') on committed takes. Controller tests (CI lane, Ogre-gated): preview drives weights and restores the pre-preview value bit-exactly on stop, record -> single-undo clip, stop-during-recording commits, no-selection refusal. 48 runnable mocap-suite tests green locally; app + UnitTests build clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Slice G of epic #869 — hardening + release readiness. - scripts/upload-mocap-models.sh (the upload-triposr-models.sh pattern): pushes the five converted graphs to HF QtMeshEditor-models under mocap/{face,pose}/ plus the Apache-2.0 MediaPipe NOTICE next to the weights. Running it is a maintainer action (needs the HF write token). - Model base-URL semantics unified: QTMESH_MOCAP_MODEL_BASE_URL / ai/mocapModelBaseUrl now point at the mocap ROOT and each predictor appends its face/ / pose/ subdir — one override serves both bundles. First-run download verified end-to-end from a clean AppData against a local server standing in for HF (all 5 graphs fetched, capture ran); QTMESH_MOCAP_NO_DOWNLOAD offline guard verified (clean error, no network). - Combined --face --body mode is now fail-soft per stream: a face too small to track in full-body footage reports its error without aborting the body recording (exit 0 when at least one stream recorded). - docs/MOCAP.md: user guide (mesh requirements incl. ARKit-52 sources + the #519 authoring path, mapping-override JSON, humanoid gate, backends, tuning, known limitations, MCP tools). - CLAUDE.md: qtmesh mocap CLI examples + subcommand list + a Performance Capture architecture section; README feature list; action.yml command description (the new-subcommand doc rule). - Release CI: macOS + Linux release lanes get the qtmultimedia aqt module and -DENABLE_MOCAP=ON (Windows/MinGW stays OFF pending Qt Multimedia verification — the ONNX/MinGW precedent). Debian control adds libqt6multimedia6. - Gamification: 'mocap' feature-cluster key added to featureCatalog() (cloud-side DISCOVERY_FEATURES coordination noted inline — the CLAUDE.md contract) + the CLI subcommand -> cluster mapping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…b 1.x
The upload script called 'huggingface-cli upload', which huggingface_hub 1.x
removed (it now errors 'no longer works, use hf'). Switched to the 'hf upload
<repo> <local> <path>' equivalent + fixed the prereq comment.
Models are now HOSTED on fernandotonon/QtMeshEditor-models under mocap/{face,pose}/:
verified the public resolve URLs return 200 with matching byte sizes, and a
clean-AppData first-run download from the live repo (no URL override) produced
a working face clip (9 keyframes, both channels) and body clip (18 tracks).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Created five standalone mirror repos (the per-model-repo convention in
THIRD_PARTY_AI_MODELS.md / sync-hf-model-repos.sh), each with its own model
card (attribution, Apache-2.0, I/O contract) + its .onnx:
QtMeshEditor-blazeface-onnx face detector
QtMeshEditor-facemesh-onnx 478 face landmarks
QtMeshEditor-faceblendshapes-onnx 52 ARKit blendshapes
QtMeshEditor-blazepose-onnx person detector
QtMeshEditor-poselandmarks-onnx 33 world landmarks
- scripts/create-mocap-mirror-repos.sh: one-time creator (hf repo create +
card + weight upload per graph).
- scripts/sync-hf-model-repos.sh: the five mocap-* mirrors added to REPOS +
FILES (aggregate mocap/{face,pose}/* is the source of truth); also fixed
its download/upload calls to 'hf' (huggingface-cli was removed in
huggingface_hub 1.x — same fix as the upload script).
- THIRD_PARTY_AI_MODELS.md: hosting-layout note lists the new mirrors.
All 5 verified live: README + correctly-sized .onnx resolve at 200. The app
still downloads from the aggregate QtMeshEditor-models repo; these mirrors are
for discoverability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ntity) is selected The section gated on PropertiesPanelController.hasEntitySelection, which is false when the Scene tree selects a node — the common case. Switched to hasMeshInSelection, which is true when a selected node has an attached entity (or an entity/sub-entity is directly selected). This is the same getResolvedEntities() resolution the controller already uses in startPreview, so the visibility gate and the capture target now agree: select the node, the section appears, and preview/record drive the entity under it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he panel
Two fixes to the live Performance Capture panel:
1. Camera permission (the 'always starting camera' bug). Qt 6's QCamera
delivers NO frames on macOS until the app has requested + been granted
camera access — there was no QCameraPermission request, so the sink stayed
silent and the state machine sat in CameraStarting forever (and the app
was never even in the TCC camera list). startPreview() now checks/requests
QCameraPermission and only enters beginPreview() on grant; Denied /
Undetermined surface a clear status instead of hanging. NSCameraUsage-
Description was already in Info.plist.
2. Full body live-drive + record (Face/Head/Body toggles in the panel).
- Worker: when Body is on, also runs PoseCapPredictor + PoseIK per frame
and marshals a BodyLiveFrame (22 canonical world quats) to the main
thread alongside the face sample.
- Live drive: maps canonical roles -> rig bones (MotionInbetween::
canonicalIndexForBone, the retarget's own mapping), drives them with the
same world-delta-vs-calibration-frame conjugation recordBody uses, with
full snapshot/restore of every driven bone's orientation + manual-control
flag. Body owns the skeleton when enabled, so it never fights the
separate head-bone path. Gated on a skinned mesh resolving >= half the
22 roles (bodyAvailable); body models download on first preview, failing
soft to face/head-only if unavailable.
- Record: buffers body frames and, on stop, writes a '<clip>_Body' skeletal
clip via RecordBodyClipCommand — a SEPARATE undo step so face and body
each Ctrl+Z cleanly.
- QML: Face/Head/Body checkboxes (each enabled only when the selection
supports it), a body-detected HUD dot, and a 'Driving: …' summary.
Body pipeline verified end-to-end via the CLI (18 tracks / 13 roles on a
Quaternius rig); 49 pure-data mocap tests green; app + UnitTests build clean
with ENABLE_MOCAP=ON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e the panel controls Camera-permission root cause: the plain dev/build .app carried CMake's EMPTY MACOSX_BUNDLE default plist — no NSCameraUsageDescription and an empty CFBundleIdentifier — so macOS refused the camera prompt entirely and the app never appeared under Privacy & Security → Camera (Info.plist.in was only configured to the INSTALL prefix + manually copied in the release job, never into the build-tree bundle). Fixes: - Configure Info.plist.in into the build tree and set it as MACOSX_BUNDLE_INFO_PLIST on the target, so every build's .app has the real plist (camera usage string + file associations). Install-prefix copy kept for the release manual-copy step. - CFBundleIdentifier: bare 'QtMeshEditor' → 'com.qtmesheditor.app' (reverse-DNS, consistent with the existing com.qtmesheditor.* UTIs) so TCC has a stable identity to grant against. Verified the built bundle now reports both. (Adhoc-signing the dev build gives it a matching code identity for the prompt; release builds are signed by the deploy pipeline.) Theming: the Face/Head/Body toggles and the camera device dropdown now use the inspector's InspectorCheckBox + ThemedComboBox components (theme colors, focus ring, disabled opacity) instead of default Qt Quick Controls styling — matches the rest of the panel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… a message) The Denied branch of the permission check dead-ended with only a status message and never actually requested access — and checkPermission() never shows the OS dialog regardless. Now Preview always routes through requestPermission() unless already Granted: on first use macOS shows the prompt (creating the camera TCC row), and only if the request comes back Denied WITHOUT a dialog do we show the open-Settings hint. This is what makes clicking Preview trigger the OS access request directly. (Paired with the prior Info.plist/bundle-id fix that gave the app a stable TCC identity — verified the OS now tracks it as com.qtmesheditor.app.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ompt' cause) Root cause of the camera prompt never appearing (verified with a minimal signed Qt probe that logged 'Could not find permission plugin for QCameraPermission'): QtMultimedia's macOS permission backends ship as STATIC Qt plugins (libqdarwin*permission.a). Linking Qt6::Multimedia does NOT pull them in — so QCameraPermission::requestPermission() found no plugin and returned Denied instantly, never reaching TCC. That's why no OS dialog showed and the app never appeared under Privacy & Security → Camera (and why the Info.plist/bundle-id/signing changes made no difference — they were necessary but not sufficient). Fix: qt_import_plugins(... QDarwinCameraPermissionPlugin QDarwinMicrophonePermissionPlugin) on macOS when ENABLE_MOCAP. Verified the QDarwinCameraPermissionHandler symbol is now in the binary. This applies to the shipped release build too (it had the same missing import). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nt for live capture The live-camera prompt is blocked by code-signing, not code: macOS silently denies camera access to ad-hoc-signed apps (verified with a minimal signed Qt probe — request returns Denied, no dialog, no TCC entry), and the macOS release job signs '--sign -' (ad-hoc), so end users would hit the same wall. - Add cfg/QtMeshEditor.entitlements (com.apple.security.device.camera + microphone + the hardened-runtime allowances Qt/QML need). macOS only shows the camera prompt for a hardened-runtime app signed with these entitlements. - Release codesign now applies the entitlements (still ad-hoc for now; falls back to the plain sign if the entitlements sign fails). The ONLY remaining step to make the prompt work for users is signing with a real Apple Developer ID + notarization (needs an Apple account + CI secret) — noted inline and in docs/MOCAP.md. - docs/MOCAP.md: honest macOS-camera-permission section (notarized build required; dev/ad-hoc builds use the CLI video path) + corrected the stale 'body capture is offline' line (body-live shipped). Verified the video->skeletal path end-to-end: a 40-frame moving-person sequence -> 18 bone tracks / 13 canonical roles -> 1.3s BodyCap clip, exported to glb (54 channels x 40 keyframes) and reimported alongside the rig's own animations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…merged) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> # Conflicts: # .gitignore # CLAUDE.md # scripts/sync-hf-model-repos.sh # src/mainwindow.cpp # src/test_main.cpp
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> # Conflicts: # THIRD_PARTY_AI_MODELS.md
P1s: - MocapController::stopPreview: block until the inference thread actually exits (loop the wait) before releasing the camera/mailbox it drains — a single wait(2000) could time out mid-inference and fall through into a use-after-free (PR #884). - MocapRecorder::recordFace: never delete a pre-existing scene-level node clip for the head take — RecordMocapClipCommand's undo doesn't snapshot node clips, so overwriting one lost it permanently. Reject the name collision (new report.headError, surfaced in CLI text + JSON) instead; a fresh name proceeds. Undo comment in the command corrected (PR #882). P2s: - VideoFrameSource: File/Camera sources now tear down the player/session (which hold the sink pointer) BEFORE the sink, instead of relying on reverse-declaration member destruction (PRs #880). - FaceCapPredictor::load: clear d->available when a reload can't find the models — isAvailable() no longer reports a stale prior success (PR #881). - MocapRecorder::recordBody: don't removeAnimation() up front — applyMotionClip replaces on success and leaves the clip intact on a retarget failure, so the pre-delete destroyed the user's clip when retarget failed (PR #883). - MocapCLI: face-only --json keeps its original top-level schema (only nest under face/body when body is included); the face success line + head warning are gated on report.ok(); after <2 pose frames, body recording is SKIPPED rather than fed the empty clip (PRs #883/#885). - MCP capture_face/body_from_video: honour the SelectionSet when entity_name is omitted (the schema's documented default), not just the first entity (PR #883). - MocapController::beginPreview: check selection drivability BEFORE downloading the face models — a plain mesh no longer triggers a ~30 MB fetch just to fail the drivability check (PR #884). - upload-mocap-models.sh: fail hard on a missing required model instead of skipping (a partial upload = broken release) (PR #885). - export-facecap-onnx.py: assert the 2px landmark parity threshold and fail on face-detection disagreements (n_ok>0 could hide per-image failures); fix Ruff E741 (l → pt) (PR #878). - export-bodycap-onnx.py: create the output dir + validate the MHR asset before torch.jit.load (PR #878). - THIRD_PARTY_AI_MODELS.md: spell out MHR's Apache-2.0 redistribution terms instead of 'No restrictions' (PR #878). 28 pure-data mocap unit tests pass; Ogre-dependent suites (recorder, predictor) run under CI's Xvfb. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 19, 2026
Closed
Closed
Closed
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds optional mocap support across build configuration, model conversion and hosting, face/body inference, animation recording, CLI/MCP automation, live camera capture, QML controls, undo integration, packaging, and documentation. ChangesPerformance capture
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 760e7a390a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| InspectorCheckBox { | ||
| text: "Body" | ||
| checked: MocapController.bodyEnabled | ||
| enabled: MocapController.bodyAvailable |
There was a problem hiding this comment.
bodyAvailable is only computed inside beginPreview() after the user clicks Preview, but this checkbox is only editable while the parent row is idle. On a humanoid rig the user has no chance to enable Body: before Preview the box is disabled because bodyAvailable is still false, after Preview it may become true but the row is locked, and startPreview() has already evaluated d->bodyEnabled == false. This makes GUI body capture unreachable unless some stale controller state already had Body enabled.
Useful? React with 👍 / 👎.
Comment on lines
+714
to
+715
| if (!entity || sample.confidence <= 0.f) | ||
| return; |
There was a problem hiding this comment.
When body capture is enabled, valid pose frames still pass through this face-confidence gate. In a full-body camera shot where the face is too small, occluded, or face driving was turned off, body.valid can be true while sample.confidence is 0, so the body live-drive block and the recording append below are skipped and body-only recording produces nothing. Gate only the face/head paths on face confidence and let the body path depend on body.valid.
Useful? React with 👍 / 👎.
Comment on lines
+524
to
+525
| bodyReport = MocapRecorder::recordBody( | ||
| entity, clipQuats, static_cast<int>(fps), bodyOptions); |
There was a problem hiding this comment.
This records the body clip at the requested decimation rate rather than the timestamps actually emitted by the video source. For example, a 24 fps input with the default --fps 30 is not decimated, but recordBody()/applyMotionClip() key the frames at 30 fps, making the exported body animation about 20% too fast while the face track uses frame.timeSec and stays correctly timed. Use the emitted sample times or the actual emitted cadence instead of static_cast<int>(fps).
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (2)
src/Mocap/MocapRecorder_test.cpp (1)
139-172: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftAdd coverage for the skinned-head and body paths.
MocapRecorder_test.cpponly usescreateFaceTestMesh(), soresolveHeadBone()never exercises the skinned-mesh branch andrecordBody()has no coverage here. Add a skeletal fixture with aHeadbone and a minimalrecordBody()happy path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/Mocap/MocapRecorder_test.cpp` around lines 139 - 172, The MocapRecorderTest fixture only covers the facial mesh path and lacks coverage for skeletal heads and body recording. Add a skeletal test fixture or helper containing a Head bone so resolveHeadBone() exercises the skinned-mesh branch, and add a minimal successful recordBody() test using that fixture and valid body input.Source: Path instructions
scripts/export-facecap-onnx.py (1)
714-719: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueOptional: validate archive members before
extractall(Zip Slip).
z.extractall(dest)trusts every entry path. Since this is an offline developer tool operating on Google-provided.taskbundles the risk is low, but validating that each member resolves insidedestis cheap hardening.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/export-facecap-onnx.py` around lines 714 - 719, Harden extract_task by validating each ZipFile member before extraction, ensuring its resolved destination path remains inside the resolved dest directory and rejecting unsafe entries; only call z.extractall(dest) after all members pass validation, while preserving the existing TFLite mapping behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/deploy.yml:
- Around line 2147-2164: Add QtMultimedia to the manual framework copy block and
the explicit framework signing loop used by the macOS deployment workflow.
Update both existing framework lists so QtMultimedia.framework is copied into
the app bundle and signed consistently with the other Qt frameworks, preserving
the current deployment and signing behavior.
In `@scripts/sync-hf-model-repos.sh`:
- Around line 86-87: Update sync_one so model and repo are declared in separate
local statements, assigning model from $1 before evaluating repo. Preserve the
existing repo construction using OWNER, REPOS, and the assigned model.
In `@src/commands/RecordMocapClipCommand.cpp`:
- Around line 80-84: Both redo methods retain a default or stale successful
report when their target entity is missing. In
src/commands/RecordMocapClipCommand.cpp lines 80-84, update
RecordMocapClipCommand::redo() to reset m_report to
MocapRecorder::FaceRecordReport, set the error to “entity '<name>' not found,”
then return; apply the identical behavior at lines 213-217 in
RecordBodyClipCommand::redo() using MocapRecorder::BodyRecordReport.
In `@src/Info.plist.in`:
- Around line 46-48: Add an NSMicrophoneUsageDescription entry to the Info.plist
alongside NSCameraUsageDescription, with a clear explanation that QtMeshEditor
uses the microphone for live performance capture. Keep the existing camera
permission entry unchanged.
In `@src/Mocap/FaceCapPredictor.cpp`:
- Around line 392-404: The stage 3 blendshape extraction must require at least
478 landmarks because kBlendshapeLandmarkSubset can reference index 477. Update
the validation in the landmark-processing flow and the guard before bsInput
construction to use landmarkCount >= 478, preventing out-of-bounds access in the
loop that indexes pts.
In `@src/Mocap/MocapCLI.cpp`:
- Around line 264-269: Update the SentryReporter::addBreadcrumb call to derive
both the breadcrumb category and CLI message from the validated face/body flags,
so body-only runs are labeled as body capture and face runs retain face
labeling. Keep the existing mesh suffix and video/frames-dir details unchanged.
In `@src/Mocap/MocapRecorder.cpp`:
- Around line 184-213: Update the bone-recording branch in MocapRecorder around
the existing skel->hasAnimation and if (!exists || options.replaceExisting)
logic: when the animation already exists and options.replaceExisting is false,
set report.headError to the same skip-on-existing-animation message used by the
node path. Preserve the current write behavior for new animations and
replacement requests, and keep the report fields consistent with the documented
FaceRecordReport::headError contract.
In `@src/Mocap/PoseCapPredictor.cpp`:
- Around line 195-200: Update PoseCapPredictor::load() to set d->available =
false before returning false from both the missing-models check and the
anchor-mismatch branch. Ensure failed reloads cannot leave isAvailable()
reporting true or allow predict() to use stale session state.
---
Nitpick comments:
In `@scripts/export-facecap-onnx.py`:
- Around line 714-719: Harden extract_task by validating each ZipFile member
before extraction, ensuring its resolved destination path remains inside the
resolved dest directory and rejecting unsafe entries; only call
z.extractall(dest) after all members pass validation, while preserving the
existing TFLite mapping behavior.
In `@src/Mocap/MocapRecorder_test.cpp`:
- Around line 139-172: The MocapRecorderTest fixture only covers the facial mesh
path and lacks coverage for skeletal heads and body recording. Add a skeletal
test fixture or helper containing a Head bone so resolveHeadBone() exercises the
skinned-mesh branch, and add a minimal successful recordBody() test using that
fixture and valid body input.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 6b95be2c-1dfe-4778-a9f7-964a85ef752e
📒 Files selected for processing (61)
.github/workflows/deploy.yml.gitignoreCLAUDE.mdCMakeLists.txtDEBIAN-control.inREADME.mdTHIRD_PARTY_AI_MODELS.mdaction.ymlcfg/QtMeshEditor.entitlementsdocs/MOCAP.mddocs/MOCAP_SPIKE.mdqml/PropertiesPanel.qmlscripts/create-mocap-mirror-repos.shscripts/export-bodycap-onnx.pyscripts/export-facecap-onnx.pyscripts/sync-hf-model-repos.shscripts/upload-mocap-models.shsrc/AppLaunchHandler.cppsrc/CLIPipeline.cppsrc/CMakeLists.txtsrc/GamificationTypes.cppsrc/Info.plist.insrc/MCPServer.cppsrc/MCPServer.hsrc/Mocap/FaceCapCanonicalData.hsrc/Mocap/FaceCapGeom.cppsrc/Mocap/FaceCapGeom.hsrc/Mocap/FaceCapGeom_test.cppsrc/Mocap/FaceCapMapper.cppsrc/Mocap/FaceCapMapper.hsrc/Mocap/FaceCapMapper_test.cppsrc/Mocap/FaceCapPose.cppsrc/Mocap/FaceCapPose.hsrc/Mocap/FaceCapPose_test.cppsrc/Mocap/FaceCapPredictor.cppsrc/Mocap/FaceCapPredictor.hsrc/Mocap/FaceCapPredictor_test.cppsrc/Mocap/MocapCLI.cppsrc/Mocap/MocapCLI.hsrc/Mocap/MocapController.cppsrc/Mocap/MocapController.hsrc/Mocap/MocapController_test.cppsrc/Mocap/MocapRecorder.cppsrc/Mocap/MocapRecorder.hsrc/Mocap/MocapRecorder_test.cppsrc/Mocap/OneEuroFilter.cppsrc/Mocap/OneEuroFilter.hsrc/Mocap/OneEuroFilter_test.cppsrc/Mocap/PoseCapPredictor.cppsrc/Mocap/PoseCapPredictor.hsrc/Mocap/PoseIKSolver.cppsrc/Mocap/PoseIKSolver.hsrc/Mocap/PoseIKSolver_test.cppsrc/Mocap/VideoFrameSource.cppsrc/Mocap/VideoFrameSource.hsrc/Mocap/VideoFrameSource_test.cppsrc/MorphAnimationManager.cppsrc/MorphAnimationManager.hsrc/commands/RecordMocapClipCommand.cppsrc/commands/RecordMocapClipCommand.hsrc/mainwindow.cpp
Comment on lines
+2147
to
+2164
| # Finally sign the entire app bundle WITH the camera/microphone | ||
| # entitlements (epic #869). NOTE: this is still an ad-hoc signature | ||
| # (`--sign -`); macOS only PROMPTS for camera access when the app | ||
| # is signed with a real Apple Developer ID + notarized. Until this | ||
| # pipeline gains a Developer ID cert (secret) + notarization step, | ||
| # the live-camera capture will be blocked on end-user machines the | ||
| # same way it is on an ad-hoc dev build (the CLI `qtmesh mocap` | ||
| # video path and file-based capture are unaffected). The | ||
| # entitlements + Info.plist NSCameraUsageDescription are in place so | ||
| # that flipping to a Developer ID signature is the ONLY remaining | ||
| # step to enable the prompt. See docs/MOCAP.md. | ||
| echo "Signing app bundle (with camera entitlements)..." | ||
| ENT="${{github.workspace}}/cfg/QtMeshEditor.entitlements" | ||
| sudo codesign --force --sign - --entitlements "$ENT" \ | ||
| ${{github.workspace}}/bin/QtMeshEditor.app \ | ||
| || sudo codesign --force --sign - ${{github.workspace}}/bin/QtMeshEditor.app \ | ||
| || echo "Failed to sign app bundle (non-fatal)" | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Sign the new QtMultimedia framework.
The ENABLE_MOCAP option introduces QtMultimedia as a new macOS dependency. However, QtMultimedia is missing from the explicit framework code-signing loop (around line 2091 in the original file) and the manual framework copy block (around line 1943).
Even if macdeployqt automatically copies QtMultimedia.framework into the bundle, the hardcoded for framework in ... loop will skip it. This leaves the framework unsigned (or retains an invalid signature), which breaks the ad-hoc bundle signature and causes library validation crashes at runtime.
Please add QtMultimedia to the for framework in ... signing list and the manual copy steps.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/deploy.yml around lines 2147 - 2164, Add QtMultimedia to
the manual framework copy block and the explicit framework signing loop used by
the macOS deployment workflow. Update both existing framework lists so
QtMultimedia.framework is copied into the app bundle and signed consistently
with the other Qt frameworks, preserving the current deployment and signing
behavior.
Comment on lines
86
to
87
| sync_one() { | ||
| local model=$1 repo="$OWNER/${REPOS[$model]}" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Fix local variable assignment dependency.
In bash, variables declared in the same local statement are evaluated simultaneously. The evaluation of repo="$OWNER/${REPOS[$model]}" will use the value of model from the outer scope (or be empty) rather than the $1 assigned just before it.
🐛 Proposed fix to split local declarations
sync_one() {
- local model=$1 repo="$OWNER/${REPOS[$model]}"
+ local model=$1
+ local repo="$OWNER/${REPOS[$model]}"
echo "=== $model → $repo"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| sync_one() { | |
| local model=$1 repo="$OWNER/${REPOS[$model]}" | |
| sync_one() { | |
| local model=$1 | |
| local repo="$OWNER/${REPOS[$model]}" | |
| echo "=== $model → $repo" |
🧰 Tools
🪛 Shellcheck (0.11.0)
[warning] 87-87: This assignment is used again in this 'local', but won't have taken effect. Use two 'local's.
(SC2318)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/sync-hf-model-repos.sh` around lines 86 - 87, Update sync_one so
model and repo are declared in separate local statements, assigning model from
$1 before evaluating repo. Preserve the existing repo construction using OWNER,
REPOS, and the assigned model.
Source: Linters/SAST tools
Comment on lines
+80
to
+84
| void RecordMocapClipCommand::redo() | ||
| { | ||
| Ogre::Entity* entity = findEntity(m_entityName); | ||
| if (!entity) | ||
| return; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
redo() reports stale/default success instead of an error when the target entity is missing, in both command classes. Each redo() early-returns on !entity without ever touching m_report, so report().ok() — the contract the GUI/CLI/MCP layers rely on — reads true from either the default-constructed report (first call) or a stale previous-take report (subsequent calls), rather than reflecting that nothing was recorded this time. MocapRecorder::recordFace/recordBody both set an explicit "no entity" error for this exact condition; the wrapper should forward that instead of dropping it.
src/commands/RecordMocapClipCommand.cpp#L80-L84: inRecordMocapClipCommand::redo(), whenfindEntityfails, setm_report = MocapRecorder::FaceRecordReport{}; m_report.error = QStringLiteral("entity '%1' not found").arg(QString::fromStdString(m_entityName));before returning.src/commands/RecordMocapClipCommand.cpp#L213-L217: apply the identical fix inRecordBodyClipCommand::redo()usingMocapRecorder::BodyRecordReport.
📍 Affects 1 file
src/commands/RecordMocapClipCommand.cpp#L80-L84(this comment)src/commands/RecordMocapClipCommand.cpp#L213-L217
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/commands/RecordMocapClipCommand.cpp` around lines 80 - 84, Both redo
methods retain a default or stale successful report when their target entity is
missing. In src/commands/RecordMocapClipCommand.cpp lines 80-84, update
RecordMocapClipCommand::redo() to reset m_report to
MocapRecorder::FaceRecordReport, set the error to “entity '<name>' not found,”
then return; apply the identical behavior at lines 213-217 in
RecordBodyClipCommand::redo() using MocapRecorder::BodyRecordReport.
Comment on lines
+46
to
+48
| <key>NSCameraUsageDescription</key> | ||
| <string>QtMeshEditor uses the camera for live performance capture.</string> | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Add NSMicrophoneUsageDescription.
The entitlements file (cfg/QtMeshEditor.entitlements) requests the com.apple.security.device.microphone entitlement, and the build explicitly imports QDarwinMicrophonePermissionPlugin. Without a corresponding NSMicrophoneUsageDescription in the Info.plist, macOS will terminate the application with a SIGKILL (privacy violation) if any code path implicitly or explicitly initializes microphone access (e.g., when setting up a QMediaCaptureSession).
🐛 Proposed fix
<key>NSCameraUsageDescription</key>
<string>QtMeshEditor uses the camera for live performance capture.</string>
+ <key>NSMicrophoneUsageDescription</key>
+ <string>QtMeshEditor uses the microphone for live performance capture.</string>
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| <key>NSCameraUsageDescription</key> | |
| <string>QtMeshEditor uses the camera for live performance capture.</string> | |
| <key>NSCameraUsageDescription</key> | |
| <string>QtMeshEditor uses the camera for live performance capture.</string> | |
| <key>NSMicrophoneUsageDescription</key> | |
| <string>QtMeshEditor uses the microphone for live performance capture.</string> | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/Info.plist.in` around lines 46 - 48, Add an NSMicrophoneUsageDescription
entry to the Info.plist alongside NSCameraUsageDescription, with a clear
explanation that QtMeshEditor uses the microphone for live performance capture.
Keep the existing camera permission entry unchanged.
Comment on lines
+392
to
+404
| const int landmarkCount = static_cast<int>(landmarkFloats / 3); | ||
| std::vector<float> pts(rawLandmarks, rawLandmarks + landmarkFloats); | ||
| for (auto& v : pts) | ||
| v /= ls; // 256-space px -> normalized crop coords (z: /256 then *w) | ||
| FaceCapGeom::projectLandmarks(pts.data(), landmarkCount, 3, rect); | ||
|
|
||
| // --- stage 3: blendshapes (146-subset pixel coords) | ||
| std::vector<float> bsInput(FaceCap::kBlendshapeInputLandmarks * 2); | ||
| for (int i = 0; i < FaceCap::kBlendshapeInputLandmarks; ++i) { | ||
| const int id = FaceCap::kBlendshapeLandmarkSubset[i]; | ||
| bsInput[i * 2 + 0] = pts[id * 3 + 0]; | ||
| bsInput[i * 2 + 1] = pts[id * 3 + 1]; | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm the max index in the C++ blendshape subset requires 478 landmarks.
fd -t f 'FaceCapCanonicalData.h' | xargs -I{} ast-grep run --pattern 'kBlendshapeLandmarkSubset' --lang cpp {} 2>/dev/null
rg -n 'kBlendshapeLandmarkSubset' -A40 $(fd -t f 'FaceCapCanonicalData.h') | rg -n '4[0-9][0-9]|477'Repository: fernandotonon/QtMeshEditor
Length of output: 2329
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== FaceCapPredictor.cpp relevant section =="
sed -n '350,420p' src/Mocap/FaceCapPredictor.cpp | cat -n
echo
echo "== FaceCapCanonicalData.h subset section =="
fd -t f 'FaceCapCanonicalData.h' | while read -r f; do
sed -n '40,55p' "$f" | cat -n
doneRepository: fernandotonon/QtMeshEditor
Length of output: 5172
Require 478 landmarks before stage 3
src/Mocap/FaceCapPredictor.cpp:43-54
count >= 468 * 3 still lets a 468-landmark tensor through, but kBlendshapeLandmarkSubset indexes up to 477. That makes pts[id * 3 + ...] read past the buffer when only 468 landmarks are returned. Guard on landmarkCount >= 478 before building bsInput.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/Mocap/FaceCapPredictor.cpp` around lines 392 - 404, The stage 3
blendshape extraction must require at least 478 landmarks because
kBlendshapeLandmarkSubset can reference index 477. Update the validation in the
landmark-processing flow and the guard before bsInput construction to use
landmarkCount >= 478, preventing out-of-bounds access in the loop that indexes
pts.
Comment on lines
+264
to
+269
| SentryReporter::addBreadcrumb( | ||
| QStringLiteral("ai.assist.mocap_face"), | ||
| QStringLiteral("cli mocap --face mesh=.%1 source=%2") | ||
| .arg(meshFi.suffix(), | ||
| framesDir.isEmpty() ? QStringLiteral("video") | ||
| : QStringLiteral("frames-dir"))); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Breadcrumb misattributes body-only runs as face capture.
This runs unconditionally once face || body is validated, but the category (ai.assist.mocap_face) and message (cli mocap --face ...) are hardcoded. A qtmesh mocap <video> --body ... invocation will emit a face-capture breadcrumb, polluting telemetry for the new feature. Derive the label from the face/body flags.
🔧 Suggested change
- SentryReporter::addBreadcrumb(
- QStringLiteral("ai.assist.mocap_face"),
- QStringLiteral("cli mocap --face mesh=.%1 source=%2")
- .arg(meshFi.suffix(),
- framesDir.isEmpty() ? QStringLiteral("video")
- : QStringLiteral("frames-dir")));
+ QStringList modes;
+ if (face) modes << QStringLiteral("--face");
+ if (body) modes << QStringLiteral("--body");
+ SentryReporter::addBreadcrumb(
+ QStringLiteral("ai.assist.mocap"),
+ QStringLiteral("cli mocap %1 mesh=.%2 source=%3")
+ .arg(modes.join(QLatin1Char(' ')),
+ meshFi.suffix(),
+ framesDir.isEmpty() ? QStringLiteral("video")
+ : QStringLiteral("frames-dir")));📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| SentryReporter::addBreadcrumb( | |
| QStringLiteral("ai.assist.mocap_face"), | |
| QStringLiteral("cli mocap --face mesh=.%1 source=%2") | |
| .arg(meshFi.suffix(), | |
| framesDir.isEmpty() ? QStringLiteral("video") | |
| : QStringLiteral("frames-dir"))); | |
| QStringList modes; | |
| if (face) modes << QStringLiteral("--face"); | |
| if (body) modes << QStringLiteral("--body"); | |
| SentryReporter::addBreadcrumb( | |
| QStringLiteral("ai.assist.mocap"), | |
| QStringLiteral("cli mocap %1 mesh=.%2 source=%3") | |
| .arg(modes.join(QLatin1Char(' ')), | |
| meshFi.suffix(), | |
| framesDir.isEmpty() ? QStringLiteral("video") | |
| : QStringLiteral("frames-dir"))); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/Mocap/MocapCLI.cpp` around lines 264 - 269, Update the
SentryReporter::addBreadcrumb call to derive both the breadcrumb category and
CLI message from the validated face/body flags, so body-only runs are labeled as
body capture and face runs retain face labeling. Keep the existing mesh suffix
and video/frames-dir details unchanged.
Comment on lines
+184
to
+213
| const QString headBone = resolveHeadBone(entity); | ||
| if (!headBone.isEmpty()) { | ||
| // author on the MESH skeleton (shared, exported); the entity's | ||
| // SkeletonInstance only mirrors it for playback | ||
| Ogre::SkeletonPtr skel = entity->getMesh()->getSkeleton(); | ||
| Ogre::Bone* bone = skel->getBone(headBone.toStdString()); | ||
| const std::string clip = (options.clipName | ||
| + QStringLiteral("_Head")).toStdString(); | ||
| const bool exists = skel->hasAnimation(clip); | ||
| if (exists && options.replaceExisting) | ||
| skel->removeAnimation(clip); | ||
| if (!exists || options.replaceExisting) { | ||
| // Express the camera-frame delta on the bone's local axes: | ||
| // rel = boneWorldBind^-1 . delta . boneWorldBind, keyed as the | ||
| // offset Ogre applies onto the binding orientation. | ||
| const Ogre::Quaternion boneWorld = bone->_getDerivedOrientation(); | ||
| Ogre::Animation* anim = skel->createAnimation( | ||
| clip, static_cast<Ogre::Real>(length)); | ||
| Ogre::NodeAnimationTrack* track = | ||
| anim->createNodeTrack(bone->getHandle(), bone); | ||
| for (int i : keys) { | ||
| auto* kf = track->createNodeKeyFrame( | ||
| static_cast<Ogre::Real>(times[i])); | ||
| kf->setRotation(boneWorld.Inverse() * deltaAt(i) * boneWorld); | ||
| } | ||
| entity->refreshAvailableAnimationState(); | ||
| report.headKeyframesWritten = static_cast<int>(keys.size()); | ||
| report.headTarget = QStringLiteral("bone:") + headBone; | ||
| } | ||
| } else if (entity->getParentSceneNode()) { |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Bone-path head recording silently no-ops on name collision, unlike the node path.
When resolveHeadBone succeeds but a "<clip>_Head" skeleton animation already exists and options.replaceExisting is false, the if (!exists || options.replaceExisting) guard at Line 195 is false, so nothing is written — but report.headError is never set. headTarget stays "none" and the caller has no way to distinguish "nothing to record" from "skipped on purpose." The node sub-path (Lines 224-228) explicitly reports this via headError; the bone sub-path should do the same for symmetry with the documented FaceRecordReport::headError contract in MocapRecorder.h.
🐛 Proposed fix to report the skip
const bool exists = skel->hasAnimation(clip);
if (exists && options.replaceExisting)
skel->removeAnimation(clip);
if (!exists || options.replaceExisting) {
// Express the camera-frame delta on the bone's local axes:
// rel = boneWorldBind^-1 . delta . boneWorldBind, keyed as the
// offset Ogre applies onto the binding orientation.
const Ogre::Quaternion boneWorld = bone->_getDerivedOrientation();
Ogre::Animation* anim = skel->createAnimation(
clip, static_cast<Ogre::Real>(length));
Ogre::NodeAnimationTrack* track =
anim->createNodeTrack(bone->getHandle(), bone);
for (int i : keys) {
auto* kf = track->createNodeKeyFrame(
static_cast<Ogre::Real>(times[i]));
kf->setRotation(boneWorld.Inverse() * deltaAt(i) * boneWorld);
}
entity->refreshAvailableAnimationState();
report.headKeyframesWritten = static_cast<int>(keys.size());
report.headTarget = QStringLiteral("bone:") + headBone;
+ } else {
+ report.headError = QStringLiteral(
+ "a head clip named '%1' already exists; head capture will "
+ "not overwrite it — record under a different clip name")
+ .arg(options.clipName + QStringLiteral("_Head"));
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const QString headBone = resolveHeadBone(entity); | |
| if (!headBone.isEmpty()) { | |
| // author on the MESH skeleton (shared, exported); the entity's | |
| // SkeletonInstance only mirrors it for playback | |
| Ogre::SkeletonPtr skel = entity->getMesh()->getSkeleton(); | |
| Ogre::Bone* bone = skel->getBone(headBone.toStdString()); | |
| const std::string clip = (options.clipName | |
| + QStringLiteral("_Head")).toStdString(); | |
| const bool exists = skel->hasAnimation(clip); | |
| if (exists && options.replaceExisting) | |
| skel->removeAnimation(clip); | |
| if (!exists || options.replaceExisting) { | |
| // Express the camera-frame delta on the bone's local axes: | |
| // rel = boneWorldBind^-1 . delta . boneWorldBind, keyed as the | |
| // offset Ogre applies onto the binding orientation. | |
| const Ogre::Quaternion boneWorld = bone->_getDerivedOrientation(); | |
| Ogre::Animation* anim = skel->createAnimation( | |
| clip, static_cast<Ogre::Real>(length)); | |
| Ogre::NodeAnimationTrack* track = | |
| anim->createNodeTrack(bone->getHandle(), bone); | |
| for (int i : keys) { | |
| auto* kf = track->createNodeKeyFrame( | |
| static_cast<Ogre::Real>(times[i])); | |
| kf->setRotation(boneWorld.Inverse() * deltaAt(i) * boneWorld); | |
| } | |
| entity->refreshAvailableAnimationState(); | |
| report.headKeyframesWritten = static_cast<int>(keys.size()); | |
| report.headTarget = QStringLiteral("bone:") + headBone; | |
| } | |
| } else if (entity->getParentSceneNode()) { | |
| const QString headBone = resolveHeadBone(entity); | |
| if (!headBone.isEmpty()) { | |
| // author on the MESH skeleton (shared, exported); the entity's | |
| // SkeletonInstance only mirrors it for playback | |
| Ogre::SkeletonPtr skel = entity->getMesh()->getSkeleton(); | |
| Ogre::Bone* bone = skel->getBone(headBone.toStdString()); | |
| const std::string clip = (options.clipName | |
| QStringLiteral("_Head")).toStdString(); | |
| const bool exists = skel->hasAnimation(clip); | |
| if (exists && options.replaceExisting) | |
| skel->removeAnimation(clip); | |
| if (!exists || options.replaceExisting) { | |
| // Express the camera-frame delta on the bone's local axes: | |
| // rel = boneWorldBind^-1 . delta . boneWorldBind, keyed as the | |
| // offset Ogre applies onto the binding orientation. | |
| const Ogre::Quaternion boneWorld = bone->_getDerivedOrientation(); | |
| Ogre::Animation* anim = skel->createAnimation( | |
| clip, static_cast<Ogre::Real>(length)); | |
| Ogre::NodeAnimationTrack* track = | |
| anim->createNodeTrack(bone->getHandle(), bone); | |
| for (int i : keys) { | |
| auto* kf = track->createNodeKeyFrame( | |
| static_cast<Ogre::Real>(times[i])); | |
| kf->setRotation(boneWorld.Inverse() * deltaAt(i) * boneWorld); | |
| } | |
| entity->refreshAvailableAnimationState(); | |
| report.headKeyframesWritten = static_cast<int>(keys.size()); | |
| report.headTarget = QStringLiteral("bone:") + headBone; | |
| } else { | |
| report.headError = QStringLiteral( | |
| "a head clip named '%1' already exists; head capture will " | |
| "not overwrite it — record under a different clip name") | |
| .arg(options.clipName + QStringLiteral("_Head")); | |
| } | |
| } else if (entity->getParentSceneNode()) { |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/Mocap/MocapRecorder.cpp` around lines 184 - 213, Update the
bone-recording branch in MocapRecorder around the existing skel->hasAnimation
and if (!exists || options.replaceExisting) logic: when the animation already
exists and options.replaceExisting is false, set report.headError to the same
skip-on-existing-animation message used by the node path. Preserve the current
write behavior for new animations and replacement requests, and keep the report
fields consistent with the documented FaceRecordReport::headError contract.
Comment on lines
+195
to
+200
| if (!QFileInfo::exists(det) || !QFileInfo::exists(lmk)) { | ||
| d->error = QStringLiteral( | ||
| "pose capture models not found in %1 — they download on first " | ||
| "use, or set QTMESH_MOCAP_MODEL_BASE_URL").arg(dir.absolutePath()); | ||
| return false; | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
load() failure paths leave d->available stale (regression vs FaceCapPredictor).
Unlike FaceCapPredictor::load() (which explicitly sets d->available = false on both the missing-models path and the anchor-mismatch path), this early return does not reset d->available. If a previous load() succeeded and a later load() is pointed at a missing/invalid dir, this returns false but isAvailable() keeps reporting true, so predict() will run against stale/torn-down session state. The anchor-mismatch branch at Lines 221-227 has the same gap.
🛡️ Suggested fix
if (!QFileInfo::exists(det) || !QFileInfo::exists(lmk)) {
+ d->available = false;
d->error = QStringLiteral(
"pose capture models not found in %1 — they download on first "
"use, or set QTMESH_MOCAP_MODEL_BASE_URL").arg(dir.absolutePath());
return false;
}And in the anchor-mismatch branch (Lines 221-227):
if (shape.size() == 3
&& shape[1] != static_cast<int64_t>(d->anchors.size())) {
d->error = QStringLiteral(
"pose detector anchor count mismatch (model %1, ours %2)")
.arg(shape[1]).arg(d->anchors.size());
+ d->available = false;
return false;
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/Mocap/PoseCapPredictor.cpp` around lines 195 - 200, Update
PoseCapPredictor::load() to set d->available = false before returning false from
both the missing-models check and the anchor-mismatch branch. Ensure failed
reloads cannot leave isAvailable() reporting true or allow predict() to use
stale session state.
…ink failure The per-suite test executables link tests/CMakeLists.txt's own qtmesh_test_common, whose explicit source list didn't include the Mocap sources while MCPServer.cpp / mainwindow.cpp in that same list reference them (undefined references to MocapController, FaceCapPredictor, RecordMocapClipCommand, … on unit-tests-linux). Add the Mocap sources (they self-guard with #ifdef ENABLE_MOCAP → empty TU when off) and link Qt6::Multimedia into the test lib under ENABLE_MOCAP (the frame sources include QCamera/QMediaPlayer headers). Mirrors how the main build wires UnitTests, and how the FaceRig sources were added for #903. The UnitTests target was unaffected (it GLOBs src recursively) — which is why the local build masked it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fixes The pose-IK body retarget drove the whole rig 180 deg yawed (hip quat stuck at ~(0,-1,0,0) every frame), producing the contorted/backwards body. Root cause: PoseIKSolver canonicalized MediaPipe world landmarks as (x,-y,-z), leaving the subject facing -Z while canonical rigs face +Z. Flip to (-x,-y,+z) so the subject faces +Z; the hip now stays upright (w~=1) and the body tracks the video. Also stabilize the hip horizontal axis (shoulder- line fallback + temporal sign guard) so a seated/occluded subject can't flip the torso frame. Live GUI path (MocapController) fixes from this session: - stopPreview() blocks until the worker thread exits before freeing the mailbox it drains (use-after-free P1); - body neutral calibrates only once the torso is tracked (Hip+Chest), not on the first barely-tracked frame, and not gated on limb roles a seated subject never resolves; - live arm/shoulder drive transports the world delta through the PARENT bind frame, not the bone's own; - Face/Head/Body checkboxes reflect the selected mesh via a selectionChanged- driven refreshMappingForSelection(), not just after the first Preview; - clear warning when Face is on but the mesh has no ARKit blendshapes. New: Load Video button + MocapController::startPreviewFromVideo() / openVideoDialog() to drive capture from a video file on macOS where the camera is blocked. VideoFrameSource gains a shared base-class mailbox so file + camera feed the same worker path. Verified headlessly (frames-dir capture then isometric render): body stands upright and tracks; face 25/51 channels move. PoseIK tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
Autonomous session while AFK — mocap faithfulness (video → mesh)Set up a fully headless capture→render→measure loop (no camera/GUI needed) and drove it to a working result. The test video is Findings & fixes
Also landed this session
Residual (documented, not blocking)Mild arms-high bias — analytic pose-IK on a 2D-lifted seated subject has no elbow-extension model. Acceptable; a learned lifter (SAM 3D Body path) is the tracked upgrade. 34 mocap unit tests pass; face + body verified headlessly. |
- MocapRecorderTest.HeadTargetIsNoneForStaticMeshWithoutNode: the node head-clip collision guard I added rejected even the recorder's own <clip>_Head under replaceExisting (default), so a re-run in the same process (NodeAnimationManager singleton persists clips across tests) wrote 0 head keys. Restore replace-on-replaceExisting (delete+recreate our own clip); keep the reject only for !replaceExisting, where undo can't restore a pre-existing user clip. - GamificationTypes.FeatureCatalogMatchesCloudContract: the epic added the 'mocap' discovery cluster, so the catalog is 26, not 25. Update the contract count (coordinated with qtmesh-cloud DISCOVERY_FEATURES). - FaceCapPredictor.EnvGatedRealInference: GTEST_SKIP -> SUCCEED()+return (the CI harness counts any skipped test as a suite failure — same convention as the SkinEvaluate / ArkitTemplate env-gated tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
CI status noteLatest commit Re-ran the job to get a clean pass under the cap. If it times out again, the SonarCloud coverage lane duration (not the mocap changes) is the cause — a CI-config concern separate from this PR. |
The live GUI preview splayed the arms out to the sides (near bind pose, wrong offset) even after the facing fix, because onSample drove bones with a WORLD-DELTA (cur * neutral^-1) applied onto the rig bind — which assumes the pose-IK and rig frames coincide. They don't (arm bones carry a large local bind), so the delta rotated about the wrong axes. Rewrite the live drive to DIRECTION-MATCH, matching the offline applyMotionClip path that renders correctly: aim each rig bone's bind-pose WORLD DIRECTION at where the pose-IK bone's direction moved since calibration (a rotation well-defined within pose-IK's own frame, so no cross-frame assumption), then convert to a parent-relative local and drop twist. Capture each bone's bind world direction (toward its first child) at snapshot time. Note: the Record button already produced the correct clip (it goes through recordBody -> applyMotionClip); only the live PREVIEW had the bad math. Panel hint updated to say Preview is a live guide and Record bakes the accurate clip. Offline body render re-verified (upright, facing forward, arms at chest tracking); 37 mocap tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… Record" This reverts commit bbad480.
…d BodyRetargeter) Root fix for the splayed/wrong live-preview arms: the live onSample path was a SEPARATE reimplementation of the body retarget and kept diverging from the recorded clip (missing the CtInv torso-frame transform, using a world-delta instead of absolute direction aiming). Extract the offline applyMotionClip direction-match into a shared, stateful AnimationMerger::BodyRetargeter (top-level class): construct once from the target skeleton (captures the bind frame, torso frame Ct, per-role bind directions, hierarchy order), then evaluateFrame(canonicalQuats, mask) runs the IDENTICAL per-frame formula — ds = CtInv·(clipQ·+Y), aim the bone's bind direction there, drop twist — returning per-bone locals. onSample now calls it, so live Preview and Record produce the same pose by construction (no per-take neutral; bind-referenced like the offline path). Removed the live path's bespoke world-delta math + calibration gate. Offline body render re-verified unchanged after the refactor; 37 pure-data mocap tests pass (AnimationMerger's Ogre-dependent suite runs under CI Xvfb). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared BodyRetargeter::evaluateFrame was replicating applyMotionClip's DIRECTION-MATCH path (path #1), which rendered inverted/splayed. Record's correct output comes from the LEGACY-TRANSPORT path (path #2): compose each joint's parent-relative articulation delta onto the rig's harvested STANDING pose (calmest frame of the existing authored clip), with the Mixamo roll correction Mc, and lock the root to standing (facing lives in the hip). - BodyRetargeter now harvests the standing pose + precomputes Mc per bone (lazily on the first frame, using it as the clip reference frame), and composes standLocal · (Mc⁻¹ · delta · Mc). Unresolved roles hold standing. - MocapController skips the dedicated head-drive when the body retargeter is active, so the two never fight over the Head bone. - Validation harness (QTMESH_MOCAP_USE_RETARGETER=1) now writes ABSOLUTE local keyframes (Ogre node keys replace, not post-multiply the reset pose) — the bindLocal⁻¹ subtraction was double-correcting and produced a false crouch. Render-verified headless: BodyRetargeter output is now bit-identical to the applyMotionClip Record path (leg keyframes match to 3 decimals) and tracks the test video's arm/torso motion upright. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The header + onSample comments still described the old direction-match algorithm (bind-referenced, twist-dropped). Update them to the shipped legacy-transport math: articulation delta composed onto the harvested standing pose with the Mc roll correction, returning absolute local. Also document that evaluateFrame's first call lazily caches the neutral reference (single-thread use). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/MeshImporterExporter.cpp (1)
3063-3069: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueInclude
<cstdio>explicitly forstd::fprintf
<cstdlib>is already present; add<cstdio>here instead of relying on the indirect<stdio.h>fromsrc/OgreXML/tinyxml.h.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/MeshImporterExporter.cpp` around lines 3063 - 3069, Explicitly include the C++ `<cstdio>` header in the translation unit containing the debug logging around `std::fprintf`. Keep the existing `QTMESH_MOCAP_DEBUG` logging unchanged and remove reliance on the indirect header from `tinyxml.h`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/MeshImporterExporter.cpp`:
- Around line 3063-3069: Explicitly include the C++ `<cstdio>` header in the
translation unit containing the debug logging around `std::fprintf`. Keep the
existing `QTMESH_MOCAP_DEBUG` logging unchanged and remove reliance on the
indirect header from `tinyxml.h`.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ebf5c1c8-9f35-4809-8ef8-2c83a46aee0b
📒 Files selected for processing (16)
qml/PropertiesPanel.qmlsrc/AnimationMerger.cppsrc/AnimationMerger.hsrc/GamificationTypes_test.cppsrc/MCPServer.cppsrc/MCPServer.hsrc/MeshImporterExporter.cppsrc/Mocap/FaceCapPredictor_test.cppsrc/Mocap/MocapController.cppsrc/Mocap/MocapController.hsrc/Mocap/MocapRecorder.cppsrc/Mocap/PoseIKSolver.cppsrc/Mocap/PoseIKSolver.hsrc/Mocap/VideoFrameSource.cppsrc/Mocap/VideoFrameSource.htests/CMakeLists.txt
🚧 Files skipped from review as they are similar to previous changes (10)
- src/Mocap/PoseIKSolver.h
- src/MCPServer.h
- src/Mocap/MocapController.h
- src/Mocap/MocapRecorder.cpp
- src/Mocap/FaceCapPredictor_test.cpp
- src/Mocap/PoseIKSolver.cpp
- src/Mocap/VideoFrameSource.h
- src/Mocap/MocapController.cpp
- src/MCPServer.cpp
- src/Mocap/VideoFrameSource.cpp
recordFace's node-TRS head path (skeleton-less mesh) created the node clip with length == the last key's time, then wrote keys AT that time. Unlike MorphAnimationManager (which auto-extends: `if (t > getLength()) setLength(t)`), NodeAnimationManager::addKeyframe REJECTS `time > getLength()` — and the double→float round-trip on the stored length can dip just below the final key time, so every boundary key is silently dropped (headKeyframesWritten == 0). This is why MocapRecorderTest.HeadTargetIsNoneForStaticMeshWithoutNode failed on the Linux Release CI lane (float rounding) while other paths passed. Pad the node clip length by 1e-3 so every key sits strictly inside the clip, and only report headTarget="node" when keys actually landed (honest report). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The length-pad in the previous commit didn't clear MocapRecorderTest.HeadTargetIsNoneForStaticMeshWithoutNode (still 0 keyframes, headTarget now honestly "none"), so the clip-boundary guard is not the (whole) cause. Add a headError explaining which node-path step failed — createClip returning false (scene animation name clash) vs every addKeyframe being rejected (node not found in the singleton SceneManager / time guard) — and echo it from the test's EXPECT messages. The next CI run will name the exact failing step instead of just "0 keyframes". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI diagnostics pinpointed it: HeadTargetIsNoneForStaticMeshWithoutNode's fixture attached the entity to an ANONYMOUS scene node (createChildSceneNode() with no name), so entity->getParentSceneNode() ->getName() was "". The node-TRS head clip re-finds the node BY NAME through NodeAnimationManager::addKeyframe, which rejects an empty nodeName — hence "clip created but 0/2 keyframes written (node '' rejected)". The editor never hits this: Manager::addSceneNode always names its nodes. Fix both sides: - test: name the fixture node (mirror production) so it exercises the real, working node-TRS path. - production: guard the node-head branch on a non-empty parent-node name and skip cleanly (headTarget stays "none") instead of authoring an orphan clip; keep the headError reporting so any future 0-keyframe case is self-explaining. Root cause was NOT the earlier clip-length boundary (that pad is retained as cheap insurance). Pre-existing failure on this branch, unrelated to the live body-retarget work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SonarCloud flagged a CRITICAL reliability bug: the destructor called stopPreview() (which touches Ogre — bone/morph snapshot-restore — and can throw), and an exception escaping a destructor calls std::terminate. Wrap it in try/catch; we're tearing down regardless. Clears the PR quality gate (new_reliability_rating 4 -> 1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…olute local The live drive (MocapController::onSample) applies the retargeter output via Bone::setOrientation(local), which REPLACES the bone's local transform. But BodyRetargeter harvested the standing pose as f0.getRotation() off a NodeAnimationTrack keyframe — and Ogre keyframes are RELATIVE (applyToNode does node->rotate(kf), accumulating onto the bind pose), not absolute. So standLocal was a relative delta being consumed as an absolute local: on any non-identity- rest bone the error compounded and drove the figure upside-down/folded. The headless baked-clip harness hid this: baked playback re-accumulates onto the reset pose, accidentally re-adding the bind, so the render looked upright while the live setOrientation path did not. Fix: store standLocal as the true absolute local (bindLocal · keyframeRotation) so it composes correctly for setOrientation AND for the Mc world-chain. Correct the validation harness to write bindLocal⁻¹ · local as its keyframe (relative, as applyToNode expects) so the baked render now reproduces EXACTLY what the live drive shows — both paths provably land at the same final local orientation. Verified: baked render upright + tracking; live and baked now share one result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The AI Model Settings "QtMeshEditor Models" download tab (driven by
AIModelCatalog) was missing the two newest features' models, so users could
only fetch them implicitly on first use or via env vars — no way to pre-download
from the UI like every other AI feature.
Add three catalog entries (all hosted on the HF models repo, verified 200):
- "ARKit Face Rig Template" (facerig/arkit_template.bin) — always available.
- "Performance Capture — Face" (mocap/face/{detector,landmarks,blendshapes})
- "Performance Capture — Body" (mocap/pose/{detector,landmarks})
Both mocap entries gated on ENABLE_MOCAP. The mocap predictors read a single
ai/mocapModelBaseUrl and append face/|pose/, so the catalog bakes those suffixes
onto the resolved base. All 12 model-base-URL keys in the codebase are now
represented in the catalog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This was referenced Jul 22, 2026
Mocap Slice D — Face video → morph weight clip: MocapRecorder + CLI 'qtmesh mocap --face' + MCP
#873
Closed
Closed
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidated PR for the performance-capture epic (#869) — supersedes the stacked slice PRs #878 (A), #880 (B), #881 (C), #882 (D), #883 (E), #884 (F), #885 (G), all closed in favour of this one. Their review findings are addressed here (see the latest commit).
What this does
Video/webcam → animation, entirely on-device and offline-capable:
All behind
-DENABLE_MOCAP(requiresENABLE_ONNX); OFF by default, clean "rebuild with -DENABLE_MOCAP" message otherwise.Surfaces
qtmesh mocap <video> --face [--body] [--head] [-o out] [--json]capture_face_from_video,capture_body_from_video,list_capture_devicesMocapController), device picker, live preview.Models / hosting
Five converted Google MediaPipe graphs (Apache-2.0, code AND weights) on the
fernandotonon/QtMeshEditor-modelsHF repo undermocap/{face,pose}/, plus per-graph mirror repos; download on first use. The MHR skeleton (SAM 3D Body quality path) is Apache-2.0; the SAM 3D Body checkpoint is an optional, license-gated, never-bundled backend (decision record inTHIRD_PARTY_AI_MODELS.md). Conversion + numerical-parity proof:scripts/export-facecap-onnx.py(landmarks ≤ 0.59px, blendshapes ≤ 0.0148, pose ≤ 1.02cm vs the Python reference). Spike/contract:docs/MOCAP_SPIKE.md; user docs:docs/MOCAP.md.Review fixes in this consolidation
Two P1s (worker-thread teardown use-after-free; node head-clip undo data loss) plus P2s across recorder/CLI/MCP/predictor/video-source and the export scripts — see the final commit message for the itemized list.
Verification
28 pure-data mocap unit tests pass locally; Ogre-dependent suites (recorder, predictor, camera source) run under CI's Xvfb. Merged current master (facerig #903) into the branch — conflicts resolved (both mocap + facerig QML singletons registered, both HF mirror sets kept).
🤖 Generated with Claude Code
Summary by CodeRabbit
qtmesh mocapCLI plus automation tools for capture device listing, live start/stop, and channel selection.