Mocap Slice C (#872): FaceCapPredictor (ONNX) + pure-data core (mapper, One-Euro, head pose, geometry) - #881
Conversation
Slice C of epic #869 (performance capture). Implements the contract proven in docs/MOCAP_SPIKE.md (Slice A) in C++. - FaceCapPredictor: three-session ONNX pipeline (BlazeFace detector -> Face Mesh V2 landmarks -> MLP-Mixer blendshapes) -> FaceSample {52 weights, head pose quat+translation, confidence}. Detector-skip tracking (next-frame ROI from the previous landmarks; detector re-runs only when presence drops). Models download on first use to AppData/ai_models/mocap/face/ (QTMESH_MOCAP_MODEL_BASE_URL / ai/mocapModelBaseUrl / QTMESH_MOCAP_NO_DOWNLOAD; UniRig multi-file pattern). Runtime I/O discovery, anchor-count sanity check, graceful degradation without ONNX/models. - FaceCapGeom (pure, QtGui-only): letterbox + inverse, SSD anchor gen (896 face / 2254 pose), TensorsToDetections decode + weighted NMS, face/pose ROI rects, rotated-crop tensor sampling (plain bilinear, BORDER_ZERO, the cv2-integer-index convention the spike parity pinned), landmark projection. - FaceCapPose (pure): weighted rigid fit via Horn's quaternion method with a self-contained 4x4 Jacobi eigensolver (no linear-algebra dependency); solveHeadPose fits the embedded canonical face model with MediaPipe's Procrustes basis weights. Convention documented in the header. - FaceCapMapper (pure, QtCore-only): canonical-52 -> mesh morph-target names via side-token expansion + normalization + alias table + JSON override sidecar; unmatched channels always reported. - OneEuroFilter (pure): scalar + quaternion (hemisphere-aligned slerp) variants. - FaceCapCanonicalData.h: generated constants (52 names in model order, the 146-landmark blendshape subset, 468 canonical vertices, Procrustes basis). 33 headless tests pass; the env-gated real-inference test (models + QTMESH_MOCAP_MODELS_DIR/QTMESH_MOCAP_TEST_IMAGE) verifies the full pipeline on a real photo: mouthSmileLeft 0.96 on the smiling MediaPipe portrait, identity head pose, confidence 1.0, and the detector-skip assertion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b437600ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| d->error = QStringLiteral( | ||
| "face capture models not found in %1 — they download on first " | ||
| "use, or set QTMESH_MOCAP_MODEL_BASE_URL").arg(dir.absolutePath()); | ||
| return false; |
There was a problem hiding this comment.
If load() succeeds once and is later called with a missing or invalid model directory, this early return reports failure but leaves d->available and the existing ONNX sessions untouched. In that scenario isAvailable() still returns true and subsequent predict() calls continue using the previous models despite the failed reload, so this branch should clear availability/tracking/sessions before returning.
Useful? React with 👍 / 👎.
|
Consolidated into #909 (single epic PR targeting master). Review findings from this PR are addressed there — see the latest commit. |
1 participant
Part of epic #869. Closes #872. Stacked on #880 (Slice B) — merge that first; this PR's base is the Slice B branch so the diff shows only Slice C.
What this adds
ONNX consumer #9: one RGB888 frame in →
FaceSample{52 blendshape weights, head pose, confidence}out, per the contract numerically proven indocs/MOCAP_SPIKE.md(#878):FaceCapPredictor— 3 sessions (detector 128²/[-1,1] → rotated 256² crop/[0,1] → 478 landmarks → 146-subset px coords → 52 scores), detector-skip tracking (MediaPipe's own strategy: ROI from previous-frame landmarks, detector re-runs only when presence < 0.5), first-use model download (~6.4 MB total) with the house env/QSettings overrides, runtime I/O-name/shape discovery, anchor-count sanity check, graceful no-ONNX/no-models degradation.FaceCapGeom— letterbox/anchors/decode/weighted-NMS/ROI/rotated-crop/projection. The crop uses plain bilinear, zero border, cv2 integer-index convention — the exact resampling the spike found to be the parity maker-or-breaker.FaceCapPose— weighted rigid fit (Horn's quaternion method + self-contained 4×4 Jacobi; no Eigen dependency added),solveHeadPoseagainst the embedded canonical face model with MediaPipe's Procrustes weights. Convention documented: +X right, +Y up, camera looks −Z, identity = facing camera.FaceCapMapper— ARKit-52 → mesh morph-target names (side-token expansionMouth_Smile_L≡mouthSmileLeft, alias table, JSON override sidecar{"map":…, "ignore":…}); unmatched channels always reported, never dropped.OneEuroFilter— scalar + quaternion (hemisphere-aligned slerp) variants, 30 fps face-capture defaults.FaceCapCanonicalData.h— generated constants (names/subset/canonical model/Procrustes basis) with provenance header.Verification
QTMESH_MOCAP_MODELS_DIR+QTMESH_MOCAP_TEST_IMAGE) run locally against the Slice A converted models on MediaPipe's portrait test photo:mouthSmileLeft 0.96 / mouthSmileRight 0.93 / jawOpen 0.12(matches the Python reference to ~0.01), identity head rotation, confidence 1.0, detector-skip asserted (2nd frame runs no detector). Full pipeline: 52 ms including session creation.🤖 Generated with Claude Code