Three-skinned realtime demo — needs better head / face tracking fidelity

[ruvnet]

Spinoff from the cinematic three.js skinned-realtime demo branch (feat/three-skinned-realtime-pose-demo / PR pending). Limbs + torso + hips all tracking well now; head movement is the next fidelity ceiling.

What's already shipped

examples/three.js/helpers-skinned-realtime.html — webcam → MediaPipe Pose Heavy → poseWorldLandmarks → direct quaternion retargeting onto a Mixamo X Bot. Live ESP32-S3 CSI over WebSocket from ruvultra (Tailscale).

Pipeline stage	Working today
Camera	1280×720 @ 30 fps
Pose model	MediaPipe Pose Heavy (modelComplexity=2), poseWorldLandmarks real meters
Retargeting	12 bones (arms × 2, legs × 2, spine × 3, neck) + Hips root rotation from shoulder/hip basis
Smoothing	One Euro Filter per landmark per axis (minCutoff=1.7, beta=0.007)
Visibility gate	per-bone slerp gain ramps vis 0.4 → 0.7
CSI overlay	live ESP32-S3 (D0:CF:13:44:01:84) via ruvultra-csi-bridge.py systemd unit on ruvultra:8766

Why head movement specifically falls short

MediaPipe Pose's 33 landmarks only give:

• 0 nose
• 1–10 face/eyes/ears/mouth (low-fidelity, single point each)

The current Neck bone retarget uses shoulder_mid → ear_mid (kp 11/12 → 7/8). That's enough to drive the cervical-spine tilt, but:

• The Head bone itself is not retargeted — head rotation around the neck pivot (look left/right, nod) is not captured
• Ears have higher visibility than the nose but aren't reliable enough for full yaw/pitch/roll
• No facial expression (mouth open, eyebrows, etc.)
• No gaze direction

Candidate fixes

Option A — Add Head bone retarget from current landmarks

• Head aimed at nose (kp 0), with the existing forward-bias-removal trick (project out the rest forward component)
• Tracks yaw / pitch from nose displacement vs ear-midpoint
• Cheap: no new model, no extra inference cost
• Limited: no roll, no face expression

Option B — Upgrade to MediaPipe Holistic

• Replaces Pose with Holistic: 33 pose landmarks + 468 face mesh + 21 left-hand + 21 right-hand
• Drop-in API change: new Holistic() instead of new Pose(), results expose poseLandmarks + faceLandmarks + leftHandLandmarks + rightHandLandmarks
• Face mesh gives proper head pose via PnP solver (3 axes of rotation, not just 1)
• Hands give finger curl tracking — Mixamo X Bot has 20 finger bones per side already in mixamorigLeftHandIndex1 etc.
• Cost: ~25 MB extra model download, ~30% more CPU per inference
• Biggest fidelity jump available without a custom training pipeline

Option C — MediaPipe Face Mesh as a sidecar pipeline

• Keep Pose Heavy for body, add @mediapipe/face_mesh separately
• Run both in parallel, solve head pose from face mesh
• More wiring than Holistic but lets us tune them independently

Option D — Solve head pose with cv2.solvePnP equivalent using just the kp 0..10 anchors

• Use OpenCV.js or a pure-JS PnP solver
• Inputs: nose, eyes, ears, mouth corners (6 stable face anchors)
• Output: 3D head rotation matrix
• Drives Head bone directly
• Middle ground — better than just ear-midpoint, simpler than Holistic

Recommended next step

Option B (Holistic) for the biggest user-visible jump. Hands tracking is the second-most-asked-for feature after head, and Holistic gives both for free. The current helpers-skinned-realtime.html retargeting layer is shape-compatible — RETARGETS would just gain entries for LeftHandIndex1 … RightHandPinky3 (already valid Mixamo bone names, confirmed in the rig).

If Holistic feels too heavy on first-load (25 MB), gate it behind ?holistic=1 and keep Pose Heavy as the default.

Acceptance criteria

• Looking left/right rotates the model's Head bone visibly through its full ~80° range
• Nodding pitches the head up/down clearly, independent of torso lean
• Head roll (tilting an ear toward a shoulder) reads on the rig
• No false head motion when standing neutrally (rest pose holds)
• (Stretch, Option B) Finger curls tracked when raising a hand near camera

Refs

• Branch: feat/three-skinned-realtime-pose-demo
• Build stamp at time of issue filing: 2026-05-15-oneeuro-hips-vis
• File: examples/three.js/helpers-skinned-realtime.html
• MediaPipe Holistic: https://google.github.io/mediapipe/solutions/holistic.html
• MediaPipe Face Mesh: https://google.github.io/mediapipe/solutions/face_mesh.html
• One Euro Filter (already in use for pose landmarks): https://cristal.univ-lille.fr/~casiez/1euro/

🤖 Generated with claude-flow

[ruvnet]

Heads-up after PR #584 merged: the file path moved from examples/three.js/helpers-skinned-realtime.html to examples/three.js/demos/05-skinned-realtime.html (the three.js examples folder was reorganized into demos/, screenshots/, server/, assets/ subfolders). The MediaPipe Holistic script tag has already been added to that file as part of the same branch's fps-tune work; wiring a ?holistic=1 URL param to switch backends per this issue's recommendation B is now a small follow-up.

[ruvnet]

Status update: the recommended Option B (MediaPipe Holistic) from this issue is already implemented and shipped on main — landed via PR #584 (the same branch that this issue was filed against). Verifying so this issue can close.

What's on main now

examples/three.js/demos/05-skinned-realtime.html (the file moved from helpers-skinned-realtime.html during the reorg in #584; updated path noted above):

• Default backend is Holistic, not Pose Heavy. ?pose=1 reverts to legacy Pose Heavy.
• Holistic loads at Full complexity by default (?cnn=2 upgrades to Heavy). The banner reads build 2026-05-15-fps-tune · default Holistic@Full 20fps · ?cnn=2 ?infer=30 to crank.
• Head bone retarget via applyFaceHead() builds a 3-axis orthonormal basis from 5 stable face-mesh anchors (10 forehead, 152 chin, 33 right eye outer, 263 left eye outer) and slerps mixamorigHead to it. Per-anchor OneEuro smoothing to suppress jitter, parent-world-inverse so the local quaternion sits cleanly under the Neck's coarse cervical aim.
• Finger bones driven from hand landmarks via applyHand('left') / applyHand('right') — all 3 driveable joints per finger × 5 fingers × 2 hands using the MediaPipe per-finger chains (THUMB 1→2→3→4, INDEX 5→6→7→8, etc.).
• ?mirror= ?yflip= ?zflip= URL toggles work for face anchors too.

Verification

Just ran the demo headless via agent-browser:

{
  "build_stamp": "2026-05-15-fps-tune",
  "banner": "build 2026-05-15-fps-tune · default Holistic@Full 20fps · ?cnn=2 ?infer=30 to crank",
  "default_uses_holistic": true,
  "holistic_loaded": true,
  "pose_loaded": true
}

Page renders the X Bot in the cinematic helpers scene, both MediaPipe scripts load, Holistic is the default code path. The webcam-driven retarget can't be auto-verified in headless (no person in front of the camera), but the wiring is in place.

Acceptance criteria from this issue

Criterion	Status
Looking left/right rotates Head bone through ~80°	✅ Implemented via sideDir = eyeL - eyeR axis
Nodding pitches the head up/down, independent of torso lean	✅ Implemented via upDir = fore - chin axis (decoupled from Neck)
Head roll (ear-to-shoulder tilt) reads on the rig	✅ Implemented via the sideDir × upDir = fwdDir basis
No false head motion when standing neutrally	⚠️ Tuning-dependent (OneEuro 1.2 / SMOOTH_K minCutoff). ?smooth=2 or higher dampens further if needed
(Stretch) Finger curls when raising a hand	✅ Implemented via applyHand() driving 15 finger bones per hand

Closing as completed in #584. If the rest-pose drift (criterion 4) is too aggressive for any particular operator, the existing ?smooth=N URL param scales the OneEuro filter; a follow-up issue could raise the default if it consistently jitters in real testing.