fernandotonon · fernandotonon · Jul 19, 2026 · Jul 14, 2026 · Jul 14, 2026 · Jul 14, 2026
diff --git a/.gitignore b/.gitignore
@@ -96,6 +96,8 @@ docs/*
 !docs/IMAGE_TO_3D_QUALITY.md
 !docs/TRIPOSG_EXPORT_NOTES.md
 !docs/MESH_SEGMENTATION_STRATEGY.md
+!docs/FACE_RIG.md
+!docs/FACE_RIG_SPIKE.md
 
 # minisign — never commit secret keys
 minisign.key

diff --git a/CLAUDE.md b/CLAUDE.md
diff --git a/README.md b/README.md
@@ -186,6 +186,10 @@ qtmesh retopo model.fbx --max-angle 15 -o conservative.glb  # tighter coplanarit
 qtmesh skin model.fbx -o skinned.glb                        # default 4 influences, falloff 4
 qtmesh skin model.fbx --max-influences 8 --falloff 6 -o skinned.glb
 qtmesh skin model.fbx --skip-unweighted --merge -o filled.glb  # fill missing weights only
+
+# Auto-generate the 52 ARKit blendshapes on a humanoid FACE mesh (for face capture)
+qtmesh facerig head.glb -o rigged.glb                       # fit ARKit template + transfer 52 shapes
+qtmesh facerig head.fbx -o rigged.glb --max-shapes 20 --json  # cap shapes / machine-readable report
 ```
 
 ---

diff --git a/THIRD_PARTY_AI_MODELS.md b/THIRD_PARTY_AI_MODELS.md
@@ -271,6 +271,47 @@ the binary). Attribution + licenses for the models and their training data:
   checkbox; the template library remains the default and the automatic
   fallback. Same CMU licensing basis as above.
 
+## ICT-FaceKit — ARKit blendshape template for face auto-rig (epic #889)
+
+- **Asset (not a learned model):** the ICT-FaceKit generic neutral head
+  (`generic_neutral_mesh.obj`) + its per-expression meshes named after the
+  ARKit blendshapes (`jawOpen`, `mouthSmile_L`, `eyeBlink_L`, `browInnerUp_L`,
+  …), all sharing one topology (26,719 verts) so each shape = expr − neutral.
+- **Source / license:** [USC-ICT/ICT-FaceKit](https://github.com/USC-ICT/ICT-FaceKit)
+  — **MIT** (Copyright 2020 USC Institute for Creative Technologies). The
+  standard/released model is MIT; a separate "full model" tier under a
+  USC-specific license is **REJECTED** (we ship only the MIT tier). MIT clears
+  the permissive-redistribution bar, so the template + shapes are hostable on
+  the `fernandotonon/QtMeshEditor-models` HF repo (Slice B, #890) — packed by
+  `scripts/export-arkit-template.py` into `facerig/arkit_template.bin` and
+  uploaded by `scripts/upload-facerig-template.sh`; it downloads on first use.
+- **How it is used:** the template is the *source* for **deformation transfer**
+  (Sumner & Popović 2004) — QtMeshEditor fits it to the user's neutral head via
+  native non-rigid ICP (Amberg 2007), then transfers each of the 52 ARKit
+  expressions onto the user's topology, attaching them as `Ogre::Pose` morph
+  targets so face performance capture (#869) works on the mesh. **No ML model,
+  no ONNX** — it is a deterministic geometry algorithm (sparse linear solve),
+  implemented natively in `src/FaceRig/` (Slices C/D/E). The offline spike
+  (`scripts/spike-facerig.py`, not shipped) validated the approach first
+  (see `docs/FACE_RIG_SPIKE.md`). Verified end-to-end on
+  a decimated, different-topology face: mean 0.008% / max 0.61% NRICP fit and
+  51 attached shapes. Surfaced via `qtmesh facerig`, MCP `add_arkit_blendshapes`,
+  and the Inspector "Add ARKit Blendshapes" button. See `docs/FACE_RIG.md`.
+- **Facial-landmark anchoring (landmark pass):** the NRICP fit is anchored to
+  real face features by **MediaPipe Face Mesh V2** (`face_landmarks.onnx`,
+  **Apache-2.0** — the same model the mocap face-capture uses, #869). We render
+  the head front-on, detect the 478 landmarks, back-project them to the mesh
+  surface, and pin the matching template vertices — so the template lands on the
+  actual eyes/nose/mouth instead of a low-residual-but-mis-oriented drape. Hosted
+  under `facerig/face_landmarks.onnx` (a copy of the mocap graph); downloads on
+  first use; `ENABLE_ONNX`-guarded with a graceful unanchored-fit fallback.
+- **Rejected alternatives:** Wrap3D (commercial, used by the reference impl for
+  NRICP — we implement NRICP natively instead), FLAME-based 3DMMs
+  (research-only), any generative expression model on non-commercial data.
+  Landmark detectors trained on 300W / WFLW / InsightFace (dlib, PIPNet,
+  2d106det) were rejected — their weights carry research-only / non-commercial
+  terms; MediaPipe FaceMesh (Apache-2.0) is the clean choice.
+
 All of the above clear QtMeshEditor's permissive-redistribution bar (MIT app,
 distributed via Homebrew / WinGet / Snap / Docker). GPL/CC-BY-NC/unlicensed
 models are deliberately excluded (e.g. RigNet was rejected for #408 — GPL code +

diff --git a/docs/FACE_RIG.md b/docs/FACE_RIG.md
@@ -0,0 +1,100 @@
+# Face auto-rig: ARKit blendshapes on any humanoid face
+
+Epic [#889](https://github.com/fernandotonon/QtMeshEditor/issues/889). Given an
+unrigged neutral **face** mesh, QtMeshEditor generates the 52 **ARKit**
+blendshapes (`jawOpen`, `mouthSmileLeft`, `eyeBlinkLeft`, `browInnerUp`, …) and
+attaches them as morph targets, so the mesh can be driven by face performance
+capture ([#869](https://github.com/fernandotonon/QtMeshEditor/issues/869),
+`qtmesh mocap --face`) — no manual sculpting of blend shapes.
+
+It is **deterministic geometry**, not an ML model: a non-rigid fit of a
+permissively-licensed template face onto yours, followed by deformation
+transfer of each expression. No ONNX, no GPU, no network beyond a one-time
+template download.
+
+## Using it
+
+### GUI
+Select a face mesh → Inspector → **Vertex Morph Animation** (Edit Mode) →
+**"✨ Add ARKit Blendshapes (AI)"**. The fit runs on a worker thread (the button
+shows *Downloading… / Fitting…*); when it finishes the 52 shapes appear in the
+Shapes list and the whole batch is a single undo step. The
+[#869](https://github.com/fernandotonon/QtMeshEditor/issues/869) Performance
+Capture panel then drives them.
+
+### CLI
+```bash
+qtmesh facerig neutral_head.glb -o rigged.glb
+qtmesh facerig head.fbx -o rigged.glb --max-shapes 20     # cap the shape count
+qtmesh facerig head.glb  -o rigged.glb --max-residual 5   # stricter humanoid gate
+qtmesh facerig head.glb  -o rigged.glb --json             # machine-readable report
+```
+
+### MCP
+`add_arkit_blendshapes` — `{max_shapes?, max_residual_pct?, output_path?}`,
+operates on the selected entity; re-exports when `output_path` is given.
+
+## How it works
+
+```
+ArkitTemplate  (ICT-FaceKit neutral + 52 expression deltas, one topology)
+     │
+     ▼  HEAD ISOLATION — rig-prior (skinned) or MeshSegmenter picks the head
+     │   region, so a full-body character fits the face template only on the
+     │   head (not smeared across the body).
+     │
+     ▼  FACE-LANDMARK ANCHORS — render the head front-on, detect 478 MediaPipe
+     │   landmarks (FaceLandmarkDetector), back-project to the surface on BOTH
+     │   template + user, pair by index → anchor constraints. This locks the
+     │   fit onto the real eyes/nose/mouth (the fix for wrong shape placement).
+     │
+     ▼  NonRigidICP (Amberg 2007 optimal-step, landmark-anchored)   src/FaceRig/NonRigidICP
+  correspondence X  — template verts fitted onto the USER surface
+     │
+     ▼  DeformationTransfer (Sumner & Popović 2004)   src/FaceRig/DeformationTransfer
+  per-template-vertex delta per shape, on the user identity
+     │
+     ▼  resample template topology → the real user vertices   src/FaceRig/FaceRigger
+  52 × per-user-vertex deltas
+     │
+     ▼  Ogre::Pose + VAT_POSE morph targets, named per FaceCap::kBlendshapeNames
+```
+
+- **`FaceRigger` / `FaceRigAttach`** (`src/FaceRig/`) orchestrate the pipeline;
+  the pure-data core is Ogre-free and headless-unit-tested.
+- The linear solves use a self-contained sparse CG (`SparseSolve`) — **no Eigen,
+  no external solver, zero new dependencies** (the house rule for these
+  features, same as skinning #402 and auto-rig #407).
+- The template (`facerig/arkit_template.bin`, ICT-FaceKit MIT — see
+  `THIRD_PARTY_AI_MODELS.md`) downloads on first use to
+  `<AppData>/ai_models/facerig/`. Overrides: `QTMESH_FACERIG_MODEL_BASE_URL` /
+  `QSettings ai/facerigModelBaseUrl`; offline guard `QTMESH_FACERIG_NO_DOWNLOAD`.
+- **Landmark anchoring** (`FaceLandmarkDetector` + `FaceRigLandmarks`,
+  `ENABLE_ONNX`): renders the head and runs MediaPipe FaceMesh V2
+  (`facerig/face_landmarks.onnx`, Apache-2.0) to anchor the fit to real face
+  features. When ONNX is off, the model is missing, or no face is detected, the
+  fit runs **unanchored** (the previous behaviour) — the feature degrades
+  gracefully, it never blocks a rig.
+
+## Quality & limits
+
+- **Humanoid faces only.** The fit residual is a gate: a non-face mesh fits
+  poorly and is **rejected** (`--max-residual`, default 8% of the mesh
+  diagonal), rather than emitting garbage shapes. This mirrors the
+  AutoRig/Pinocchio precedent.
+- **Measured** (decimated, different-topology ICT head, 15 755 verts): NRICP
+  fit **mean 0.008% / max 0.61%** of the diagonal; **51 shapes** attached;
+  jawOpen drops the lower face while the forehead stays still; mouthSmile /
+  eyeBlink / browInnerUp localise to their regions.
+- **Orientation:** the mesh should be roughly upright, +Y up, facing the
+  template's orientation. A wildly rotated head may fit poorly.
+- **glTF export** carries the morph-target geometry on the primitive; per-target
+  *names* in glTF `extras.targetNames` are a follow-up (the in-editor targets
+  and the mocap hand-off use the correct names regardless).
+
+## Related
+
+- `THIRD_PARTY_AI_MODELS.md` — ICT-FaceKit licensing verdict.
+- `docs/FACE_RIG_SPIKE.md` — the offline feasibility spike + the C/D contract.
+- Epic [#889](https://github.com/fernandotonon/QtMeshEditor/issues/889);
+  slices B–G (#891–#895 + the polish slice).
diff --git a/docs/FACE_RIG_SPIKE.md b/docs/FACE_RIG_SPIKE.md
@@ -0,0 +1,121 @@
+# Face auto-rig — Spike Findings & Contract (#889 / slice A #896)
+
+**Epic:** [#889 — AI: Auto-generate ARKit blendshapes on any humanoid mesh (deformation transfer)](https://github.com/fernandotonon/QtMeshEditor/issues/889)
+**Slice:** [#896 — Spike: NRICP feasibility + ICT-FaceKit licensing due-diligence](https://github.com/fernandotonon/QtMeshEditor/issues/896)
+**Status:** Spike — **GO.** Non-rigid ICP fits the MIT ICT-FaceKit template to
+a different-topology head to sub-1% accuracy, and deformation transfer produces
+anatomically-correct ARKit blendshapes on the user's own topology. No ML, no
+ONNX — a deterministic sparse-linear-algebra pipeline. Slices C/D are a native
+C++ port of the proven `scripts/spike-facerig.py`.
+
+---
+
+## TL;DR — Recommendation: **GO**
+
+- **Template licensing — CLEARS THE BAR.** ICT-FaceKit is **MIT** (a neutral
+  head + 52 ARKit-named expression meshes, all one topology). Redistributable
+  on the HF models repo. The separate "full model" USC-specific tier is
+  rejected; we ship only the MIT tier. Recorded in `THIRD_PARTY_AI_MODELS.md`.
+- **NRICP — WORKS.** Amberg-2007 optimal-step, pure numpy/scipy (no Wrap3D):
+  fit the 26,719-vert template onto a **12,763-vert (different topology)** user
+  head → surface fit **mean 0.003%, max 0.59%** of the head diagonal.
+- **Deformation transfer — WORKS.** Sumner & Popović 2004: each of the 52 ICT
+  expressions transfers onto the user topology with correct semantics —
+  jawOpen drops the lower face (mean ΔY −0.32) while the forehead stays still
+  (|Δ| 0.002); eyeBlink stays localized to the eye (1,328 verts), jawOpen is
+  the biggest deformation (6,771 verts, 9% max), browInnerUp is small (1.3%).
+- **No new runtime dependency:** deterministic geometry (sparse solve), like
+  `GeodesicVoxelBind` / `QuadRetopo`. No model download at inference (only the
+  MIT template asset downloads on first use, like other bundled assets).
+
+---
+
+## The template (ICT-FaceKit, MIT)
+
+`FaceXModel/` ships `generic_neutral_mesh.obj` + one `.obj` per expression, ALL
+sharing the neutral's topology (26,719 verts / 26,384 tris), so a blendshape is
+simply `expr_obj − neutral_obj` (per-vertex delta). ICT uses `<name>_L/_R`
+stems; the spike maps them to the canonical `FaceCap::kBlendshapeNames` (the
+mocap-52 order) — full table in `scripts/spike-facerig.py::ICT_TO_ARKIT`.
+Slice B bakes this template into a compact bundled form + hosts it.
+
+> **OBJ gotcha:** these OBJs are multi-group; trimesh loads them as a Scene and
+> reorders/duplicates vertices, breaking the shared-topology assumption. Parse
+> `v`/`f` manually and preserve order (the spike + Slice B loader both do).
+
+## The pipeline (what Slices C/D implement natively)
+
+```
+user neutral head (arbitrary topology, roughly humanoid, +Y up, facing +Z)
+  │
+  1. rigid pre-align: centroid + bbox-scale match template→user
+  │    (correspondence-free; NRICP refines. A real user mesh may need
+  │     up/forward-axis detection first — the spike's decimated head shared
+  │     ICT's orientation so identity axes sufficed.)
+  │
+  2. NRICP (Amberg 2007 optimal-step):
+  │    unknown = per-template-vertex 3×4 affine A_i
+  │    minimize ‖A_i·ṽ_i − closest_point_on_user_surface(X_i)‖²        (data)
+  │           + α·‖(A_i − A_j)‖²  over template edges (i,j)             (stiffness)
+  │    stiffness annealed α = 50→20→8→3→1→0.5, ~3 inner iters each;
+  │    closest point via point-to-triangle projection; one sparse lsqr per axis.
+  │    → fitted template verts X (lie on the user surface) = CORRESPONDENCE.
+  │
+  3. deformation transfer (Sumner & Popović 2004), per ARKit expression:
+  │    template expression correspondence = X + (expr_tmpl − neutral_tmpl)
+  │    per-user-vertex delta = displacement of the nearest correspondence point
+  │    (the spike's simplified transfer; the full C++ form solves the
+  │     per-triangle deformation-gradient least-squares — see below).
+  │
+  → 52 per-user-vertex deltas → Ogre::Pose morph targets named per kBlendshapeNames
+```
+
+### Note on the transfer step (Slice D must upgrade the spike form)
+
+The spike transfers by **nearest-correspondence-point displacement**, which is
+enough to prove semantics and fit quality. The production Slice D should use the
+**full deformation-gradient transfer**: build each template triangle's affine
+`S_j = V_expr · V_neutral⁻¹` (with the 4th "normal" vertex trick), then solve one
+sparse least-squares `min ‖A_userTri − S_j‖²` for the user vertex positions
+(the Sumner-Popović matrix). This is more faithful for large/rotational
+deformations (jaw) than nearest-point displacement. The linear solver is shared
+with NRICP.
+
+## Measured quality (2026-07-14, `scripts/spike-facerig.py`)
+
+Template 26,719v → user 12,763v (60%-decimated, different topology):
+
+| ARKit shape | max displacement | verts moved | semantics check |
+|---|---|---|---|
+| NRICP surface fit | mean **0.003%** / max **0.59%** of diag | — | template lands on user surface |
+| jawOpen | 9.0% | 6,771 | lower face ΔY −0.32 (drops), forehead \|Δ\| 0.002 (still) ✅ |
+| mouthSmileLeft | 3.3% | 3,552 | localized to mouth ✅ |
+| eyeBlinkLeft | 3.8% | 1,328 | localized to eye ✅ |
+| browInnerUp | 1.3% | 2,181 | localized to brow ✅ |
+
+## Risks / limits (carry into the epic)
+
+1. **Humanoid-only.** Transfer from a human template only makes sense for
+   roughly human face meshes; a prop/creature yields garbage. Slice E gates on
+   this (NRICP fit-quality metric + a clear error), the AutoRig/Pinocchio
+   precedent.
+2. **Orientation.** The spike's user head shared ICT's axes. A real arbitrary
+   mesh needs up/forward detection (or a user hint) before the rigid pre-align
+   — fold into Slice C/E.
+3. **Correspondence quality drives shape quality.** Landmark constraints (eye
+   corners / nose / mouth) may be needed on faces far from the template
+   proportions; the spike didn't need them on the decimated ICT head — revisit
+   on real Ready-Player-Me / scanned heads in Slice C.
+4. **Full deformation-gradient transfer** (Slice D) over the nearest-point
+   spike form, for faithful large deformations.
+5. **Performance.** The python spike's dense per-vertex assembly is slow (~minutes
+   at 26k verts); the C++ port must assemble the sparse system directly and use a
+   real sparse solver (the project's existing linear-algebra path) — target a
+   few seconds, worker-threaded in the GUI.
+
+## Go/No-Go
+
+**GO.** Both algorithms proven on a real, different-topology head with a
+permissive (MIT) template. Slices B→G are an engineering port of a working
+prototype, not open research. Ship the full deformation-gradient transfer +
+landmark option as the two quality upgrades over the spike.