AI Image-to-3D (TripoSR/ONNX) — full feature, all slices #765–#769 (#764)

.gitignore

@@ -92,6 +92,7 @@ website/dist/
 website/node_modules/
 docs/*
 !docs/AUTO_UPDATER_DESIGN.md
+!docs/IMAGE_TO_3D_SPIKE_764.md
 !docs/MESH_SEGMENTATION_STRATEGY.md
 
 # minisign — never commit secret keys
@@ -109,3 +110,39 @@ mined_training_data/
 .meshseg_work/
 .venv-meshseg/
 __pycache__/
+
+# Local scratch: CLI/tool test-output dirs + stray asset dumps (never commit — a
+# `git add -A` once swept ~600 of these in and crashed the asset-scan CI job).
+# Anchored to the repo root (/) so tracked tools/*-vat-demo/ assets are unaffected.
+.sentry-native/
+/anim/
+/anim2/
+/quad/
+/quad2/
+/hip[0-9]*/
+/lod[0-9]*/
+/pbr/
+/pbr[0-9]*/
+/paint/
+/paint[0-9]*/
+/pose/
+/pose[0-9]*/
+/rumba/
+/rumba[0-9]*/
+/rumba[0-9]*.material
+/rumba[0-9]*.mesh
+/simplify[0-9]*/
+/scene.material
+/scene.scene.glb
+/screenshot.png
+/i
+/Boss_*.png
+/Ch14_*.png
+/Hip Hop Dancing.*
+/Rumba Dancing*.skeleton
+/fbx_linux_out.png
+/mage*.png
+/mage.*
+/monk_*.png
+/multiview_bake_*.png
+/multiview_bake_*i.png

THIRD_PARTY_AI_MODELS.md

@@ -19,6 +19,51 @@ the binary). Attribution + licenses for the models and their training data:
   `encoder.onnx` + `decoder.onnx` via ONNX Runtime (`src/UniRigPredictor.cpp`),
   downloading them on first use to `AppData/ai_models/unirig/`.
 
+## TripoSR — image-to-3D mesh generation (epic #764)
+
+- **Model:** TripoSR single-image 3D reconstruction (DINO ViT tokenizer +
+  triplane transformer + NeRF decoder), exported to ONNX as an encoder
+  (image → triplane) + decoder (triplane + points → density/color) pair.
+- **Source:** Tripo AI + Stability AI — *"TripoSR: Fast 3D Object Reconstruction
+  from a Single Image"* (arXiv 2403.02151).
+  https://github.com/VAST-AI-Research/TripoSR — code **MIT**.
+  Weights: https://huggingface.co/stabilityai/TripoSR — **MIT** (code AND weights).
+- MIT code+weights is the deciding factor: it clears QtMeshEditor's permissive-
+  redistribution bar (Homebrew / Snap / WinGet / Docker), the same reason UniRig
+  (#408) passed. Non-commercial SF3D / Stable-Fast-3D was rejected on license.
+- The host-side iso-surface step (density grid → mesh) is a native, from-scratch
+  Lorensen marching cubes (`src/ImageTo3D/MarchingCubes.{h,cpp}`, public-domain
+  tables — no vendored/GPL dependency); TripoSR's own `torchmcubes` is torch/GPU-only.
+- The ONNX export is produced by `scripts/export-triposr-onnx.py` (one-time,
+  offline developer tool — not shipped). The app runs the resulting encoder +
+  `triposr_decoder.onnx` via ONNX Runtime (`src/ImageTo3D/MeshGenPredictor.cpp`),
+  downloading them on first use to `AppData/ai_models/triposr/`.
+- **Encoder size tiers** (all the SAME MIT weights, just re-precisioned by the
+  export script — no separate license): `triposr_encoder.onnx` (fp32, ~1.68 GB) and
+  `triposr_encoder_int8.onnx` (~430 MB, ORT dynamic quantization). The user picks
+  the tier; each downloads on demand. (fp16 was dropped — TripoSR's attention has a
+  hardcoded Cast-to-float32 the ONNX fp16 converters can't rewrite into a loadable
+  graph; int8 is smaller anyway.)
+- **Hosted** on the `fernandotonon/QtMeshEditor-models` HF repo:
+  `triposr/triposr_encoder.onnx`, `triposr/triposr_encoder_int8.onnx`,
+  `triposr/triposr_decoder.onnx`, `rembg/u2net.onnx` (uploaded via
+  `scripts/upload-triposr-models.sh`). First use downloads them; if ever absent the
+  feature reports a clean "not yet hosted" state (no crash) — the RigNet precedent.
+
+## U²-Net — background removal for image-to-3D (epic #764)
+
+- **Model:** U²-Net salient-object detection (`u2net.onnx`) — the default
+  foreground-segmentation model shipped by [rembg](https://github.com/danielgatis/rembg).
+- **Source:** Qin et al., *"U²-Net: Going Deeper with Nested U-Structure for
+  Salient Object Detection"* (Pattern Recognition 2020),
+  https://github.com/xuebinqin/U-2-Net — code **Apache-2.0**; the released ONNX
+  weights are redistributed by rembg under the same permissive terms.
+- Used only as a **pre-process** for TripoSR image-to-3D (`src/BackgroundRemover.cpp`):
+  isolate the subject so the reconstruction sees a clean background. Downloads on
+  first use to `AppData/ai_models/rembg/u2net.onnx` (override
+  `QTMESH_REMBG_MODEL_BASE_URL` / `QSettings ai/rembgModelBaseUrl`; offline guard
+  `QTMESH_REMBG_NO_DOWNLOAD`). Falls back to the raw image when unavailable.
+
 ## PBRify_Remix — PBR map synthesis (issue #404)
 
 - Three SPAN models from https://github.com/Kim2091/PBRify_Remix — **CC0-1.0**,

action.yml

@@ -8,7 +8,7 @@ branding:
 
 inputs:
   command:
-    description: 'Subcommand: scan, info, validate, convert, fix, anim, lod, pose, turntable, isometric'
+    description: 'Subcommand: scan, info, validate, convert, fix, anim, lod, pose, turntable, isometric, generate3d'
     required: true
   input-file:
     description: 'Directory or file to scan (relative to workspace). Defaults to . (workspace root).'

docs/IMAGE_TO_3D_SPIKE_764.md

@@ -0,0 +1,241 @@
+# Image-to-3D (TripoSR via ONNX) — Spike Findings & Design (#764 / slice A #765)
+
+**Epic:** [#764 — AI: Image-to-3D mesh generation (TripoSR via ONNX)](https://github.com/fernandotonon/QtMeshEditor/issues/764)
+**Slice:** [#765 — Spike: TripoSR ONNX export + marching-cubes proof](https://github.com/fernandotonon/QtMeshEditor/issues/765) (de-risk first)
+**Status:** Spike — **GO** (both risks retired; see the go/no-go at the bottom).
+
+This is the deliverable for #765's acceptance criteria: (1) the TripoSR network
+exports to ONNX and loads under our ONNX Runtime, (2) a host-side marching-cubes
+routine produces a valid closed mesh from a synthetic SDF under a confirmed
+permissive license, and (3) this note records the tensor contract, the MC choice,
+and the go/no-go so slices B–E can proceed with a known interface.
+
+---
+
+## TL;DR — Recommendation: **GO**
+
+- **Marching cubes** — DONE and verified. Native Lorensen implementation in
+  `src/MarchingCubes.{h,cpp}` (zero new dependencies, public-domain tables),
+  proven watertight on a sphere (Euler χ = 2, 0 boundary/non-manifold edges,
+  vertices on-surface to 0.0004 vs a 0.043 cell) and a box (AABB matches ±0.5
+  exactly). Tests: `src/MarchingCubes_test.cpp`.
+- **ONNX export** — DONE (proven against the real weights). TripoSR splits cleanly
+  into two exportable graphs: encoder `image[1,3,512,512] → scene_codes[1,3,40,64,64]`
+  (~1.68 GB) and decoder `scene_codes + points[1,P,3] → density[1,P,1], color[1,P,3]`
+  (~180 KB). Both export at opset 17 and load+run under ONNX Runtime 1.20.1; the
+  decoder's `grid_sample` traced and matched (round-trip match=True). No
+  autoregressive loop (unlike UniRig #408), so the export is simpler than the prior
+  ONNX consumers. Two version pins + one ViT-pos-encoding monkeypatch were needed
+  (documented below) — no blocker.
+- **License** — CLEARS THE BAR. TripoSR is **MIT for code AND weights**
+  (`stabilityai/TripoSR`), redistributable via Homebrew/Snap/WinGet/Docker. See
+  `THIRD_PARTY_AI_MODELS.md`.
+
+---
+
+## Why TripoSR (model selection)
+
+Same reasoning the epic records: MIT code+weights is the deciding factor (safe for
+the project's permissive redistribution), it's fast/small (CPU-runnable), and it
+fits the existing "drop-in offline ONNX tool" pattern. Non-commercial
+SF3D/Stable-Fast-3D and heavier CRM/MeshLRM/Unique3D pipelines were rejected — the
+same license wall that excluded LAFAN1 (#409) and GPL TetGen (#402).
+
+---
+
+## The pipeline (confirmed from `tsr/system.py`, `tsr/models/*`)
+
+```
+image (PIL/np)
+  │  ImagePreprocessor: RGB → float /255 → resize to (cond_image_size, cond_image_size)
+  │                     NB: plain [0,1], NO ImageNet mean/std normalization
+  ▼
+DINO ViT tokenizer (transformers ViTModel, facebook/dino-vitb16)  ──┐
+  ▼                                                                 │  ENCODER
+learned triplane tokens (tokenizer) → transformer backbone          │  (image → scene_codes)
+  (cross-attends the image tokens) → post_processor                 │
+  ▼                                                                 │
+scene_codes  =  the triplane  [1, 3, Ct, Ht, Wt]  ──────────────────┘
+  │
+  │  renderer.query_triplane(decoder, points, scene_code):          ──┐
+  │    points (-radius..radius) → scaled to (-1,1)                    │  DECODER
+  │    → 3× F.grid_sample(triplane) → concat features                 │  (triplane + points
+  │    → NeRF MLP (ReLU) → density (1ch) + features (3ch)             │   → density, color)
+  ▼                                                                   │
+density grid  (queried at resolution³ points in [0,1]³ world box)  ───┘
+  │
+  │  isosurface: marching cubes on  -(density - threshold)  at iso 0
+  ▼                                                                     HOST-SIDE C++
+triangle mesh (+ optional per-vertex color from a second query_triplane pass)
+```
+
+### Key constants (measured at export against the real `stabilityai/TripoSR` weights)
+| Thing | Value | Source |
+|-------|-------|--------|
+| `cond_image_size` | **512** | `TSR.cfg` (printed by the export) |
+| Image normalization | `/255` only, **no mean/std** | `tsr/utils.py ImagePreprocessor` |
+| `renderer.radius` | **0.87** (query-point half-extent) | `TSR.renderer.cfg` |
+| Triplane (`scene_codes`) | **`[1, 3, 40, 64, 64]`** (3 planes × 40 ch × 64²) | measured |
+| Marching-cubes resolution | 256 (default; 128 for a fast/preview path) | `extract_mesh` |
+| Density threshold | **25.0** | `extract_mesh(threshold=25.0)` |
+| Query-point range | `(-0.87, +0.87)`, scaled to `(-1,1)` for grid_sample | `query_triplane` |
+| MC field sign | surface at `-(density - threshold) = 0`, i.e. `density ≥ threshold` inside | `MarchingCubeHelper.forward` |
+
+### Grid ordering (matters for slice B/C)
+TripoSR builds grid vertices with `meshgrid(x, y, z, indexing="ij")` reshaped to
+`[-1,3]` — **x slowest, z fastest** — then swaps the extracted-vertex axes `[2,1,0]`
+and divides by `(resolution − 1)`. Our native MC consumes a **row-major
+`field[z*ny*nx + y*nx + x]` (x fastest)** grid. Slice B must fill the density grid
+in whatever order it queries the decoder and hand `MarchingCubes::extract` a
+consistent `[nx,ny,nz]` layout + `gridMin/gridMax` world box; the export script
+prints the reference grid ordering so the C++ side can match.
+
+---
+
+## The tensor contract (target for `MeshGenPredictor::predict`, slice B #766)
+
+Produced by `scripts/export-triposr-onnx.py` (offline dev tool — NOT shipped, NOT
+wired into CMake/CI; mirrors `export-rmib-onnx.py` / `export-unirig-onnx.py`).
+
+**`triposr_encoder.onnx`** (~1.68 GB — full DINO ViT + triplane transformer)
+| | name | dtype | shape |
+|--|------|-------|-------|
+| in  | `image`       | float32 | `[1, 3, 512, 512]` (RGB in `[0,1]`, no mean/std) |
+| out | `scene_codes` | float32 | `[1, 3, 40, 64, 64]` (the triplane) |
+
+**`triposr_decoder.onnx`** (~180 KB — the NeRF MLP + grid_sample)
+| | name | dtype | shape |
+|--|------|-------|-------|
+| in  | `scene_codes` | float32 | `[1, 3, 40, 64, 64]` |
+| in  | `points`      | float32 | `[1, P, 3]` (world coords in `(-0.87, 0.87)`) |
+| out | `density`     | float32 | `[1, P, 1]` (post `density_act`; threshold at 25.0) |
+| out | `color`       | float32 | `[1, P, 3]` (sigmoid features; optional vertex color) |
+
+The decoder is a **per-point graph** with a dynamic `P` axis, so the C++ side tiles
+the `resolution³` grid through it in chunks (bounded memory), fills the density
+grid, then runs `MarchingCubes::extract(field = density − threshold, isoLevel = 0)`.
+Vertex color is a second decoder pass over the extracted vertices only.
+
+**Measured export (`--verify` against `stabilityai/TripoSR`):**
+- Both graphs exported (opset 17, legacy TorchScript exporter, `dynamo=False`).
+- `grid_sample` in the decoder traced and ran under ONNX Runtime 1.20.1 with no
+  custom op — the key feasibility question. **Decoder ORT round-trip: match=True**
+  (`atol=1e-2`), i.e. the density path (which determines the surface) is exact.
+- Encoder ORT round-trip reported `match=False` only because the script's tight
+  `atol=1e-3` is unrealistic against a `scene_codes` tensor whose values span
+  ±~1370; the encoder is **deterministic in ORT** (repeat-run max|diff| = 0.0) and
+  the relative error is negligible. Slice B should compare with a relative
+  tolerance, not `1e-3` absolute.
+
+**Export gotchas (recorded for reproducibility):**
+- `tsr` imports `torchmcubes` (torch/GPU MC we don't use) at module load — stub it
+  or install it; the export never calls it.
+- Pin **`transformers==4.35.0`** (TripoSR's requirement): newer transformers renamed
+  the ViT state-dict keys (`encoder.layer.N.*` vs `layers.N.attention.q_proj`) and
+  the checkpoint won't load otherwise.
+- The DINO ViT interpolates its positional embedding 224 → 512 via
+  `nn.functional.interpolate(bicubic)`, which **does not trace** (`upsample_bicubic2d`
+  rejects the traced dynamic `output_size`). Since the input size is fixed, the
+  export script precomputes the interpolated table once and monkeypatches
+  `interpolate_pos_encoding` to return that constant. Slice E's production export
+  reuses this.
+
+---
+
+## Marching cubes — the host-side iso-surface step
+
+**Decision: native, ported into `src/` — NOT vendored.** The codebase had no
+iso-surface code (confirmed: nothing in `src/` or `src/dependencies/`). TripoSR
+uses `torchmcubes` (GPU/torch — not usable in the C++ app). We ship a from-scratch
+Lorensen–Cline marching cubes:
+
+- `src/MarchingCubes.h/.cpp` — pure-data (no Ogre, no Qt-singleton; same shape as
+  `PbrMapSynth` / `SkinWeights`, so it unit-tests without a GL context).
+- The 256-entry edge mask + 256×16 triangle tables are the canonical **public-domain**
+  marching-cubes tables (Paul Bourke's widely-mirrored tabulation) — authored into
+  our `.cpp`, nothing fetched. Zero new dependency, matching the native-heuristic
+  stance of SkinWeights (#402, avoided GPL TetGen) and QuadRetopo (#401).
+- API: `extract(field, nx, ny, nz, isoLevel, gridMin, gridMax) → {positions, indices}`
+  in world space, with edge-hash vertex welding (adjacent triangles share vertices,
+  so downstream normal accumulation and export are clean).
+- Inside-positive convention: caller passes `field = density − threshold`,
+  `isoLevel = 0` — the sign/threshold bookkeeping lives at the call site.
+
+### Verified (offline, no GL — `src/MarchingCubes_test.cpp`)
+| SDF | Result |
+|-----|--------|
+| Sphere R=0.6 @ 48³ | V=3744, T=7484; **0 boundary edges, 0 non-manifold, Euler χ = 2** (watertight); max radial deviation 0.0004 (cell 0.0426) |
+| Box H=0.5 @ 40³ | AABB = `[-0.5,0.5]³` to within a cell |
+| Empty / null / degenerate grid | empty mesh, no crash |
+| Iso-level shift | higher iso → smaller surface (threshold param works) |
+
+> **macOS local caveat:** `UnitTests` aborts at startup on macOS because
+> `test_main` requires a working GL context (`tryInitOgre()` — a known project
+> limitation). The MC tests are pure-data and run on **Linux CI (Xvfb)**; locally
+> they were verified by compiling `MarchingCubes.cpp` standalone against the same
+> assertions (all pass).
+
+### Expected costs (slice B/C planning)
+- Vertex counts scale ~with the surface area in cells: a res-256 character ≈ tens
+  of thousands of triangles (TripoSR's own default). Res 128 is a good fast/preview
+  tier.
+- The dominant cost is the **decoder grid query** (resolution³ points through the
+  MLP), not the MC pass. Chunked decoder inference + optional coarse-grid
+  early-out are slice-B tuning knobs.
+
+---
+
+## C++ load-proof (`src/MeshGenSpike_test.cpp`)
+
+An `ENABLE_ONNX`-guarded test opens the exported encoder+decoder with the **exact**
+`Ort::Session` setup the shipping predictors use (`UniRigPredictor.cpp` ~820-843:
+`ORT_ENABLE_ALL`, CoreML EP in try/catch on `__APPLE__`, wide-string path on
+`_WIN32`) and asserts the I/O node counts + tensor ranks match the contract above.
+It **skips** unless the exported `.onnx` files are present in the AppData cache
+(`ai_models/triposr/`), because the model is not hosted yet (slice E) — the same
+"covered on CI when the model is available" convention as UniRig/PBR and the
+"rignet.onnx not yet hosted" precedent. It compiles and links against the real ORT
+headers today (validating the API slice B clones); it turns green the moment a
+developer drops the exported models in the cache.
+
+**Verified on this machine** (macOS arm64) with a standalone build of the same
+`Ort::Session` setup against the freshly-exported models:
+```
+ENCODER: in[0]=[-1,3,512,512]  out[0]=rank-5 triplane
+DECODER: in=[1,3,40,64,64]+[1,P,3]  out=density[1,P,1]+color[1,P,3]
+DECODER RAN: density=[1,512,1]   → LOAD-PROOF PASS
+```
+(`UnitTests` itself can't run on macOS — `test_main` requires GL; the gtest runs on
+Linux CI. The encoder's exported output dims show as dynamic `-1` because batch was
+marked dynamic and shape inference propagated through the triplane transformer; the
+concrete `[1,3,40,64,64]` is confirmed by the Python `--verify` run and the decoder
+accepting it. The committed gtest asserts ranks, not the dynamic dims, so it is
+robust to this.)
+
+---
+
+## Risks & mitigations
+
+| Risk | Assessment | Mitigation |
+|------|-----------|------------|
+| ONNX export of the transformer backbone | LOW — no AR loop, standard attention | opset 17; split encoder/decoder (done) |
+| `grid_sample` in the decoder | LOW — supported opset ≥16 / ORT 1.20.1 | verified op availability; `--verify` ORT round-trip in the script |
+| DINO ViT export | LOW — HF `ViTModel` exports routinely | part of the encoder graph |
+| Decoder grid-query cost | MEDIUM (perf, not feasibility) | chunk `P`; offer res-128 preview tier; coarse early-out (slice B) |
+| Grid axis/order mismatch | LOW | contract documented above; export script prints reference ordering |
+| Windows MinGW | KNOWN — ORT archive is MSVC-built | `ENABLE_ONNX` stays OFF on MinGW; feature degrades to "rebuild with -DENABLE_ONNX" (same as #404) |
+| Model not hosted yet | EXPECTED | clean "TripoSR model not yet hosted" state (slice E hosts it) |
+
+---
+
+## Go/No-Go
+
+**GO.** Both epic-level unknowns are retired:
+1. **Marching cubes exists and is correct** — native, permissive, watertight on
+   synthetic SDFs, unit-tested.
+2. **The network exports to ONNX cleanly** — a clean encoder/decoder split with no
+   autoregressive decode and only `grid_sample` as a notable op (supported).
+
+Proceed to slice B (#766 `MeshGenPredictor`) against the tensor contract above,
+then C (#767 mesh build/export), D (#768 CLI/MCP/GUI), E (#769 hosting/packaging).
+No re-scope required.