Epic: AI — Local-AI-assisted 3D workflows

[fernandotonon]

Overview

Add a coherent set of local, self-contained AI/automation features to QtMeshEditor that meaningfully improve common 3D content tasks (LOD, retopo, UVs, skinning, texturing, PBR map authoring, motion). The goal is to keep the existing offline-first design — no cloud calls, no Python runtime — by reusing the patterns already in the codebase (SDManager/SDWorker for sd.cpp, LLMManager for llama.cpp, the CLI/MCP/QML triad) and adding a small number of well-chosen native C++ libraries plus an ONNX Runtime backend for narrow ML tasks.

This epic groups every child issue under the ai-assist label and the AI: title prefix so it is clearly separated from the rest of the roadmap.

Why these tasks (and not full 3D-asset generation)

Full text-to-mesh generation (TripoSR, Hunyuan3D, etc.) is PyTorch-only today — there is no mature triposr.cpp/shape.cpp analogous to sd.cpp. Porting one is a multi-week effort with uncertain quality.

By contrast, the tasks in this epic are already solved or nearly solved by native C++ libraries (meshoptimizer, xatlas, Instant Meshes, libigl, Pinocchio) or have small ONNX-exportable models that run fast on CPU/CoreML on the user's machine. Each task delivers a real UX win for indie game devs without inventing new ML infrastructure.

Architecture

A new singleton, AIAssistManager (mirrors SDManager/LLMManager), owns:

• A worker thread (AIAssistWorker) for any blocking ML/algorithmic work.
• A unified ONNX Runtime context (loaded lazily, CoreML EP on macOS, CPU EP elsewhere) reused across child features that need it.
• Progress + completion signals consumed by QML inspector panels and the MCP server.
• Model storage in <AppData>/ai_models/ matching existing conventions.

Every feature must ship across the three surfaces this project already supports:

1. GUI — QML inspector entry (mode-aware where it fits the active editor mode work in Redesign editor UI around mode-based workflows and contextual panels #391).
2. CLI — qtmesh subcommand or flag (e.g. qtmesh lod --algo meshopt, qtmesh skin --algo biharmonic).
3. MCP — JSON-RPC tool exposed by MCPServer.cpp.

This parity is non-negotiable and matches the slice-E PBR preset work that just landed.

Dependencies (to be added once, shared across child issues)

Dep	Why	License	Approx size
meshoptimizer	LOD, vertex cache opt, simplification	MIT	~200 KB
xatlas	Auto-UV unwrap	MIT	~150 KB
Instant Meshes	Quad retopology	BSD	bundled source
libigl (header-only subset)	Biharmonic skinning weights, mesh utils	MPL2	header-only
Pinocchio (auto-rigger)	Skeleton placement v1	MIT-ish	~small C++ source, vendored
ONNX Runtime	RigNet, DeepBump, motion in-betweening	MIT	~30 MB binary
Real-ESRGAN ncnn	Texture upscaling	BSD	already-known port

CMake options (-DENABLE_AI_ASSIST=ON default ON, -DENABLE_ONNX=ON default OFF until first ONNX feature lands) must keep all of this opt-in at build time so CI/Docker images can keep current footprint.

Child Issues

Listed in value-to-effort order.

Tier 1 — Native C++, no ML, ship within days each

• AI: meshoptimizer-backed LOD generation #398 — AI: meshoptimizer-backed LOD generation
• AI: vertex cache + overdraw + fetch optimization on export #399 — AI: vertex cache + overdraw + fetch optimization on export
• AI: xatlas auto-UV unwrap (qtmesh uv --unwrap) #400 — AI: xatlas auto-UV unwrap (qtmesh uv --unwrap)
• AI: Instant Meshes quad retopology (qtmesh retopo) #401 — AI: Instant Meshes quad retopology (qtmesh retopo)
• AI: libigl bounded biharmonic skinning weights #402 — AI: libigl bounded biharmonic skinning weights

Tier 2 — Leverage existing sd.cpp / LLM infrastructure

• AI: Mesh-aware texture generation (depth-conditioned sd.cpp) #403 — AI: Mesh-aware texture generation (depth-conditioned sd.cpp)
• AI: PBR map synthesis from albedo (DeepBump-style, ONNX) #404 — AI: PBR map synthesis from albedo (DeepBump-style, ONNX) — lands ONNX Runtime infra
• AI: Real-ESRGAN texture upscaling #405 — AI: Real-ESRGAN texture upscaling
• AI: LLM-assisted material preset prompts #406 — AI: LLM-assisted material preset prompts

Tier 3 — ONNX Runtime + small models

• AI: Pinocchio auto-rigging (classical, native C++) #407 — AI: Pinocchio auto-rigging (classical, native C++)
• AI: RigNet auto-rigging (ONNX, ML) #408 — AI: RigNet auto-rigging (ONNX, ML)
• AI: Animation in-betweening (ONNX) #409 — AI: Animation in-betweening (ONNX)
• AI: Mesh segmentation / part labeling (ONNX) #410 — AI: Mesh segmentation / part labeling (ONNX)

Tier 4 — Research/ambitious

• AI: Text-to-motion via MDM (ONNX) — research/ambitious #411 — AI: Text-to-motion via MDM (ONNX) — research/ambitious

Explicitly out of scope

• Text/image-to-mesh generation (TripoSR/Hunyuan3D/Trellis). No mature C++ port exists. Will be reconsidered once a *.cpp analogue ships in the open-source ecosystem. A subprocess-based Python integration is not acceptable for this epic — it breaks the offline, single-binary distribution that QtMeshEditor ships today.
• Cloud-API-backed features. Everything in this epic must run locally.

Acceptance Criteria (epic-level)

• AIAssistManager skeleton merged with worker-thread + signal pattern matching SDManager.
• ENABLE_AI_ASSIST and ENABLE_ONNX CMake options exist and keep all new deps opt-in.
• At least all Tier 1 child issues closed.
• At least one Tier 2 feature shipped (target: mesh-aware texturing).
• Every shipped feature has GUI + CLI + MCP parity.
• Every shipped feature emits Sentry breadcrumbs (ai.assist.* category).
• Every shipped feature has unit tests guarded for headless CI.
• Documentation in CLAUDE.md updated under a new "AI Assist" section.

Notes for implementers

• Reuse existing patterns; do not invent a new threading model, settings system, or model-download flow — SDManager/LLMManager already have working ones.
• Model files belong in <AppData>/ai_models/<feature>/ and must be downloadable on demand (mirror ModelDownloader).
• ONNX Runtime should be wrapped in a tiny internal facade so individual features don't each link directly against the full API surface.
• Keep CLI flags consistent with existing subcommands (--json, --verbose, --no-telemetry).
• All MCP tools must follow the existing JSON-RPC 2.0 conventions and run on the main thread via QSocketNotifier (no BlockingQueuedConnection).