Skip to content

AI: PBR map synthesis from albedo (DeepBump-style, ONNX) #404

Description

@fernandotonon

Parent epic: #397

Goal

Predict normal, roughness, and height maps from a single diffuse texture using a small ONNX model (DeepBump-style UNet, ~5–20 MB).

Why

Material authoring in QtMeshEditor currently leans on the slice-E PBR presets, but if a user provides only a diffuse (or generates one via SDManager), they still have to find or hand-author the supporting PBR maps. This task closes that loop.

Scope

  • Land ONNX Runtime as a CMake dependency behind ENABLE_ONNX (off by default, on for our release builds). This is the first ONNX consumer; the wrapper facade lives in AIAssistManager.
  • Vendor a permissively-licensed small UNet (DeepBump or equivalent) and host it via the existing ModelDownloader flow.
  • CoreML execution provider on macOS, CPU EP elsewhere. CUDA EP can come later.
  • GUI: in the Material Editor, a "Generate PBR maps from diffuse" button on each texture slot.
  • CLI: qtmesh material --texture diffuse.png --generate-pbr -o material.material writes normal/roughness/height alongside.
  • MCP: generate_pbr_maps tool.
  • Cache outputs next to the source diffuse; re-run on demand.

Acceptance Criteria

  • Given a diffuse texture, produces plausible normal, roughness, and height maps.
  • Generated maps wire into the existing slice-E preset templates correctly.
  • First-run downloads the model with progress and offline-fails gracefully.
  • CLI and MCP parity.
  • Sentry breadcrumb ai.assist.pbr_synth recorded.

Effort

~5–7 days, including landing the ONNX Runtime infrastructure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-assistLocal-AI-assisted 3D workflows (epic prefix: AI:)enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions