Skip to content

AI: Mesh-aware texture generation (depth-conditioned sd.cpp) #403

Description

@fernandotonon

Parent epic: #397

Goal

Generate textures that fit the mesh by conditioning sd.cpp on rendered depth + normal maps from the current viewport, then bake the result back to UVs. The single biggest user-visible AI win for the texturing workflow.

Why

The existing SDManager produces flat 2D textures that often don't fit the mesh's UV layout or geometry. Depth/normal-conditioned generation (ControlNet-style) produces textures where lighting and details follow the mesh shape, which is what indie devs actually want.

Scope

  • Render mesh depth + normal maps from the active viewport using Ogre RTT.
  • Feed those into sd.cpp as ControlNet conditioning images (sd.cpp supports ControlNet).
  • Bake the generated texture to the mesh's UVs and assign it to the active material's diffuse slot.
  • GUI: a new "Generate from mesh" action in Material Mode that opens a prompt dialog and shows progress through SDManager's existing progress signal.
  • CLI: qtmesh material model.fbx --generate-texture "rusty bronze armor" -o out.fbx.
  • MCP: generate_mesh_texture tool with prompt, model, controlnet_strength params.
  • Pair with the PBR map synthesis task to optionally produce normal/roughness/height alongside the diffuse.

Acceptance Criteria

  • Generated textures show silhouette and lighting that match the mesh shape.
  • Works on at least one humanoid and one prop test mesh.
  • Progress + cancellation work through the existing SDManager worker pattern.
  • CLI and MCP variants produce the same output as the GUI for the same params.
  • Documented limitation: requires an existing UV layout (suggest running qtmesh uv --unwrap first if needed).
  • Sentry breadcrumb ai.assist.mesh_texture recorded.

Dependencies

Effort

~7 days.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-assistLocal-AI-assisted 3D workflows (epic prefix: AI:)enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions