build: include libmtmd in Apple XCFramework (opt-in LLAMA_BUILD_MTMD)

[theabecaster]

The Apple xcframework from build-xcframework.sh is built with LLAMA_BUILD_TOOLS=OFF, so no mtmd_* symbols end up in it. That means Swift / Obj-C consumers on iOS, macOS, tvOS, visionOS can't use vision even though the sources cross-compile fine.

This adds an opt-in LLAMA_BUILD_MTMD option (default OFF). When on and full tools are off, only the tools/mtmd subdirectory gets added so just the library target builds. The CLI exe targets in tools/mtmd/CMakeLists.txt are now gated on LLAMA_BUILD_TOOLS — without that, standalone visionOS xcodebuild fails with install TARGETS given no BUNDLE DESTINATION for MACOSX_BUNDLE executable target 'llama-mtmd-cli'.

build-xcframework.sh turns the option on, adds libmtmd.a to combine_static_libraries, copies mtmd.h and mtmd-helper.h into the framework Headers/ dir, and lists them in the modulemap. The C++ sections in mtmd.h are #ifdef __cplusplus-guarded, so Clang's pure-C module precompile never sees them; Swift consumers import llama and call the mtmd C API directly.

Tested:

• build-xcframework.sh passes on macos-15 for all 8 platform slices
• nm ios-arm64/llama.framework/llama | grep -c ' _mtmd_' → 52
• Swift consumer links against the produced framework and calls mtmd_default_marker, mtmd_context_params_default, mtmd_bitmap_init, mtmd_input_chunks_init on the macOS slice; typechecks on iphoneos / iphonesimulator / xros with the same modulemap
• Legacy path (LLAMA_BUILD_MTMD=OFF or LLAMA_BUILD_TOOLS=ON) unchanged — full-tools build still pulls mtmd in via tools/

Disclosure: I used Claude to help draft the CMake wording on the original version. Design, debugging, and on-device verification are mine.

[ggml-gh-bot]

Hi @theabecaster, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[theabecaster]

Flagging the disclosure line at the bottom of the PR body / commit message — I used Claude for CMake wording, but the investigation and fix are mine.

For context on motivation: I'm building an iOS app that needs vision-capable on-device inference. Linking against the released xcframework failed because the mtmd_* symbols aren't in the binary; tracing it led back to LLAMA_BUILD_TOOLS=OFF in build-xcframework.sh. I worked around it locally with an extern "C" shim, but the root fix belongs upstream so other Apple-platform consumers don't hit the same wall. Happy to iterate on naming, defaults, or split the commit if preferred.

[danbev]

Manually verified using llama-swift-macos.

[ngxson]

Nice, very useful option. Can you have a look? @ggerganov

[ngxson]

I think it's not just isolated to XCFramework, but seems like in my use case, wasm binding also need this.

Probably better to clarify that this can be useful for building llama and mtmd for binding, so it's not just isolated for XCFramework use case

[theabecaster]

Only ggml-ci-win-intel-vulkan is red, and it timed out on test-chat-template, test-thread-safety, and test-arg-parser (1500s ctest cap, 76 min wall). Same job is failing on a bunch of unrelated PRs from the same window — webgpu FA fix, reasoning budget, NVFP4 repost, TP fix — so the Intel Arc runner is just slow.

My change is Apple-only: build-xcframework.sh, the top-level CMakeLists.txt, and tools/mtmd/CMakeLists.txt for an opt-in XCFramework build. Nothing in there is reachable from a Windows Vulkan ctest run.

Can someone retrigger?

[CISC]

Can someone retrigger?

No need, that CI is just broken ATM.

[enriqxxx]

Strong +1 on this — the missing libmtmd in the Apple xcframework is the single
thing keeping our iOS build text-only.

Real-world data point: we already run qwen3-vl-4b doing on-device image
understanding on Android (Snapdragon, with mtmd compiled into the JNI build) and
it works well. The engine is clearly cross-platform — the only reason our iOS app
is still text-only is exactly this: the released Apple xcframework ships libllama
but not libmtmd, so the mtmd_* symbols aren't linkable from Swift.

The opt-in LLAMA_BUILD_MTMD=OFF default here looks clean and non-breaking — would
love to see it land. Happy to test the resulting xcframework on a real iPhone
(Metal) with qwen3-vl + mmproj if that helps move it forward. Is there anything
blocking beyond the rebase/CI?

[enriqxxx]

For a bit more context on viability: our Android build compiles libmtmd directly
into the JNI/CMake build (Snapdragon preset — Adreno OpenCL + Hexagon) and runs
qwen3-vl-4b with its mmproj for real on-device image understanding. So the
cross-platform path is proven end-to-end — iOS is the only target where we can't
link the mtmd_* symbols, because the released Apple xcframework doesn't ship
libmtmd. That's exactly what this PR fixes, and the opt-in flag keeps the default
framework lean for text-only consumers.

[enriqxxx]

@ggerganov Following up from our side (production iOS app, currently text-only solely because the published xcframework ships libllama but not libmtmd — same motivation as my earlier comments, not repeating the Android/qwen3-vl context here).

We're validating the currently shipping head (87b8ed4) directly, since the standing approvals predate the header-copy + umbrella approach. danbev approved commit 21dcc2d ("Manually verified using llama-swift-macos") — that was the shim-era revision, before your "We can just copy the mtmd headers?" note led to dropping the shim. The header copy + umbrella "Headers" modulemap that ships at head landed afterward, so no review on record has build-tested the artifact as it stands today. We're filling exactly that gap: build the 87b8ed4 xcframework with build-xcframework.sh, then from a vanilla Swift target (no C++ interop, no shim) do a bare import llama and call mtmd_default_marker(), mtmd_bitmap_init(...), mtmd_input_chunks_init(), followed by an on-device qwen3-vl-4B + mmproj vision run on iPhone/Metal. I'll attach the build log, the no-shim Swift snippet, and device output/timings once the run completes — flagging now that those numbers are pending, not yet posted.

While doing this I hit one concrete, separable improvement worth folding into this PR:

MTMD_VIDEO defaults ON, so every Apple slice compiles in an ffmpeg/ffprobe subprocess pipeline that can't function at runtime on the six non-macOS slices. tools/mtmd/CMakeLists.txt:3 sets MTMD_VIDEO ON ("requires ffmpeg binary in PATH"), and build-xcframework.sh never overrides it, so all seven slices (the 7 setup_framework_structure calls / 7 -framework entries in the -create-xcframework step) inherit it. That pulls in mtmd-helper.cpp:40 #include "sheredom/subprocess.h" (under #ifdef MTMD_VIDEO), and on every non-Windows target (#if !defined(_WIN32), i.e. all seven Apple slices) subprocess.h:234 includes <spawn.h> and calls posix_spawn/posix_spawnp. It links fine (the build succeeds, consistent with danbev's pass), but on the six sandboxed non-macOS slices (iOS / iOS-sim / visionOS / visionOS-sim / tvOS / tvOS-sim) it can never spawn a binary at runtime — and tvOS even marks posix_spawn unavailable. It's reachable not only via the explicit mtmd_helper_video_* API but as the silent last-resort fallback inside the public mtmd_helper_bitmap_init_from_buf (mtmd-helper.cpp:553-571), so an unrecognized buffer triggers a spawn attempt that always fails. (It's also the kind of process-spawning API that can draw App Store static-analysis attention.) macOS is the one slice where the pipeline could actually work (ffmpeg in PATH). One-line fix, ABI-safe:

 LLAMA_BUILD_SERVER=OFF
+MTMD_VIDEO=OFF
 GGML_METAL=ON

     -DLLAMA_BUILD_MTMD=${LLAMA_BUILD_MTMD}
+    -DMTMD_VIDEO=${MTMD_VIDEO}

Safe because image (stb_image) and audio (miniaudio) decoding are compiled unconditionally and unaffected, and the full mtmd_helper_video_* ABI is still emitted — the definitions live outside the #ifdef, so with MTMD_VIDEO OFF the bodies just degrade to a logged error / GGML_ASSERT (mtmd-helper.cpp:983,1015,1026,1034,1044) and no symbol disappears. If you'd rather keep video on for the macOS slice only, that's easy too — happy to PR whichever you prefer.

One smaller, forward-looking note, not a blocker: the umbrella sweeps in mtmd.h/mtmd-helper.h, whose C++ regions (<map>/<string>/<vector>/<memory> and namespace mtmd, all #ifdef __cplusplus-guarded) land in an un-partitioned framework module llama. The default import llama is unaffected — ClangImporter builds the module as Obj-C with __cplusplus undefined, so those regions are elided (which is why the symbol-resolution tests pass). It'd only matter for a consumer enabling Swift C++ interop or importing from an ObjC++/C++ TU — notably, danbev's own bridge header routes around exactly this with a separate shim ("the mtmd headers contain c++ ... clang [is] unable to import them"). If you'd want that path clean without changing today's default behavior, gating the two headers behind a requires cplusplus submodule is the idiomatic shape:

framework module llama {
    umbrella "Headers"
    exclude header "mtmd.h"
    exclude header "mtmd-helper.h"
    link "c++"
    link framework "Accelerate"
    link framework "Metal"
    link framework "Foundation"
    export *

    explicit module mtmd {
        requires cplusplus
        header "mtmd.h"
        header "mtmd-helper.h"
        export *
    }
}

I haven't reproduced a break with the current modulemap (our consumer uses the C API), so treat this strictly as optional hygiene.

Glad to PR the MTMD_VIDEO=OFF change, share the full per-slice build logs, and post the device-matrix vision results here.

[theabecaster]

what are we missing for this? @ggerganov

[CISC]

This broke Release:
https://github.com/ggml-org/llama.cpp/actions/runs/28149340046/job/83399746540