Skip to content

mtmd: add unlimited-ocr (converter, full MHA) - #24969

Merged
ngxson merged 1 commit into
ggml-org:masterfrom
sfallah:sf/unlimited-ocr
Jun 24, 2026
Merged

mtmd: add unlimited-ocr (converter, full MHA)#24969
ngxson merged 1 commit into
ggml-org:masterfrom
sfallah:sf/unlimited-ocr

Conversation

@sfallah

@sfallah sfallah commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Overview

  • Adds baidu/Unlimited-OCR to mtmd: converts and runs the model in llama.cpp.
  • Converter-only, no C++ changes.
  • Registers UnlimitedOCRForCausalLM (text + mmproj). Writes the decoder sliding_window
    to the GGUF as metadata; the decoder runs full MHA and ignores it for now.
  • Parity test: single-page vs the HF reference (transformers 4.46.3), full MHA, within tolerance.

Scope: conversion + full-MHA inference. Follow-up PR (stacked on this one) implements R-SWA in the decoder.

How to run

GGUF models: sabafallah/Unlimited-OCR-GGUF

build/bin/llama-mtmd-cli -hf sabafallah/Unlimited-OCR-GGUF:bf16 \
  --image tools/mtmd/test-1.jpeg -p "document parsing." \
  --chat-template deepseek-ocr \
  --temp 0 --flash-attn off --no-warmup \
  -n 4096 -c 16384 \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 35 \
  --dry-penalty-last-n 128 --dry-sequence-breaker none

Additional information

  • Unlimited-OCR is trained with R-SWA (Reference Sliding Window Attention). This PR runs
    the decoder as full MHA.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - I used AI assistance for code review, debugging, implementation checks, and testing. I have reviewed the submitted changes and take responsibility for the full contents of this PR.

@sfallah
sfallah requested review from a team and CISC as code owners June 24, 2026 11:11
@github-actions github-actions Bot added examples python python script changes labels Jun 24, 2026
@ngxson
ngxson merged commit 894bb27 into ggml-org:master Jun 24, 2026
5 checks passed
Geminihaha pushed a commit to Geminihaha/llama.cpp that referenced this pull request Jun 25, 2026
papamoose pushed a commit to papamoose/llama.cpp that referenced this pull request Jun 27, 2026
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
baylitoo added a commit to baylitoo/slm_benchmark that referenced this pull request Aug 4, 2026
* feat(detect): map unlimited-ocr / deepseek-ocr archs to the vision_ocr family

Unlimited-OCR is the DeepSeek-OCR architecture; its GGUF general.architecture
is "unlimited-ocr" (llama.cpp mtmd support in ggml-org/llama.cpp#24969, merged
2026-06-24). Add unlimited-ocr / deepseek-ocr / deepseek2-ocr → vision_ocr so
the inspect verdict reports "supported" (deploy now) instead of needs_family.
Requires a llama-server built after that merge — a serving image rebuild.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(detect): honest runtime-support caveat on the inspect verdict

"supported" only means a family contract exists — the node's llama-server
must also be new enough to LOAD the arch, a separate gate. resolve_family now
carries a runtime_note for archs whose serving support landed recently
(unlimited-ocr / deepseek-ocr → "needs a llama-server built after
llama.cpp#24969; rebuild the serving image if it won't load"), inspect_repo
surfaces it, and the HF search panel shows it as an amber caveat under the
verdict. A user isn't misled into a deploy that fails at load on an old build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
baylitoo added a commit to baylitoo/slm_benchmark that referenced this pull request Aug 4, 2026
…ible) (#87)

The llama-builder stage clones+builds llama.cpp in ONE cached RUN, so a plain
`docker compose build serving` reuses the old llama-server even after upstream
HEAD moves — a new arch (Unlimited-OCR / deepseek-ocr, ggml-org/llama.cpp#24969)
never gets picked up. New optional LLAMA_REF build arg: set a commit/tag to
shallow-fetch and build llama.cpp at that pin — the changed ARG busts the layer
cache (forcing a rebuild) AND makes the build reproducible. Empty = current
behavior (latest HEAD). Wired on the serving service (it holds the llama-server
that serves deployed models) via ${LLAMA_REF:-} and documented in .env.example.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python python script changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants