Blog: Getting structured data out of images with Granite Vision 4.1

[planetf1]

Closes #60

What this is showing off

Vision models return prose. The point of this post is that they don't have to.

The blog demonstrates Mellea's extraction pattern — pass a format= Pydantic model to
m.instruct(), get a typed Python object back instead of a string. No JSON prompt
engineering, no json.JSONDecodeError handlers, no post-processing regex. The return type
is the contract, and constrained decoding enforces it.

It then builds up two layers of validation on top:

• requirements= — plain-English semantic constraints (date format, positive totals).
  The model retries with the failed requirement injected into the repair prompt.
• IVR validation_fn — programmatic arithmetic check (line items × quantities = subtotal).
  The failure reason gets fed back into the repair prompt verbatim.

The receipt image is a clean, synthetically-rendered (PIL) receipt with a grey smudge box
placed over part of the subtotal digit, giving the validation layers something concrete to catch.
Reviewer note: because it's a clean digital render rather than a photographed receipt (no skew,
no paper texture, no lighting), the demo is a light test of the vision model's visual robustness —
it mainly exercises the extraction/validation pipeline, not OCR under real-world noise. The
pipeline and its value proposition are real either way; readers wanting a harder vision test
would need a photographed source image instead.

Demo sticks to Ollama throughout — no mlx-vlm/vLLM setup needed to follow along.

---

Status: Ready for review — targeting publish 2026-07-28

All three original blockers are resolved:

1. Ollama ships llama.cpp ≥9630 — Ollama 0.32.4 bundles llama.cpp b10091, well past this.
2. granite-vision-4.1 lands in the Ollama library — the short library name still 404s, but
   this is moot: IBM has published an official pre-quantized GGUF repo,
   ibm-granite/granite-vision-4.1-4b-GGUF,
   and ollama pull hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M is a first-class,
   permanent, supported Ollama feature — not a temporary workaround.
3. Assumed mellea model name — moot; the blog now uses the hf.co/... reference directly
   as model_id everywhere, so there's no naming assumption to confirm.

Code and API usage verified against mellea 0.7.0 (current PyPI latest) and re-run live against
the model above — all four code blocks pass, output matches what's printed in the post.

• Remove the EDITORIAL NOTE block from the blog post
• Flip PR from draft → ready for review
• Ping ajbozarth for final review pass

---

Receipt arithmetic (verified)

Item	Qty	Unit	Line
Cold brew coffee	3	$4.75	$14.25
Grain bowl	2	$12.95	$25.90
Granola bar	4	$2.95	$11.80
Oat milk add-on	3	$0.75	$2.25
Avocado toast	1	$11.50	$11.50
Blueberry muffin	2	$3.95	$7.90
Subtotal			$73.60
Tax (8.5%)			$6.26
Total			$79.86

IVR check: sum(qty × unit_price) = $73.60. Smudged subtotal scenario: model misreads
as e.g. $70.60 → check fires → repair prompt receives "line items sum to 73.60, subtotal shows 70.60".

---

Handover note

• Model path changed: pulling directly from IBM's official GGUF repo on Hugging Face
  (ollama pull hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M) — Ollama auto-pairs the
  mmproj, granite4_vision projector loads correctly. This replaces the old gguf-forge/mlx-vlm
  workaround path entirely; that tool was only ever a stand-in until a real GGUF existed.
• All four code blocks re-verified today against this exact pull, mellea 0.7.0.
• Added model_options={ModelOption.CONTEXT_WINDOW: 4096} to the structured-extraction and
  backend-swap examples — Ollama otherwise defaults to the model's full 131072-token context
  (~9 GB); capping at 4096 drops that to ~2.5 GB with no change in output quality.
• ajbozarth reviewed an earlier version; those comments are addressed.
• I've done final validation/testing directly against this model now.

---

Test plan

• npm run dev — confirm post renders at /blogs/granite-vision-structured-extraction
• Receipt image displays correctly (smudge on subtotal visible)
• Code syntax highlighting correct on all blocks
• Run all four code blocks against hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M

[psschwei]

Don't have a strong opinion here, but assuming it takes a while to get the vision model into Ollama should we consider using vllm in the blog instead?

[ajbozarth]

I'll walk through the blog and try it out myself when I have bandwidth, but to start heres a small review from Claude:

---

Code checks out against current mellea source — APIs, imports, signatures, and the RepairTemplateStrategy switch all verify. Front matter and asset are good. Snippet syntax checks pass; live execution skipped (model not in Ollama yet, per the editorial note). de-llmify score 1.

A few inline notes below. Pre-publish blockers (editorial note removal, Ollama availability) are already tracked in the PR description.

[planetf1]

Don't have a strong opinion here, but assuming it takes a while to get the vision model into Ollama should we consider using vllm in the blog instead?

vllm tends to target more the operational audience rather than developer I think but also doesn't have good macOS support. Whilst there's a fork supporting mlx, it doesn't support vision models. I'd actually suggest sticking with the mlx-vlm workaround if the gguf takes a while for mac. But we could add vllm as the linux workaround. Ollama remains the least friction, so will look into likely timescales.

[planetf1]

Thanks @psschwei @ajbozarth for comments. There are still two areas I'm working on

• the likely timescale for ollama support (sometimes new model releases require changes). If long, fallback could be to add vllm for linux users
• refining the actual scenario - I'm not happy with the run-through currently. Needs a bit more work on the scenario, prompts & mellea constructs. (hence draft - but wanted to share the overall idea and get general review)

[psschwei]

vllm tends to target more the operational audience rather than developer I think but also doesn't have good macOS support. Whilst there's a fork supporting mlx, it doesn't support vision models. I'd actually suggest sticking with the mlx-vlm workaround if the gguf takes a while for mac. But we could add vllm as the linux workaround.

Sorry, I misread your original post, thought I read mlx-vllm. Mostly was just trying to see if ollama is a hard requirement or if serving the model another way would be fine.

Ollama remains the least friction, so will look into likely timescales.

My understanding is that it is being actively worked on, but may take a few weeks (apparently getting vision models working with ollama is not very straightforward).

Though, and not saying we should just posing as an idea, what if we target colab for running the code? The vision model is 4B which might fit on a free instance, and if not would only take a few credits (<$1) to run for an hour. It would also reduce the setup headaches even more (simply click a cell or two and everything is up and running).

[ajbozarth]

The updates addressing my previous review look good, but I'm going to hold off on further review until you've finalized this and moved it out of draft

[psschwei]

closing and reopening to get the DCO bot to fire
edit: didn't work 😕

[planetf1]

granite 4.1 vision support is being worked on -> https://github.com/gabe-l-hart/llama.cpp/tree/Granite4Vision. Thought is to hold this blog until that is merged/released - it will be another nice boost for granite vision given ollama support

[planetf1]

PR for llama.cpp is at ggml-org/llama.cpp#23545

We could update blog to use llama.cpp directly and not wait for ollama to adopt the new version. llama release multiple times per day. lag for ollama is likely 0-4 days.

[psschwei]

I would just wait the extra week for ollama

[gabe-l-hart]

Hey guys! Blog looks great and should hopefully be unblocked soon. With the recent changes for Ollama to depend on llama-server directly, we should be able to get this into Ollama quickly after the llama.cpp PR is merged.

One note:

Watch https://ollama.com/library for granite-vision-4.1. When it lands: remove the
editorial note, verify ollama pull granite-vision-4.1 works, flip to ready.

The naming convention for Ollama models is slightly different from HF, so the landing zone will be https://ollama.com/library/granite4.1-vision (and https://ollama.com/ibm/granite4.1-vision which is the org we can fully control without getting the Ollama team to create the model repo for us).

[gabe-l-hart]

The llama.cpp PR is officially merged! ggml-org/llama.cpp#23545

[planetf1]

Nice — just tested with b9570. Converted to GGUF and ran the blog's receipt extraction against llama-server, works as expected. Waiting on Ollama to pick it up before we publish.

[ajbozarth]

Waiting on Ollama to pick it up before we publish.

Once upstream ollama is ready and you update the draft to match I'll take another pass at review. I'd suggest tentatively setting this to 2026-06-17 is a safe target date IIUC

[gabe-l-hart]

I think the 17th may be ambitious. Ollama is still a few versions behind on llama.cpp, so we'll be blocked on starting the publication until they bump past the point where the llama.cpp support was added. Once done, we'll need to get the conversion pipelines in place and then get the Ollama team to make a placeholder model, all of which can take a while based on their schedules / responsiveness.

[ajbozarth]

I think the 17th may be ambitious. Ollama is still a few versions behind on llama.cpp

That's fair, I didn't look into how long the llama.cpp -> Ollama cycle was

[planetf1]

Quick update: The llama.cpp fix for the n_head=0 divide-by-zero crash (commit b9509) that enables Gemma 4 12B multimodal models is now available in Ollama v0.30.8 (released June 4, 2026). I've verified works correctly and the blog's receipt extraction examples run as expected. Ready to publish once we update the editorial note in the post.

[planetf1]

Quick update: The llama.cpp fix for the n_head=0 divide-by-zero crash (commit b9509) that enables Gemma 4 12B multimodal models is now available in Ollama v0.30.8 (released June 4, 2026).

However we don't yet have a gguf version on hf or in ollama repo. We could document how to do a conversion, but I wonder if we might see a vision gguf pushed to hf @gabe-l-hart ?

[planetf1]

Local testing status — 2026-06-15

Tested the model locally while the Ollama blocker is open.

Method: built an Ollama Modelfile from the Q4_K_M GGUF + mmproj-f16 in the HF cache, then fell back to homebrew llama-server when Ollama failed.

Ollama 0.30.8: can't load the model. The granite4_vision projector type (declared in clip.projector_type in the mmproj GGUF) is not present in Ollama 0.30.8's bundled llama-server binary. The b9509 fix in 0.30.8 addressed a different issue (n_head=0 divide-by-zero crash). granite4_vision projector support landed in llama.cpp build 9630.

Homebrew llama-server 9630: works correctly. Tested the Q4_K_M quant against the receipt image — all 6 line items extracted with correct quantities, unit prices, subtotal ($73.60), and total ($79.86).

So: the blog code and the ollama pull instructions are fine as written. Waiting on Ollama to ship llama.cpp ≥9630.

[planetf1]

Follow-up: full code verification + Q4 provenance note

Ran all four code blocks from the post against llama-server 9630 via uv run:

Block	Result	Notes
Naïve description	✓	Prose, correct items + totals
format=Receipt	✓	All fields populated, total 79.86
requirements=	✓	1 repair triggered (date not ISO on first attempt), passed on second
validation_fn IVR	✓	1 repair triggered (arithmetic off on first attempt), passed on second with delta 0.0000

The IVR repair loop fired in both blocks that use it, which is the core behaviour the post is demonstrating — so that's a useful confirmation, not just a pass/fail.

Q4 provenance: the GGUF I tested against (ibm-granite--granite-vision-4.1-4b.Q4_K_M.gguf) is a local conversion — general.name in the GGUF metadata is a git commit hash, not an IBM release tag. The HF repo currently only publishes full safetensors weights. When the official GGUF lands (either on HF or in the Ollama library), a retest would be worth doing.

No code changes needed. Blocking on Ollama shipping llama.cpp ≥9630.

[psschwei]

FYI: to avoid the DCO failure on bot commits, you can add the following to your settings.json:

  "attribution": {
    "commit": "",
    "pr": ""
  },

[gabe-l-hart]

@planetf1 It looks like the next version of Ollama should support G4V (https://github.com/ollama/ollama/blob/v0.30.9-rc0/LLAMA_CPP_VERSION). Support for G4V was added in b9534, so v0.30.8 was still before that (b9509)

[gabe-l-hart]

For official GGUFs, we're actively working through that. There seems to be a bug in the implementation for 4.0 that @mrutkows and I were trying to debug on Friday. I didn't find it, so we'll proceed with 4.1 and keep digging on 4.0 in the background.

[planetf1]

Thanks @gabe-l-hart - I'll be going with 4.1 anyway, and @psschwei useful tip. added (previously was only in md files so not so rigidly adhered to)

[planetf1]

@gabe-l-hart any news on getting the models on the ollama library?

[ajbozarth]

I did an in depth review after the latest updates and pending the items outlined in the EDITORIAL NOTE this is good to go. In my testing following Nigels gguf-forge steps the model outputted garbage, so we'll want to re-test once the real model is on ollama.

@gabe-l-hart whenever the model is available on ollama if you could ping here and I'll clean this up and do final testing if @planetf1 is still out (he's out until July 16th)

[psschwei]

The official GGUF is on HF.

So in theory we could run the model using something like this:

ollama run hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M

haven't tested outside of that command

[planetf1]

Picking this back up. HF model is finalized, and IBM shipped an official GGUF repo — ollama pull hf.co/ibm-granite/granite-vision-4.1-4b-GGUF:Q4_K_M works directly, no workarounds needed. All 4 code blocks re-verified. Doing final validation now, will ping when ready.

[gabe-l-hart]

There is still a bug in the llama.cpp implementation that we should have a PR up for soon, but the 4.1 model works well most of the time, even without the fix, so we should be able to move forward with this PR to the website.

We haven't pushed an official Ollama model yet, but that's on the TODO list soon. In the meantime, the command from @psschwei to run directly from HF should work now that Ollama cleanly uses embedded jinja2 templates in GGUFs.

[ajbozarth]

I'll do a full review pass today

[ajbozarth]

LGTM feel free to merge tomorrow when you start your day @planetf1

[planetf1]

Thanks for the update @gabe-l-hart and for the approval @ajbozarth - publishing