Streamable chat parsing by Rocketknight1 · Pull Request #45847 · huggingface/transformers

Rocketknight1 · 2026-05-08T15:19:40Z

Work in progress PR for streamable chat parsing. This introduces a new improved kind of response schema, which I'm calling response_template. Response templates are a lot easier to write, since they avoid most of the messy regexes of the older system. More importantly, though, using a region-based system means that model authors can write templates that enable streaming parsing, emitting fields as they're ready, rather than needing the whole message to be ready at once.

TODO:

Merge/cleanup content types
Can we merge coerce into content types?
Correct handling of tokens/strings inputs, batched lists, etc.
Final docs cleanup, ensure examples match API
Lots and lots of general testing
Handling of multiple tool calls - do we need to emit arrays differently?
Explicit option for "multiple literals" as a match without needing a regex, avoids regex buffer
Properly document the weird content args

HuggingFaceDocBuilderDev · 2026-05-08T15:34:25Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Rocketknight1 · 2026-05-15T16:24:43Z

This should now be ready for review! It's a lot, but thankfully I think you can skim a lot of it. It's also a little ugly right now, because we need to maintain legacy support for the older parse_response() method, but once this is in we can begin deprecating that very quickly. That cleanup, and support for batched inputs, will come in a follow-up PR (batched handling will also be needlessly messy because of the legacy code right now, but it'll be much cleaner to add once it's removed)

The main things to review are:

The doc
The public-facing changes in response_parser.py, processing_utils.py and tokenizer_utils.py
The test coverage

Most other files are agent-written boilerplate changes to support the new API (the pipeline + Gemma conversion script changes), or agent-written internal mechanics that are mostly dense and should be well-covered by tests (content_parsers.py, response_templates.py and the second half of response_parser.py)

pcuenca

Quick look at the docs + gemma 4 testing.

pcuenca · 2026-05-18T11:47:36Z

+The `json` parser also accepts dialect arguments (`unquoted_keys`, `string_delims`) for models that emit JSON with cute
+quirks that completely break the standard parser. The model authors who are responsible for this being necessary
+know who they are and should feel an appropriate amount of shame.


This comment should never be resolved. LOVE IT. 😂

Each schema becomes a JSON file under response_parser/schemas/, loaded once at import. schemas.py shrinks to a thin loader plus a comment explaining why this code exists at all: llama.cpp's chat-auto-parser introspects the GGUF chat template at runtime so per-family schemas disappear, but that module lives in llama.cpp's common/ tree and is not reachable from llama-cpp-python today. When a binding appears the whole response_parser/ package gets deleted in one commit. transformers pinned to >=5.8.1,<5.9 because HF is mid-refactor of the schema DSL (PR huggingface/transformers#45847 renames response_schema to response_template with a new shape). A casual minor bump would break extraction.

pcuenca · 2026-05-19T14:10:44Z

-                        },
-                    },
-                },
+            "open_pattern": r"<\|tool_call>call:(?P<name>\w+)",


Note this is Python-only syntax for named groups, so we'd need a translation layer to ICU somewhere for downstream implementations, cc @xenova. Perhaps we should accept both in transformers and document ICU as the standard?

Ah, I wasn't aware! Is there a universal way to do it, or is there no universal way to make named groups work? I might even prefer unnamed groups and putting the names in a separate field rather than causing cross-language problems

I didn't find a universal way to do it, unfortunately. This is Python and this is ICU (screenshots with the relevant fragments below).

Yeah, perhaps unnamed groups is the best option if it doesn't make things too confusing.

Hmmn, it might be possible to handle this universally by just having other language implementations strip the <> angle brackets if their regex engine treats them as part of the name!

The main problem is the P, the regex has to be rewritten without.

Ah, I'm sorry, I misread that bit! I'll update the docs

pcuenca · 2026-05-20T14:44:43Z

+    "close": "<|END_ACTION|>",
+    "content": "json",
+    "transform_each": True,
+    "transform": {"type": "function", "function": {"name": "{tool_name}", "arguments": "{parameters}"}},


Perhaps we could show a fragment of a rendered Cohere tool call so readers understand it contains tool_name and parameters.

Added a little example to clarify!

Rocketknight1 · 2026-06-09T14:48:40Z

I think we've gotten all the reviews we're going to get! Pinging core maintainers @ArthurZucker @Cyrilvallez @vasqu if any of you wants a veto before I merge this 😅

ArthurZucker

TY!
IDK how much we can trust AI for the core part, I don't have the time to deep dive into what it's doing. The format is the most important, we can improve what's inside later on!

Examples are really important when defining the standards imo! Let's make sure we add more when explaining.

"content": "text", appears a lot, could be default in lots of cases no?
But that's a nit.
Format wise I like json, I think it should be alright to express stuff. Cross pateform is super important so it helps. !

ArthurZucker · 2026-06-18T14:37:06Z

+A field should have **either** `open` or `open_pattern`, but not both, and the same is true for `close` and `close_pattern`.
+
+A field with **neither** `open` nor `open_pattern` is the **implicit** field: it's active whenever no explicit
+region is open, so it captures leftover text. At most one field can be implicit. This is most often used when `content`
+does not have special token tags, it's just written as plaintext after the other fields.
+
+In addition to opening and closing delimiters, you can also specify `repeats`, which indicates that the field is a list
+and the delimiters can match multiple times. This is most common for parallel tool calling, when a model emits
+multiple tool calls simultaneously.
+
+Finally, you can specify `optional: false` for fields that must be present. If parsing finishes and a non optional field
+was never opened, we raise an error instead of silently omitting the field.
+
+The end of generation will close and finalize any open regions, even if their closing delimiter was not seen.


small examples here again can help otherwise its hard to visualize

Scattered a few small examples here!

ArthurZucker · 2026-06-18T14:38:24Z

+type conversion if required. They do not have any `content_args`, except for `text` which supports the arg `strip`,
+which strips whitespace from the start and end of the captured text, and defaults to `true`.
+
+#### json


i would put small example for each type here

ArthurZucker · 2026-06-18T14:39:50Z

+    "content": "json",
+    "transform": {"type": "function", "function": "{content}"},
+},
+```


i would show again how that would render

ArthurZucker · 2026-06-18T14:40:47Z

+    "content": "json",
+    "transform": {"type": "function", "function": {"name": "{name}", "arguments": "{content}"}},
+},
+```


again would like to see example!!!

like out -> becomes what with this

ArthurZucker · 2026-06-18T14:44:28Z

+_LAX_OPEN, _LAX_CLOSE = "\x01", "\x02"
+
+
+def _json(text: str, args: dict) -> Any:


if anything is vendored let's credit !

None of this was copied from elsewhere, I think! A lot of it was agent-written because it was deep internals

ArthurZucker · 2026-06-18T14:45:06Z

+    return out
+
+
+CONTENT_PARSERS = {


can people easily register custom parsers?

There isn't a facility for that right now! Custom parsers introduce remote code execution issues, and the range of parsers we have right now covers every case I know of. I think if a new model needs a new parser, we should just accept a PR to add a new parser to transformers, or to expand the args for an existing parser to support the new model. But hopefully that should be quite rare.

ArthurZucker · 2026-06-18T14:50:18Z

+            reloaded = AutoTokenizer.from_pretrained(tmpdir)
+        self.assertEqual(reloaded.response_template, ernie_template)
+
+    def test_tokenizer_parse_response(self):


these are very nice!

vasqu

Sorry for the delays, also some comments from my side. My only real concern is the batching capability being a limitation now - I don't expect the stream based parsing to be fully async but it would be nice if we had some simple version for batches 🤔

vasqu · 2026-06-18T20:00:20Z

So we are starting with gemma4 as the first one?

Yeah, Gemma4 is the only model that has the old-style response_schema. I'm hoping for a very, very short deprecation period where we merge this PR, then move Gemma4 to a new response_template, then totally deprecate/remove the old code.

vasqu · 2026-06-18T20:40:50Z

+            task="text-generation",
+            model="hf-internal-testing/tiny-gpt2-with-chatml-template",
+        )
+        text_generator.tokenizer.chat_template = (


Just fmi, are overriding the templates to simplify or why is it necessary?

In this case I'm overriding the template to prefill part of the assistant message, to check that parsing still works!

vasqu · 2026-06-18T20:43:16Z


-    def test_batched_inputs(self):
+    def test_batched_inputs_raise(self):
+        # Batched parsing is no longer supported: `parse_response` accepts a single sequence only


That is a huge limitation imo. We really should aim to enable batching 🤔 maybe not with streaming at the same time (async) but at least in a simpler variation where we can safely process the complete text

This is very temporary! I was basically trying to avoid code clutter while we deprecate the old response_schema. After it's gone, clean batching is much simpler. However, if you prefer I can try to implement batching for both, do the deprecation, then simplify after we deprecate

Yea I think, this is the biggest point to me. The rest are rather small issues or nits. But it really would be nice to support batching from the get-go even if it comes with some additional overhead at the beginning

vasqu · 2026-06-18T20:45:33Z

They are currently not integrated into CI ig? Should we add them to slow ci? Just unsure how long these would take 👀

In general no complaints here, solid integration tests

I think the CI is running these tests already! I think anything in tests/utils/* ends up in tests_non_model

vasqu

Ok, I think this looks pretty ready. We can push for efficiency in follow up PRs imo (meaning for batching) but I think this is probably still gonna see a lot of changes regardless. Maybe we should note in the docs that this is experimental / in development so things are subjected to change?

My points are smaller in nature 🤗 nice job!

vasqu · 2026-06-22T17:40:03Z

+The model authors responsible for the existence of `unquoted_keys` and `string_delims` know who
+they are and should feel an appropriate amount of shame.


😂 but we should refrain from formulating it as this lol

vasqu · 2026-06-22T17:46:55Z

+parsed JSON dict slots in directly as the value of `function`. A placeholder must be the entire string: mixing
+text and placeholders (`"abc {name} def"`) is not permitted. They're not f-strings!
+
+You can abuse `transform` quite a lot, which becomes necessary when the model output has a wildly different format


wording :D abusing -> versatile

vasqu · 2026-06-22T17:48:59Z

+model authors, this shouldn't be a problem, but if you are a developer writing an implementation of response parsing
+in another language, you should be aware of our implementation details. This section is dedicated to everyone
+who had to implement an entire Jinja parser to get non-Python chat templating to work - we hope that if you
+follow the simple guidelines below, then response templates should be much less painful:


nit: do we want to wrap this in a details / summary section

I think it's okay down at the bottom under the 'for developers' block, but we can change later if it's annoying!

vasqu · 2026-06-22T17:51:55Z

        # Decode inputs and outputs the same way to remove input text from generated text if present
        skip_special_tokens = skip_special_tokens if skip_special_tokens is not None else True
-        if getattr(self.tokenizer, "response_schema", False):
+        if getattr(self.tokenizer, "response_template", None) or getattr(self.tokenizer, "response_schema", None):


not really related to this PR but we should probably refactor these common parts out at some point 🤔 because we copy paste this through anything generation related

vasqu · 2026-06-22T18:30:06Z

+                )
+        else:
+            # Explicit schema argument: detect new-style by the presence of a top-level `fields` key.
+            use_new_template = isinstance(schema, dict) and "fields" in schema


hmm, seems a bit brittle as check no?

vasqu · 2026-06-22T18:31:08Z

+            if isinstance(prefix, str):
+                prefix_texts, prefix_batched = [prefix], False
+            elif isinstance(prefix, (list, tuple)) and (not prefix or isinstance(prefix[0], str)):
+                prefix_texts, prefix_batched = list(prefix), True
+            else:
+                decoded = self.decode(prefix)
+                prefix_texts, prefix_batched = ([decoded], False) if isinstance(decoded, str) else (decoded, True)
+            if not prefix_batched:
+                prefixes = prefix_texts * len(responses)  # broadcast the single prefix to every response


its the same as for responses nit maybe to refactor a bit? but tbh very big nit

vasqu · 2026-06-22T18:32:12Z

+        spec's `start_anchor`). Generated chunks fed via `stream.feed()` are then classified
+        correctly even when the chat template emitted assistant-turn content (e.g., `<think>\\n`)
+        that the model continues from."""
+        tmpl = response_template if response_template is not None else getattr(self, "response_template", None)


vasqu · 2026-06-22T18:33:49Z

+            parsed = [_template_parse_response(text, schema, prefix=pfx) for text, pfx in zip(responses, prefixes)]
+        else:
+            parsed = [recursive_parse(text, schema) for text in responses]
+        return parsed if batched else parsed[0]


ok yea it's a foreach as we heavily rely on the stream parser. idk how chat templates are handled currently but let's aim to make this efficient in follow up PRs, e.g. some multi threading pool, async streamer etc

I think chat templates are also just a foreach because there is no parallel Jinja! I'm hopeful that the region matching is pretty fast though, especially when run on a whole message at once.

But yeah if performance is annoying later we can definitely refactor to speed it up a lot without breaking BC

vasqu · 2026-06-22T18:35:35Z

+            "content": "text",
+        },
+    },
+}


you prolly already thought of this but let's make these be loaded with auto tokenizer when we upload the templates one by one

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

github-actions · 2026-06-24T13:40:57Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: gemma4

Rocketknight1 · 2026-06-24T13:53:03Z

I think most of the review comments have now been addressed, going to merge! There are a few nits in docs or performance that we can move to follow-up work but nothing that will break BC, so I think merging is safe at this point!

* scaffolding * Let's do an initial commit * Let's do an initial commit * More cleanup * More cleanup * Add the prefill * One more stash commit * One more stash commit * Renames/docstring cleanup * More renames/docstring cleanup * More renames/docstring cleanup * Docstring cleanup * make fix-repo * Doc cleanup * Doc cleanup * make fix-repo * No more coerce, merge content types * Cleanup: Support tokens/arrays/lists correctly for input and prefix * Add prefix to docs too * Better initial_events handling, better region handling * Allow multiple literal matches * Allow emitting dirty regions * Doc updates * Drop `assemble` to just use `transform` * Lots more doc updates * Lots more doc updates * Remove batched handling, lots of cleanup * Use the new method in test_serve * Small doc update * Update docs/source/en/chat_response_parsing.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Update docs/source/en/chat_response_parsing.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Update docs/source/en/chat_response_parsing.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Update src/transformers/models/gemma4/convert_gemma4_weights.py Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Mention the extra * Fix the example stream so it doesn't refer to the previous example * Remove the `strip` arg for numerical/bool types because it was meaningless * Fix convert script bracket bug * Add a regex compatibility section * Update regex compatibility section * jmespath: defeated * jmespath: defeated * Flesh out the Cohere example * Update the docs with proper content_args * Make start_anchor mandatory * Update gemma4 template! * Proper buffering * Corrected prefix handling * Type fixes * make fix-repo * Pipeline updates, batching changes * Post-rebase fixes * Tiny reorder on the version check * Fix licences + no variable abbreviations * Cleanups, remove assertion * Refactor load_response_template * Lots of little doc examples * Fix docs to explain prefix a bit better * Add batching * Add batching * Docs cleanup * Docs cleanup, big block of vasqu comments * Apply suggestions from code review Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * More comment fixes --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

Rocketknight1 marked this pull request as ready for review May 8, 2026 15:21

Rocketknight1 force-pushed the streamable_chat_parsing branch 4 times, most recently from 9ec6ddc to fd4f211 Compare May 15, 2026 15:12

SunMarc mentioned this pull request May 18, 2026

[serve] Support for reasoning #45690

Merged

pcuenca reviewed May 18, 2026

View reviewed changes

Rocketknight1 force-pushed the streamable_chat_parsing branch from 3b0b423 to 3bd988b Compare May 19, 2026 13:12

pcuenca reviewed May 19, 2026

View reviewed changes

qgallouedec mentioned this pull request May 19, 2026

Integrate the new response parsing API huggingface/trl#5791

Merged

8 tasks

pcuenca mentioned this pull request May 19, 2026

Experimental: Response Parser 🔥 huggingface/swift-transformers#362

Open

pcuenca reviewed May 19, 2026

View reviewed changes

Comment thread docs/source/en/chat_response_parsing.md Outdated

pcuenca reviewed May 20, 2026

View reviewed changes

Rocketknight1 force-pushed the streamable_chat_parsing branch 2 times, most recently from 189e804 to 1f8619e Compare May 28, 2026 14:56

SunMarc mentioned this pull request May 28, 2026

Update serve chat parsing #46267

Merged

Rocketknight1 force-pushed the streamable_chat_parsing branch 2 times, most recently from dbae352 to 5210577 Compare June 2, 2026 13:28

Rocketknight1 force-pushed the streamable_chat_parsing branch from 01b9c65 to 462064d Compare June 9, 2026 14:45

Rocketknight1 mentioned this pull request Jun 12, 2026

serve: Gemma 4 non-thinking responses returned as reasoning_content with empty content #46561

Closed

Rocketknight1 added 6 commits June 15, 2026 12:26

scaffolding

714d7e8

Let's do an initial commit

4332521

Let's do an initial commit

85c6cc4

More cleanup

0d6272b

More cleanup

7b793c5

Add the prefill

6aec091

Rocketknight1 added 4 commits June 15, 2026 12:26

make fix-repo

c8aec83

Pipeline updates, batching changes

5e1e347

Post-rebase fixes

73f3d3a

Tiny reorder on the version check

fdfc1d4

Rocketknight1 force-pushed the streamable_chat_parsing branch from b49a3a6 to fdfc1d4 Compare June 15, 2026 11:26

ArthurZucker approved these changes Jun 18, 2026

View reviewed changes

vasqu reviewed Jun 18, 2026

View reviewed changes

Rocketknight1 added 7 commits June 19, 2026 16:51

Fix licences + no variable abbreviations

8e2bed7

Cleanups, remove assertion

4b3ce3a

Refactor load_response_template

7936bb1

Lots of little doc examples

bc6a303

Fix docs to explain prefix a bit better

e3333ae

Add batching

19062d8

Add batching

6d1bdf9

vasqu approved these changes Jun 22, 2026

View reviewed changes

Rocketknight1 and others added 4 commits June 22, 2026 20:27

Docs cleanup

d2d5652

Docs cleanup, big block of vasqu comments

7906572

Apply suggestions from code review

315974f

Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

More comment fixes

a4dfd0b

Rocketknight1 enabled auto-merge June 24, 2026 13:53

Rocketknight1 added this pull request to the merge queue Jun 24, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 24, 2026

Rocketknight1 added this pull request to the merge queue Jun 24, 2026

Merged via the queue into main with commit 4ab0479 Jun 24, 2026
101 checks passed

Rocketknight1 deleted the streamable_chat_parsing branch June 24, 2026 14:42

qgallouedec mentioned this pull request Jul 1, 2026

Fix add_response_schema tests for the new parse_response prefix requirement huggingface/trl#6236

Merged

		_LAX_OPEN, _LAX_CLOSE = "\x01", "\x02"


		def _json(text: str, args: dict) -> Any:

		The model authors responsible for the existence of `unquoted_keys` and `string_delims` know who
		they are and should feel an appropriate amount of shame.

Uh oh!

Conversation

Rocketknight1 commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented May 8, 2026

Uh oh!

Rocketknight1 commented May 15, 2026

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rocketknight1 commented Jun 9, 2026

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Rocketknight1 commented May 8, 2026 •

edited

Loading