feat: validated streaming refactor POC + Phase 2 epic breakdown

[ajbozarth]

Pull Request

Issue

Related to #1013

Description

Draft — POC + planning artifacts for review, not a merge candidate.

This PR exists to share a proof-of-concept and a proposed epic breakdown ahead of Phase 2 planning. It is intentionally not a finished feature: the code is a parallel prototype, not wired into the framework, with no test suite or telemetry integration yet. The goal is to give reviewers something concrete to react to before the work is decomposed into issues.

What's here, and why:

• A working POC of the streaming refactor (mellea/stdlib/streaming_poc.py + ModelOutputThunk.__aiter__). It demonstrates that validated streaming can be a single-task async for primitive rather than the current background-task orchestration — validating the core architectural claim behind the proposed refactor. It lives alongside the existing stream_with_chunking rather than replacing it.
• A planning doc (docs/dev/streaming-phase2-plan.md) decomposing the remaining validated-streaming work into separately-shippable items with dependencies and open decisions, plus the origin proposal (docs/dev/streaming-simplification.md) it grew from.
• A runnable verification harness (docs/examples/streaming/validated_streaming_poc.py, # pytest: skip) that exercises the POC end-to-end without Ollama, since nothing else drives the unplugged prototype.

What this PR is not: it does not refactor the existing code, is not API-stable, and may be substantially reworked or discarded as the plan is finalized. Treat the docs as the primary artifact and the code as evidence.

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

POC scope: the prototype is deliberately untested/unintegrated. The verification harness is # pytest: skip (drives an unplugged module, pokes internals). Real tests come when the refactor is picked up as actual work.

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

• Component
• Requirement
• Sampling Strategy
• Tool

[ajbozarth]

cc @psschwei @jakelorocco @planetf1 @nrfulton @HendrikStrobelt this is the POC and design doc that will drive the streaming validation call next Tuesday for early review

[planetf1]

This is a good simplification of the original design — the two-task split isn't needed, since nothing ended up using the independent events() stream it was built to support. The vocabulary (chunking, validation, event types) all carries over; only the concurrency mechanism underneath changes.

A few things worth deciding explicitly ahead of Tuesday rather than leaving open:

1. Item 1/2 gating. Doing item 1 without item 2 leaves a rough edge — on early exit, full_text stops at the last delta instead of the last full chunk, until item 2 merges. Makes sense to do both before either goes in, but as separate PRs rather than one big one, so risk and review stay manageable.

2. Shim vs. hard break. A shim keeps the old orchestrator running alongside the new one — back to the complexity item 1 removes. Given how new the API is, a hard break seems better.

3. A vs. C fork (item 2). Multi-modal matters, so C over A. But item 1 should still just yield plain strings — mix the two changes now and we'll end up breaking things twice.

4. MOT.__aiter__ as a protocol change. __aiter__/__anext__ makes MOT look like a normal async iterator — but it isn't one, since calling it twice just splits the same stream between two callers, not two independent runs. Nothing stops that today beyond a docstring, so item 1 should add a real guard: mark it consumed, raise if a second caller starts.

Making it genuinely support parallel consumers is a bigger job — buffering, per-consumer cursors, deciding what a late joiner sees. Worth checking if anyone actually needs that before building it; otherwise the guard's enough.

[ajbozarth]

Some notes from todays call that will drive the work moving forward:

• Handling events: we will create an example that will demo a events plugin (the plugin could be in the example or src depending on how I write it) showing how it can be used to consume and surface streaming events for multiple streams in parallel
• My Phase 2 plan doc conflated two separate issues: where does chunking belong and stream_parsed_repr
  • stream_parsed_repr is just a version of parsed_repr that supports streaming and is completely separate from chunking and for now) validation
  • where does chunking belong: the doc purposes moving it into the MOT which would be the most straight forward solution based on todays implementation, but is not the direction we intent to take it. Instead chunking with be handled on the requirements side of streaming that the MOT or Streamer.
• Initial plans for chunking, will be included in the refactor work, either as a operate commit, or follow up:
  • We will need to reintroduce the handling for the streaming boundary that exists in the current code, how/where this is handled determines who holds the state
  • Each Requirement will include a ChunkingStrategy instead of Streamer
  • Initially a helper function on Requirement base class will handle chunking as we do currently and will be wired into stream_validate
  • The above imp would have Requirement own the state, this is still an open discussion and this implementation is meant to be an initial pitch before continuing that discussion and might be changed in later itterations
• The EPIC plan details will be put partially on hold until after the refactor is done, but as of now the sub issues look like:
  • the refactor: Streamer, events example, move chunking to Requirement
  • stream_parsed_repr: a parsed_repr for streaming (most likely addresses multi modal as well, not separate)
  • stream_instruct: after discussion rather than a separate function like the existing example this may just be adding an optional sampling param to stream just we do validate

Next steps:

I will expand this POC to complete the refactor and that will drive further discussion on chunk ownership and state. I will also rewrite the EPIC plan pitch to address the above and that will drive a future sync.

If I missed anything or got any details wrong post here and I'll make sure to factor them in.

[ajbozarth]

@psschwei @jakelorocco @planetf1 @nrfulton @HendrikStrobelt heads up I have update the phase 2 epic #1013 and split out subissues based on Tuesdays call. the first sub issue will replace this POC and the reaming sub issues has on-going discussion points spun out of open items from our call if you want to take a look and give input before I get to them