Add reasoning and step reporting event surfacing - #26
Merged
Conversation
12 tasks
contextablemark
force-pushed
the
feature/reasoning
branch
from
April 26, 2026 02:28
abfeeb1 to
32c66c7
Compare
Emit AG-UI REASONING_* events when the agent streams reasoning content (extended thinking), and STEP_STARTED/STEP_FINISHED events from OpenClaw's onItemEvent callback for multi-step progress. - Bump @ag-ui/core + @ag-ui/encoder to ^0.0.52 for REASONING events - Wire onReasoningStream/onReasoningEnd into replyOptions - Wire onItemEvent for step lifecycle events - Default on, opt-out via surfaceReasoning/surfaceSteps channel defaults - Auto-close open reasoning blocks before RUN_FINISHED Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
contextablemark
force-pushed
the
feature/reasoning
branch
from
April 26, 2026 03:17
32c66c7 to
fc02b7b
Compare
This was referenced Apr 26, 2026
mikehole
added a commit
to Prodigi-Group/clawg-ui
that referenced
this pull request
Jun 5, 2026
…cument, binary)
Upstream currently drops array-form UserMessage.content silently: only string
content is forwarded to the agent, and any message with an image/audio/video/
document/binary attachment falls into the 400 "empty prompt" path. This commit
adds canonical support for every AG-UI multimodal content-part type and wires
extracted attachments into the OpenClaw agent runner's MediaPath* contract.
Spec provenance (@ag-ui/core):
- ImageInputContentSchema, AudioInputContentSchema, VideoInputContentSchema,
DocumentInputContentSchema (added in 0.0.52) all share the same shape:
{ type, source: { type: "data" | "url", value, mimeType? }, metadata? }
- BinaryInputContentSchema (present since 0.0.43) keeps its flat shape:
{ type: "binary", mimeType, data?, url?, filename?, id? }
What changed:
- extractAndSaveAttachments walks user messages, writes every inline-base64
payload (source.value for the source-based types; data or data: URI in
binary.data/binary.url) to os.tmpdir() under clawg-ui-<uuid>.<ext>
- Type-guards isSourcePart / isBinaryPart replace loose casts
- mimeType -> file-extension map covers common image/audio/video/document
mimetypes with a safe fallback
- extractTextContent now flattens array content and appends a categorized
marker ([user attached: 2 images, 1 document]) so the downstream prompt is
non-empty and the LLM has a signal about attachments
- ctxPayload gets MediaPath / MediaUrl / MediaType (single) plus
MediaPaths / MediaUrls / MediaTypes arrays for multi-attachment — same
shape the msteams channel uses
- finally-block cleanup unlinks every temp file regardless of dispatch
success or failure (the one real weakness in the prior image-only
fork code this replaces)
- Request body cap raised from 1 MB to 25 MB for multi-part base64 payloads
Security posture:
- http(s) URLs on source.value or binary.url are rejected with 400
invalid_request_error up-front — deferring SSRF, redirect handling and
mid-stream size enforcement to a follow-up that can design a proper
URL-fetch surface. data: URIs (base64) are accepted.
Does not bump @ag-ui/core (still ^0.0.43). Runtime casts + type-guards cover
the 0.0.52 shapes without a dep bump; the bump is being proposed in PR contextablemark#26
(feature/reasoning) and can land independently.
Tests (17 new cases in src/http-handler.test.ts "Multimodal content parts"):
- Parameterized happy path: image / audio / video / document
- binary with inline data, with data: URI in url, non-image mimetype
- multiple mixed attachments -> MediaPaths arrays in input order
- categorized marker in the body
- http(s) URL rejection (source + binary.url)
- no route / dispatch side-effects when rejected
- attachment parts on non-user roles ignored
- malformed parts skipped, valid ones still processed
- temp files unlinked after successful dispatch
- temp files unlinked when dispatcher throws
All 87 existing tests still pass.
Open question for the maintainer: MediaPath / MediaType injection is verified
end-to-end only for image/* (matching the msteams precedent this code models
on). Audio/video/document follow the same injection path with their native
mimeType — if the current OpenClaw agent runner gates on image mimetypes and
drops others, happy to narrow scope to images-only, add a runner-side fix in
OpenClaw, or coordinate.
Follow-ups:
- Remote http(s) URL attachment fetching (separate SSRF / size-enforcement design)
- @ag-ui/core bump to 0.0.52 (being proposed in PR contextablemark#26)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
REASONING_START,REASONING_MESSAGE_START/CONTENT/END,REASONING_ENDwhen the agent streams thinking content. UsesonReasoningStream/onReasoningEndcallbacks from OpenClaw's reply pipeline.STEP_STARTED/STEP_FINISHEDfromonItemEvent, giving CopilotKit clients visibility into multi-step agent progress.@ag-ui/coreto^0.0.52— uses newREASONING_*events (THINKING is deprecated, removed in AG-UI v1.0.0).surfaceReasoning: false/surfaceSteps: falsein channel defaults.RUN_FINISHEDfor robustness.Notes from rebase onto
main(was:feat/a2ui-support)onItemEventcallback parameter was over-narrowed (itemId: string,phase: string). Loosened to optional to match openclaw'sGetReplyOptionssignature; null-check applied inside.pnpm-lock.yaml(7241 lines). Tracked separately — see follow-up PR.## Unreleasedalongside theX-OpenClaw-Session-Keyentry. Version inpackage.jsonleft at0.6.4— bumped at release time.Test plan
pnpm test— 92 tests pass (5 new: reasoning flow, no-reasoning, auto-close, step events, step dedup)pnpm build— clean compile (after type fix and rebase pulled in OpenClaw 2026.4.16 plugin-sdk compat frommain)REASONING_*events in SSE streamSTEP_STARTED/STEP_FINISHEDin SSE stream🤖 Generated with Claude Code