Skip to content

fix(slack): stop inlining PDFs as DataContent on vision-capable models - #638

Merged
Aaronontheweb merged 1 commit into
devfrom
claude-wt-urgent-slack-fix
Apr 13, 2026
Merged

fix(slack): stop inlining PDFs as DataContent on vision-capable models#638
Aaronontheweb merged 1 commit into
devfrom
claude-wt-urgent-slack-fix

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Summary

Fixes the urgent production incident where Slack PDF uploads to ArdyBot fail with LLM provider error (500): Invalid url format: data:application/pdf;base64.

Root cause: SlackThreadBindingActor was computing inlinePdfs = InputModalities.HasFlag(ModelModality.Image) — conflating "vision-capable" with "accepts inline PDFs". A PDF dropped into a Slack DM against a llama.cpp / vLLM / Ollama endpoint via OpenAiCompatibleChatClient got wrapped as DataContent(application/pdf), which ToMessage unconditionally serialized as an image_url content part with a data:application/pdf;base64,... URL. No OpenAI-compatible upstream accepts that.

Fix: PDFs are now unconditionally path-only on all providers. The file still lands in {sessionDir}/inbox/ and the announcement line already points the agent at AttachmentNotes.ModelMissingPdf"current model has no native PDF support; use shell_execute (e.g., pdftotext) to extract text". The agent reads the file via tool calls instead of the runtime shoving bytes through a wire format the endpoint doesn't support.

Images are unchanged: vision-capable models still inline as image_url, because that's the only way they can be "seen" on OpenAI-compatible wire formats.

Reproduces on session D0AC6CKBK5K/1776091017.523089 (2026-04-13).

Changes

  • src/Netclaw.Channels.Slack/SlackThreadBindingActor.cs — delete inlinePdfs from live ingress and historical backfill paths; simplify ResolveInlineDecision to two parameters.
  • src/Netclaw.Actors.Tests/Channels/SlackAttachmentIngressTests.cs — flip Pdf_in_dm_on_vision_capable_model_* into a regression lock asserting no DataContent(application/pdf) ever reaches the chat client.

Three prior PRs (#601, #607, #626) plumbed PDFs through ingress → scanner → inbox → LLM input assembly, which is what unmasked this far-end gating bug. None of them touched the modality gate or the provider client.

Out of scope (deliberate)

  • Enabling native inline PDFs on providers that actually support them (Anthropic document blocks via DataContent through the Anthropic SDK, OpenAI file content parts) — those go through different client paths and need their own plumbing. Separate PR if/when someone wants that feature.
  • Revisiting whether Qwen3.5-27B-UD-Q4_K_XL.gguf's detected modalities are accurate. Orthogonal.

Test plan

  • dotnet build clean
  • dotnet test — full suite, 2500+ tests pass including the flipped Pdf_in_dm_is_saved_to_inbox_path_only_and_never_inlined regression
  • dotnet slopwatch analyze — 0 issues
  • Manual end-to-end: rebuild netclawd, re-send the same PDF to ArdyBot in Slack. Expected: announcement line arrives with inlined="false" and the "current model has no native PDF support" note; agent runs shell_execute pdftotext ... against inbox/Logo Design Services.pdf and answers the user. No 500.

SlackThreadBindingActor was computing `inlinePdfs` by checking the
ModelModality.Image flag — conflating "model is vision-capable" with
"model accepts inline PDFs". When a user DMed a PDF to a Slack bot
backed by a llama.cpp / vLLM / Ollama endpoint via
OpenAiCompatibleChatClient, the PDF bytes were wrapped as a DataContent
and serialized as an `image_url` content part with a
`data:application/pdf;base64,...` URL. Every OpenAI-compatible upstream
rejects that with "Invalid url format: data:application/pdf;base64",
failing the turn before the agent ever got a chance to process the
file.

PDFs now stay path-only on all providers. The file still lands in
`{sessionDir}/inbox/` via the existing attachment pipeline and the
announcement line already points the agent at
`AttachmentNotes.ModelMissingPdf` — "use shell_execute (e.g.,
pdftotext) to extract text". The agent reads the file via tool calls
instead of the runtime trying to squeeze it through a wire format no
OpenAI-compatible endpoint supports.

Images are unchanged: vision-capable models still receive inline
`image_url` content parts, which is the only way they can be "seen" on
OpenAI-compatible wire formats.

Repros on session D0AC6CKBK5K/1776091017.523089 (2026-04-13).

- Delete `inlinePdfs` from both the live ingress path and
  MergeGapWithLiveContents; simplify ResolveInlineDecision to two
  parameters.
- Flip SlackAttachmentIngressTests.Pdf_in_dm_on_vision_capable_model_*
  into a regression lock asserting no DataContent(application/pdf) ever
  reaches the chat client.

Closes the urgent production incident. Follow-up work (enabling native
inline PDFs on providers that actually support them — Anthropic
document blocks, OpenAI file parts) is intentionally out of scope.
@Aaronontheweb
Aaronontheweb merged commit 1fa5d75 into dev Apr 13, 2026
4 checks passed
@Aaronontheweb
Aaronontheweb deleted the claude-wt-urgent-slack-fix branch April 13, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant