Enable file_read to return parsed content for multimodal file types (images, audio, video)

[Aaronontheweb]

Problem

Currently, file_read only returns text content for files. When passed images, PDFs, audio, or other multimodal content, it returns raw binary data that the agent cannot meaningfully process.

This creates a gap: the LLM/agent system is multimodal-capable (can process images, audio, etc.), but the file_read tool cannot deliver that content in a usable form.

Expected Behavior

file_read should attempt to parse and extract meaningful content from file types the multimodal LLMs can process:

• Images (PNG, JPEG, WebP, etc.) → extract text via OCR, describe image content, or provide metadata
• Audio (MP3, WAV, etc.) → extract transcript via Whisper or similar
• Video (MP4, WebM, etc.) → extract keyframes and/or transcript
• PDFs → extract text content (not just binary data)

This would allow the agent to use file_read for multimodal files instead of requiring binary raw data output.

Current Behavior

file_read("image.png") → returns raw binary PNG data
file_read("document.pdf") → returns raw binary PDF data
file_read("audio.mp3") → returns raw binary audio data

The agent receives binary data and cannot interpret it without external tools.

Suggested Approach

1. Detect file MIME type from extension or magic bytes
2. For text-based formats (PDF, images, audio, video), attempt appropriate parsing:
   • PDFs: Extract text content
   • Images: Run OCR (Tesseract or similar) to extract text
   • Audio: Transcribe via Whisper
   • Video: Extract keyframes + transcript
3. Return parsed text content in the response
4. Fall back to binary data if parsing fails (with a warning)

This should be configurable — users could opt out of image/audio processing if they want raw binary.

Use Cases

• Reviewing screenshots attached to sessions (like Slack/Discord screenshots for PII review)
• Processing PDF attachments (contracts, documents)
• Transcribing voice memos or audio notes
• Analyzing charts, diagrams, and screenshots without external tooling