feat: store AI response output on ai.request and search.delegate spans (#529)

[buger]

Summary

• Added onResult callback parameter to withSpan() in simpleTelemetry.js — enriches spans with result data before they close
• ai.request spans now capture ai.output (the AI's text response) and ai.output_length
• search.delegate spans now capture search.delegate.output (the delegate's synthesized answer) and search.delegate.output_length
• Added truncateForSpan() helper that preserves head + tail of long text (first ~2K and last ~2K chars) instead of front-only truncation, giving better context in traces
• Also applied head+tail truncation to ai.input on ai.request spans

Fixes #529

Test plan

• 5 new truncateForSpan unit tests (short text, falsy input, head+tail preservation, omitted count accuracy, custom maxLen)
• 4 new withSpan onResult tests (callback invocation, error resilience, no-call-on-error, truncation behavior)
• Updated existing search-delegate.test.js to expect the new onResult parameter
• All 61 telemetry tests pass
• All 15 search-delegate tests pass
• Full npm test suite: 2977 tests pass
• Build succeeds

🤖 Generated with Claude Code

[probelabs]

Summary

This PR enhances telemetry spans by capturing AI response output on ai.request and search.delegate spans. It introduces an onResult callback mechanism to enrich spans with result data before they close, and adds a truncateForSpan() helper that preserves both head and tail of long text for better trace context.

Files Changed Analysis

File	Changes	Description
npm/src/agent/simpleTelemetry.js	+25/-3	Core changes: new truncateForSpan() helper, withSpan() now accepts onResult callback
npm/src/agent/ProbeAgent.js	+8/-3	Uses truncateForSpan() for input, adds onResult callback to capture AI output
npm/src/tools/vercel.js	+7/-0	Adds onResult callback to search.delegate span to capture delegate output
npm/tests/unit/simpleTelemetry.test.js	+113/-1	9 new tests for truncateForSpan and withSpan onResult behavior
npm/tests/unit/search-delegate.test.js	+2/-1	Updated mock expectation for new onResult parameter

Total: +155 additions, -8 deletions across 5 files

Architecture & Impact Assessment

What This PR Accomplishes

• Enables observability of AI response content in traces by capturing output text and length on spans
• Provides better debugging context by preserving both the beginning and end of long text outputs

Key Technical Changes

1. truncateForSpan(text, maxLen=4096) - New utility function:

   • Returns text as-is if ≤ maxLen
   • For longer text: shows first half + separator + last half
   • Separator format: ... [N chars omitted] ...
   • Handles falsy inputs gracefully (returns empty string)

2. withSpan(spanName, fn, attributes, onResult) - Extended API:

   • New optional 4th parameter onResult(span, result) callback
   • Called after successful function execution, before span closes
   • Callback errors are silently caught to prevent breaking execution flow
   • Not called if the spanned function throws an error

3. Span attribute additions:

   • ai.request spans: ai.output (truncated text), ai.output_length (full length)
   • search.delegate spans: search.delegate.output (truncated text), search.delegate.output_length (full length)

Affected System Components

flowchart TD
    subgraph Telemetry Layer
        ST[simpleTelemetry.js]
        TFS[truncateForSpan helper]
        WS[withSpan API]
    end
    
    subgraph AI Layer
        PA[ProbeAgent.js]
        AIR[ai.request spans]
    end
    
    subgraph Search Layer
        VJ[vercel.js]
        SD[search.delegate spans]
    end
    
    ST --> TFS
    ST --> WS
    PA --> TFS
    PA --> WS
    PA --> AIR
    VJ --> TFS
    VJ --> WS
    VJ --> SD
    
    WS -->|onResult callback| AIR
    WS -->|onResult callback| SD

Loading

Scope Discovery & Context Expansion

Direct Impact

• Telemetry consumers: Any system reading span attributes will now see additional output fields
• Trace storage: Slightly increased storage per span (up to ~4KB per output field)

Related Areas to Consider

• Other span types in the codebase that might benefit from similar output capture
• Trace export/visualization tools that display these attributes
• Rate limiting or cost attribution based on output lengths

Test Coverage

• 5 unit tests for truncateForSpan (short text, falsy input, head+tail preservation, omitted count accuracy, custom maxLen)
• 4 unit tests for withSpan onResult (callback invocation, error resilience, no-call-on-error, truncation behavior)
• Updated existing search-delegate.test.js mock expectations
• Full test suite: 2977 tests passing

References

• npm/src/agent/simpleTelemetry.js:5-19 - truncateForSpan() function
• npm/src/agent/simpleTelemetry.js:481 - withSpan() signature with onResult parameter
• npm/src/agent/simpleTelemetry.js:494-501 - onResult callback invocation logic
• npm/src/agent/ProbeAgent.js:4331 - AI input truncation using truncateForSpan
• npm/src/agent/ProbeAgent.js:4342-4348 - ai.request span onResult callback
• npm/src/tools/vercel.js:554-561 - search.delegate span onResult callback
• npm/tests/unit/simpleTelemetry.test.js:7-49 - truncateForSpan tests
• npm/tests/unit/simpleTelemetry.test.js:324-393 - withSpan onResult tests

Metadata

• Review Effort: 2 / 5
• Primary Label: enhancement

---

Powered by Visor from Probelabs

Last updated: 2026-03-18T16:10:41.454Z | Triggered by: pr_opened | Commit: d521d3c

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

Security Issues (1)

Severity	Location	Issue
🟠 Error	system:0	ProbeAgent execution failed: Cannot read properties of null (reading 'length')

Performance Issues (1)

Severity	Location	Issue
🟠 Error	system:0	ProbeAgent execution failed: Error: Failed to get response from AI model. No output generated. Check the stream for errors.

Quality Issues (1)

Severity	Location	Issue
🟠 Error	system:0	ProbeAgent execution failed: Error: Failed to get response from AI model. No output generated. Check the stream for errors.

---

Powered by Visor from Probelabs

Last updated: 2026-03-18T16:07:23.918Z | Triggered by: pr_opened | Commit: d521d3c

💡 TIP: You can chat with Visor using /visor ask <your question>