.NET: [Bug]: Microsoft.Agents.AI.Tools.Shell.HeadTailBuffer can reorder and silently drop UTF-8 runes at the head/tail boundary

[Lightczx]

Description

HeadTailBuffer may reorder UTF-8 characters and silently discard output when a multi-byte rune does not fit in the remaining head capacity.

The returned result can report Truncated == false even though data was removed.

This affects command output containing non-ASCII characters near the head/tail boundary.

Code Sample

HeadTailBuffer buffer = new(20);

buffer.AppendLine("aaaaaaa\U0001F600\U0001F600\U0001F600");

var (text, truncated) = buffer.ToFinalString();

The input, including the appended newline, is exactly 20 UTF-8 bytes:
aaaaaaa: 7 bytes
Three emoji runes: 12 bytes
Newline: 1 byte

Expect:
text == "aaaaaaa😀😀😀\n"
truncated == false

Actual:
text == "aaaaaaa\n😀😀"
truncated == false

The first emoji is silently discarded, and the newline is moved before the remaining emoji runes.

Package Versions

Microsoft.Agents.AI.Tools.Shell 1.12.0-preview.260629.1

.NET Version

.NET 10

[github-actions]

🤖 Automated triage reproduction notes (agent-authored — trust but verify)

Agent analysis

Repro: HeadTailBuffer.AppendInternal() in dotnet/src/Microsoft.Agents.AI.Tools.Shell/HeadTailBuffer.cs lines 57-84 allows single-byte runes to enter _head after multi-byte runes were already routed to _tail (because head had room for 1 byte but not 4), breaking character order; additionally the tail eviction loop (line 78) fires unconditionally even when _totalBytes <= _cap, silently dropping data while ToFinalString() reports truncated=false. Minimal repro: construct HeadTailBuffer(20) and call AppendLine("aaaaaaa\U0001F600\U0001F600") — expects all 20 bytes preserved in order but gets reordered output with one emoji lost.

• Failing test: dotnet/tests/Microsoft.Agents.AI.Tools.Shell.UnitTests/HeadTailBufferBug7112Tests.cs
• Files examined: dotnet/src/Microsoft.Agents.AI.Tools.Shell/HeadTailBuffer.cs, dotnet/tests/Microsoft.Agents.AI.Tools.Shell.UnitTests/HeadTailBufferTests.cs
• Tests run: HeadTailBufferBug7112Tests.AppendLine_ExactlyAtCap_WithMultiByteRunes_PreservesAllContent, HeadTailBufferBug7112Tests.AppendLine_ExactlyAtCap_WithMultiByteRunes_DoesNotReorderCharacters, HeadTailBufferBug7112Tests.AppendLine_ExactlyAtCap_NoBytesLost
• Reported version: 1.12.0-preview.260629.1

[jstar0]

I'd like to take this on. I plan to add boundary regressions for exact-cap and overflow cases, then adjust the append logic so that once a rune spills into the tail, later runes cannot re-enter the head out of order, and tail eviction only occurs when the total byte count exceeds the cap. Could a maintainer confirm this direction and assign the issue before I start?

[moonbox3]

@jstar0 thanks for taking this, direction looks good. We'll await a PR.