Skip to content

Fix ToolTask output loss: increase EOF pipe timeout from 2s to 30s#13767

Open
huulinhnguyen-dev wants to merge 5 commits into
dotnet:mainfrom
huulinhnguyen-dev:dev/huulinhnguyen/fix-tooltask-eof-timeout
Open

Fix ToolTask output loss: increase EOF pipe timeout from 2s to 30s#13767
huulinhnguyen-dev wants to merge 5 commits into
dotnet:mainfrom
huulinhnguyen-dev:dev/huulinhnguyen/fix-tooltask-eof-timeout

Conversation

@huulinhnguyen-dev
Copy link
Copy Markdown
Contributor

Fixes #13734

Context

ToolTask.WaitForProcessExit had a 2-second timeout for AsyncStreamReader to flush stdout/stderr after process exit. On loaded CI machines the timeout expired before the reader finished, and the result was silently discarded — causing final output lines to be lost.

Changes Made

  • Increased EOF wait timeout from 2s to 30s in WaitForProcessExit
  • Captured WaitHandle.WaitAll return value and drain queues on timeout
  • Added a MessageImportance.Low diagnostic when timeout fires

Testing

  • All 379 Microsoft.Build.Utilities.UnitTests tests pass
  • ToolTaskDoesNotHangWhenGrandchildInheritsPipeHandles verifies no regression on grandchild pipe scenario
  • ToolTaskCapturesAllOutputWithFix verifies end-to-end output capture still works

Notes

Fix stays within the existing Wave18_6 gate. No new infrastructure added.

@huulinhnguyen-dev huulinhnguyen-dev marked this pull request as ready for review May 15, 2026 06:57
Copilot AI review requested due to automatic review settings May 15, 2026 06:57
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses intermittent loss of final stdout/stderr lines captured by ToolTask under Wave18_6, by giving the async pipe readers more time to reach EOF after the tool process has exited and emitting a low-importance diagnostic when EOF still doesn’t arrive.

Changes:

  • Increased the post-exit EOF wait from 2s to 30s in ToolTask.WaitForProcessExit and acted on the WaitAll timeout result.
  • On EOF-wait timeout, drains any already-delivered stdout/stderr and logs a MessageImportance.Low diagnostic (ToolTask.PipeEOFTimeout).
  • Added the new localized resource string across Strings.resx and the localized .xlf files.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Utilities/ToolTask.cs Extends EOF wait and adds a low-importance diagnostic + drain behavior on timeout (Wave18_6 path).
src/Utilities/Resources/Strings.resx Adds ToolTask.PipeEOFTimeout resource string used by the new diagnostic message.
src/Utilities/Resources/xlf/Strings.cs.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.de.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.es.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.fr.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.it.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.ja.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.ko.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.pl.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.pt-BR.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.ru.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.tr.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.zh-Hans.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.
src/Utilities/Resources/xlf/Strings.zh-Hant.xlf Adds localized entry stub for ToolTask.PipeEOFTimeout.

Comment thread src/Utilities/Resources/Strings.resx
Comment thread src/Utilities/ToolTask.cs
@JanProvaznik
Copy link
Copy Markdown
Member

JanProvaznik commented May 15, 2026

this pr improved the logging #13760 can you verify the flaky test output matches expectations with this better diagnosability?

we'd like to have confirmation before commiting this candidate fix

@huulinhnguyen-dev

@rainersigwald
Copy link
Copy Markdown
Member

What is the behavior of a task that does launch a long-lived child that inherits the stdout handle? Does it now hang for 30s?

@huulinhnguyen-dev
Copy link
Copy Markdown
Contributor Author

Updated the grandchild test to use a longer-lived process (ping -n 40, ~39s) with a Stopwatch assertion proving the task returns within 35s — confirming the 30s EOF timeout is a hard bound. The diagnostic message ("Pipe EOF not received within 30s") also appears in the log, addressing the diagnosability ask from #13760.

Comment thread src/Utilities.UnitTests/ToolTask_Tests.cs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ToolTask drops final stdout/stderr lines after Wave18_6 EOF wait — flaky ToolTaskCanChangeCanonicalErrorFormat

5 participants