Skip to content

Fix stdio send hang and bare send failure when the server exits at once - #2006

Open
azixaka wants to merge 1 commit into
modelcontextprotocol:mainfrom
azixaka:fix/stdio-exit-races
Open

azixaka wants to merge 1 commit into
modelcontextprotocol:mainfrom
azixaka:fix/stdio-exit-races

Conversation

@azixaka

@azixaka azixaka commented Nov 8, 2026

Copy link
Copy Markdown

Fixes #2005

Problem

When a stdio server exits right after it starts, the client's first send races the transport's own cleanup. On Unix, two things can go wrong. Both were reproduced on Linux; macOS uses the same Unix pipe code.

1. The send never returns

When the server exits, the read loop sees EOF, and CleanupAsync closes the server's stdin (#1838). If the first send is still in PipeStream.WriteAsync at that moment, the write is creating the pipe's socket in SafePipeHandle.CreatePipeSocket. It finds the pipe disposed, so it disposes the new socket. But it still holds a reference on the pipe handle, and SafeSocketHandle.CloseAsIs waits for a release that only that handle's ReleaseHandle performs. The thread waits on itself:

System.Threading.SpinWait.SpinOnce()
System.Net.Sockets.SafeSocketHandle.CloseAsIs(bool)
System.Net.Sockets.Socket.Dispose()
Microsoft.Win32.SafeHandles.SafePipeHandle.CreatePipeSocket(bool)
System.IO.Pipes.PipeStream.WriteAsyncCore(...)
ModelContextProtocol.Client.StreamClientSessionTransport.SendMessageAsync(...)
ModelContextProtocol.Client.StdioClientSessionTransport.SendMessageAsync(...)

That part looks like a runtime bug. The transport can avoid it anyway, by never closing stdin while a send is writing to it.

2. The send fails with a bare IOException: Failed to send message.

SendMessageAsync catches the IOException and calls GetUnexpectedExitExceptionAsync. That returns null unless Process.HasExited is already true. A server that has just exited has closed its pipes but may not have been reaped yet. So the caller gets the bare send failure instead of "MCP server process exited unexpectedly", with the exit code and stderr.

Repro

16 workers each connected over and over to a server that writes to stderr and exits at once (sh -c 'echo … >&2; exit 3'). On 2.3.0, all 16 workers hung within 30,000 connects, in each of two runs. Those runs also had 152 and 1,844 connects that ended in the bare send failure. With this change on 2.3.0, 1,136,147 connects ran in 3 minutes, with no worker hung and no bare send failure.

StdioClientSessionTransport.cs is unchanged from v2.3.0 to v2.4.0, and on main.

Fix

  • CleanupAsync now takes the send lock, waiting at most ShutdownTimeout, before it closes stdin. So stdin is never disposed under a write. If a send still holds the lock after that, stdin is left open, and the server is left to the existing process-tree kill. _sendLock in StreamClientSessionTransport becomes private protected for this.
  • When a send fails with an IOException, SendMessageAsync now waits for the process to exit, at most ShutdownTimeout, before it builds the exit exception.

Tests

Added to StdioClientTransportTests. Both tests run on Unix only, with the existing stderr-callback guard, because both races are in the Unix pipe and process handling.

  • SendMessageAsync_ServerExited_ThrowsWithExitDetails: the server closes its stdin, writes to stderr, and exits a second later. That holds open the window between the failed send and the observed exit. The send must throw the "exited unexpectedly" exception with the server's stderr, and the transport must complete with exit code 3. The test checks the exit code on StdioClientCompletionDetails rather than in the message. The message can still lose the exit code when the transport's cleanup disposes the Process first, and this PR doesn't change that. On main, the test fails every time with "Failed to send message.".
  • ConnectSendAndDispose_ServerExitsImmediately_CompletesWithExitDetails: 16 workers each run 200 cycles of connect, send and dispose against sh -c 'echo <id> >&2; exit 3'. All must complete within TestConstants.DefaultTimeout, and no send may fail without the exit details. With the fix, it takes 0.4 to 0.6 s. On main the outcome is probabilistic, because it depends on hitting the race. Locally it failed in 3 of 3 runs, with 5, 8 and 9 of the 16 workers hung. In a stack capture during one such run, every hung thread had the stack above. With 100 cycles per worker, it failed in 7 of 7 runs, with 1 to 8 workers hung.

Validation, on Linux x64 with the .NET 10 SDK (only the .NET 10 runtime is installed, so the tests ran on net10.0):

  • dotnet build and dotnet build -c Release: 0 warnings, 0 errors.
  • Both new tests, 30 consecutive runs with the fix: 30 of 30 passed.
  • StdioClientTransportTests, Debug and Release: 56 passed.
  • dotnet test tests/ModelContextProtocol.Tests -f net10.0 --filter "(Execution!=Manual)": 3,676 passed, 0 failed, 5 skipped. The skipped tests need an external API key, or are Windows-only or skipped by attribute.

Notes

  • If a send is still blocked after ShutdownTimeout, for example because the server has stopped reading its stdin, stdin is not closed. Dispose then waits for the exit and kills the server as before, so in that case it can take up to one more ShutdownTimeout than it does today.
  • The wait after a failed send is bounded by ShutdownTimeout, not by the send's cancellationToken, the same as the existing WaitForProcessExitAsync calls.

When a stdio server exits right after it starts, the read loop sees EOF
and CleanupAsync closes the server's stdin while the client's first send
may still be writing to it. On Unix, disposing the pipe while that write
is creating its socket in SafePipeHandle.CreatePipeSocket hangs the
write forever in SafeSocketHandle.CloseAsIs. Take the send lock, for at
most ShutdownTimeout, before closing stdin. A send still holding it
after that leaves stdin open and the server to the process-tree kill.

A send that failed with an IOException checked for the server's exit
before a server that had just exited might have been reaped, so it
threw a bare "Failed to send message." instead of the exit code and
stderr. Wait for the process to exit, for at most ShutdownTimeout,
before building the exit exception.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stdio client send can hang forever, or fail with a bare "Failed to send message.", when the server exits at once

1 participant