Skip to content

Stdio client send can hang forever, or fail with a bare "Failed to send message.", when the server exits at once #2005

Open

Description

@azixaka

Summary

When a stdio server exits right after it starts (a bad argument, a missing dependency, a crash during startup), the client's first send races the transport's own cleanup. On Unix, two things can go wrong (reproduced on Linux; macOS uses the same Unix pipe code):

  1. The send never returns. Its thread waits forever in SafeSocketHandle.CloseAsIs, under SafePipeHandle.CreatePipeSocket.
  2. The send fails with a bare IOException: Failed to send message. The caller should get the "MCP server process exited unexpectedly" exception, with the exit code and the server's stderr.

Version checked: 2.4.0. StdioClientSessionTransport.cs is unchanged from 2.3.0 to 2.4.0, and on main.

Repro

Sixteen workers, each connecting over and over to a stdio server that writes a line to stderr and exits at once (sh -c 'echo … >&2; exit 3'):

Package Connects Workers hung Bare "Failed to send message."
2.3.0, run 1 under 30,000 16 of 16 152
2.3.0, run 2 under 30,000 16 of 16 1,844
2.3.0 with the fix below 1,136,147 in 3 minutes 0 0

A smaller version runs at transport level: StdioClientTransport.ConnectAsync, one SendMessageAsync, then DisposeAsync, 200 times in each of 16 parallel workers. On main it left 5, 8 and 9 of the 16 workers hung in three local runs (Linux x64, .NET 10).

Stack of a hung send:

System.Private.CoreLib!System.Threading.Thread.Sleep(int32)
System.Private.CoreLib!System.Threading.SpinWait.SpinOnce()
System.Net.Sockets!System.Net.Sockets.SafeSocketHandle.CloseAsIs(bool)
System.Net.Sockets!System.Net.Sockets.Socket.Dispose(bool)
System.Net.Sockets!System.Net.Sockets.Socket.Dispose()
System.IO.Pipes!Microsoft.Win32.SafeHandles.SafePipeHandle.CreatePipeSocket(bool)
System.IO.Pipes!System.IO.Pipes.PipeStream+<WriteAsyncCore>d__83.MoveNext()
...
ModelContextProtocol.Core!ModelContextProtocol.Client.StreamClientSessionTransport+<SendMessageAsync>d__10.MoveNext()
...
ModelContextProtocol.Core!ModelContextProtocol.Client.StdioClientSessionTransport+<SendMessageAsync>d__7.MoveNext()

Cause, as far as we can read it

  1. Since Close stdio input before waiting for server exit #1838, CleanupAsync closes the server's stdin before waiting for the exit. When the server exits at once, the read loop sees EOF and runs that cleanup while the first send may still be in PipeStream.WriteAsync. On Unix, the first write creates the pipe's socket in SafePipeHandle.CreatePipeSocket. If the pipe is disposed during that call, CreatePipeSocket disposes the new socket while it still holds a reference on the pipe handle. SafeSocketHandle.CloseAsIs then waits for the socket handle to be released, which happens only in the pipe handle's ReleaseHandle, after CreatePipeSocket returns. The thread waits on itself. That part looks like a runtime bug, but the transport can avoid it by never closing stdin while a send is writing to it.
  2. SendMessageAsync catches the IOException and calls GetUnexpectedExitExceptionAsync, which returns null unless Process.HasExited is already true. A server that has just exited has closed its pipes but may not have been reaped yet. So the exit check fails, and the bare exception is rethrown.

Suggested fix

  1. In CleanupAsync, take the send lock, waiting at most ShutdownTimeout, before closing stdin. If a send still holds it after that, leave stdin open; the existing process-tree kill still ends the server.
  2. In SendMessageAsync, after an IOException, wait for the process to exit, at most ShutdownTimeout, before building the exit exception.

Together these are about 20 lines in StdioClientSessionTransport and StreamClientSessionTransport. I have a PR ready with both changes and regression tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions