Skip to content

grpc-js: write callback never called when a streaming call ends while the write is held by flow control #3114

Open

Description

@realies

Problem description

On a client-streaming or bidi call, if a write() is waiting on HTTP/2 flow control when the call ends (call.cancel(), deadline, or the server sending a status), its callback is never called. status (and error for bidi) are emitted, but the pending write callback and any writes queued behind it never settle, so code that awaits a write hangs forever.

Reproduction steps

Server handler that never reads the request stream; client writes 64 KiB messages until one callback stays pending, writes once more, then cancels:

const grpc = require('@grpc/grpc-js');
const id = (x) => x;
const path = '/probe/upload';
const server = new grpc.Server();
server.addService(
  { upload: { path, requestStream: true, responseStream: false, requestSerialize: id, requestDeserialize: id, responseSerialize: id, responseDeserialize: id } },
  { upload(call) { call.sendMetadata(new grpc.Metadata()); /* never reads, so flow control fills up */ } },
);
server.bindAsync('127.0.0.1:0', grpc.ServerCredentials.createInsecure(), (err, port) => {
  const client = new grpc.Client(`127.0.0.1:${port}`, grpc.credentials.createInsecure());
  const call = client.makeClientStreamRequest(path, id, id, () => {});
  call.on('error', (e) => console.log('error event:', e.code));
  call.on('status', (s) => console.log('status event:', s.code));
  call.once('metadata', async () => {
    // Write until one write's callback stays pending under backpressure.
    for (let i = 0; ; i++) {
      const done = await Promise.race([
        new Promise((resolve) => call.write(Buffer.alloc(65536), resolve)).then(() => true),
        new Promise((resolve) => setTimeout(resolve, 200, false)),
      ]);
      if (!done) { console.log(`write #${i} is pending; cancelling`); break; }
    }
    const pending = new Promise((resolve) => call.write(Buffer.alloc(1), (e) => resolve(e ? e.message : 'ok')));
    call.cancel();
    const result = await Promise.race([pending, new Promise((r) => setTimeout(r, 3000, 'still pending after 3 s'))]);
    console.log('write callback:', result);
    process.exit(0);
  });
});

Output on 1.14.6:

write #15 is pending; cancelling
status event: 1
write callback: still pending after 3 s

Same with a deadline instead of cancel(), with the server ending the call with OK or an error status, and on bidi calls. Master (6741e13) behaves the same.

Environment

  • Linux x86_64, Node 24.22.1
  • @grpc/grpc-js 1.14.6 and master 6741e13

Additional context

Expected: every pending or queued write callback is called once, with an error, when the call ends. For bidi calls the readable side should stay as it is now: responses already buffered and end still delivered.

Destroying the stream on status settles the callbacks, but on bidi it drops responses the client hasn't read yet, so I'm not sure what shape you'd want for the fix. Happy to send a PR once the expected behaviour is settled.

Activity

  1. murgatroid99 commented on Oct 28, 2026

    @murgatroid99
    Member

    This looks like the problem that is fixed by nodejs/node#66016, so updating to a newer version of Node should fix this. Note that this apparently introduces a bug with retries as noted in #3112, but there is a pending fix for that in #3111.

  2. realies commented on Nov 3, 2026

    @realies
    Author

    Thanks, but I don't think that's it. nodejs/node#66016 shipped in Node 24.22.0, and the repro above was already on 24.22.1. I've also just run it on 24.22.2 and 26.13.0 with grpc-js 1.14.6, and the output is the same: after cancel() the write callback is still pending.

    I don't think the pending write is held in Node's http2 stream. The _write callback is handed down to the call layers inside grpc-js, and when the call ends with a status nothing calls it back. The Node fix settles callbacks for writes the http2 stream holds, and this one never gets back to the public stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions