Skip to content

feat!: irpc error reorg - #164

Merged
rklaehn merged 31 commits into
mainfrom
fix/no-truncated
Nov 9, 2026
Merged

rklaehn merged 31 commits into
mainfrom
fix/no-truncated

Conversation

@rklaehn

@rklaehn rklaehn commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

The final error reorg before irpc 1.0

Errors with rpc

// channel
enum SendError          { ReceiverClosed, Rpc { source: RpcSendError } }  // also writing a request
enum oneshot::RecvError { SenderClosed,   Rpc { source: RpcRecvError } }
enum mpsc::RecvError    {                 Rpc { source: RpcRecvError } }

// rpc
enum RpcSendError {
    MaxSizeExceeded,
    EncodeFailed { source: postcard::Error },
    Closed,
    Remote { code: ErrorCode },
    ConnectionClosed { close: ConnectionClose },
    ConnectionFailed { source: io::Error },
}
enum RpcRecvError {
    MaxSizeExceeded,
    DecodeFailed { source: postcard::Error },
    Remote { code: Option<ErrorCode> },
    ConnectionClosed { close: ConnectionClose },
    ConnectionFailed { source: io::Error },
}
enum ReadRequestError {
    MaxSizeExceeded,
    Malformed { source: io::Error },
    DecodeFailed { source: postcard::Error },
    ConnectionClosed { close: ConnectionClose },
    ConnectionFailed { source: noq::ConnectionError },
}
enum ConnectionClose { Remote { code: Option<ErrorCode>, reason: Vec<u8> }, Local }
enum ErrorCode {
    RequestAborted, RequestMaxSizeExceeded, RequestEncodeFailed, RequestDecodeFailed,
    MessageMaxSizeExceeded, MessageEncodeFailed, MessageDecodeFailed, MessageAborted,
    Other(VarInt),
}

// lib
enum RequestError {
    Other { source: AnyError },
    Connect { source: noq::ConnectError },
    ConnectionClosed { close: ConnectionClose },
    ConnectionFailed { source: noq::ConnectionError },
}

Errors without rpc

enum SendError          { ReceiverClosed }
enum oneshot::RecvError { SenderClosed }
enum mpsc::RecvError    {}  // a local receiver ends with Ok(None)
enum RequestError       {}  // a local request cannot fail

Changes

  • Local first. Each channel error has the variant a local channel has, plus one Rpc arm for everything only a remote channel can hit. Something that happens normally on a working connection (a peer dropped its sender or receiver, a cancelled send) looks like the local case.
  • Split by direction. RpcSendError and RpcRecvError only have the variants their side can hit.
  • Three kinds of remote failure. Remote is the peer stopping or resetting a stream on a working connection, with its code. ConnectionClosed is a close on purpose, with the code and reason. ConnectionFailed is a broken connection, including a custom transport's own failures. An unknown code is ErrorCode::Other(n) instead of an io error.
  • Codes are VarInt. ErrorCode::to_code, from_code, Other and HandlerError::close_connection use VarInt, the type that streams and connections carry, re-exported as irpc::rpc::VarInt. Every ErrorCode can be sent.
  • No WriteError. Writing a request fails like any other send, so it returns SendError. Error::Write, ErrorCode::Closed and From<ErrorCode> for VarInt are gone.
  • Later sends repeat the error. After the receiver closed or the connection broke, later sends get the same error. Closed is for a sender that closed itself after a bad message.
  • remote_code() on every error returns the peer's code, whether it came from a stream or from the close of the connection.
  • Empty enums without rpc need fix: support empty enums in stack_error n0-error#68, patched to n0-error main until it is released.

@rklaehn rklaehn changed the title feat!: treat a cancelled send like a dropped sender feat!: only connection failures are Io errors Nov 3, 2026
@rklaehn
rklaehn changed the base branch from fix/io-wire-only to main November 3, 2026 10:53
@rklaehn
rklaehn changed the base branch from main to fix/cancelled-send-is-end November 6, 2026 07:44
@rklaehn
rklaehn force-pushed the fix/cancelled-send-is-end branch from bc73002 to fd0df49 Compare November 6, 2026 08:58
rklaehn added a commit that referenced this pull request Nov 6, 2026
#167)

A cancelled send, for example from a timeout or a `select!`, closes a
remote mpsc sender and all its clones, and the receiver gets an io
error. Locally, a cancelled send is just not sent and the channel keeps
working.

This makes the remote mpsc send cancel safe like the local one. The
sender keeps the encoded message and how much of it was written in its
state instead of in the send future, like the receiver does since #149.
Each mpsc frame is now the postcard encoding of `(message, MessageEnd)`
under one size prefix. If a send is cancelled after it wrote part of its
frame, the next send on the sender or any clone writes the rest with the
last byte encoded as `MessageEnd::Aborted`, and the receiver drops that
message with a debug log.

| Cancelled send | Result |
|---|---|
| before any byte was written | not sent, sender usable |
| after the first byte, sender keeps sending | not delivered, sender
usable |
| after the first byte, sender dropped | not delivered, the stream ends
mid-message |

The last row is an io error for the receiver today, and `Ok(None)` with
#164.

Costs: one byte per mpsc message, and a cancel after the first byte
still sends the rest of that message. Requests and oneshot keep their
framing.

## Compatibility

No new ALPN needed. Released irpc versions (checked 0.11 to 0.17) decode
the body with `postcard::from_bytes`, which ignores trailing bytes, so
they read new frames and ignore the marker. They also deliver an aborted
message, which gives the behavior of #165 for that peer. A new receiver
decodes the message with `take_from_bytes`: no byte left means an old
peer and counts as complete, one byte is the marker, more is
`DecodeFailed`. This relies on message types knowing their own length,
which postcard types do.

## Options

1. **Close the channel on cancel** (main, and #164 with local-like
errors): no wire change, but a timeout on one send kills the sender and
all its clones.
2. **Finish the message with the next send** (#165, stacked on #164): no
wire change, the channel survives, but a cancelled send may or may not
be delivered, possibly much later, so a retry can deliver it twice. Its
first commit finishes the message in a spawned task instead, which needs
a runtime in `Drop`.
3. **Mark the message as aborted** (this PR): the channel survives and a
cancelled send is never delivered, as for a local channel. The
cancellation is visible on the wire but not in the API. Compatible with
older peers, which fall back to option 2.
4. **COBS framing** (#168): never delivered, and an abort costs at most
2 bytes instead of the rest of the message, but changes the wire format.

Not options: resetting the stream with `MessageAborted` drops earlier
messages that the receiver has not read yet (`RESET_STREAM_AT` would fix
that, but noq does not have it), and chunked framing is too much for the
message sizes irpc is meant for.

Tests: `mpsc_cancelled_sends_keep_clones_usable` (replaces
`mpsc_sender_clone_drop_error`),
`mpsc_partly_cancelled_send_is_not_delivered`,
`mpsc_partly_cancelled_send_is_dropped_with_sender`,
`mpsc_unstarted_cancelled_send_is_not_sent`, `mpsc_bad_message_end`, and
for compatibility `mpsc_frame_with_extra_bytes` plus
`tests/mpsc_old_peer.rs`, which runs both directions against a copy of
the irpc 0.17.0 mpsc framing; tests that write frames by hand now add
the marker.

## Breaking changes

- a cancelled remote mpsc send no longer closes the sender and its
clones.
@rklaehn
rklaehn force-pushed the fix/cancelled-send-is-end branch from fd0df49 to c332e76 Compare November 6, 2026 09:17
rklaehn added a commit that referenced this pull request Nov 6, 2026
Stacked on #167.

A cancelled send can leave a stream that is finished (FIN) in the middle
of a message. The receiver reported that as an io error. Now it is the
same as a sender that was dropped:

- mpsc: the receiver delivers all complete messages, then returns
`Ok(None)`. The partial message is logged at debug level. With #167 this
happens when a sender is dropped right after a partly cancelled send.
- oneshot: the receiver returns `SenderClosed`, also for a reset with
`MessageAborted`.

A reset or a lost connection in the middle of a message is still an
error.

Together with #167 this finishes the cancellation work. Split out of
#164, which keeps the error refactor.

Tests: `mpsc_truncated_message`, `mpsc_truncated_size`,
`oneshot_truncated_message`, `oneshot_truncated_size`,
`oneshot_cancelled_send_is_sender_closed` (was
`oneshot_cancelled_send_resets_stream`), and #167's
`mpsc_partly_cancelled_send_is_dropped_with_sender` now expects the
earlier messages instead of an io error. Negative cases:
`{mpsc,oneshot}_reset_inside_message_is_an_error`,
`{mpsc,oneshot}_connection_closed_inside_message_is_an_error`.

## Breaking changes

- mpsc: `Ok(None)` instead of `Io(UnexpectedEof)` for a stream finished
in the middle of a message.
- oneshot: `SenderClosed` instead of `Io(UnexpectedEof)` for a stream
finished in the middle of the message, and instead of `Remote { code:
MessageAborted }` for a cancelled send.
@rklaehn rklaehn changed the title feat!: only connection failures are Io errors feat!: irpc error reorg Nov 9, 2026
@rklaehn
rklaehn requested a review from Frando November 9, 2026 07:05
The `Remote` variants of `SendError`, `{mpsc,oneshot}::RecvError`, and
`WriteError` held an `ErrorCode`, so a stream that the peer reset or
stopped with an application code (16 and up) or an unknown reserved code
became an `Io` error, and `remote_code()` returned `None`. A code is
something the peer chose to send, not an io failure.

Add `rpc::RemoteCode`, either `Irpc(ErrorCode)` or `Other(u64)`. It is
never 0, which irpc reports as a closed channel. The `Remote` variants
and `remote_code()` now use it.
Code 0 means no error: the peer's application closed or dropped the
stream. It never comes from a network failure, so it should not be an
`Io` error. Before, a reset with code 0 was an `Io` error on both
receivers, and a stop with code 0 was an `Io` error on `WriteError`.

* `oneshot::RecvError::SenderClosed` now also covers a reset with code 0.
* added `mpsc::RecvError::SenderClosed` for a reset with code 0. Unlike
  the end of the stream, it can drop messages that were not read yet.
* added `WriteError::ReceiverClosed` for a stop with code 0 while the
  client writes a request.
* removed `ErrorCode::Closed`. Code 0 is not an error code, and
  `RemoteCode` never holds it.
A connection that the peer closed with an application close, or that
this side closed, was an `Io` error on `SendError`,
`{mpsc,oneshot}::RecvError`, and `WriteError`. The code was only
reachable by digging it out of the io error with `remote_code()`. A close
is something an application chose to do, not an io failure.

* added `rpc::ConnectionClose`: `Remote { code, reason }` for a close by
  the peer, with `code: None` for code 0, or `Local`.
* added a `ConnectionClosed { close }` variant to `SendError`,
  `{mpsc,oneshot}::RecvError`, and `WriteError`. `Io` is now only for a
  connection that failed, for example a timeout or a reset.
* after a send fails with `ConnectionClosed`, later sends on the sender
  and its clones get the same error, as for `ReceiverClosed` and `Remote`.
`ReadRequestError::Connection` and `RequestError::Connection` held the
raw `noq::ConnectionError`, so a connection that the peer closed with a
code looked the same as one that failed.

* added `ReadRequestError::ConnectionClosed` for a peer that closes the
  connection with a code other than 0. A close with code 0 or by this
  side still makes `read_request` return `None`.
* added `RequestError::ConnectionClosed` for opening a request on a
  connection that was closed on purpose.
* `Connection` is now only for a connection that failed.
After a send failed with a local error or an io error, later sends on
the sender and its clones got `Io(BrokenPipe)`. That is the state of
this sender, not an io failure, so it gets its own variant. Later sends
after an error from the peer still get that same error.
A size prefix is a varint, and a peer can send a varint with more bytes
than fit in a u64. Such a prefix is larger than MAX_MESSAGE_SIZE in any
case, so the receivers and `read_request` now return `MaxSizeExceeded`
and stop the stream with the code for it. Before, it was an `Io` error
on a channel and `Malformed` for a request.
`ErrorCode` cannot be exactly the codes that irpc defines anyway: a peer
with a newer version of irpc can send a code that this version does not
know. So instead of a separate `RemoteCode`, `ErrorCode` gets an
`Other(u64)` variant for any code that this version does not know, an
application code or a newer irpc code. It is never 0.

* `ErrorCode::to_code` now returns `u64`, and `from_code` takes a `u64`
  and returns `Other` for an unknown code and `None` for 0.
* the `Remote` variants and `remote_code()` use `ErrorCode` again, as in
  #138.
* removed `From<ErrorCode> for VarInt`: `Other` can hold a value that is
  not a valid `VarInt`.
After a send failed with an io error, for example because the connection
timed out, later sends on the sender and its clones got `Closed`, which
is for a sender that closed itself after a bad or cancelled send. They
now get the same io error again, as for the other errors from the peer.
A local channel only ends when its other side is dropped. Everything
else, a message that does not encode, decode or fit, a code from the
peer, a closed or failed connection, only happens on a remote channel,
and was spread over the variants of each channel error.

Each channel error now has its variant for the local case, and wraps
the rest in an `Rpc` variant with the new `rpc::RpcError`:

* `SendError { ReceiverClosed, Rpc }`
* `oneshot::RecvError { SenderClosed, Rpc }`
* `mpsc::RecvError { Rpc }`. A local mpsc receiver has no errors.

`RpcError` has `MaxSizeExceeded`, `EncodeFailed`, `DecodeFailed`,
`Remote`, `ConnectionClosed`, `Closed`, and `Io`. Its `Remote` holds an
`Option<ErrorCode>`, like `ConnectionClose`: `None` is a reset or stop
with code 0 where that is not the same as a local channel, a reset of an
mpsc stream that can drop unread messages.

Without the rpc feature, the channel errors only have their local
variant, and the io conversions into them are gone.
n0-error 1.0.1 cannot derive stack_error for an empty enum, so patch it
to main (n0-computer/n0-error#68) until that is released.
Writing the initial request fails in the same ways as sending a channel
item, so RemoteSender::write returns SendError, and Error::Write goes.
The request still resets its stream with the Request* codes.
Each side now only has the variants it can hit: a sender cannot fail to
decode, and a receiver cannot fail to encode or be closed by an earlier
send. A stop on the send side always has a code, since code 0 is
ReceiverClosed, so RpcSendError::Remote holds an ErrorCode, not an
Option.
The rpc errors' Io and the request errors' Connection both mean that
the connection broke, next to ConnectionClosed for a connection that was
closed on purpose. Name them the same, so every connection-related
variant starts with Connection, and Remote is the peer acting on a stream
of a working connection.
@rklaehn
rklaehn changed the base branch from fix/cancelled-send-is-end to main November 9, 2026 07:15

@Frando Frando left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the new organization, it looks very clean now. I didn't spot anything wrong so far, am about half-though. Posting a few doc nits already, will continue the review in a bit.

Comment thread src/channel.rs Resolved
Comment thread src/channel/mpsc.rs Resolved
Comment thread src/rpc.rs Resolved
@rklaehn
rklaehn merged commit b792c2b into main Nov 9, 2026
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants