[Enhancement] Stream Load blocks its libevent worker in on_header (begin_txn + FE plan) and while the body pipe is full, stalling every other connection on that loop #68973

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.1.4 (cloud mode, arm64). On 4.1.4, StreamLoadAction::_handle also blocks in ctx->future.get() and commit. On master, #53144 already moved that into a callback. The other two blocking points are unchanged on master:

• be/src/service/http/action/stream_load.cpp:244-296 → _on_header :386-402: begin_txn (FE loadTxnBegin, :394) and, in _process_put, the FE streamLoadPut RPC (:826-831) plus the fragment start, all run synchronously from libevent's header callback (ev_http_server.cpp:72-79).
• be/src/service/http/action/stream_load.cpp:405-445: on_chunk_data calls body_sink->append, which blocks in StreamLoadPipe::_append on _put_cond while the pipe is full (be/src/io/fs/stream_load_pipe.cpp:200-212; the pipe size is kMaxPipeBufferedBytes).

Description

Each libevent worker (webserver_num_workers, default 128) serves many keep-alive connections. When a callback blocks, every other connection on that worker waits, including other loads' body reads and unrelated endpoints such as /api/health or /metrics.

1. Header. Before any body byte is read, on_header performs two synchronous FE RPCs (begin_txn, then streamLoadPut) and starts the fragment. While one load waits on the FE, the other connections on that loop stall.
2. Full pipe. When the scanner is slower than the network, on_chunk_data blocks the loop thread until the scanner drains the pipe. Back-pressure should apply to that one connection, not to the whole worker.

What we measured: an off-CPU profile with 32 concurrent senders showed EvHttpServer threads blocked for 91 s per 2 s window in StreamLoadPipe::_append. On 4.1.4 we also saw the finish wait (fixed on master): Stream Load receive p50 rose to 3.0–3.7 s at only 4–5 Gbit/s, because body reads queued behind a neighbouring load's write and commit. We do not have a production measurement of the header part alone.

Solution

We have a patch that keeps each load's work off the loop. Both changes are behind mutable configs:

1. Pause instead of block. When the pipe is full, stop reading that connection, and resume once the scanner has drained the pipe to half. Other connections on the loop keep being served. Reads are resumed only while the body is still being read, because libevent aborts if reading is re-enabled in the wrong state. Cancelling the pipe also resumes reads.
2. Header off the loop. On the loop, on_header only parses and validates the headers. It pauses reads and hands begin_txn, the plan and the fragment start to a thread pool. Body bytes that arrive with the headers are held until the pipe exists. The result is posted back to the loop, which resumes reads. On error, the load rolls back and replies exactly as the inline path does. A load that times out while queued fails before begin_txn. Add a HeaderWaitTimeMs timing to the reply.

How to Reproduce?

1. Set webserver_num_workers low (e.g. 2) to make collisions likely.
2. Send several concurrent keep-alive stream loads, where one load's scanner is slow (a large Arrow/JSON body into a wide table), or make begin_txn slow (for example with an FE debug point or a busy FE).
3. Meanwhile, time a second load or curl /api/health on the same port. Its latency follows the slow load's header phase or the time its pipe is full.

Anything Else?

Unit tests cover both changes. They show that a full pipe, or a slow begin_txn (via a debug point), on one loop no longer stalls a second request on that loop, and that the old path does. A small A/B on a test box (2 event loops, 8 clients) showed no throughput change (541k vs 530k rows/s, within noise). So we expect the gain to be in tail latency and isolation, not in peak throughput.

Trade-offs: a pipe can exceed its capacity by up to one network read before reads pause. Errors found during the FE work arrive after 100-continue has been sent. The http_stream endpoint (HttpStreamAction) still runs this work on the loop.

Related: #53144 (closes #53139) already moved the finish/commit wait off the loop on master; it is not in branch-4.1, where we saw it too. The full-pipe stall was reported as part B of #67777, which was closed by a metrics fix (#67945) that did not change Stream Load.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct