Repository navigation
[Bug] BE aborts when BetaRowsetReader iterator init throws a std::exception (rethrown by DorisCallOnce, never converted to Status) #68915
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
4.1.4 (cloud mode, arm64). master is the same: BetaRowsetReader::_init_iterator_once (be/src/storage/rowset/beta_rowset_reader.cpp:324-326) returns _init_iter_once.call(...) with no try/catch, DorisCallOnce::call (be/src/util/once.h) stores and rethrows any exception, and RETURN_IF_CATCH_EXCEPTION (be/src/common/exception.h:67) only catches doris::Exception.
What's Wrong?
A BE aborted three times with std::terminate on a scanner thread, each time while a long full-table scan (an auto-analyze SELECT min/max over a large table, under heavy ingest) was being cancelled at its timeout. Stack (from a terminate handler that logs the rethrown exception):
std::bad_function_call
std::terminate <- std::rethrow_exception <- DorisCallOnce<Status>::call (util/once.h)
<- BetaRowsetReader::_init_iterator_once <- BetaRowsetReader::_next_batch
<- VCollectIterator::Level0Iterator::next <- BlockReader::next_block_with_aggregation
<- OlapScanner::_get_block_impl <- ScannerScheduler::_scanner_scan
_init_iterator() threw std::bad_function_call (an empty std::function was called) while initializing segment iterators. DorisCallOnce saves the exception and rethrows it on this call and every later call. Nothing on the scan path catches a std::exception: RETURN_IF_CATCH_EXCEPTION and the scanner guards only catch doris::Exception, so the exception escapes to std::terminate and the whole BE goes down.
We don't know the root throw site (which std::function was empty); the rethrow loses the original stack.
What You Expected?
A failure while initializing a rowset reader's iterators should fail that read (or the query) with an error status, not abort the process.
How to Reproduce?
We can't reproduce the original empty std::function deterministically. To show the missing guard: inject a throw std::bad_function_call() (or any std::exception) in BetaRowsetReader::_init_iterator via a debug point, then query the table: the BE terminates instead of returning an error.
Anything Else?
What we're doing as hardening:
BetaRowsetReader::_init_iterator_onceconverts adoris::Exceptionorstd::exceptionfrom the call into aStatus, on the first call and on later calls that hit the stored exception.DorisCallOnceitself stays unchanged._init_iteratorandLazyInitSegmentIterator::initlog the step that threw, with tablet, rowset and segment ids, so the next occurrence identifies the emptystd::function.
A unit test with a debug point that calls an empty std::function in _init_iterator gets a status containing bad_function_call from both the first and the second next_batch.
#68675 fixed a related cached-failure site (BetaRowset::get_segment_num_rows), but not this path. #61077 fixed a different std::bad_function_call (a data race in AnalysisFactoryMgr); we don't know whether that's the same root cause.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Oct 11, 2026
Breakwater-GitHub-Analysis-Slot: slot_db219bfe8e2e
This warrants high-priority BE/storage crash triage. The exception-to-Status boundary gap is supported by the source; the origin of the production
std::bad_function_callremains unproven. These are two separate investigation items, and a focused hardening PR need not wait for a deterministic reproduction of the original empty callable.I inspected the public
4.1.4source at4accb51bc1f27d84980668f3a2eaf0ac96fb3cbaand master atdd456ab7a894f54c6553415223c6d5c9c7d4e035. The issue currently has no labels. This was a read-only source investigation; I did not run the proposed injection test or reproduce the production crashes.Verified mechanism
BetaRowsetReader::_init_iterator_oncedirectly returns_init_iter_once.call(...)._init_iteratorbuilds segment iterators, constructs the merge/union iterator, and calls itsinit. Its non-OK Status handling does not handle an exception that bypasses the return.DorisCallOnce::callcatches all exceptions, storesexception_ptr, publishes the called flag, and rethrows. Later calls rethrow the saved exception without rerunning initialization. This is deliberate behavior, with synchronized publication; the supplied stack does not establish a race in DorisCallOnce itself._next_batchusesRETURN_IF_ERRORaround initialization. That checks a returned Status, not a thrown exception. The scanner'sRETURN_IF_CATCH_EXCEPTIONandASSIGN_STATUS_IF_CATCH_EXCEPTIONcatch onlydoris::Exception. A plainstd::bad_function_calltherefore escapes these guards. This supports the reported terminate path, conditional on that exception being thrown.PR scope and validation
Wrapping the entire
_init_iter_once.call(...)invocation is a reasonable narrow boundary: it handles both the initial throw and a later cached rethrow. Catchdoris::Exceptionbeforestd::exception, preserve Doris error codes and the existing memory-error mapping, and retainwhat()plus reader identity for standard exceptions. Return a non-OK Status so_next_batchstops before using a partially initialized iterator. Keep DorisCallOnce unchanged and do not retry initialization on that same reader: initialization mutates state before completion.There is a scope limit worth documenting.
VUnionIterator::next_batchinitializes subsequent segments after initial rowset initialization has succeeded. Those calls occur outside_init_iterator_once. Logging inLazyInitSegmentIterator::initfollowed by rethrow will provide diagnostics but will not contain such later exceptions. Include a later-segment injection case, and either cover that boundary or explicitly track it as a follow-up; do not describe the narrow patch as protecting every scan-time exception.For the proposed test, assert that the first and second
next_batchon the same reader return non-OK Status values containing the exception message, and that the injected initializer ran only once. Also check ordinary returned errors, Doris exception code preservation, successful reads, and query-level error propagation without BE termination. Record the exact tested commit and commands in the PR.Related fixes and remaining evidence
#68675 changes failed segment-row-count caching on a long-lived BetaRowset. It does not change this reader's exception boundary, and its retry policy should not be copied into partially initialized reader state. #61077 fixes an unsynchronized AnalysisFactoryMgr registry initialization check. In the inspected
4.1.4and master code,create()already callsinitialise()unconditionally before registry lookup. There is no evidence here that the previously fixed race is the current throw site.To establish the original cause, please attach:
Segment::new_iterator, and inner iterator initialization, with tablet/rowset/segment IDs and exception type/message. In a controlled reproduction, breaking on the standard-library bad-function-call throw helper, or usingcatch throw, can capture the original stack before DorisCallOnce caches/rethrows it.Timeout, heavy ingest, cloud mode, and arm64 are reported correlations, not verified causes. The current evidence supports containment work, but does not identify an empty callable, cancellation race, or architecture-specific defect.
yiguolei commentedon Oct 16, 2026
@matanper Very curious about this problem. The detail stack about the std::bad_function_call. I think we should find the stack and solve the problem.
matanper commentedon Oct 18, 2026
Unfortunately we failed to reproduce it. All three occurrences happened during automatic statistics collection (full min/max scans) on BEs under heavy concurrent load, and the exception was captured only at the rethrow in DorisCallOnce, so the original throw site was lost.