[Bug] @incr visibility-wait timeout in shared-nothing mode is reported without error code 5101 #70279

Activity

1. Doris-Breakwater commented on Nov 5, 2026

   

   Doris-Breakwater

   on Nov 5, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_fb0669e965d4

   Triage: the error-classification inconsistency is supported by the source at 72e91896bbdd8760f8bd4e488e5672406b40ce52 (high confidence). This is an FE incremental-read/transaction-fence issue with a connector retry consequence. The issue is currently open with no labels. I verified the pinned revision and the cited connector revision; I did not reproduce this on a cluster, run tests, or verify the claim that today's master is unchanged. No implementation changes were made.

   Verified mechanism

   • Strongly consistent shared-nothing reads take the transaction-watermark drain and publisher-lock synchronization path. Both deadline expiration in waitForPreviousTransactions() and failure to obtain the publisher read lock in time throw a plain UserException. That exception defaults to ERR_UNKNOWN_ERROR, rather than 5101. Bounded shared-storage reads instead use TSOService.waitForReadableWindow(), which creates an IncrWindowNotReadyException with 5101 for a visibility deadline and 5100 for a future window or master change. See the fence implementation, classic timeout sites, and shared-storage classification.
   • The missing classification persists across component boundaries. Planning wraps the exception; StmtExecutor specifically preserves the code of IncrWindowNotReadyException, while other planning errors take the generic-error path. The master fence RPC likewise serializes structured window-error fields only for that exception type. Flight emits UNAVAILABLE with doris-error-code metadata only when query state holds 5100/5101; otherwise it emits INTERNAL. Consequently, changing only the message, or assigning a code to an otherwise plain UserException, would not complete the fix through these paths. See planner-error handling, master RPC handling, and Flight metadata.
   • The cited connector retries the same statement only when ADBC details contain doris-error-code=5101. Its source.changelog.visible-wait-timeout controls the client retry budget; it cannot make an unclassified FE timeout eligible for retry. 5100 is not retried by this connector revision, so returning 5100 for an actual visibility timeout would not address the reported failure. See connector retry logic.

   Reproduction qualification

   The classic conflict check considers non-final transactions touching the requested tables with transaction ID at or below the watermark; it does not compare their commit TSO with the requested window. This supports the proposed long-load blocker, provided its transaction is registered before the fence captures the watermark. It does not by itself establish that this conservative drain is incorrect. See transaction predicate.

   The identical slow-load scenario need not time out in shared-storage mode: its bounded wait filters registered commit TSOs by the window end and target tables. A transaction outside that range may correctly leave the read unblocked. To demonstrate shared-storage 5101, hold an in-range registered transaction unfinished. Also choose the reproduction's end timestamp at or below the master's current TSO physical time; otherwise classic validation can reject the query before reaching the visibility wait.

   Recommended next steps

   1. Classify the actual classic visibility deadlines as 5101 using the existing typed window-error propagation, or an equivalent mechanism verified through planning, RPC, and Flight. Preserve the drain, publisher synchronization, and useful transaction/table diagnostics. Keep interruption, transaction-check failure, and transport failure distinguishable from a visibility deadline.
   2. Include follower execution in scope: classic follower journal replay has additional generic timeout exits. The fence RPC currently grants extra response time only to bounded cloud reads, so a classic follower can encounter a socket timeout before receiving a typed master timeout. Cover these boundaries explicitly rather than considering the two master-side throws sufficient.
   3. Add regression coverage for classic drain timeout and publisher-lock timeout, direct and follower execution, query-state code preservation, and Flight metadata. Verify the connector retries an unchanged window within its budget, succeeds after the blocker clears, and stops when the budget expires. Existing typed-error tests exercise cloud paths and do not establish classic coverage.

   Still needed for runtime confirmation: an executable table DDL/load script and exact timestamped SQL; confirmation that the blocking transaction was registered before fence acquisition; actual change_visible_timeout_ms and consistency settings; connected FE role; FE query ID/logs with transaction states; connector build/options and full ADBC error details showing metadata. These are needed to validate the reproduction and observed failover, but the source is already sufficient to justify an error-propagation fix. There is no evidence here establishing data loss or the frequency of this failure in deployed workloads.