[Bug] SHOW QUERY STATS leaks FE Thrift clients #70317

Activity

1. Doris-Breakwater commented on Nov 6, 2026

   

   Doris-Breakwater

   on Nov 6, 2026

   Collaborator

   More actions

   The reported client lifecycle defect is confirmed by source inspection. I checked the exact master commit 87b462e4f58104140f098f5f4a8b5d58bec70149 and the public 2.1.8-rc01 tag. Both borrow a peer FE client in QueryStatsUtil.getStats() without any corresponding return or invalidation. This supports treating the issue as an FE resource leak with potential cluster availability impact. I have not run a multi-FE reproduction or independently observed worker exhaustion.

   The relevant evidence is:

   • In master QueryStatsUtil.getStats(), normal completion, status rejection, empty results, missing-field validation exceptions, and RPC exceptions all leave the borrowed client outstanding. The 2.1.8-rc01 implementation has the same defect.
   • ClientPool uses unlimited total and per-address active-client limits for frontendPool. Its 128 idle-client limit does not cap clients that were never returned. GenericPool delegates ownership release to explicit return/invalidation; invalidation destroys the transport.
   • The reported worker symptom is consistent with the blocking TThreadPoolServer setup. In Config.java at this commit, thrift_server_type defaults to THREAD_POOL, and thrift_client_timeout_ms defaults to zero. Persistent connections can therefore retain workers waiting for their next request. Actual exhaustion depends on the deployed server type, timeout, traffic, and worker limit; it is not established merely by finding the leak.

   The next implementation step should be cleanup in a per-peer finally block, using the same address key as the borrow. Mark the RPC as completed immediately after getQueryStats() returns, before validating status or payload. Return the client on every completed RPC, including rejected status, empty payload, and subsequent local validation exceptions. Invalidate an acquired client if the RPC throws; do neither if no client was acquired. Preserve the existing result filtering and continuation to other peers.

   Suggested regression coverage:

   • Valid results, missing/non-OK status, empty payloads, and missing expected payload fields: exactly one return and no invalidation per acquired client. Cover each payload-validation branch.
   • RPC exception: exactly one invalidation and no return. Borrow failure: no cleanup of an unacquired client. Verify processing continues to the next live peer; self and dead peers are skipped.
   • Repeated sequential requests against two FEs: after warm-up, connections plateau instead of growing with request count, and outstanding borrows return to baseline after requests finish. Healthy idle pooled connections may remain open; expecting zero TCP connections would be incorrect.

   To tie the deployment symptoms to this verified defect, please add the exact FE version/build commit on each node, the precise SQL variant and request frequency/concurrency, FE topology, and effective thrift_server_type, thrift_client_timeout_ms, and thrift_server_max_worker_threads. Capture connection counts before/after a known number of requests, filtering both the peer RPC port and the requesting FE address—the supplied port-only command also counts unrelated clients. A peer FE thread dump showing the accumulating Thrift workers and corresponding FE timeout/rejection logs would establish the exhaustion mechanism. These details are not prerequisites for fixing the demonstrated ownership bug, but are needed to confirm the full operational impact and backport applicability.

   Until a verified fix is deployed, pause or reduce automated polling of the affected command to limit additional leaks. This does not release clients already leaked. Backport selection should follow verification against the exact target build; this investigation does not establish a fixed release.

   Breakwater-GitHub-Analysis-Slot: slot_274008bc5f42