[Bug] BE rs_normal scanner threads leak on every external catalog query; doris_max_remote_scanner_thread_pool_thread_num has no effect #67898

Activity

1. Doris-Breakwater commented on Sep 12, 2026

   

   Doris-Breakwater

   on Sep 12, 2026

   Collaborator

   More actions

   Initial triage: the evidence is internally consistent and this should be treated as a high-severity availability/resource-exhaustion problem. It is also a near-exact duplicate of #65416, which reports the same unbounded rs_normal growth for Paimon-on-S3 on 4.1.1-rc01, including ineffective BE and workload-group caps. This report adds a much better controlled reproducer, so the two issues should be linked and investigated together.

   What is verified from the reported revision

   I checked the exact build commit, 7126cf65d96ebc43fce0906f51e92c1a2ccf24a6:

   • rs_normal is the remote scan scheduler owned by the normal workload group (construction path). Having only one workload group narrows the problem, but does not bypass this workload-group-owned scheduler.
   • The raw default of doris_max_remote_scanner_thread_pool_thread_num is indeed -1 (config), but -1 is an automatic-sizing sentinel, not an unlimited value: it becomes max(512, detected_cores * 10) (calculation). Thus /api/show_config = -1 and the documented effective floor of 512 are compatible, although the documentation should explain the sentinel.
   • With the shown local size of 48, an explicit BE value of 256 should yield a 256 maximum unless the normal workload group supplies a positive max_remote_scan_thread_num, which overrides the BE fallback (override path).
   • The task executor only creates a worker when num_threads + pending_start < max_threads (cap check), and idle workers above the minimum are supposed to retire after 500 ms (retirement path). A single correctly accounted pool therefore cannot intentionally contain 3,192 or 35,000 live workers.

   This confirms that the observed behavior violates the intended cap/lifecycle. It does not yet identify which invariant is failing. The remaining possibilities include scheduler accounting/teardown, multiple same-named scheduler owners, or non-Doris/JVM-side threads retaining or inheriting the native name. The thread-name histogram alone cannot distinguish them, so I would not assign a Paimon or scheduler root cause yet.

   Highest-value missing evidence

   Please collect the following on a freshly restarted BE, immediately before and after the 20-query reproducer:

   1. SHOW WORKLOAD GROUPS output for normal, especially max_remote_scan_thread_num and min_remote_scan_thread_num.
   2. The BE INFO lines containing [topic_publish_wg] ... thread info=. This revision already logs remote-pool accounting as r_scan num:<accounted>, real_num:<thread-set-size>, min_num:<...>, max_num:<...> (format). The before/after values are the most direct test of whether the executor itself sees the leaked threads.
   3. /threadz and /threadz?group=thread%20pool before/after. Comparing Doris-registered thread counts with /proc/<pid>/task will show whether all rs_normal TIDs were created by Doris's thread wrapper.
   4. For 5 newly created and 5 old rs_normal TIDs: /proc/<pid>/task/<tid>/status, wchan, and stack (if permitted), plus native backtraces. Also provide jcmd <pid> Thread.print -l or jstack <pid> and map those decimal TIDs to Java nid=0x.... This will establish whether they are idle in the scheduler, blocked in JNI/Paimon/Hadoop/S3 code, or absent from the JVM.
   5. The Paimon table type (append-only vs primary-key), file format (ORC/Parquet), bucket mode, approximate split count for the test query, and whether every query completed normally or any were cancelled/timed out.

   Suggested next steps

   • Link/consolidate with BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes #65416 and route jointly to BE execution/workload scheduling and Lakehouse/Paimon maintainers. No labels are currently set; kind/bug and area/catalog are appropriate, plus the repository's high-severity/availability label if applicable.
   • Use r_scan num/real_num/max_num plus representative stacks to choose the owner before changing code. If real_num stays at or below 256 while /proc grows, investigate JNI/plugin-created threads or native-name inheritance. If real_num itself exceeds max_num, the task-executor accounting invariant is directly broken. If several pool instances appear, trace workload-group scheduler creation/destruction.
   • Run one staging A/B on the same table with SET enable_paimon_cpp_reader=true if that reader supports this table. Growth only on the default JNI path would isolate Paimon/JNI lifecycle from the common remote scheduler; growth on both paths would point back toward scheduler ownership/accounting.
   • The reported build predates Paimon observability and cleanup work in [improvement](paimon) Add Paimon JNI scanner observability #65354, [opt](paimon) control paimon jni reader async file reader #65365, and [fix](be) Harden JNI reader split lifecycle #65503. A repeat of the exact 20-query test on 4.1.4-rc04 is useful because that source contains the relevant async-reader metrics/control and iterator/JNI cleanup changes. This is a diagnostic upgrade test, not a claim that the issue is already fixed.
   • Until ownership is confirmed, rolling BE restarts before thread/RSS exhaustion are the only demonstrated mitigation here. The 256 cap and allocator tuning should not be presented as fixes for the observed behavior.

   Breakwater-GitHub-Analysis-Slot: slot_52e24c6dcbe1
2. wangmingzhou1986 commented on Sep 14, 2026

   

   wangmingzhou1986

   on Sep 14, 2026

   Author

   More actions

   Thanks for the very precise triage — it pointed straight at the right discriminator. I collected the requested evidence, and it resolves the ambiguity: the leaking threads are not Doris scheduler threads at all.

   TL;DR

   The rs_normal name in /proc/<pid>/task/*/comm is inherited comm, not a Doris thread. In jstack those same TIDs are AWS SDK v2 sdk-ScheduledExecutor-<n>-<m> threads. Doris's remote scan pool is correctly bounded the whole time (r_scan real_num:8, cap 256). What actually leaks is AWS SDK async clients created per scan and never closed — roughly 2 SDK ScheduledExecutor instances (10 threads) per query per BE.

   So your first branch is the right one:

   If real_num stays at or below 256 while /proc grows, investigate JNI/plugin-created threads or native-name inheritance.

   Both conditions hold simultaneously.

   Build: doris-4.1.3-rc02-7126cf65d96 (same as reported). JVM: OpenJDK 17.0.2+8-86.

   ---

   1. SHOW WORKLOAD GROUPS for normal

   Name   scan_thread_num  max_remote_scan_thread_num  min_remote_scan_thread_num
   normal      -1                    -1                          -1
   

   All three are -1 at the workload-group level, i.e. the group does not override the BE fallback. Consistent with your reading of the override path.

   2. [topic_publish_wg] ... thread info= — the decisive measurement

   Identical before and after the 20-query reproducer, on both BEs:

   thread info=scheduler: p_normal_simple_scheduler, info:
     [exec num:16, real_num:16, min_num:16, max_num:16],
     [l_scan num:48, real_num:48, min_num:48, max_num:48],
     [r_scan num:8, real_num:8, min_num:8, max_num:256],
     [mem_tab_flush num:8, real_num:8, min_num:8, max_num:64]
   

   r_scan real_num never moves off 8, while /proc grows by hundreds. The executor's accounting is not broken, and the explicit BE value of 256 is in effect as max_num — it simply does not govern the threads that are actually leaking.

   This also corrects the framing of my original report: it is not that doris_max_remote_scanner_thread_pool_thread_num "has no effect". It bounds the remote scan scheduler correctly; it has no effect on this leak because the leaked threads were never in that pool. The issue title is misleading and should probably be amended.

   3. Controlled before/after (freshly restarted BE)

   Reproducer: 20 × SELECT COUNT(*) FROM paimon_lake.store.sys_user. All 20 completed normally (0 cancelled, 0 timed out).

   doris-be-0 — restarted immediately before BEFORE:

   metric	BEFORE	AFTER	Δ
   /proc/<pid>/task total	1,783	1,993	+210
   comm == rs_normal	8	217	+209
   JVM threads (jstack)	27	244	+217
   sdk-ScheduledExecutor threads	0	200	+200
   distinct sdk-ScheduledExecutor-<n> instances	0	40	+40
   VmRSS	1,008 MB	1,236 MB	+228 MB
   r_scan real_num (BE self-report)	8	8	0
   /threadz rows	16	16	0

   doris-be2-0 — not restarted, ~15 h uptime, same workload:

   metric	BEFORE	AFTER	Δ
   /proc/<pid>/task total	39,148	39,350	+202
   distinct SDK instances	7,473	7,513	+40
   r_scan real_num	8	8	0

   Rate: +2 SDK executor instances = +10 threads, per query, per BE. Identical on a clean BE and on one that already holds 37 k threads — the leak rate does not decay.

   /threadz and /threadz?group=thread pool are unchanged across the run, i.e. Doris-registered pools are stable while /proc grows.

   4. Thread identity — this is where it becomes unambiguous

   On the 15 h BE (37,537 threads total, 35,757 with comm == rs_normal):

   jstack thread-name histogram (top):
     35745  sdk-ScheduledExecutor-<n>-<m>
        10  Thread-<n>
         6  GC Thread#
         6  G1 Refine#
         1  org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner
         1  Timer for 's3a-file-system' metrics system
         1  idle-connection-reaper
         ... (VM/Service/Sweeper/Finalizer, 1 each)
   
   jstack total JVM threads : 35,789
   distinct sdk-ScheduledExecutor instances : 7,149  (ids 0..7148, ~5 threads each)
   occurrences of "rs_normal" / "scan" / "Scanner" in jstack : 0
   

   Mapping the decimal TIDs to nid=0x… as you asked — 5 lowest and 5 highest rs_normal TIDs, all ten are JVM threads:

   TID 10000 (0x2710) -> "sdk-ScheduledExecutor-1096-3" #6135 daemon
   TID 10001 (0x2711) -> "sdk-ScheduledExecutor-1100-3" #6136 daemon
   TID 10002 (0x2712) -> "sdk-ScheduledExecutor-1091-3" #6137 daemon
   TID 10003 (0x2713) -> "sdk-ScheduledExecutor-1088-3" #6138 daemon
   TID 10004 (0x2714) -> "sdk-ScheduledExecutor-1093-2" #6139 daemon
   TID  9995 (0x270b) -> "sdk-ScheduledExecutor-1089-2" #6109 daemon
   TID  9996 (0x270c) -> "sdk-ScheduledExecutor-1099-2" #6131 daemon
   TID  9997 (0x270d) -> "sdk-ScheduledExecutor-1097-1" #6094 daemon
   TID  9998 (0x270e) -> "sdk-ScheduledExecutor-1092-2" #6133 daemon
   TID  9999 (0x270f) -> "sdk-ScheduledExecutor-1077-3" #6134 daemon
   

   /proc/<pid>/task/<tid>/status for those TIDs:

   Name:   rs_normal [work        <- truncated at TASK_COMM_LEN (16)
   State:  S (sleeping)
   wchan:  futex_wait_queue_me
   

   (/proc/<pid>/task/<tid>/stack is not readable in our container — no CAP_SYS_ADMIN. The Java-side stack below makes it unnecessary.)

   Every single one of the 35,745 is WAITING (parking), with an identical stack:

   "sdk-ScheduledExecutor-1096-3" #6135 daemon prio=5 os_prio=0 cpu=0.29ms elapsed=46067.04s nid=0x2710 waiting on condition
      java.lang.Thread.State: WAITING (parking)
   	at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)
   	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.2/LockSupport.java:341)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(...)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(...:1177)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(...:1062)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(...:1122)
   	at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)
   

   cpu=0.29ms after elapsed=46,067s — these threads have done essentially nothing since creation. They are parked scheduler threads of abandoned AWS SDK async clients.

   Why they are named rs_normal

   Linux threads inherit comm from the creating thread unless explicitly renamed. These executors are constructed on a Doris rs_normal [worker] thread (inside the Paimon JNI call), and the JDK does not rename ScheduledThreadPoolExecutor worker threads at the OS level. So /proc shows rs_normal [work for threads that have nothing to do with the Doris scheduler. A /proc-based thread-name histogram is actively misleading here — exactly the caveat you raised.

   Scope of the leak

   Note Timer for 's3a-file-system' metrics system and idle-connection-reaper appear exactly once. So S3AFileSystem itself is effectively a singleton, but the SDK clients underneath are not — 7,149 of them accumulated in ~15 h. The lifecycle problem is on the client, not on the filesystem object.

   5. Table / query characteristics

   Table	Paimon primary-key table (primaryKeys: [user_id])
   Partition keys	none
   File format	parquet, zstd
   Bucket	4
   Columns	24
   Storage	S3 (MinIO), s3a:// via Hadoop
   Query	SELECT COUNT(*), small table, few splits
   Outcome	20/20 completed normally, none cancelled or timed out

   The leak therefore does not require large scans or failures — 20 trivial counts on a tiny table are enough.

   6. enable_paimon_cpp_reader A/B — attempted, but not a clean result

   I ran the same 20 queries with SET enable_paimon_cpp_reader=true. All 20 failed at:

   doris::PaimonCppReader::init_reader()
   

   presumably because the C++ reader does not support this primary-key table in this build. Thread and SDK-instance deltas were exactly 0.

   I want to be explicit that this is not evidence that the C++ path is leak-free — the queries never reached the read path. It is only consistent with the leak living behind the JNI reader. A proper A/B needs an append-only table that the C++ reader actually supports; I can run that if it would help.

   ---

   Where this leaves the investigation

   • The task-executor accounting invariant is not broken (num == real_num == 8, max_num == 256).
   • There are not multiple same-named scheduler owners.
   • It is native-name inheritance plus a genuine JNI-side client lifecycle leak: each Paimon scan appears to build an AWS SDK async client (with its own 5-thread ScheduledExecutor) that is never close()d, so both the threads and their per-thread allocator caches accumulate until rss hits mem_limit.

   That suggests ownership sits with the Lakehouse/Paimon JNI reader (S3 client construction/close per scan or per split), not with BE execution/workload scheduling. Linking with #65416 still makes sense — the /proc symptom is identical and would have the same explanation.

   Two small notes on the mitigation guidance in the original report, for anyone who finds this later:

   • Rolling BE restarts remain the only demonstrated mitigation. Confirmed again here: a restart took this BE from 37,537 threads / 5,449 MB to 1,783 threads / 1,008 MB.
   • SHOW BACKENDS reports Alive: true right up to the point where every query fails on MEM_LIMIT_EXCEEDED, so it must not be used as the health signal. Tracking thread count gives hours of warning; tracking memory percentage alone does not.

   Happy to collect anything else — an append-only-table C++ reader A/B, a repeat on 4.1.4-rc04, or jcmd VM.native_memory / allocator breakdowns.
3. wangmingzhou1986 commented on Sep 14, 2026

   

   wangmingzhou1986

   on Sep 14, 2026

   Author

   More actions

   Resolved in 4.1.4. Upgraded the cluster today and the leak is gone.

   Measurement

   Same reproducer as before (20 × SELECT COUNT(*) on a Paimon primary-key table via external catalog), both BEs freshly restarted:

   metric, per BE per 20 queries	4.1.3 (doris-4.1.3-rc02-7126cf65d96)	4.1.4 (doris-4.1.4-rc04-ad35a140c7f)
   distinct sdk-ScheduledExecutor-<n> instances	+40	+0
   threads in /proc/<pid>/task	+200	+1 (noise)

   BE              threads before  after   Δ      SDK instances before  after   Δ
   doris-be-0             1860     1861    +1                       1      1    +0
   doris-be2-0            1851     1852    +1                       1      1    +0
   

   The SDK executor instance count stays at 1 on both BEs instead of growing by 2 per query. That is exactly the mechanism identified earlier (AWS SDK async clients created per Paimon scan and never close()d), so this issue can be closed as fixed in 4.1.4.

   One small note for anyone searching: the published image apache/doris:be-4.1.4 self-reports as doris-4.1.4-rc04-ad35a140c7f. There is no separate -rc04 tag on Docker Hub — the rc04 build is the 4.1.4 release.

   A compatibility caveat worth documenting for Paimon-catalog users

   4.1.4 bumps the bundled Paimon from 1.3.1 to 1.4.2, and the java_extensions directory is renamed paimon-scanner → paimon-connector. Two things followed from that in our environment:

   1. Lake-format compatibility is fine. Our tables are written by Paimon 1.3.1 (Flink CDC) and now read by 1.4.2. Before upgrading I verified this in isolation — a throwaway pod running the 4.1.4 image with sleep (no Doris process), using its bundled Paimon 1.4.2 through the Java API to read production tables directly:

   • 3 static tables matched field-by-field (11×24, 1×16, 2159×10)
   • a 57.8M-row primary-key table, both engines pinned to the same snapshot, matched exactly: 57,783,725 from FOR VERSION AS OF 2333 on 4.1.3 and 57,783,725 from the 1.4.2 direct read. After the upgrade the same query still returns 57,783,725.

   (Comparing row counts across engines on a live table is meaningless — the table commits every ~30 s, so each engine pins a different snapshot. Pinning both to one snapshot is what makes the comparison valid.)

   2. FE and BE must be on the same Paimon version. The documented upgrade order is BE first, then FE. During that window — BE on 4.1.4, FE still on 4.1.3 — every external-catalog query fails:

   [INTERNAL_ERROR]failed to init reader, err: [JNI_ERROR]RuntimeException:
   java.io.InvalidClassException: org.apache.paimon.table.CatalogEnvironment;
   local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2
   

   The FE serializes Paimon split/table objects and the BE deserializes them; CatalogEnvironment's serialVersionUID changed 1 → 2 between Paimon 1.3.1 and 1.4.2. So for deployments using Paimon external catalogs, the BE-then-FE window is a planned outage, not a transparent rolling upgrade. Ours lasted about 6 minutes. It might be worth a line in the upgrade docs.

   Happy to run anything else if it would help confirm.
4. Asthenia0412 commented on Sep 14, 2026

   

   Asthenia0412

   on Sep 14, 2026

   Contributor

   More actions

   Hi @wangmingzhou1986, I've submitted the master-side fix in #67963: Paimon is bumped 1.3.1 -> 1.4.2 (same as the 4.1.4 release you verified), with the paimon-scanner code adapted to the 1.4.2 API. This should fix the AWS SDK async client thread leak on master. Thanks for the detailed triage!
5. Asthenia0412

   added a commit that references this issue on Sep 14, 2026

   [fix](fe) Upgrade Paimon to 1.4.2 to fix AWS SDK async client thread …

   ...

   This issue will close once commit 7f1620b is merged into the 'master' branch.

   7f1620b
6. wangmingzhou1986 commented on Sep 15, 2026

   

   wangmingzhou1986

   on Sep 15, 2026

   Author

   More actions

   @Asthenia0412 thanks — that matches what we measured. Confirming the fix from the release side, in case it helps the PR review.

   We upgraded the whole cluster (2 FE + 2 BE) from doris-4.1.3-rc02-7126cf65d96 to doris-4.1.4-rc04-ad35a140c7f and re-ran the identical 20-query reproducer on freshly restarted BEs:

   per BE, per 20 external-catalog queries	4.1.3	4.1.4
   distinct sdk-ScheduledExecutor-<n> instances	+40	+0
   threads in /proc/<pid>/task	+200	+1 (noise)

   So the Paimon 1.3.1 → 1.4.2 bump is indeed what fixes it, and the three JNI-cleanup PRs suggested earlier in triage were not the relevant change. Since #67963 is the same bump for master, I'd expect it to close this.

   Two things we verified before upgrading that may save someone else the trouble:

   1. Lake-format compatibility (1.4.2 reader over 1.3.1-written tables) holds. Our tables are written by Flink CDC with Paimon 1.3.1. Before touching production we ran the 4.1.4 image as a throwaway sleep pod (no Doris process) and used its bundled Paimon 1.4.2 through the Java API to read production tables directly. Three static tables matched field-by-field, and a 57.8M-row primary-key table matched exactly when both engines were pinned to the same snapshot: 57,783,725 from FOR VERSION AS OF 2333 on 4.1.3, and 57,783,725 from the 1.4.2 direct read. After the upgrade the same query still returns 57,783,725.

   Worth noting for anyone repeating this: comparing row counts across engines on a live table is meaningless — ours commits every ~30 s, so each engine pins a different snapshot and the numbers differ in both directions. Pinning both sides to one snapshot is what makes the comparison valid.

   2. FE and BE must be on the same Paimon version — the documented BE-then-FE order is an outage window for Paimon catalogs. With BE on 4.1.4 and FE still on 4.1.3, every external-catalog query fails:

   [INTERNAL_ERROR]failed to init reader, err: [JNI_ERROR]RuntimeException:
   java.io.InvalidClassException: org.apache.paimon.table.CatalogEnvironment;
   local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2
   

   The FE serializes Paimon split/table objects for the BE to deserialize, and CatalogEnvironment's serialVersionUID changed 1 → 2 between Paimon 1.3.1 and 1.4.2. Ours was down about 6 minutes. For deployments using Paimon external catalogs this is a planned outage rather than a transparent rolling upgrade — might be worth a line in the upgrade docs alongside this change.

   Happy to test anything else on 4.1.4 if useful.