[Bug] Arrow Flight SQL causes FE Direct Memory leak until OOM ("Cannot reserve ... direct buffer memory", Internal; Prepare) #65305

Search before asking

• I had searched in the issues and found no similar issues.

Version

Apache Doris 4.1.2
(JDK 17.0.19, MaxDirectMemorySize=12g)

What's Wrong?

When a client keeps executing queries over Arrow Flight SQL (Python ADBC / adbc_driver_flightsql), the FE Direct Memory grows monotonically and is never released, until the FE can no longer allocate and every query fails with:

adbc_driver_manager.InternalError: INTERNAL: [FlightSQL] Unknown error:
java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory
(allocated: 12883979585, limit: 12884901888) (Internal; Prepare)

Key characteristics:

1. Java Heap is healthy — the leak is in Direct Memory, not the heap.

jstat -gcutil <fe_pid> on the affected FE:

 S0     S1     E      O      M     CCS     YGC     YGCT     FGC    FGCT     CGC      CGCT       GCT
0.00  61.42  0.10  17.66  99.69  98.94  12833   97.970    7     1.381   674882   3878.319  5658.417

• Old Gen (O) stays low (~14–18%): no heap leak.
• After we added -XX:+ExplicitGCInvokesConcurrent, Full GC (FGC) stays low, but Concurrent GC (CGC) explodes to 600k+ — i.e. the JVM is constantly running GC trying (and failing) to reclaim Direct Memory.

2. Direct Memory reaches exactly MaxDirectMemorySize and stays there.

allocated: 12883979585  ≈  limit: 12884901888  (12 GB)

3. io.netty.buffer.PoolSubpage accumulates massively on the FE that serves Arrow Flight traffic.

jcmd <fe_pid> GC.class_histogram on the abnormal FE:

io.netty.buffer.PoolSubpage        265337 instances    19,104,264 bytes
[Lio.netty.buffer.PoolSubpage;       3175 instances     6,355,568 bytes
[B (byte arrays)                   376158 instances   117,217,360 bytes

On a healthy FE (little/no Arrow Flight traffic), for comparison:

io.netty.buffer.PoolSubpage        <fill: run jcmd GC.class_histogram on a healthy FE> instances

In our environment the abnormal FE shows ~40–60x more PoolSubpage than idle FEs. Please fill in the healthy-FE numbers before submitting.

4. The failure always happens in the Prepare phase, and it strongly correlates with queries that error out. Our FE fe.audit.log shows a large number of State=ERR Arrow Flight queries (note Client=0.0.0.0:0, i.e. Arrow Flight), e.g. querying non-existent tables:

|Client=0.0.0.0:0|User=prod|Db=prod1|State=ERR|ErrorCode=1105|
ErrorMessage=errCode = 2, detailMessage = Table [xxx] does not exist in database [prod1].|
...|IsQuery=true|StmtType=SELECT|CommandType=Query|
Stmt= select ... from xxx tmp1 join ... where ...

These error queries have PeakMemoryBytes=0 and fail within ~1ms during analysis, yet the FE Direct Memory keeps climbing — suggesting buffers allocated on the Arrow Flight Prepare/error path are not released when the statement fails.

What You Expected?

Arrow Flight SQL statements — especially failed / cancelled ones — should release all Direct Memory (Arrow buffers / Netty PooledByteBuf) allocated during the Prepare phase, so that FE Direct Memory stays bounded under continuous query load.

How to Reproduce?

1. Enable Arrow Flight SQL on FE/BE.
2. From an ADBC client, continuously execute a large number of distinct SQL statements over Arrow Flight (each statement text is unique — note Doris Arrow Flight does not support parameter binding, so clients inline literals, making every query a new Prepare).
3. Include a portion of statements that error out (e.g. SELECT ... FROM a_nonexistent_table ...).
4. Monitor FE Direct Memory (e.g. jcmd <pid> GC.class_histogram watching io.netty.buffer.PoolSubpage, and the JVM Direct Memory usage).

Observed: FE Direct Memory increases monotonically and is not reclaimed by GC, eventually hitting MaxDirectMemorySize and failing every query with Cannot reserve ... direct buffer memory (Internal; Prepare).

Anything Else?

Client-side mitigations we already tried — none stop the growth, which indicates the retention is on the FE server side, not a client connection leak:

• Small Arrow connection pool (maxconnections=4).
• Periodic whole-pool rebuild by time and by query count (close + recreate all Flight sessions).
• Disabling the pool entirely (open a fresh connection per query, close it immediately after each query).
• -XX:+ExplicitGCInvokesConcurrent + larger MaxMetaspaceSize — this only removed a Full-GC storm; Direct Memory still leaks.

Even when the Flight session is explicitly closed after every single query, FE Direct Memory is not released — so the leaked buffers appear to be strongly referenced on the FE, not tied to session lifecycle.

Environment: 3 FE nodes (32 GB each). Only the FE that actually receives Arrow Flight traffic accumulates the leak; the other FEs (same shared metadata) stay healthy — ruling out a catalog/tablet/replica metadata leak and pointing at the Arrow Flight request-processing path.

Possibly related:

• [JAVA] Use prepared statement leads Memory leak arrow-java#50 ("Use prepared statement leads Memory leak", still open)
• A very similar report in the Doris Chinese forum on 2.1.10 (FE Direct Memory keeps growing with Arrow Flight SQL until create prepared statement failed, Failure allocating buffer / OutOfDirectMemoryError).

We could not find an existing PR that fixes this specific FE Arrow Flight Prepare-phase Direct Memory leak (the arrow memleak fixes we found, e.g. #51929, are BE-side and unrelated). Could the maintainers confirm whether this is known and in which version it is fixed?

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct