[morningman]

What problem does this PR solve?

Issue Number: None

Related PR: #69298, #69566, #66530

Problem Summary:

In short

A statement's connector state could outlive the statement in two ways on master. A split producer still inside a read when the statement ends went on to load values into the already-closed connector statement scope, such as an Iceberg table borrowed from the table cache, and nothing ever closed them. EXPLAIN REFRESH MATERIALIZED VIEW plans on a context of its own that has no query id, so its scans registered their query-finish cleanup under "" and its connector scope was never closed: each such EXPLAIN over an external base table left one registry entry and one open scope behind, without bound. This PR makes a closed scope refuse to load, and finishes the explain context's own statement when the EXPLAIN ends.

This was found while checking master against branch-4.x's #66473 (scan reuse, ported as #66530). branch-4.x has neither the connector statement scope nor the query-finish registry, and its statement resources already refuse registration once closed.

Background

• Every statement has a connector statement scope (ConnectorStatementScopeImpl, hung on the StatementContext). Connectors keep what the statement shares in it: the ConnectorMetadata of each catalog, Iceberg table leases borrowed from the table cache (returning one is what lets the cache close the table's FileIO when it evicts it), Iceberg tracked tables, write transactions, and the query's task groups since [improvement](datasource) Bound and fairly schedule external scan planning #69298. closeAll() releases them when the statement ends.
• A scan registers two query-finish callbacks under the query id when it starts planning splits: one stops its split producers, the other (QueryFinishCleanup) releases a read transaction and closes the scope. [improvement](datasource) Bound and fairly schedule external scan planning #69298 runs them at the end of every attempt, also for EXPLAIN and for plans that fail; [fix](fe) Finish a failed prepared EXECUTE like a failed COM_QUERY #69566 does the same for prepared EXECUTE.
• Batch and streaming scans plan on split producers. A producer calls into the connector off the statement thread, for example Iceberg's streamSplits, which resolves its table through the scope.

The problem, and what it cost

1. Late producers load into the closed scope. Stopping a scan only flags its producers (SplitProducer.close()), and closing the query's task groups cancels queued work but deliberately does not interrupt metadata IO. A producer already inside getNextBatch keeps going after closeAll(). closeAll() had also cleared the scope, so the producer found nothing and loaded again: Iceberg borrowed a new table lease, Hive partition batches built a new ConnectorMetadata. Both stayed in the closed scope forever, so the lease was never returned. A value whose load was in flight while closeAll() ran was missed the same way. This happens whenever a statement ends while a producer is still reading: an EXPLAIN of a large table (EXPLAIN also starts batch scans), a LIMIT that finishes early, a KILL, a timeout. Nothing fails and no result is wrong; the FE just keeps the leases, metadata and FileIO alive.
2. EXPLAIN REFRESH MATERIALIZED VIEW leaks its own statement. It plans on an MTMV context of its own (RefreshMTMVCommand.getExplainConnectContext), and that context had no query id, so the connector sessions of its scans had "" as their query id and registered their cleanup under "". The end of the outer statement finishes only the outer query id; nothing finishes "", and it could not be finished at the end of one statement anyway, since every context without a query id shares it. The explain context's StatementContext was never closed either.

Before this PR	Effect
A producer reading when the statement ends	Borrows a new table lease (or builds a new metadata) into the closed scope; never released
EXPLAIN REFRESH MATERIALIZED VIEW over an external base table	One permanent registry entry under "" and one open connector scope per execution

How this PR fixes it

• ConnectorStatementScopeImpl: after closeAll() the statement is over, so computeIfAbsent (and with it computeIfAbsentForPlanning and getOrCreateMetadata) does not run the loader and throws CancellationException, as the query's task groups already do once closed; Iceberg's planning paths treat it as "the query stopped" and do not fall back to the SDK. A load that raced the close is released by its caller, because closeAll() could not see it. closeAll() now removes each entry before releasing its value, still finalizing transactions first, so every value is released exactly once. getIfPresent returns nothing once closed.
• StatementContext.getConnectorStatementScopeForNewSession(): a session built on a statement whose scope is already closed (the lineage of an INSERT, after the insert finished its query, say) captures ConnectorStatementScope.NONE and the direct task scheduler, as offline work does, instead of failing in or loading into the closed scope. Sessions captured before the close keep the closed scope, so a producer holding one stops at its next load.
• ExplainCommand: an explain context of its own gets a query id, and when the EXPLAIN ends, whether planning succeeded or not, it runs that query id's query-finish callbacks and closes the context's StatementContext, as MTMVTask does for a refresh. The thread-local context is still restored if that cleanup throws.
• Comments that said EXPLAIN never registers a query-finish cleanup and that closeAll() runs after the producers have quiesced are corrected; the SPI javadoc of computeIfAbsent states the closed-scope behavior.

Results

	Before	After
Producer resolving its table after the statement closed the scope	New lease borrowed, never returned	CancellationException, nothing borrowed
Value whose load raced closeAll()	Kept in the closed scope, never closed	Closed once by its caller
Session built after the scope closed	Loads task groups into the closed scope	Offline session (no scope, direct scheduler)
EXPLAIN REFRESH MATERIALIZED VIEW	Entry under "" and open scope left behind	Its callbacks run once, its scope closed

New tests:

• ConnectorStatementScopeTest: closedScopeRefusesToLoadAndHandsOutNothing, valueLoadedWhileTheScopeClosesIsReleasedOnceByItsCaller (the loader blocks on another thread while closeAll() runs), sessionBuiltAfterTheScopeClosesRunsAsOfflineWork.
• SplitProducerTest.readThatOutlivesItsStatementLoadsNothingIntoTheClosedScope: a producer is stopped inside getNextBatch, the scope closes, then the read resolves its table.
• ExplainRefreshMTMVCommandTest.testExplainCommandFinishesTheStatementOfItsOwnContext: planning registers a callback under the explain context's query id and keeps a closeable in its scope; after the EXPLAIN, both succeeding and failing, the callback ran once and the value was closed.

Each new test was checked against a mutation of the code it guards; every mutation turns its test red:

Mutation	Red test
Drop the closed check before loading	closedScopeRefusesToLoadAndHandsOutNothing (the loader ran twice), readThatOutlivesItsStatementLoadsNothingIntoTheClosedScope (the late read loaded once)
Drop the closed check after loading	valueLoadedWhileTheScopeClosesIsReleasedOnceByItsCaller (the caller got the value)
Let a new session capture the closed scope	sessionBuiltAfterTheScopeClosesRunsAsOfflineWork (building the session throws: the scope cannot load engine:query-task-groups)
Do not finish the explain context	testExplainCommandFinishesTheStatementOfItsOwnContext (callback never ran)
Give the explain context no query id	testExplainCommandFinishesTheStatementOfItsOwnContext (planned without a query id)

Release note

None

Check List (For Author)

• Test: Unit Test
  • FE UT: 51 related test classes in fe-core, 457 tests, all pass with TZ=Asia/Shanghai: ConnectorStatementScopeTest, SplitProducerTest, ExplainRefreshMTMVCommandTest, StmtExecutorTest, ConnectorSessionImplTest, QueryTaskGroupsTest, QueryFinishCallbackRegistryTest, MysqlConnectProcessorExecuteCleanupTest, ExecuteCommandTest, StatementContextTest, TaskProcessorTest, ConnectProcessorRetryTest, PluginDrivenScanNode*Test (36 classes), PluginDrivenMetadataTest, PluginDrivenMissingTableTest, NereidsPlannerConnectorPlanningTest, SplitAssignmentTest, SplitStreamingConcurrencyTest, CloudInsertProfileRetryTest; fe-core Checkstyle passes
• Behavior changed: Yes. A connector statement scope refuses to load once the statement has closed it (CancellationException); a connector session built after that runs without a statement scope.
• Does this need documentation: No

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morningman]

run buildall

[morningman]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: workspace routing discovery unauthorized (401)
Workflow run: https://github.com/apache/doris/actions/runs/40629912807

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[hello-stephen]

TPC-H: Total hot run time: 28811 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f8374ca590bbeff28f03d38a9656f880a115c36, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17578	5202	5073	5073
q2	2090	298	260	260
q3	10257	1370	844	844
q4	4677	458	335	335
q5	7414	788	528	528
q6	180	186	149	149
q7	740	790	595	595
q8	9246	1306	1038	1038
q9	5538	4232	4211	4211
q10	6831	1287	1016	1016
q11	424	245	226	226
q12	632	390	285	285
q13	18095	3546	3001	3001
q14	269	274	246	246
q15	q16	717	718	671	671
q17	1177	741	572	572
q18	7036	6171	6137	6137
q19	1108	944	598	598
q20	394	347	226	226
q21	4434	2516	2510	2510
q22	390	336	290	290
Total cold run time: 99227 ms
Total hot run time: 28811 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6031	5890	5957	5890
q2	744	589	560	560
q3	5660	5771	5368	5368
q4	2055	2184	1584	1584
q5	5841	5747	5767	5747
q6	236	193	142	142
q7	2314	1964	1830	1830
q8	2847	2475	2474	2474
q9	7560	7518	7492	7492
q10	3672	3576	3170	3170
q11	581	420	410	410
q12	703	656	509	509
q13	3320	3581	3100	3100
q14	288	288	279	279
q15	q16	673	680	623	623
q17	7829	7026	6885	6885
q18	13015	12293	12848	12293
q19	901	781	807	781
q20	2166	2128	1927	1927
q21	5746	4646	4740	4646
q22	481	440	423	423
Total cold run time: 72663 ms
Total hot run time: 66133 ms

[hello-stephen]

TPC-DS: Total hot run time: 151527 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f8374ca590bbeff28f03d38a9656f880a115c36, data reload: false

query5	4300	596	430	430
query6	422	212	195	195
query7	4816	443	226	226
query8	333	166	147	147
query9	8803	3235	3206	3206
query10	480	308	255	255
query11	6029	3900	3606	3606
query12	155	90	86	86
query13	1238	446	316	316
query14	6165	3974	3678	3678
query14_1	3467	3431	3470	3431
query15	205	200	184	184
query16	936	432	411	411
query17	852	661	539	539
query18	2420	432	316	316
query19	205	168	138	138
query20	87	84	82	82
query21	221	132	106	106
query22	13016	12967	12854	12854
query23	13432	12988	12478	12478
query23_1	12695	12568	12705	12568
query24	7344	851	481	481
query24_1	487	500	509	500
query25	568	428	362	362
query26	1246	266	145	145
query27	2746	457	279	279
query28	4600	2050	2066	2050
query29	1587	613	476	476
query30	297	214	187	187
query31	1015	936	821	821
query32	148	92	97	92
query33	532	311	249	249
query34	927	882	546	546
query35	749	764	655	655
query36	810	813	735	735
query37	120	105	87	87
query38	1780	1763	1702	1702
query39	804	804	745	745
query39_1	732	746	744	744
query40	230	120	102	102
query41	71	69	69	69
query42	87	87	91	87
query43	349	370	313	313
query44	1372	715	737	715
query45	182	180	165	165
query46	842	971	584	584
query47	2971	2951	2646	2646
query48	302	322	218	218
query49	574	397	297	297
query50	679	304	210	210
query51	10599	10649	10197	10197
query52	76	88	69	69
query53	185	206	150	150
query54	226	191	184	184
query55	70	65	67	65
query56	222	229	212	212
query57	1650	1630	1545	1545
query58	286	269	258	258
query59	2125	2143	2005	2005
query60	271	244	224	224
query61	143	143	142	142
query62	378	325	283	283
query63	200	163	158	158
query64	2752	955	760	760
query65	3329	3259	3274	3259
query66	1788	410	297	297
query67	20247	20144	20096	20096
query68	2957	1037	621	621
query69	397	298	253	253
query70	848	778	785	778
query71	292	230	220	220
query72	2651	2607	2281	2281
query73	519	533	309	309
query74	4613	4551	4377	4377
query75	2316	2294	1919	1919
query76	2324	1028	622	622
query77	338	374	281	281
query78	9684	9667	9128	9128
query79	1018	850	545	545
query80	1234	455	374	374
query81	578	317	278	278
query82	572	140	106	106
query83	299	192	206	192
query84	321	117	105	105
query85	861	439	388	388
query86	396	235	231	231
query87	2001	1996	1811	1811
query88	3797	2800	2767	2767
query89	320	279	243	243
query90	1928	176	172	172
query91	166	151	126	126
query92	100	90	89	89
query93	1008	1020	576	576
query94	621	311	274	274
query95	578	327	297	297
query96	690	507	238	238
query97	2445	2421	2332	2332
query98	166	152	151	151
query99	757	756	658	658
Total cold run time: 235537 ms
Total hot run time: 151527 ms

[hello-stephen]

ClickBench: Total hot run time: 24.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f8374ca590bbeff28f03d38a9656f880a115c36, data reload: false

query1	0.01	0.00	0.00
query2	0.09	0.05	0.05
query3	0.27	0.08	0.09
query4	1.63	0.09	0.10
query5	0.23	0.24	0.24
query6	1.17	0.68	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.34	0.35
query10	0.57	0.61	0.59
query11	0.19	0.14	0.14
query12	0.19	0.15	0.14
query13	0.50	0.49	0.50
query14	1.05	1.01	1.00
query15	0.62	0.61	0.61
query16	0.32	0.34	0.35
query17	1.16	1.10	1.10
query18	0.22	0.19	0.19
query19	2.14	2.06	1.99
query20	0.02	0.02	0.01
query21	15.50	0.20	0.14
query22	4.91	0.06	0.05
query23	16.31	0.33	0.13
query24	3.01	0.40	0.40
query25	0.11	0.06	0.06
query26	0.99	0.21	0.14
query27	0.04	0.04	0.03
query28	3.86	0.85	0.38
query29	12.52	4.16	3.26
query30	0.27	0.14	0.09
query31	2.78	0.55	0.32
query32	3.22	0.60	0.50
query33	3.34	3.18	3.18
query34	15.37	4.02	3.43
query35	3.40	3.34	3.37
query36	0.57	0.45	0.41
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 97.61 s
Total hot run time: 24.34 s

[morningman]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: workspace routing discovery unauthorized (401)
Workflow run: https://github.com/apache/doris/actions/runs/40654235162

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[morningman]

/review

[github-actions]

Reviewed all 8 changed files at head 0f8374ca590bbeff28f03d38a9656f880a115c36 using the supplied aggregate diff. No actionable new findings and no existing blocking inline comments. The review converged in 1 round: both complete-review agents and the independent risk reviewer returned NO_NEW_VALUABLE_FINDINGS; the main reviewer independently verified the investigated mechanisms. No additional user focus was supplied.

Critical checkpoint conclusions:

1. Goal and implementation: closed scopes refuse retained-session loads; conditional removal assigns release ownership for a value published during closure. A separate EXPLAIN REFRESH context receives its own query id before planning and is finished on success or failure. The added unit tests cover these intended behaviors by inspection.
2. Scope, clarity and reuse: the changes are focused on statement lifetime and reuse existing query-finish callbacks, StatementContext cleanup and a common release helper.
3. Concurrency and locking: traced executor cleanup against retained streaming/partition-batch producers and task groups. The volatile flag and conditional map removals prevent duplicate ownership in the reviewed interleavings. Metadata IO remains inside the existing map loader; same-bin cleanup waiting also existed through the previous cache.clear(). No introduced lock-order defect was established; cleanup is not claimed to be wait-free.
4. Lifecycle: retained producers keep their original scope; later independent sessions select NONE/DIRECT. Operation-local cross-statement scopes remain explicitly owned. Retries reset scope, prepared executions replace their context, and deferred Arrow Flight retains its existing cleanup boundary. EXPLAIN's fresh context is closed before restoring the previous thread-local, including exceptional exits. No cross-translation-unit initialization is involved.
5. Configuration: no new or modified configuration items.
6. Compatibility: the SPI edit is documentation only; public signatures, recorded API surface and stored formats do not change. The newly documented cancellation behavior was traced through its consumers.
7. Thrift/protobuf: no definition changes or field-id review required.
8. Parallel paths: checked synchronous scans, streaming and partition-batch producers, ordinary EXPLAIN, COMPLETE/INCREMENTAL refresh explanations, attempt retries, prepared execution and explicit background scopes. No missing equivalent fix was identified.
9. Conditions: closed-scope checks address concrete late producer loads; context identity distinguishes ownership. Existing target-table and thread-local restoration guards retain their purpose.
10. Test coverage: inspected the new JUnit 5 closed-scope, in-flight publication, offline-session, late-producer and EXPLAIN success/failure cases, plus adjacent reset/transaction-order coverage. Concurrency tests use latches and bounded waits. Real connector integration, populated-bin stress and cleanup-failure execution were assessed through code paths, not executed.
11. Test results: no regression result files changed. Assertions match the reviewed ownership and callback ordering; this review does not attest that tests pass.
12. Observability and errors: cancellation includes the scope key; per-value release and callback failures retain existing warning isolation. No new critical distributed operation requires additional metrics here.
13. Transactions and persistence: transaction holders are created/begun on the executor thread, rather than by surviving read producers. The reviewed reachable cleanup paths preserve transaction finalization before shared metadata close. No EditLog, replay or failover protocol changes.
14. Writes and atomicity: no new write execution path; EXPLAIN teardown does not execute a refresh or commit writes. No new crash-consistency or atomicity issue identified.
15. FE/BE passing: no new transmitted variables or send paths.
16. Performance and memory: lookups add bounded closed-state checks, cleanup remains linear in scope entries, and EXPLAIN adds one query id and callback drain. No additional scan materialization or new retained state was identified.
17. Further recheck: reconsidered late fresh sessions, multiple loader waiters, planning-key cleanup, transaction ordering, callback initialization timing and failure restoration. Both initial risk areas were dismissed with code evidence; no unresolved candidate or uncovered changed file remains.

Validation limit: static review only. No builds, tests, benchmarks, runtime probes, source modifications or submodule initialization were performed. Convergence describes completion of the static review; it does not establish runtime validation.

Review converged after 1 round(s).

[924060929]

External scans share metadata, table leases and task groups through a statement scope. This change prevents late producers from loading into a closed scope and gives an independently owned EXPLAIN REFRESH context its own query id and deterministic cleanup.

Reviewed all 8 changed files and relevant FE call chains against the supplied merge base. No actionable new P0/P1/P2 findings. Scope ownership, concurrent load/close handling, transaction teardown ordering, retries, prepared execution, cancellation propagation and the added tests were inspected. No wire-format or FE/BE parameter changes.

APPROVE within the FE static-review scope. The supplied current-head P0 checks succeeded, and the existing same-head AI review reports no blocking findings. FE coverage failed; required-check rules are unknown, so this is not confirmation that all merge requirements passed. No builds or tests were run.

Automated FE-focused static review (GPT-6.1 Sol, medium). No build or tests were run.