[morningman]

What problem does this PR solve?

Issue Number: None

Related PR: #69229

Problem Summary:

In short. Doris rejects every write to a format-v1 Iceberg table whose current partition spec holds a void field, the field Iceberg keeps for each partition field dropped from a v1 table. This PR pins such a spec as it is, skips void fields wherever the write path looks for a partition source column, and lets BE write NULL for a void field without resolving its source. The problem came up in the end-to-end checks for #69229.

Background. Dropping a partition field from a v1 table does not remove it from the spec: Iceberg replaces it with a void transform, which yields NULL for every row (v2 removes the field). The column the field came from can be dropped afterwards, which Iceberg allows because a void field never reads it. Before planning a write, Doris pins the schema and the partition spec in IcebergWriteSchemaContext, and at commit time it checks that the table still has the pinned spec with the same definition.

The problem, and what it cost.

Current spec of the v1 table	Spark 4.0 with Iceberg 1.10	Doris master	Doris with this PR
[void(region)], region kept or dropped	Writes	INSERT and INSERT OVERWRITE fail: pinned spec 1 is not available with the same definition; retry the statement	Written like an unpartitioned table and committed under spec 1
[void(region), identity(category)], region dropped	Reads, but cannot write: its partition key needs an accessor for every source field	Planning fails: Iceberg partition field 1000 references source field 2 outside pinned schema 1	Written under spec 1 with region NULL, routed by category
[void(region), identity(category)], region kept	Writes	Works	Works
The tables above after an upgrade to format v2	As above	DELETE, UPDATE and MERGE fail the same way	Work

• bindPartitionSpec replaces every spec whose isPartitioned() is false with an unpartitioned stand-in that has the same spec id. An all-void spec is not partitioned in that sense but still has fields, so the stand-in's JSON never equals the table's spec and the commit-time check always fails.
• validateWriterMetadataSources requires the source column of every partition field, void included, and BE VIcebergTableWriter::_to_iceberg_partition_columns resolves the source of every field as well.

How this PR fixes it.

• IcebergWriteSchemaContext gives the unpartitioned stand-in only to a spec without fields and binds every other spec as it is, so the pinned definition equals the table's. Its source check skips void fields.
• IcebergWritePlanProvider skips void fields in the INSERT routing (getWriteDistribution) and in the MERGE insert partitioning (getWritePartitioning). They are NULL for every row; without the skip, a dropped source turns the write random, and a table with many partitions can exceed table_sink_partition_write_max_partition_nums_per_writer on one writer.
• BE VIcebergTableWriter no longer resolves a source for a void field and fills a NULL column for it, so the file lands under region=null/... and commits NULL for the field, as Spark writes the same spec while the column exists. Iceberg itself allows dropping that column, because a void field never reads it.
• An all-void spec stays unpartitioned for the write: the sink carries no spec, BE writes unpartitioned files, and the commit builds the data files with DataFiles.builder(spec) and no partition values, the path Spark's unpartitioned writer takes. A dynamic overwrite replaces the whole table, since ReplacePartitions checks isUnpartitioned().
• During a rolling upgrade, a new FE with an old BE still fails for a void field whose source column was dropped, with the old BE's outside writer schema error before any file is written, so no exec version gate is needed.

Branch-4.1 and branch-4.2 have the same code in fe-core IcebergWriteSchemaContext and in BE.

Results.

• Unit tests: the new IcebergWritePlanProviderTest#writeSchemaPinsV1VoidPartitionFields and #writeRoutingSkipsV1VoidPartitionFields, IcebergConnectorTransactionTest#insertCommitsNullForV1VoidPartitionFields (pins, commits and checks the committed data file for an all-void spec and for a void field whose source was dropped) and VIcebergTableWriterTest.VoidPartitionFieldNeedsNoSourceColumn pass, with all 1625 tests of fe-connector-iceberg and 91 BE tests of the Iceberg writer, partition, MERGE / DELETE sink and related suites. Reverting each of the six changes (the stand-in, the source check, the INSERT routing, the MERGE partitioning, the BE source resolution, the BE NULL column) fails the tests that cover it.
• Regression test: test_iceberg_write_v1_void_partition creates three v1 tables in Spark (void only, void only with the column dropped, void plus identity with the column dropped), writes them with Doris INSERT and INSERT OVERWRITE, upgrades them to merge-on-read v2 and runs DELETE, UPDATE and MERGE, comparing every step with Spark, including Spark's partition pruning on the identity field. With the master connector the first Doris INSERT fails with the pinned spec 1 error; with this PR the suite passes, as do all 43 suites of external_table_p0/iceberg/write. Spark only reads the void plus identity table, which it cannot write.
• Manual test, on a local Hadoop catalog with the same three shapes: INSERT, INSERT OVERWRITE, and DELETE / UPDATE / MERGE after a v2 upgrade return the expected rows, and the Iceberg API shows every new data and delete file under the current spec with a NULL void value (region=null/category=x/ for the identity table, the table root for the void-only tables).

Release note

Doris can write format-v1 Iceberg tables whose partition spec contains void fields left by dropped partition fields, also after their source columns were dropped.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morningman]

run buildall

[morningman]

/review

[github-actions]

Reviewed all eight changed paths at a2a6965. No new actionable findings and no existing inline blocking findings. The FE, BE, and separate risk-integration reviewers all returned NO_NEW_VALUABLE_FINDINGS on the same empty inline set; required coverage and all candidate/risk dispositions are complete. The review converged after one round.

Critical-checkpoint conclusions:

• Goal and tests: the change preserves the real void-containing spec for schema/commit validation, skips source lookup for constant-null fields, and routes by active fields. Added FE pinning/routing/commit tests, BE two-row routing, and Spark/Doris regression cases target this behavior. Runtime success was not independently verified.
• Scope and reuse: the implementation is small and focused, reusing the existing partition transform, nullable column, sink, and commit abstractions.
• Concurrency and locking: changed FE loops use pinned metadata/local lists; transformed BE columns belong to the writer task. No new shared state, thread, lock, blocking critical section, or lock-order edge was found.
• Lifecycle and initialization: statement contexts, spec-field references, temporary columns, sorted-writer captures, table leases and file cleanup keep their existing ownership boundaries. No new cycle or cross-translation-unit static initialization dependency was introduced.
• Configuration: no configuration item or dynamic-update behavior changes.
• Compatibility: no stored format, SPI signature, or new wire dialect changes. All-void writes keep the actual FE spec ID while using the existing unpartitioned BE mode; the pinned SDK retains that ID in its DataFiles builder. An old BE still rejects a mixed void spec with a dropped source before creating a writer; the new support requires the fixed BE.
• Thrift/protobuf: no definitions, field IDs, or enum values changed.
• Parallel paths: INSERT, whole-table/static/dynamic overwrite, REWRITE and MERGE data writes share the relevant fixed path. DELETE commits retain original per-file spec IDs/tuples. Sorted/unsorted writers and Parquet/ORC were traced.
• Conditional checks and errors: comments explain why void sources are omitted and NULLs materialized. Nullable defaults short-circuit value/type access, valid static partitions admit active identity fields only, and missing nonvoid sources still fail explicitly.
• Test coverage: retained/dropped source, void-only/mixed identity, spec pinning, route fields, commit NULLs, Spark reads/pruning, overwrite and upgraded-v2 DELETE/UPDATE/MERGE are covered by the added tests. Existing missing-source rejection remains a negative guard. Multiple/reordered/nonstring void fields, static hybrids, branch/evolution and rollout interactions were checked statically.
• Expected results: all 12 ordered query tags match the output sections, and expected rows/spec IDs agree with the SQL state transitions. Runtime results and automatic output-generation provenance were not verified in this invocation.
• Observability: existing failure messages and writer profile counters cover the added constant-null path; no additional logging/metric requirement was identified.
• Persistence and failover: no new edit-log or FE-local persistent state. Existing schema/format/table/spec/order fences and SDK metadata CAS/retry remain the publication boundary. Native visible-version and MoW bitmap paths are outside these changes.
• Writes and atomicity: filtering hash inputs does not filter file partition tuples. Full specs reach data writers/commit conversion; historical delete fragments are grouped by their original spec. Existing transaction staging, conflict checks, abort and file-cleanup paths remain intact.
• FE-to-BE values: no new transmitted variable; existing partition-spec JSON retains the complete mixed spec while routing metadata carries only active fields.
• Performance and memory: new NULL materialization is linear in rows times void fields and uses existing column allocators/lifetimes. No new unbounded ownership, redundant metadata IO or asymptotic regression was found.
• Other risks: source/type lookup, static overrides, filtered-distribution/full-tuple alignment, all-void metadata and historical delete-spec concerns were verified and dismissed with evidence. No unresolved introduced-correctness candidate remains.

No additional user focus was supplied. A pre-existing support limit remains: historical files with non-null partition values whose source was dropped can reject row-ID DML until rewritten; the regression explicitly rewrites them first. The unchanged scan guard was checked on both base and head. This is not a new inline finding.

Validation was static only, as required: no builds, tests, or repository source edits were performed. The review covers the complete supplied diff; it does not claim runtime validation of the author-reported results.

Review converged after 1 round(s).

[hello-stephen]

TPC-H: Total hot run time: 28480 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2a696517049b7939df2c935d1747df92c075fc2, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17576	5189	5120	5120
q2	2131	318	274	274
q3	10208	1411	821	821
q4	4681	464	335	335
q5	7424	796	530	530
q6	182	182	150	150
q7	754	808	598	598
q8	9257	1337	1074	1074
q9	5477	4171	4189	4171
q10	6820	1288	1043	1043
q11	445	248	227	227
q12	632	398	276	276
q13	18096	3654	3045	3045
q14	280	266	250	250
q15	q16	738	737	663	663
q17	1195	754	580	580
q18	7059	6184	6136	6136
q19	1228	971	597	597
q20	390	333	225	225
q21	5303	2117	2287	2117
q22	358	301	248	248
Total cold run time: 100234 ms
Total hot run time: 28480 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5654	5633	5635	5633
q2	682	572	522	522
q3	5208	5387	4927	4927
q4	1978	2158	1534	1534
q5	5335	5171	5789	5171
q6	254	200	161	161
q7	2408	2082	1919	1919
q8	3266	2788	2700	2700
q9	8050	7986	8072	7986
q10	3847	3657	3248	3248
q11	595	427	426	426
q12	710	671	521	521
q13	3467	3775	3307	3307
q14	328	331	291	291
q15	q16	683	730	641	641
q17	8227	7570	7266	7266
q18	13063	12177	12908	12177
q19	898	797	781	781
q20	2124	2136	1941	1941
q21	5663	4573	4711	4573
q22	482	484	399	399
Total cold run time: 72922 ms
Total hot run time: 66124 ms

[hello-stephen]

TPC-DS: Total hot run time: 150935 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2a696517049b7939df2c935d1747df92c075fc2, data reload: false

query5	4329	598	433	433
query6	448	200	187	187
query7	4802	445	227	227
query8	319	160	160	160
query9	8804	3079	3121	3079
query10	505	299	246	246
query11	5955	3853	3568	3568
query12	147	90	88	88
query13	1252	437	308	308
query14	6105	3941	3803	3803
query14_1	3403	3401	3434	3401
query15	210	204	182	182
query16	938	443	408	408
query17	863	658	553	553
query18	2421	430	312	312
query19	201	177	134	134
query20	89	84	94	84
query21	210	128	107	107
query22	12930	12996	13019	12996
query23	13447	12931	12414	12414
query23_1	12494	12485	12638	12485
query24	7382	838	474	474
query24_1	484	483	470	470
query25	528	424	357	357
query26	1251	260	143	143
query27	2773	446	273	273
query28	4656	1955	1958	1955
query29	1599	629	455	455
query30	307	224	187	187
query31	1024	957	819	819
query32	145	100	91	91
query33	536	308	263	263
query34	911	857	514	514
query35	735	777	655	655
query36	820	794	712	712
query37	130	104	91	91
query38	1838	1724	1654	1654
query39	788	800	772	772
query39_1	746	727	741	727
query40	222	117	100	100
query41	71	68	71	68
query42	90	86	85	85
query43	353	364	305	305
query44	1379	694	687	687
query45	181	176	162	162
query46	853	972	587	587
query47	2601	3065	2834	2834
query48	308	338	225	225
query49	620	398	298	298
query50	701	282	208	208
query51	10501	10228	10535	10228
query52	78	84	70	70
query53	185	211	149	149
query54	236	193	179	179
query55	72	67	67	67
query56	213	213	226	213
query57	1717	1631	1546	1546
query58	287	263	255	255
query59	2086	2128	1892	1892
query60	284	245	225	225
query61	148	147	148	147
query62	386	327	289	289
query63	198	163	165	163
query64	2749	945	784	784
query65	3328	3269	3284	3269
query66	1764	413	324	324
query67	20115	20084	20011	20011
query68	3036	1027	598	598
query69	400	288	244	244
query70	856	765	771	765
query71	299	234	216	216
query72	2723	2615	2247	2247
query73	531	543	292	292
query74	4607	4497	4349	4349
query75	2338	2265	1914	1914
query76	2320	1035	614	614
query77	344	374	271	271
query78	9967	9765	9160	9160
query79	979	920	534	534
query80	735	426	340	340
query81	565	316	269	269
query82	572	157	114	114
query83	275	200	186	186
query84	309	124	100	100
query85	959	444	374	374
query86	354	232	235	232
query87	2010	1960	1807	1807
query88	3698	2689	2649	2649
query89	338	277	243	243
query90	1703	176	167	167
query91	160	152	128	128
query92	103	93	90	90
query93	1056	1001	582	582
query94	496	307	277	277
query95	583	389	303	303
query96	679	505	231	231
query97	2438	2434	2295	2295
query98	169	155	144	144
query99	780	767	653	653
Total cold run time: 234185 ms
Total hot run time: 150935 ms

[hello-stephen]

ClickBench: Total hot run time: 24.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2a696517049b7939df2c935d1747df92c075fc2, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.10	0.10
query5	0.26	0.25	0.24
query6	1.17	0.67	0.67
query7	0.05	0.01	0.01
query8	0.05	0.03	0.03
query9	0.41	0.33	0.33
query10	0.57	0.58	0.57
query11	0.20	0.15	0.14
query12	0.19	0.15	0.15
query13	0.49	0.51	0.49
query14	1.05	1.01	1.00
query15	0.62	0.61	0.60
query16	0.34	0.34	0.35
query17	1.10	1.09	1.14
query18	0.21	0.19	0.20
query19	2.09	1.94	1.91
query20	0.02	0.01	0.01
query21	15.51	0.20	0.13
query22	4.96	0.06	0.05
query23	16.26	0.33	0.12
query24	3.05	0.52	0.42
query25	0.10	0.05	0.04
query26	1.06	0.22	0.17
query27	0.05	0.03	0.05
query28	3.52	0.80	0.36
query29	12.49	4.23	3.35
query30	0.28	0.13	0.10
query31	2.78	0.56	0.32
query32	3.21	0.60	0.51
query33	3.20	3.26	3.22
query34	15.32	4.01	3.37
query35	3.37	3.35	3.34
query36	0.55	0.46	0.41
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 97.01 s
Total hot run time: 24.29 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	66.44% (31834/47913)
Line Coverage	51.20% (335152/654631)
Region Coverage	46.56% (270855/581723)
Branch Coverage	48.31% (127741/264444)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.18% (35831/46428)
Line Coverage	62.42% (404269/647621)
Region Coverage	58.90% (341394/579619)
Branch Coverage	59.85% (157884/263797)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 0.00% (0/537) 🎉
Increment coverage report
Complete coverage report

[github-actions]

PR approved by anyone and no changes requested.

[github-actions]

PR approved by at least one committer and no changes requested.