Skip to content

[fix](binlog) Derive table stream schema from base table - #67130

Closed
luwei16 wants to merge 2 commits into
apache:masterfrom
luwei16:fix/doris-28093-dynamic-stream-schema
Closed

[fix](binlog) Derive table stream schema from base table#67130
luwei16 wants to merge 2 commits into
apache:masterfrom
luwei16:fix/doris-28093-dynamic-stream-schema

Conversation

@luwei16

@luwei16 luwei16 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #67109

Related PR: N/A

Problem Summary: Table Streams persisted a copy of their base table schema at creation time. Light schema changes updated the base table but left Stream metadata stale. Keep the persisted Stream schema empty, derive every Stream schema API from the current base table under its read lock, and fail explicitly when the base table is unavailable.

Release note

Table Stream metadata now follows the current base table schema after supported schema changes.

Check List (For Author)

  • Test: Unit Test
    • CreateTableStreamTest
    • ExplainTableStreamPlanTest
    • TableStreamBaseTableInfoTest
  • Behavior changed: Yes (Table Stream schema metadata is derived dynamically from the base table.)
  • Does this need documentation: No

@luwei16
[fix](fe) Derive table stream schema from base table
de20fba 
### What problem does this PR solve?

Issue Number: close apache#67109

Related PR: N/A

Problem Summary: Table Streams persisted a copy of their base table schema at creation time. Light schema changes updated the base table but left Stream metadata stale. Keep the persisted Stream schema empty, derive every Stream schema API from the current base table under its read lock, and fail explicitly when the base table is unavailable.

### Release note

Table Stream metadata now follows the current base table schema after supported schema changes.

### Check List (For Author)

- Test: Unit Test
    - `CreateTableStreamTest`
    - `ExplainTableStreamPlanTest`
    - `TableStreamBaseTableInfoTest`
- Behavior changed: Yes (Table Stream schema metadata is derived dynamically from the base table.)
- Does this need documentation: No
@hello-stephen

hello-stephen commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16

luwei16 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 25, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17082 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de20fba4353163f092b939d5d440ebf3db72e977, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18071	3111	3058	3058
q2	2449	273	227	227
q3	10354	904	517	517
q4	4738	245	202	202
q5	7688	554	382	382
q6	141	118	94	94
q7	545	504	383	383
q8	9375	853	929	853
q9	3849	2439	2411	2411
q10	6524	859	731	731
q11	620	194	184	184
q12	656	260	197	197
q13	18179	1579	1158	1158
q14	159	153	143	143
q15	q16	431	402	375	375
q17	1317	880	878	878
q18	3251	2254	2271	2254
q19	1205	894	762	762
q20	362	279	205	205
q21	5316	1837	1876	1837
q22	332	267	231	231
Total cold run time: 95562 ms
Total hot run time: 17082 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3435	3353	3369	3353
q2	517	400	362	362
q3	2257	2398	2219	2219
q4	1199	1170	898	898
q5	2205	2157	2107	2107
q6	169	130	87	87
q7	1081	912	873	873
q8	1619	1416	1426	1416
q9	3184	3136	3150	3136
q10	1893	1813	1657	1657
q11	367	280	253	253
q12	462	437	352	352
q13	1509	1539	1159	1159
q14	178	184	174	174
q15	q16	398	404	358	358
q17	3673	3376	3294	3294
q18	4994	4657	4878	4657
q19	926	830	840	830
q20	1002	938	853	853
q21	3670	3003	3151	3003
q22	403	359	321	321
Total cold run time: 35141 ms
Total hot run time: 31362 ms

@hello-stephen

hello-stephen commented Aug 25, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82782 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de20fba4353163f092b939d5d440ebf3db72e977, data reload: false

query5	4328	417	326	326
query6	422	139	129	129
query7	5026	434	231	231
query8	698	131	112	112
query9	8649	2903	2911	2903
query10	448	226	191	191
query11	5431	1044	929	929
query12	130	73	71	71
query13	1196	445	331	331
query14	6094	2225	2096	2096
query14_1	1995	1983	1970	1970
query15	175	114	113	113
query16	3267	387	378	378
query17	828	470	373	373
query18	2173	335	253	253
query19	245	144	109	109
query20	73	74	73	73
query21	742	107	88	88
query22	5838	5627	5482	5482
query23	7189	6312	6058	6058
query23_1	6091	5970	6267	5970
query24	7433	1099	800	800
query24_1	769	772	761	761
query25	437	301	268	268
query26	1267	228	135	135
query27	2746	444	261	261
query28	4633	1514	1538	1514
query29	932	600	396	396
query30	409	165	137	137
query31	836	417	354	354
query32	132	79	70	70
query33	502	214	192	192
query34	1040	846	495	495
query35	408	403	344	344
query36	583	563	518	518
query37	131	75	75	75
query38	1036	855	844	844
query39	515	501	482	482
query39_1	510	483	475	475
query40	229	84	75	75
query41	87	48	49	48
query42	69	71	73	71
query43	258	247	219	219
query44	1067	543	560	543
query45	120	104	98	98
query46	796	861	559	559
query47	836	761	724	724
query48	302	275	226	226
query49	626	236	192	192
query50	771	256	190	190
query51	8324	8314	8173	8173
query52	65	66	60	60
query53	194	197	145	145
query54	220	174	152	152
query55	86	58	54	54
query56	214	161	159	159
query57	1044	667	644	644
query58	207	161	146	146
query59	1225	1226	1101	1101
query60	273	193	166	166
query61	118	111	114	111
query62	375	199	196	196
query63	166	138	139	138
query64	2332	662	559	559
query65	1677	1623	1663	1623
query66	1803	257	211	211
query67	9821	10297	9758	9758
query68	2794	1204	722	722
query69	564	226	198	198
query70	666	619	614	614
query71	263	174	172	172
query72	2634	1718	1584	1584
query73	676	609	338	338
query74	1561	1248	1156	1156
query75	1193	1121	981	981
query76	1871	735	536	536
query77	331	266	219	219
query78	3875	3730	3226	3226
query79	2183	815	565	565
query80	1325	331	293	293
query81	587	159	136	136
query82	668	131	96	96
query83	287	210	193	193
query84	348	109	88	88
query85	869	359	290	290
query86	414	180	176	176
query87	1037	993	889	889
query88	2846	2120	2135	2120
query89	319	204	181	181
query90	1977	131	128	128
query91	134	121	99	99
query92	80	70	64	64
query93	1429	1088	657	657
query94	645	278	224	224
query95	523	273	227	227
query96	805	556	280	280
query97	1042	1090	1090	1090
query98	146	141	142	141
query99	547	352	312	312
Total cold run time: 187696 ms
Total hot run time: 82782 ms

@hello-stephen

hello-stephen commented Aug 25, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de20fba4353163f092b939d5d440ebf3db72e977, data reload: false

query1	0.02	0.00	0.00
query2	0.36	0.03	0.03
query3	0.76	0.11	0.11
query4	2.18	0.11	0.10
query5	0.18	0.16	0.16
query6	1.88	0.68	0.70
query7	0.04	0.01	0.00
query8	0.05	0.03	0.02
query9	0.83	0.21	0.21
query10	0.36	0.35	0.34
query11	0.32	0.11	0.11
query12	0.22	0.12	0.12
query13	0.32	0.31	0.32
query14	0.46	0.47	0.46
query15	0.40	0.35	0.36
query16	0.22	0.23	0.25
query17	0.68	0.69	0.66
query18	0.19	0.17	0.17
query19	1.60	1.18	1.19
query20	0.02	0.01	0.01
query21	16.69	0.15	0.12
query22	4.77	0.05	0.04
query23	16.91	0.26	0.10
query24	18.12	0.34	0.31
query25	0.12	0.05	0.03
query26	0.72	0.17	0.13
query27	0.04	0.03	0.03
query28	3.69	0.58	0.30
query29	12.66	3.17	2.62
query30	0.25	0.11	0.12
query31	2.89	0.36	0.16
query32	3.58	0.33	0.23
query33	1.38	1.43	1.39
query34	15.40	2.14	1.75
query35	1.72	1.76	1.70
query36	0.47	0.29	0.29
query37	0.27	0.04	0.04
query38	0.04	0.03	0.02
query39	0.30	0.02	0.02
query40	0.22	0.08	0.07
query41	0.35	0.02	0.03
query42	0.26	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 111.98 s
Total hot run time: 14.78 s

@hello-stephen

hello-stephen commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 58.97% (23/39) 🎉
Increment coverage report
Complete coverage report

@luwei16 luwei16 added the binlog label Aug 27, 2026
@luwei16
[chore](fe) Merge master into table stream schema fix
bf6fc38 
### What problem does this PR solve?

Issue Number: close apache#67109

Related PR: apache#67173

Problem Summary: Sync the branch with the latest master and resolve the overlapping table stream dynamic-schema implementation. Keep the upstream base-table ID lookup and stream LSN support while retaining schema generation under the base-table read lock, explicit failure when the base table is unavailable, and an empty persisted stream schema.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - CreateTableStreamTest
    - ExplainTableStreamPlanTest
    - TableStreamBaseTableInfoTest
- Behavior changed: No (merge conflict resolution preserves the intended PR behavior)
- Does this need documentation: No
@luwei16

luwei16 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.00% (21/25) 🎉
Increment coverage report
Complete coverage report

@luwei16

luwei16 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.00% (21/25) 🎉
Increment coverage report
Complete coverage report

@luwei16

luwei16 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 84.00% (21/25) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 50.00% (15/30) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 50.00% (15/30) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 3.84% (15/391) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16953 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf6fc3886d25b7f886a57f7eb560456b88a89586, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17552	3012	3024	3012
q2	2093	265	252	252
q3	10210	857	510	510
q4	4670	256	201	201
q5	7677	610	386	386
q6	134	117	92	92
q7	519	509	392	392
q8	9231	905	909	905
q9	3484	2460	2401	2401
q10	6524	841	725	725
q11	388	198	180	180
q12	611	263	199	199
q13	18126	1549	1166	1166
q14	158	155	143	143
q15	q16	426	399	378	378
q17	1372	902	831	831
q18	3179	2331	2305	2305
q19	1109	893	755	755
q20	371	290	198	198
q21	4823	1693	1834	1693
q22	319	262	229	229
Total cold run time: 92976 ms
Total hot run time: 16953 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3345	3298	3288	3288
q2	500	394	388	388
q3	2278	2423	2274	2274
q4	1216	1189	904	904
q5	2228	2173	2168	2168
q6	167	123	86	86
q7	1050	961	881	881
q8	1596	1416	1417	1416
q9	3252	3223	3197	3197
q10	1947	1847	1662	1662
q11	363	273	263	263
q12	455	429	347	347
q13	1484	1524	1168	1168
q14	187	165	167	165
q15	q16	392	401	369	369
q17	3708	3320	3301	3301
q18	4973	4601	4888	4601
q19	959	870	850	850
q20	1042	994	838	838
q21	3872	3272	3199	3199
q22	394	339	330	330
Total cold run time: 35408 ms
Total hot run time: 31695 ms

@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf6fc3886d25b7f886a57f7eb560456b88a89586, data reload: false

query5	4249	393	343	343
query6	377	148	125	125
query7	4957	437	219	219
query8	293	128	117	117
query9	8686	2870	2864	2864
query10	389	216	185	185
query11	5375	1050	941	941
query12	112	70	69	69
query13	1184	447	310	310
query14	6132	2255	2133	2133
query14_1	2041	2027	2030	2027
query15	177	125	113	113
query16	930	363	346	346
query17	796	460	383	383
query18	2331	343	261	261
query19	155	132	102	102
query20	70	66	72	66
query21	199	101	86	86
query22	5448	5391	5307	5307
query23	6818	6474	6192	6192
query23_1	6219	6190	6015	6015
query24	7294	1095	788	788
query24_1	768	786	786	786
query25	408	272	240	240
query26	1234	236	123	123
query27	2784	418	257	257
query28	4732	1496	1481	1481
query29	909	438	335	335
query30	248	156	132	132
query31	836	393	334	334
query32	129	73	71	71
query33	445	215	178	178
query34	994	822	503	503
query35	413	413	350	350
query36	552	569	530	530
query37	116	76	69	69
query38	1000	868	859	859
query39	493	478	470	470
query39_1	460	468	471	468
query40	198	88	79	79
query41	54	53	51	51
query42	76	71	70	70
query43	236	243	210	210
query44	1018	545	537	537
query45	109	104	105	104
query46	753	809	514	514
query47	764	787	716	716
query48	302	306	226	226
query49	537	236	197	197
query50	765	269	200	200
query51	7963	8030	7949	7949
query52	70	69	67	67
query53	196	200	154	154
query54	250	177	182	177
query55	79	63	59	59
query56	198	179	165	165
query57	802	644	676	644
query58	220	190	194	190
query59	1268	1266	1114	1114
query60	237	220	175	175
query61	115	145	135	135
query62	338	207	200	200
query63	165	144	142	142
query64	2710	652	592	592
query65	1689	1633	1657	1633
query66	1898	269	233	233
query67	10208	10055	9779	9779
query68	3024	1177	676	676
query69	334	226	198	198
query70	675	646	638	638
query71	246	166	158	158
query72	2370	1753	1579	1579
query73	668	646	360	360
query74	1999	1235	1148	1148
query75	1195	1122	965	965
query76	2364	714	527	527
query77	250	251	204	204
query78	3910	3721	3273	3273
query79	2781	830	591	591
query80	1589	322	289	289
query81	500	160	138	138
query82	616	121	97	97
query83	273	208	190	190
query84	289	112	89	89
query85	803	359	296	296
query86	468	203	180	180
query87	1027	982	898	898
query88	2863	2109	2093	2093
query89	266	198	175	175
query90	1990	127	136	127
query91	132	130	103	103
query92	89	65	70	65
query93	1454	1136	744	744
query94	605	244	218	218
query95	536	240	237	237
query96	819	581	279	279
query97	1113	1082	1045	1045
query98	164	151	134	134
query99	422	354	324	324
Total cold run time: 179262 ms
Total hot run time: 82736 ms

@hello-stephen

hello-stephen commented Aug 31, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf6fc3886d25b7f886a57f7eb560456b88a89586, data reload: false

query1	0.00	0.00	0.01
query2	0.08	0.04	0.04
query3	0.25	0.12	0.11
query4	1.60	0.11	0.10
query5	0.17	0.16	0.16
query6	1.26	0.67	0.71
query7	0.03	0.01	0.01
query8	0.05	0.03	0.02
query9	0.29	0.21	0.21
query10	0.34	0.35	0.34
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.31	0.30	0.31
query14	0.47	0.47	0.46
query15	0.36	0.34	0.34
query16	0.25	0.23	0.22
query17	0.66	0.71	0.71
query18	0.17	0.16	0.15
query19	1.18	1.17	1.13
query20	0.02	0.01	0.01
query21	15.45	0.18	0.13
query22	5.02	0.05	0.04
query23	16.19	0.25	0.10
query24	3.03	0.30	0.26
query25	0.11	0.04	0.04
query26	0.80	0.16	0.11
query27	0.04	0.02	0.03
query28	3.62	0.52	0.27
query29	12.48	3.15	2.56
query30	0.25	0.11	0.12
query31	2.77	0.36	0.18
query32	3.54	0.33	0.24
query33	1.38	1.53	1.54
query34	15.38	2.19	1.77
query35	1.78	1.74	1.75
query36	0.45	0.30	0.29
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.12	0.08	0.08
query41	0.08	0.03	0.02
query42	0.03	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 90.49 s
Total hot run time: 14.85 s

@morningman morningman mentioned this pull request Sep 1, 2026
Open
35 of 69 tasks
@luwei16 luwei16 changed the title [fix](fe) Derive table stream schema from base table [fix](binlog) Derive table stream schema from base table Sep 1, 2026
@luwei16 luwei16 closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

binlog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Table Stream] Stream metadata retains a dropped column after light DROP COLUMN

2 participants

@luwei16 @hello-stephen