[fix](compaction) Count empty rowsets in approximate compaction score by Yukang-Lian · Pull Request #66610 · apache/doris

Yukang-Lian · 2026-08-10T08:28:53Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Cloud cumulative compaction scheduling maintains an approximate score between metadata synchronizations. The online write path used the raw segment count, so a zero-segment rowset added no score even though metadata synchronization counts every empty rowset as one. Empty rowsets could therefore accumulate without increasing tablet scheduling priority.

This PR clamps the online contribution of a load rowset to at least one and adds the same contribution for zero-segment delete-predicate rowsets, which bypass CloudRowsetBuilder. Non-empty PREPARED rowsets keep the existing segment-based estimate because their versions have not been assigned yet.

The unit test models that PREPARED, versionless lifecycle and verifies both cases: a zero-segment rowset contributes one, while a three-segment rowset still contributes three.

Release note

Fix cloud cumulative compaction scheduling for empty rowsets.

Check List (For Author)

Test
- Regression test
- Unit Test
  - ./run-be-ut.sh --run --filter='CloudCompactionTest.*' -j80 (18 tests passed)
  - build-support/check-format.sh
  - git diff --check
- Manual test
- No need to test or manual test.
Behavior changed:
- No.
- Yes. Zero-segment rowsets now contribute one to the online approximate cumulative compaction score.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Cloud cumulative compaction scheduling updates an approximate score before rowset metadata is synchronized. Zero-segment rowsets previously added zero to this score, although synchronized metadata counts each empty rowset as one. As a result, empty rowsets could accumulate without raising tablet scheduling priority. Count every zero-segment load and delete rowset as one while preserving the existing segment-based estimate for non-empty PREPARED rowsets. ### Release note Fix cloud cumulative compaction scheduling for empty rowsets. ### Check List (For Author) - Test - [x] Unit Test - Behavior changed: - [x] Yes. Zero-segment rowsets now add one to the approximate cumulative compaction score. - Does this need documentation? - [x] No.

hello-stephen · 2026-08-10T08:28:59Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Yukang-Lian · 2026-08-10T08:44:13Z

run buildall

hello-stephen · 2026-08-10T11:05:20Z

TPC-H: Total hot run time: 29315 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d746cbf3a347ba46f34d37d91c69c3e3d4f3e761, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17635	4138	4146	4138
q2	2074	331	206	206
q3	10240	1415	814	814
q4	4682	481	343	343
q5	7532	846	548	548
q6	180	174	143	143
q7	743	798	603	603
q8	9326	1544	1615	1544
q9	5271	4066	4052	4052
q10	6739	1635	1340	1340
q11	542	378	330	330
q12	759	579	465	465
q13	18074	3346	2763	2763
q14	264	257	239	239
q15	q16	736	743	663	663
q17	1035	1094	942	942
q18	6666	5623	5598	5598
q19	1334	1225	1047	1047
q20	845	697	583	583
q21	6293	2806	2636	2636
q22	469	374	318	318
Total cold run time: 101439 ms
Total hot run time: 29315 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5246	5031	4799	4799
q2	306	349	222	222
q3	5025	5199	4725	4725
q4	2233	2281	1467	1467
q5	4744	4468	4513	4468
q6	230	179	136	136
q7	1966	1749	1531	1531
q8	2371	2059	2061	2059
q9	7210	6899	6779	6779
q10	4251	4211	3828	3828
q11	533	410	366	366
q12	716	717	495	495
q13	2955	3236	2794	2794
q14	276	281	266	266
q15	q16	671	692	614	614
q17	1269	1224	1231	1224
q18	12134	10986	11902	10986
q19	1112	1092	1096	1092
q20	2218	2200	1912	1912
q21	5417	4696	4645	4645
q22	518	465	406	406
Total cold run time: 61401 ms
Total hot run time: 54814 ms

hello-stephen · 2026-08-10T11:15:35Z

TPC-DS: Total hot run time: 158398 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d746cbf3a347ba46f34d37d91c69c3e3d4f3e761, data reload: false

query5	4357	589	445	445
query6	488	215	202	202
query7	4878	591	343	343
query8	323	163	153	153
query9	8787	4063	4075	4063
query10	450	359	297	297
query11	5846	2170	1993	1993
query12	161	104	101	101
query13	1267	600	418	418
query14	6070	4383	4003	4003
query14_1	3810	3783	3785	3783
query15	206	199	185	185
query16	1009	468	449	449
query17	932	684	586	586
query18	2432	460	342	342
query19	204	185	142	142
query20	106	103	101	101
query21	227	161	133	133
query22	13128	13153	12824	12824
query23	15860	15039	14683	14683
query23_1	14694	14590	14545	14545
query24	7502	1687	1228	1228
query24_1	1239	1256	1245	1245
query25	532	426	344	344
query26	1311	349	215	215
query27	2624	560	379	379
query28	4587	2044	2009	2009
query29	1067	597	474	474
query30	343	263	220	220
query31	1179	1114	1053	1053
query32	103	59	67	59
query33	529	301	241	241
query34	1196	1160	612	612
query35	753	750	646	646
query36	774	794	693	693
query37	158	109	93	93
query38	1827	1782	1693	1693
query39	826	814	788	788
query39_1	779	777	779	777
query40	256	164	143	143
query41	63	62	63	62
query42	98	93	93	93
query43	316	319	273	273
query44	1442	791	783	783
query45	187	180	168	168
query46	1053	1165	734	734
query47	1507	1555	1431	1431
query48	378	395	291	291
query49	583	399	298	298
query50	1064	408	350	350
query51	10614	10578	10550	10550
query52	90	90	76	76
query53	273	288	196	196
query54	282	238	218	218
query55	77	71	69	69
query56	294	314	283	283
query57	1016	980	943	943
query58	277	251	238	238
query59	1558	1634	1397	1397
query60	307	272	258	258
query61	154	152	166	152
query62	412	327	270	270
query63	246	201	206	201
query64	2994	1141	970	970
query65	3900	3811	3837	3811
query66	1837	497	368	368
query67	20225	20296	19920	19920
query68	3248	1525	1007	1007
query69	409	304	280	280
query70	867	783	753	753
query71	361	347	332	332
query72	3160	2663	2281	2281
query73	844	809	449	449
query74	4627	4543	4307	4307
query75	2366	2335	1990	1990
query76	2316	1145	762	762
query77	340	359	281	281
query78	11248	11228	10553	10553
query79	1382	1081	750	750
query80	1269	527	462	462
query81	535	333	282	282
query82	684	179	138	138
query83	385	358	303	303
query84	330	163	131	131
query85	974	627	525	525
query86	409	234	226	226
query87	1990	1975	1834	1834
query88	3745	2819	2802	2802
query89	389	334	288	288
query90	1982	207	196	196
query91	201	196	162	162
query92	64	62	54	54
query93	1721	1563	969	969
query94	708	366	319	319
query95	801	507	567	507
query96	1075	809	360	360
query97	2470	2465	2349	2349
query98	199	187	185	185
query99	741	737	615	615
Total cold run time: 245905 ms
Total hot run time: 158398 ms

hello-stephen · 2026-08-10T11:20:32Z

ClickBench: Total hot run time: 24 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d746cbf3a347ba46f34d37d91c69c3e3d4f3e761, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.04	0.04
query3	0.25	0.14	0.14
query4	1.62	0.14	0.14
query5	0.25	0.22	0.22
query6	1.15	0.84	0.81
query7	0.04	0.00	0.00
query8	0.05	0.04	0.04
query9	0.38	0.31	0.32
query10	0.56	0.55	0.59
query11	0.19	0.14	0.15
query12	0.18	0.14	0.14
query13	0.47	0.47	0.47
query14	0.99	0.98	1.01
query15	0.61	0.60	0.59
query16	0.33	0.34	0.33
query17	1.11	1.14	1.09
query18	0.20	0.22	0.20
query19	2.04	1.96	1.99
query20	0.02	0.01	0.02
query21	15.45	0.18	0.14
query22	4.97	0.06	0.05
query23	16.16	0.32	0.13
query24	2.91	0.44	0.33
query25	0.11	0.06	0.03
query26	0.74	0.20	0.15
query27	0.05	0.04	0.03
query28	3.54	0.82	0.34
query29	12.46	4.08	3.18
query30	0.27	0.15	0.15
query31	2.78	0.58	0.31
query32	3.23	0.59	0.50
query33	3.24	3.24	3.23
query34	15.85	4.00	3.31
query35	3.27	3.18	3.22
query36	0.55	0.41	0.43
query37	0.09	0.06	0.07
query38	0.05	0.03	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.64 s
Total hot run time: 24 s

github-actions · 2026-08-11T07:25:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-08-11T07:25:27Z

PR approved by anyone and no changes requested.

hello-stephen · 2026-08-11T07:49:57Z

skip check_coverage

…#66610) Problem Summary: Cloud cumulative compaction scheduling maintains an approximate score between metadata synchronizations. The online write path used the raw segment count, so a zero-segment rowset added no score even though metadata synchronization counts every empty rowset as one. Empty rowsets could therefore accumulate without increasing tablet scheduling priority. This PR clamps the online contribution of a load rowset to at least one and adds the same contribution for zero-segment delete-predicate rowsets, which bypass `CloudRowsetBuilder`. Non-empty PREPARED rowsets keep the existing segment-based estimate because their versions have not been assigned yet. The unit test models that PREPARED, versionless lifecycle and verifies both cases: a zero-segment rowset contributes one, while a three-segment rowset still contributes three. ### Release note Fix cloud cumulative compaction scheduling for empty rowsets.

…apache#66610) Problem Summary: Cloud cumulative compaction scheduling maintains an approximate score between metadata synchronizations. The online write path used the raw segment count, so a zero-segment rowset added no score even though metadata synchronization counts every empty rowset as one. Empty rowsets could therefore accumulate without increasing tablet scheduling priority. This PR clamps the online contribution of a load rowset to at least one and adds the same contribution for zero-segment delete-predicate rowsets, which bypass `CloudRowsetBuilder`. Non-empty PREPARED rowsets keep the existing segment-based estimate because their versions have not been assigned yet. The unit test models that PREPARED, versionless lifecycle and verifies both cases: a zero-segment rowset contributes one, while a three-segment rowset still contributes three. ### Release note Fix cloud cumulative compaction scheduling for empty rowsets.

…#66610) Problem Summary: Cloud cumulative compaction scheduling maintains an approximate score between metadata synchronizations. The online write path used the raw segment count, so a zero-segment rowset added no score even though metadata synchronization counts every empty rowset as one. Empty rowsets could therefore accumulate without increasing tablet scheduling priority. This PR clamps the online contribution of a load rowset to at least one and adds the same contribution for zero-segment delete-predicate rowsets, which bypass `CloudRowsetBuilder`. Non-empty PREPARED rowsets keep the existing segment-based estimate because their versions have not been assigned yet. The unit test models that PREPARED, versionless lifecycle and verifies both cases: a zero-segment rowset contributes one, while a three-segment rowset still contributes three. ### Release note Fix cloud cumulative compaction scheduling for empty rowsets.

…action score #66610 (#66656) Cherry-picked from #66610 --------- Co-authored-by: Jamie <lianyukang@selectdb.com>

Yukang-Lian marked this pull request as ready for review August 10, 2026 08:44

Yukang-Lian requested a review from gavinchou as a code owner August 10, 2026 08:44

Yukang-Lian changed the title ~~[fix](cloud) Count empty rowsets in approximate compaction score~~ [fix](compaction) Count empty rowsets in approximate compaction score Aug 11, 2026

luwei16 approved these changes Aug 11, 2026

View reviewed changes

luwei16 added dev/3.1.x dev/4.0.x dev/4.1.x labels Aug 11, 2026

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 11, 2026

github-actions Bot added the reviewed label Aug 11, 2026

gavinchou approved these changes Aug 11, 2026

View reviewed changes

Yukang-Lian merged commit e43a387 into apache:master Aug 11, 2026
38 of 39 checks passed

github-actions Bot mentioned this pull request Aug 11, 2026

branch-4.0: [fix](compaction) Count empty rowsets in approximate compaction score #66610 #66655

Closed

github-actions Bot mentioned this pull request Aug 11, 2026

branch-4.1: [fix](compaction) Count empty rowsets in approximate compaction score #66610 #66656

Merged

Yukang-Lian mentioned this pull request Sep 7, 2026

branch-3.1: [fix](compaction) Count empty rowsets in approximate compaction score #66610 #67591

Closed

1 of 4 tasks

yiguolei pushed a commit that referenced this pull request Sep 22, 2026

branch-4.1: [fix](compaction) Count empty rowsets in approximate comp…

a334145

…action score #66610 (#66656) Cherry-picked from #66610 --------- Co-authored-by: Jamie <lianyukang@selectdb.com>

yiguolei removed the dev/4.1.x label Sep 22, 2026

yiguolei added the dev/4.1.5-merged label Sep 22, 2026

Yukang-Lian removed dev/3.1.x dev/4.0.x labels Sep 22, 2026

yiguolei pushed a commit that referenced this pull request Sep 23, 2026

branch-4.1: [fix](compaction) Count empty rowsets in approximate comp…

3520924

…action score #66610 (#66656) Cherry-picked from #66610 --------- Co-authored-by: Jamie <lianyukang@selectdb.com>

Navigation Menu

Repository navigation

[fix](compaction) Count empty rowsets in approximate compaction score - #66610

[fix](compaction) Count empty rowsets in approximate compaction score#66610
Yukang-Lian merged 1 commit into
apache:masterfrom
Yukang-Lian:codex/fix-cloud-empty-rowset-score

Yukang-Lian commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

Yukang-Lian commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

hello-stephen commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Yukang-Lian commented Aug 10, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

Yukang-Lian commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

hello-stephen commented Aug 10, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

hello-stephen commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants