[compaction](config) Add a config to control whether to prune rows with delete sign=1 in base compaction by bobhan1 · Pull Request #48241 · apache/doris

bobhan1 · 2025-02-24T09:29:34Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-02-24T09:29:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-02-24T11:14:20Z

run buildall

doris-robot · 2025-02-24T12:50:04Z

TPC-H: Total hot run time: 31510 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24376acd8325eb8087eb5131569ac9513a604ccc, data reload: false

------ Round 1 ----------------------------------
q1	17602	5393	5098	5098
q2	2050	303	163	163
q3	10480	2380	758	758
q4	10229	1045	519	519
q5	7585	2492	2301	2301
q6	195	169	135	135
q7	908	768	613	613
q8	9303	1384	1149	1149
q9	4893	4618	4541	4541
q10	6829	2323	1877	1877
q11	481	276	256	256
q12	345	357	218	218
q13	17761	3673	3079	3079
q14	229	233	211	211
q15	503	470	461	461
q16	624	610	582	582
q17	579	867	336	336
q18	6828	6260	6234	6234
q19	1208	970	543	543
q20	305	324	190	190
q21	2873	2141	1943	1943
q22	369	330	303	303
Total cold run time: 102179 ms
Total hot run time: 31510 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5136	5168	5150	5150
q2	230	335	227	227
q3	2184	2695	2279	2279
q4	1390	1807	1363	1363
q5	4208	4117	4158	4117
q6	216	164	123	123
q7	1912	1818	1800	1800
q8	2644	2637	2604	2604
q9	7269	7015	7129	7015
q10	2979	3205	2752	2752
q11	584	503	485	485
q12	669	776	647	647
q13	3492	3897	3233	3233
q14	271	310	271	271
q15	514	461	465	461
q16	644	689	651	651
q17	1169	1610	1341	1341
q18	7521	7328	7358	7328
q19	803	797	846	797
q20	2006	2074	1890	1890
q21	5372	4992	4848	4848
q22	649	573	537	537
Total cold run time: 51862 ms
Total hot run time: 49919 ms

doris-robot · 2025-02-24T13:01:29Z

TPC-DS: Total hot run time: 183137 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24376acd8325eb8087eb5131569ac9513a604ccc, data reload: false

query1	960	391	369	369
query2	6524	1869	1836	1836
query3	6796	222	215	215
query4	26570	23728	23204	23204
query5	4337	659	501	501
query6	305	204	183	183
query7	4611	502	303	303
query8	291	242	234	234
query9	8642	2575	2570	2570
query10	454	306	258	258
query11	15446	15037	14881	14881
query12	159	106	101	101
query13	1680	548	403	403
query14	8976	6075	6206	6075
query15	214	197	178	178
query16	7230	627	453	453
query17	1240	683	555	555
query18	1938	394	287	287
query19	181	192	168	168
query20	122	145	115	115
query21	202	119	100	100
query22	4280	4341	4152	4152
query23	34014	32881	32866	32866
query24	8444	2398	2377	2377
query25	513	446	379	379
query26	1238	271	146	146
query27	2717	474	333	333
query28	4322	2416	2409	2409
query29	705	555	414	414
query30	227	183	153	153
query31	906	859	760	760
query32	72	63	61	61
query33	567	360	305	305
query34	766	846	506	506
query35	802	814	721	721
query36	965	949	887	887
query37	115	99	101	99
query38	4072	4195	4028	4028
query39	1503	1389	1390	1389
query40	208	117	97	97
query41	53	51	54	51
query42	125	111	106	106
query43	503	488	478	478
query44	1278	799	783	783
query45	174	193	161	161
query46	855	1021	640	640
query47	1722	1759	1672	1672
query48	389	407	303	303
query49	773	504	415	415
query50	695	719	418	418
query51	4162	4183	4223	4183
query52	108	104	92	92
query53	228	249	188	188
query54	476	485	399	399
query55	79	85	81	81
query56	278	248	270	248
query57	1109	1156	1042	1042
query58	242	240	232	232
query59	2602	2807	2517	2517
query60	289	297	258	258
query61	125	119	114	114
query62	780	722	657	657
query63	233	186	189	186
query64	4261	1075	753	753
query65	3228	3151	3148	3148
query66	1144	433	339	339
query67	16009	15457	15381	15381
query68	5575	790	514	514
query69	483	293	260	260
query70	1217	1151	1069	1069
query71	396	291	259	259
query72	5756	3604	3927	3604
query73	682	735	356	356
query74	9546	9133	9258	9133
query75	3147	3179	2736	2736
query76	3189	1161	738	738
query77	507	369	328	328
query78	9931	10004	9270	9270
query79	2240	843	600	600
query80	601	529	454	454
query81	502	290	242	242
query82	465	125	93	93
query83	162	173	154	154
query84	237	94	73	73
query85	783	339	297	297
query86	379	308	284	284
query87	4483	4512	4319	4319
query88	4214	2190	2198	2190
query89	395	321	282	282
query90	1855	200	189	189
query91	132	139	104	104
query92	75	58	54	54
query93	1867	1013	573	573
query94	700	414	304	304
query95	343	268	254	254
query96	506	543	275	275
query97	2776	2784	2732	2732
query98	238	207	196	196
query99	1470	1430	1285	1285
Total cold run time: 270086 ms
Total hot run time: 183137 ms

doris-robot · 2025-02-24T13:06:46Z

ClickBench: Total hot run time: 30.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24376acd8325eb8087eb5131569ac9513a604ccc, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.23	0.06	0.07
query4	1.61	0.10	0.10
query5	0.41	0.41	0.40
query6	1.15	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.53	0.52
query10	0.57	0.57	0.58
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.73	2.69	2.75
query15	0.93	0.84	0.85
query16	0.38	0.38	0.38
query17	1.01	1.04	1.04
query18	0.20	0.19	0.20
query19	1.93	1.77	1.98
query20	0.02	0.01	0.02
query21	15.35	0.91	0.55
query22	0.75	1.13	0.61
query23	15.04	1.36	0.65
query24	7.47	1.46	0.56
query25	0.54	0.11	0.23
query26	0.53	0.16	0.13
query27	0.05	0.05	0.05
query28	9.67	0.82	0.44
query29	12.53	4.01	3.28
query30	0.25	0.08	0.06
query31	2.85	0.57	0.38
query32	3.22	0.55	0.47
query33	2.98	3.03	3.05
query34	15.77	5.08	4.51
query35	4.56	4.53	4.52
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.14
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.55 s
Total hot run time: 30.3 s

doris-robot · 2025-02-24T13:43:56Z

TeamCity be ut coverage result:
Function Coverage: 44.51% (11753/26406)
Line Coverage: 34.01% (98270/288948)
Region Coverage: 33.09% (50325/152098)
Branch Coverage: 28.73% (25298/88042)
Coverage Report: http://coverage.selectdb-in.cc/coverage/24376acd8325eb8087eb5131569ac9513a604ccc_24376acd8325eb8087eb5131569ac9513a604ccc/report/index.html

bobhan1 · 2025-02-25T01:58:34Z

run buildall

doris-robot · 2025-02-25T02:25:24Z

TPC-H: Total hot run time: 31722 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8e50133958c35a66452511d6eacd5ce9b82527cf, data reload: false

------ Round 1 ----------------------------------
q1	17589	5287	5091	5091
q2	2048	297	170	170
q3	10448	1344	732	732
q4	10276	1018	539	539
q5	8260	2404	2380	2380
q6	192	170	136	136
q7	897	768	619	619
q8	9318	1294	1142	1142
q9	5002	4652	4834	4652
q10	6822	2328	1886	1886
q11	462	282	257	257
q12	348	357	218	218
q13	17762	3697	3147	3147
q14	237	225	215	215
q15	508	469	458	458
q16	627	618	584	584
q17	583	856	341	341
q18	6863	6178	6165	6165
q19	1633	943	557	557
q20	318	326	193	193
q21	2757	2201	1940	1940
q22	354	339	300	300
Total cold run time: 103304 ms
Total hot run time: 31722 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5194	5164	5198	5164
q2	241	340	233	233
q3	2157	2709	2309	2309
q4	1440	1857	1405	1405
q5	4217	4115	4141	4115
q6	201	179	125	125
q7	1874	1813	1784	1784
q8	2593	2652	2639	2639
q9	7284	7136	7128	7128
q10	3022	3204	2720	2720
q11	581	516	499	499
q12	653	780	592	592
q13	3568	3904	3257	3257
q14	274	298	264	264
q15	518	462	452	452
q16	661	691	642	642
q17	1140	1605	1342	1342
q18	7522	7423	7138	7138
q19	826	958	1092	958
q20	1966	2114	1916	1916
q21	5340	5044	4783	4783
q22	653	585	532	532
Total cold run time: 51925 ms
Total hot run time: 49997 ms

doris-robot · 2025-02-25T02:36:50Z

TPC-DS: Total hot run time: 184359 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8e50133958c35a66452511d6eacd5ce9b82527cf, data reload: false

query1	981	393	383	383
query2	6543	1848	1849	1848
query3	6787	211	206	206
query4	26177	23775	23174	23174
query5	4323	668	482	482
query6	296	193	182	182
query7	4622	534	295	295
query8	312	254	228	228
query9	8651	2583	2570	2570
query10	450	298	264	264
query11	15552	15109	15047	15047
query12	159	114	110	110
query13	1667	511	403	403
query14	9104	6677	6760	6677
query15	210	194	160	160
query16	7240	637	485	485
query17	1187	694	564	564
query18	1958	390	299	299
query19	189	176	150	150
query20	118	132	114	114
query21	208	115	101	101
query22	4052	4372	4289	4289
query23	33967	33125	33206	33125
query24	7732	2365	2388	2365
query25	517	444	369	369
query26	1221	269	149	149
query27	2158	493	337	337
query28	3937	2411	2402	2402
query29	741	517	410	410
query30	231	179	161	161
query31	946	862	773	773
query32	77	65	67	65
query33	557	363	304	304
query34	779	847	510	510
query35	790	820	753	753
query36	967	979	877	877
query37	121	96	72	72
query38	4220	4115	4094	4094
query39	1506	1384	1386	1384
query40	216	120	113	113
query41	61	57	69	57
query42	131	112	103	103
query43	480	505	461	461
query44	1312	806	804	804
query45	180	175	171	171
query46	872	1036	646	646
query47	1746	1777	1686	1686
query48	374	413	306	306
query49	802	519	465	465
query50	680	738	398	398
query51	4174	4264	4115	4115
query52	113	108	100	100
query53	242	263	191	191
query54	494	486	414	414
query55	83	85	80	80
query56	296	261	295	261
query57	1132	1134	1065	1065
query58	257	231	239	231
query59	2636	2798	2555	2555
query60	295	271	266	266
query61	112	117	121	117
query62	850	727	686	686
query63	232	188	199	188
query64	4393	987	685	685
query65	3199	3138	3178	3138
query66	1137	401	301	301
query67	16019	15354	15345	15345
query68	8074	750	502	502
query69	459	293	255	255
query70	1194	1105	1078	1078
query71	419	298	274	274
query72	5673	3665	3871	3665
query73	761	742	352	352
query74	9343	9332	9056	9056
query75	3303	3201	2653	2653
query76	3264	1167	746	746
query77	744	352	279	279
query78	9922	10033	9302	9302
query79	2498	794	616	616
query80	590	522	438	438
query81	485	270	244	244
query82	669	122	92	92
query83	180	217	156	156
query84	238	98	78	78
query85	786	341	300	300
query86	332	279	253	253
query87	4578	4612	4448	4448
query88	3548	2177	2198	2177
query89	387	317	290	290
query90	1900	197	249	197
query91	131	137	109	109
query92	72	62	59	59
query93	1219	1026	592	592
query94	699	395	306	306
query95	349	272	267	267
query96	485	560	266	266
query97	2831	2822	2733	2733
query98	232	206	203	203
query99	1353	1401	1244	1244
Total cold run time: 270489 ms
Total hot run time: 184359 ms

doris-robot · 2025-02-25T02:42:08Z

ClickBench: Total hot run time: 31.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8e50133958c35a66452511d6eacd5ce9b82527cf, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.41	0.42	0.40
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.53
query10	0.57	0.58	0.57
query11	0.16	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.61
query14	2.69	2.70	2.70
query15	0.92	0.85	0.85
query16	0.39	0.38	0.39
query17	1.02	1.03	1.00
query18	0.21	0.19	0.19
query19	1.92	1.82	2.05
query20	0.02	0.01	0.01
query21	15.36	0.89	0.53
query22	0.74	1.25	0.84
query23	14.81	1.41	0.62
query24	7.23	1.20	1.17
query25	0.55	0.30	0.06
query26	0.47	0.16	0.13
query27	0.05	0.05	0.04
query28	10.38	0.86	0.43
query29	12.56	3.88	3.21
query30	0.26	0.08	0.06
query31	2.83	0.60	0.40
query32	3.23	0.54	0.47
query33	2.98	3.01	3.00
query34	15.72	5.18	4.51
query35	4.64	4.57	4.57
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.15	0.13
query41	0.07	0.03	0.02
query42	0.03	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.82 s
Total hot run time: 31.04 s

doris-robot · 2025-02-25T03:07:45Z

TeamCity be ut coverage result:
Function Coverage: 44.55% (11764/26406)
Line Coverage: 34.03% (98321/288959)
Region Coverage: 33.14% (50412/152112)
Branch Coverage: 28.78% (25347/88060)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8e50133958c35a66452511d6eacd5ce9b82527cf_8e50133958c35a66452511d6eacd5ce9b82527cf/report/index.html

dataroaring

LGTM

github-actions · 2025-02-25T14:03:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-25T14:03:18Z

PR approved by anyone and no changes requested.

zhannngchen

LGTM

…th delete sign=1 in base compaction (#48241) Add a config to control whether to prune rows with delete sign=1 in base compaction for correctness test

…th delete sign=1 in base compaction (apache#48241) Add a config to control whether to prune rows with delete sign=1 in base compaction for correctness test

… to prune rows with delete sign=1 in base compaction (#48241) (#48620) pick #48241

…th delete sign=1 in base compaction (apache#48241) Add a config to control whether to prune rows with delete sign=1 in base compaction for correctness test

… to prune rows with delete sign=1 in base compaction (#48241) (#48888) pick #48241

…th delete sign=1 in base compaction (apache#48241) Add a config to control whether to prune rows with delete sign=1 in base compaction for correctness test

dataroaring reviewed Feb 24, 2025

View reviewed changes

Comment thread be/src/common/config.cpp Resolved

bobhan1 force-pushed the config-prune-delete-sign-in-base-compaction branch from 728d234 to 4113ad7 Compare February 24, 2025 10:23

bobhan1 added 3 commits February 25, 2025 09:58

update

870c6c6

update

c92b921

add case

8e50133

bobhan1 force-pushed the config-prune-delete-sign-in-base-compaction branch from 24376ac to 8e50133 Compare February 25, 2025 01:58

dataroaring approved these changes Feb 25, 2025

View reviewed changes

dataroaring added dev/2.1.x dev/3.0.x labels Feb 25, 2025

github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 25, 2025

zhannngchen approved these changes Feb 28, 2025

View reviewed changes

zhannngchen merged commit 9e1aa77 into apache:master Feb 28, 2025

github-actions Bot mentioned this pull request Feb 28, 2025

branch-3.0: [compaction](config) Add a config to control whether to prune rows with delete sign=1 in base compaction #48241 #48491

Closed

github-actions Bot added the dev/2.1.x-conflict label Feb 28, 2025

bobhan1 mentioned this pull request Mar 4, 2025

branch-2.1-pick: [compaction](config) Add a config to control whether to prune rows with delete sign=1 in base compaction (#48241) #48620

Merged

yiguolei pushed a commit that referenced this pull request Mar 5, 2025

branch-2.1-pick: [compaction](config) Add a config to control whether…

74b85d6

… to prune rows with delete sign=1 in base compaction (#48241) (#48620) pick #48241

yiguolei added dev/2.1.9-merged and removed dev/2.1.x dev/2.1.x-conflict labels Mar 5, 2025

bobhan1 mentioned this pull request Mar 10, 2025

branch-3.0-pick: [compaction](config) Add a config to control whether to prune rows with delete sign=1 in base compaction (#48241) #48888

Merged

dataroaring pushed a commit that referenced this pull request Mar 13, 2025

branch-3.0-pick: [compaction](config) Add a config to control whether…

0648241

… to prune rows with delete sign=1 in base compaction (#48241) (#48888) pick #48241

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Mar 13, 2025

hanke580 mentioned this pull request Oct 21, 2026

[Bug] Sequence column: base/full compaction drops delete-sign rows, so a later write with a smaller sequence value brings a deleted row back #69409

Open

2 of 3 tasks

Conversation

bobhan1 commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Feb 24, 2025

Uh oh!

Uh oh!

bobhan1 commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

bobhan1 commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

doris-robot commented Feb 25, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 25, 2025

Uh oh!

github-actions Bot commented Feb 25, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobhan1 commented Feb 24, 2025 •

edited

Loading