Repository navigation
[fix](cloud-compaction) prevent EMPTY_CUMULATIVE / BASE-CUMU races on the same tablet - #64619
Merged
Merged
Conversation
added 2 commits
June 17, 2026 16:28
… the same tablet
Bug
---
On the meta-service, start_compaction_job only rejected a new job when its
type strictly equalled an in-flight job's type. This left two races:
1. EMPTY_CUMULATIVE was treated as a different type from CUMULATIVE. While a
real CUMULATIVE [v_lo, v_hi] was still running, an EMPTY_CUMULATIVE could
be accepted and committed, advancing cumulative_point past v_hi. A
subsequent BASE compaction could then pull rowsets in [v_lo, v_hi] as
input and race with the in-flight CUMULATIVE on the same rowsets.
2. With check_input_versions_range=true, BASE and CUMULATIVE were never
cross-checked against each other, so overlapping input ranges across the
two types could be accepted concurrently.
Fix
---
* Normalize EMPTY_CUMULATIVE to CUMULATIVE for conflict detection so they
belong to the same conflict family.
* Extend the version-range conflict check to the whole rowset compaction
family (BASE / CUMULATIVE / EMPTY_CUMULATIVE / FULL) instead of same-type
only. Non-overlapping ranges across types are still allowed.
* Keep version_in_compaction notification scoped to the same family so BE
retry semantics are unchanged.
Behaviour matrix (new -> active, OK = accept, BUSY = JOB_TABLET_BUSY)
---------------------------------------------------------------------
before after
EMPTY_CUMU vs CUMU OK (race) BUSY
CUMU vs EMPTY_CUMU OK BUSY
BASE vs CUMU overlap OK (race) BUSY
CUMU vs BASE overlap OK (race) BUSY
BASE vs CUMU disjoint OK OK (unchanged)
same-type / FULL / STOP_TOKEN / idempotent same-id : unchanged
Tests
-----
* EmptyCumulativeBlockedByCumulativeTest
* BaseCumulativeCrossTypeConflictTest
…nflict predicate Followup of the previous commit. The conflict predicate (`may_conflict_by_type`) and the notification predicate (`is_same_conflict_family || FULL`) were no longer complementary. Concretely, a cross-family conflict (BASE vs CUMULATIVE with overlapping version ranges) returned JOB_TABLET_BUSY but did NOT surface the in-flight version range via `version_in_compaction`, leaving BE blind to ranges it should avoid when retrying. This regressed the BE retry hit rate for the very case Plan D was meant to catch. Fix --- * Use `may_conflict_by_type` for the notification loop, matching the conflict predicate so every relevant in-flight range is reported (including BASE ranges to a CUMULATIVE submitter, and vice versa). * Replace the `DCHECK_EQ(c.input_versions_size(), 2)` inside the loop with a defensive `if (c.input_versions_size() != 2) continue;` guard. Under the current registration flow an EMPTY_CUMULATIVE cannot coexist with a real CUMULATIVE in the active list (it is rejected upfront by the same-family check), so the DCHECK is not reachable in steady state. The guard is kept as defense-in-depth against (a) legacy persisted job_pb from older MS versions where same-type comparison let them coexist, and (b) future registration changes that might relax that invariant - we don't want a silent abort or a fabricated [0, 0] range leaking into the response. Tests ----- Updated `ParallelCumuCompactionTest` Step 8 to assert the new behaviour: a CUMULATIVE submission that conflicts with both BASE and CUMULATIVE jobs now receives all three ranges in `version_in_compaction`, not just the same-family ones.
liutang123
requested review from
gavinchou,
liaoxin01 and
luwei16
as code owners
June 17, 2026 11:42
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review summary: no blocking issues found.
Code-review checkpoint conclusions:
- Goal/test: the PR aims to prevent EMPTY_CUMULATIVE from racing an active CUMULATIVE and to reject overlapping BASE/CUMULATIVE rowset ranges. The meta-service conflict checks now cover those paths, and the added MetaServiceJobTest cases exercise the production request shapes for EMPTY, CUMULATIVE, and BASE.
- Scope: the implementation is focused on start_compaction_job conflict predicates plus targeted unit tests.
- Concurrency/transactions: the job-list decision still happens inside the existing meta-service transaction on the tablet job key. Same-job idempotency is preserved; no new locks or threads are introduced.
- Lifecycle/static/config: no new dynamic config, storage/protocol field, or cross-TU static initialization concern was added.
- Parallel paths: BASE, CUMULATIVE, EMPTY_CUMULATIVE, FULL, STOP_TOKEN, and schema-change blocking interactions were checked. FULL and STOP_TOKEN behavior remains unchanged; index-change compaction uses the same start path.
- Data correctness/persistence: the change prevents the overlapping rowset-compaction registrations that could violate cumulative-point/version consistency. No new persisted metadata format or replay path is introduced.
- Tests: added unit tests cover EMPTY-vs-CUMULATIVE blocking, BASE/CUMULATIVE overlap rejection, non-overlap acceptance, symmetry, and version_in_compaction response contents.
- Observability/performance: no new high-cardinality logs or expensive operations were introduced; the extra checks are over the existing active compaction list.
- RPC boundary: start_tablet_job already has RPC_PREPROCESS/RPC_RATE_LIMIT and continues to set MetaServiceCode on return paths.
- User focus: no additional user-provided review focus was present.
I did not run local Cloud UT in this review runner. GitHub shows formatter/style checks passing; Cloud UT and compile were still pending when reviewed.
Non-blocking cleanup: the PR description still says version_in_compaction is scoped to the same family, while the final patch reports all rowset-family ranges. Consider updating the description before merge so it matches the second commit.
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
luwei16
approved these changes
Jun 22, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 22, 2026
… the same tablet (#64619) Bug --- BE config enable_parallel_cumu_compaction to true. --- On the meta-service, start_compaction_job only rejected a new job when its type strictly equalled an in-flight job's type. This left two races: 1. EMPTY_CUMULATIVE was treated as a different type from CUMULATIVE. While a real CUMULATIVE [v_lo, v_hi] was still running, an EMPTY_CUMULATIVE could be accepted and committed, advancing cumulative_point past v_hi. A subsequent BASE compaction could then pull rowsets in [v_lo, v_hi] as input and race with the in-flight CUMULATIVE on the same rowsets. 2. With check_input_versions_range=true, BASE and CUMULATIVE were never cross-checked against each other, so overlapping input ranges across the two types could be accepted concurrently. Fix --- * Normalize EMPTY_CUMULATIVE to CUMULATIVE for conflict detection so they belong to the same conflict family. * Extend the version-range conflict check to the whole rowset compaction family (BASE / CUMULATIVE / EMPTY_CUMULATIVE / FULL) instead of same-type only. Non-overlapping ranges across types are still allowed. * Keep version_in_compaction notification scoped to the same family so BE retry semantics are unchanged. Behaviour matrix (new -> active, OK = accept, BUSY = JOB_TABLET_BUSY) --------------------------------------------------------------------- before after EMPTY_CUMU vs CUMU OK (race) BUSY CUMU vs EMPTY_CUMU OK BUSY BASE vs CUMU overlap OK (race) BUSY CUMU vs BASE overlap OK (race) BUSY BASE vs CUMU disjoint OK OK (unchanged) same-type / FULL / STOP_TOKEN / idempotent same-id : unchanged Tests ----- * EmptyCumulativeBlockedByCumulativeTest * BaseCumulativeCrossTypeConflictTest The cluster log: ``` 1. start cc -------------- start cc(42326-42474) ------------ RuntimeLogger I20260616 06:05:58.687474 1715 cloud_cumulative_compaction.cpp:111] start CloudCumulativeCompaction, tablet_id=1763693245218, range=[42326-42474]|job_id=7c02be46-86a3-43b7-9687-9c93b1f3affe|input_rowsets=5|input_rows=427170|input_segments=5|input_rowsets_data_size=52916937|input_rowsets_index_size=0|input_rowsets_total_size=52916937|tablet_max_version=42475|cumulative_point=42326|num_rowsets=27|cumu_num_rowsets=6 -------------- meta service record this cc -------------- I20260616 06:05:58.687247 3747094 meta_service_helper.h:174] begin start_tablet_job remote_caller=10.2.18.57:52036 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: CUMULATIVE input_versions: 42326 input_versions: 42474 base_compaction_cnt: 91 cumulative_compaction_cnt: 3590 id: "7c02be46-86a3-43b7-9687-9c93b1f3affe" expiration: 1781647558 lease: 1781561238 check_input_versions_range: true } } request_ip: "10.2.18.57:9050" meta_service.10.2.16.38.INFO:I20260616 06:05:58.688797 3747093 meta_service_job.cpp:272] (1753070360)compaction job to save job={"initiator":"10.2.18.57:9050","type":"CUMULATIVE","input_versions":["42326","42474"],"base_compaction_cnt":"91","cumulative_compaction_cnt":"3590","id":"7c02be46-86a3-43b7-9687-9c93b1f3affe","expiration":"1781647558","lease":"1781561238","check_input_versions_range":true} 2. DELETE trigger the increase of comulative point. --------- meta service record the EMPTY_CUMULATIVE job. The comulative point has become to 42476---------- I20260616 06:05:58.792258 1621135 meta_service_helper.h:174] begin start_tablet_job remote_caller=10.2.18.57:33586 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: EMPTY_CUMULATIVE base_compaction_cnt: 91 cumulative_compaction_cnt: 3590 id: "92277d99-b14f-42df-bfd8-c26e75ff8052" lease: 1781561178 } } request_ip: "10.2.18.57:9050" I20260616 06:05:58.793903 1621138 meta_service_job.cpp:272] (1753070360)compaction job to save job={"initiator":"10.2.18.57:9050","type":"EMPTY_CUMULATIVE","base_compaction_cnt":"91","cumulative_compaction_cnt":"3590","id":"92277d99-b14f-42df-bfd8-c26e75ff8052","lease":"1781561178"} I20260616 06:05:58.796713 1621137 meta_service_helper.h:174] begin finish_tablet_job remote_caller=10.2.18.57:60878 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" action: COMMIT job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: EMPTY_CUMULATIVE input_cumulative_point: 42326 output_cumulative_point: 42476 base_compaction_cnt: 91 cumulative_compaction_cnt: 3590 id: "92277d99-b14f-42df-bfd8-c26e75ff8052" lease: 1781561178 } } request_ip: "10.2.18.57:9050" --------- BE log the EMPTY_CUMULATIVE -------------- RuntimeLogger I20260616 06:05:58.801268 1715 cloud_cumulative_compaction.cpp:533] do empty cumulative compaction to update cumulative point|job_id=92277d99-b14f-42df-bfd8-c26e75ff8052|tablet_id=1763693245218|input_cumulative_point=42326|output_cumulative_point=42476 RuntimeLogger I20260616 06:05:58.801329 1715 cloud_cumulative_compaction.cpp:539] tablet stats=idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } data_size: 25302189990 num_rows: 303319387 num_rowsets: 27 num_segments: 43 base_compaction_cnt: 91 cumulative_compaction_cnt: 3591 cumulative_point: 42476 last_base_compaction_time_ms: 1781539519000 last_cumu_compaction_time_ms: 1781561158000 index_size: 0 segment_size: 25302189990 RuntimeLogger W20260616 06:05:58.801383 1715 cloud_storage_engine.cpp:529] failed to submit compaction task for tablet: 1763693245218, err: [E-2010]cumulative compaction meet delete version 3. start bc -------------- be record the base compaction (2 ～ 42431) RuntimeLogger I20260616 06:06:01.042435 1715 cloud_base_compaction.cpp:84] start CloudBaseCompaction, tablet_id=1763693245218, range=[2-42431]|job_id=a8e92687-2211-4d74-82aa-d6e99c3fc360|input_rowsets=21|input_rows=303315005|input_segments=39|input_rowsets_data_size=25294029764|input_rowsets_index_size=0|input_rowsets_total_size=25294029764 -------------- meta service record the bc -------------- I20260616 06:06:01.043434 3747099 meta_service_helper.h:174] begin start_tablet_job remote_caller=10.2.18.57:52174 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: BASE input_versions: 2 input_versions: 42431 base_compaction_cnt: 91 cumulative_compaction_cnt: 3591 id: "a8e92687-2211-4d74-82aa-d6e99c3fc360" expiration: 1781647561 lease: 1781561241 } } request_ip: "10.2.18.57:9050" RuntimeLogger I20260616 06:06:13.279381 1708 cloud_cumulative_compaction.cpp:208] finish CloudCumulativeCompaction, tablet_id=1763693245218, cost=14587ms, range=[42326-42474]|job_id=7c02be46-86a3-43b7-9687-9c93b1f3affe|input_rowsets=5|input_rows=427170|input_segments=5|input_rowsets_data_size=52916937|input_rowsets_index_size=0|input_rowsets_total_size=52916937|output_rows=427170|output_segments=1|output_rowset_data_size=45147495|output_rowset_index_size=0|output_rowset_total_size=45147495|tablet_max_version=42475|cumulative_point=42476|num_rowsets=23|cumu_num_rowsets=0|local_read_time_us=1407|remote_read_time_us=0|local_read_bytes=5717690|remote_read_bytes=0 4. cc complete and -------------- meta service record the cc and drop rowsets whitch version between 42326 and 42474 RuntimeLogger I20260616 06:06:13.279381 1708 cloud_cumulative_compaction.cpp:208] finish CloudCumulativeCompaction, tablet_id=1763693245218, cost=14587ms, range=[42326-42474]|job_id=7c02be46-86a3-43b7-9687-9c93b1f3affe|input_rowsets=5|input_rows=427170|input_segments=5|input_rowsets_data_size=52916937|input_rowsets_index_size=0|input_rowsets_total_size=52916937|output_rows=427170|output_segments=1|output_rowset_data_size=45147495|output_rowset_index_size=0|output_rowset_total_size=45147495|tablet_max_version=42475|cumulative_point=42476|num_rowsets=23|cumu_num_rowsets=0|local_read_time_us=1407|remote_read_time_us=0|local_read_bytes=5717690|remote_read_bytes=0 I20260616 06:06:13.269352 3747098 meta_service_helper.h:174] begin finish_tablet_job remote_caller=10.2.18.57:52070 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" action: COMMIT job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: CUMULATIVE input_cumulative_point: 42476 output_cumulative_point: 42475 num_input_rowsets: 5 num_input_segments: 5 num_output_rowsets: 1 num_output_segments: 1 size_input_rowsets: 52916937 size_output_rowsets: 45147495 num_input_rows: 427170 num_output_rows: 427170 input_versions: 42326 input_versions: 42474 output_versions: 42474 output_rowset_ids: "020000000008d8252c4a4e54ca9c6c96f347dc1417ad5db8" txn_id: 8054901679135295045 id: "7c02be46-86a3-43b7-9687-9c93b1f3affe" index_size_input_rowsets: 0 segment_size_input_rowsets: 52916937 index_size_output_rowsets: 0 segment_size_output_rowsets: 45147495 } } request_ip: "10.2.18.57:9050" 5. bc complete -------------- bc complete and generate new rs (2-42431)-------------- RuntimeLogger I20260616 06:54:02.973222 1699 cloud_base_compaction.cpp:293] finish CloudBaseCompaction, tablet_id=1763693245218, cost=2881925ms range=[2-42431]|job_id=a8e92687-2211-4d74-82aa-d6e99c3fc360|input_rowsets=21|input_rows=303315005|input_segments=39|input_rowsets_data_size=25294029764|input_rowsets_index_size=0|input_rowsets_total=25294029764|output_rows=303315005|output_segments=24|output_rowset_data_size=25176822135|output_rowset_index_size=0|output_rowset_total_size=25176822135|local_read_time_us=1319383|remote_read_time_us=0|local_read_bytes=1154464868|remote_read_bytes=0 ``` The version (2-42431) generated by bc conficts with the version (42326-42474) gegerated by cc. The compaction info: ```json { "rowsets": [ "[0-1] 0 DATA NONOVERLAPPING 0200000000000000ffffffffffea4868ffffffffffffffe8 0", "[2-42431] 24 DATA NONOVERLAPPING 020000000008d8262c4a4e54ca9c6c96f347dc1417ad5db8 23.45 GB", "[42326-42474] 1 DATA NONOVERLAPPING 020000000008d8252c4a4e54ca9c6c96f347dc1417ad5db8 43.06 MB", "[42475-42475] 0 DELETE OVERLAP_UNKNOWN 020000000008d8242c4a4e54ca9c6c96f347dc1417ad5db8 0" ], "missing_rowsets": [ "[42432-42325]" ] } ``` ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: liutang123 <liulijia1029@google.com>
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 22, 2026
… the same tablet (#64619) Bug --- BE config enable_parallel_cumu_compaction to true. --- On the meta-service, start_compaction_job only rejected a new job when its type strictly equalled an in-flight job's type. This left two races: 1. EMPTY_CUMULATIVE was treated as a different type from CUMULATIVE. While a real CUMULATIVE [v_lo, v_hi] was still running, an EMPTY_CUMULATIVE could be accepted and committed, advancing cumulative_point past v_hi. A subsequent BASE compaction could then pull rowsets in [v_lo, v_hi] as input and race with the in-flight CUMULATIVE on the same rowsets. 2. With check_input_versions_range=true, BASE and CUMULATIVE were never cross-checked against each other, so overlapping input ranges across the two types could be accepted concurrently. Fix --- * Normalize EMPTY_CUMULATIVE to CUMULATIVE for conflict detection so they belong to the same conflict family. * Extend the version-range conflict check to the whole rowset compaction family (BASE / CUMULATIVE / EMPTY_CUMULATIVE / FULL) instead of same-type only. Non-overlapping ranges across types are still allowed. * Keep version_in_compaction notification scoped to the same family so BE retry semantics are unchanged. Behaviour matrix (new -> active, OK = accept, BUSY = JOB_TABLET_BUSY) --------------------------------------------------------------------- before after EMPTY_CUMU vs CUMU OK (race) BUSY CUMU vs EMPTY_CUMU OK BUSY BASE vs CUMU overlap OK (race) BUSY CUMU vs BASE overlap OK (race) BUSY BASE vs CUMU disjoint OK OK (unchanged) same-type / FULL / STOP_TOKEN / idempotent same-id : unchanged Tests ----- * EmptyCumulativeBlockedByCumulativeTest * BaseCumulativeCrossTypeConflictTest The cluster log: ``` 1. start cc -------------- start cc(42326-42474) ------------ RuntimeLogger I20260616 06:05:58.687474 1715 cloud_cumulative_compaction.cpp:111] start CloudCumulativeCompaction, tablet_id=1763693245218, range=[42326-42474]|job_id=7c02be46-86a3-43b7-9687-9c93b1f3affe|input_rowsets=5|input_rows=427170|input_segments=5|input_rowsets_data_size=52916937|input_rowsets_index_size=0|input_rowsets_total_size=52916937|tablet_max_version=42475|cumulative_point=42326|num_rowsets=27|cumu_num_rowsets=6 -------------- meta service record this cc -------------- I20260616 06:05:58.687247 3747094 meta_service_helper.h:174] begin start_tablet_job remote_caller=10.2.18.57:52036 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: CUMULATIVE input_versions: 42326 input_versions: 42474 base_compaction_cnt: 91 cumulative_compaction_cnt: 3590 id: "7c02be46-86a3-43b7-9687-9c93b1f3affe" expiration: 1781647558 lease: 1781561238 check_input_versions_range: true } } request_ip: "10.2.18.57:9050" meta_service.10.2.16.38.INFO:I20260616 06:05:58.688797 3747093 meta_service_job.cpp:272] (1753070360)compaction job to save job={"initiator":"10.2.18.57:9050","type":"CUMULATIVE","input_versions":["42326","42474"],"base_compaction_cnt":"91","cumulative_compaction_cnt":"3590","id":"7c02be46-86a3-43b7-9687-9c93b1f3affe","expiration":"1781647558","lease":"1781561238","check_input_versions_range":true} 2. DELETE trigger the increase of comulative point. --------- meta service record the EMPTY_CUMULATIVE job. The comulative point has become to 42476---------- I20260616 06:05:58.792258 1621135 meta_service_helper.h:174] begin start_tablet_job remote_caller=10.2.18.57:33586 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: EMPTY_CUMULATIVE base_compaction_cnt: 91 cumulative_compaction_cnt: 3590 id: "92277d99-b14f-42df-bfd8-c26e75ff8052" lease: 1781561178 } } request_ip: "10.2.18.57:9050" I20260616 06:05:58.793903 1621138 meta_service_job.cpp:272] (1753070360)compaction job to save job={"initiator":"10.2.18.57:9050","type":"EMPTY_CUMULATIVE","base_compaction_cnt":"91","cumulative_compaction_cnt":"3590","id":"92277d99-b14f-42df-bfd8-c26e75ff8052","lease":"1781561178"} I20260616 06:05:58.796713 1621137 meta_service_helper.h:174] begin finish_tablet_job remote_caller=10.2.18.57:60878 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" action: COMMIT job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: EMPTY_CUMULATIVE input_cumulative_point: 42326 output_cumulative_point: 42476 base_compaction_cnt: 91 cumulative_compaction_cnt: 3590 id: "92277d99-b14f-42df-bfd8-c26e75ff8052" lease: 1781561178 } } request_ip: "10.2.18.57:9050" --------- BE log the EMPTY_CUMULATIVE -------------- RuntimeLogger I20260616 06:05:58.801268 1715 cloud_cumulative_compaction.cpp:533] do empty cumulative compaction to update cumulative point|job_id=92277d99-b14f-42df-bfd8-c26e75ff8052|tablet_id=1763693245218|input_cumulative_point=42326|output_cumulative_point=42476 RuntimeLogger I20260616 06:05:58.801329 1715 cloud_cumulative_compaction.cpp:539] tablet stats=idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } data_size: 25302189990 num_rows: 303319387 num_rowsets: 27 num_segments: 43 base_compaction_cnt: 91 cumulative_compaction_cnt: 3591 cumulative_point: 42476 last_base_compaction_time_ms: 1781539519000 last_cumu_compaction_time_ms: 1781561158000 index_size: 0 segment_size: 25302189990 RuntimeLogger W20260616 06:05:58.801383 1715 cloud_storage_engine.cpp:529] failed to submit compaction task for tablet: 1763693245218, err: [E-2010]cumulative compaction meet delete version 3. start bc -------------- be record the base compaction (2 ～ 42431) RuntimeLogger I20260616 06:06:01.042435 1715 cloud_base_compaction.cpp:84] start CloudBaseCompaction, tablet_id=1763693245218, range=[2-42431]|job_id=a8e92687-2211-4d74-82aa-d6e99c3fc360|input_rowsets=21|input_rows=303315005|input_segments=39|input_rowsets_data_size=25294029764|input_rowsets_index_size=0|input_rowsets_total_size=25294029764 -------------- meta service record the bc -------------- I20260616 06:06:01.043434 3747099 meta_service_helper.h:174] begin start_tablet_job remote_caller=10.2.18.57:52174 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: BASE input_versions: 2 input_versions: 42431 base_compaction_cnt: 91 cumulative_compaction_cnt: 3591 id: "a8e92687-2211-4d74-82aa-d6e99c3fc360" expiration: 1781647561 lease: 1781561241 } } request_ip: "10.2.18.57:9050" RuntimeLogger I20260616 06:06:13.279381 1708 cloud_cumulative_compaction.cpp:208] finish CloudCumulativeCompaction, tablet_id=1763693245218, cost=14587ms, range=[42326-42474]|job_id=7c02be46-86a3-43b7-9687-9c93b1f3affe|input_rowsets=5|input_rows=427170|input_segments=5|input_rowsets_data_size=52916937|input_rowsets_index_size=0|input_rowsets_total_size=52916937|output_rows=427170|output_segments=1|output_rowset_data_size=45147495|output_rowset_index_size=0|output_rowset_total_size=45147495|tablet_max_version=42475|cumulative_point=42476|num_rowsets=23|cumu_num_rowsets=0|local_read_time_us=1407|remote_read_time_us=0|local_read_bytes=5717690|remote_read_bytes=0 4. cc complete and -------------- meta service record the cc and drop rowsets whitch version between 42326 and 42474 RuntimeLogger I20260616 06:06:13.279381 1708 cloud_cumulative_compaction.cpp:208] finish CloudCumulativeCompaction, tablet_id=1763693245218, cost=14587ms, range=[42326-42474]|job_id=7c02be46-86a3-43b7-9687-9c93b1f3affe|input_rowsets=5|input_rows=427170|input_segments=5|input_rowsets_data_size=52916937|input_rowsets_index_size=0|input_rowsets_total_size=52916937|output_rows=427170|output_segments=1|output_rowset_data_size=45147495|output_rowset_index_size=0|output_rowset_total_size=45147495|tablet_max_version=42475|cumulative_point=42476|num_rowsets=23|cumu_num_rowsets=0|local_read_time_us=1407|remote_read_time_us=0|local_read_bytes=5717690|remote_read_bytes=0 I20260616 06:06:13.269352 3747098 meta_service_helper.h:174] begin finish_tablet_job remote_caller=10.2.18.57:52070 original_client_ip=10.2.18.57:9050 request=cloud_unique_id: "1:1753070360:jYdIZgSo" action: COMMIT job { idx { table_id: 1753072815281 index_id: 1753072815282 partition_id: 1763693245203 tablet_id: 1763693245218 } compaction { initiator: "10.2.18.57:9050" type: CUMULATIVE input_cumulative_point: 42476 output_cumulative_point: 42475 num_input_rowsets: 5 num_input_segments: 5 num_output_rowsets: 1 num_output_segments: 1 size_input_rowsets: 52916937 size_output_rowsets: 45147495 num_input_rows: 427170 num_output_rows: 427170 input_versions: 42326 input_versions: 42474 output_versions: 42474 output_rowset_ids: "020000000008d8252c4a4e54ca9c6c96f347dc1417ad5db8" txn_id: 8054901679135295045 id: "7c02be46-86a3-43b7-9687-9c93b1f3affe" index_size_input_rowsets: 0 segment_size_input_rowsets: 52916937 index_size_output_rowsets: 0 segment_size_output_rowsets: 45147495 } } request_ip: "10.2.18.57:9050" 5. bc complete -------------- bc complete and generate new rs (2-42431)-------------- RuntimeLogger I20260616 06:54:02.973222 1699 cloud_base_compaction.cpp:293] finish CloudBaseCompaction, tablet_id=1763693245218, cost=2881925ms range=[2-42431]|job_id=a8e92687-2211-4d74-82aa-d6e99c3fc360|input_rowsets=21|input_rows=303315005|input_segments=39|input_rowsets_data_size=25294029764|input_rowsets_index_size=0|input_rowsets_total=25294029764|output_rows=303315005|output_segments=24|output_rowset_data_size=25176822135|output_rowset_index_size=0|output_rowset_total_size=25176822135|local_read_time_us=1319383|remote_read_time_us=0|local_read_bytes=1154464868|remote_read_bytes=0 ``` The version (2-42431) generated by bc conficts with the version (42326-42474) gegerated by cc. The compaction info: ```json { "rowsets": [ "[0-1] 0 DATA NONOVERLAPPING 0200000000000000ffffffffffea4868ffffffffffffffe8 0", "[2-42431] 24 DATA NONOVERLAPPING 020000000008d8262c4a4e54ca9c6c96f347dc1417ad5db8 23.45 GB", "[42326-42474] 1 DATA NONOVERLAPPING 020000000008d8252c4a4e54ca9c6c96f347dc1417ad5db8 43.06 MB", "[42475-42475] 0 DELETE OVERLAP_UNKNOWN 020000000008d8242c4a4e54ca9c6c96f347dc1417ad5db8 0" ], "missing_rowsets": [ "[42432-42325]" ] } ``` ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: liutang123 <liulijia1029@google.com>
Closed
Closed
mymeiyi
added a commit
that referenced
this pull request
Aug 18, 2026
…l cumulative compaction (#65915) When enable parallel cumulative compaction, the cu compaction with higher version range may finish before the cu compaction with lower range. For example, cu1: v10-v20, cu2: v21-v30. The cu2 may finish before cu1. When cu2 finish, it may set cu point to v31, does not consider if cu1 finish or fail. This pr correct the cu point advancement. related to #64619
yiguolei
pushed a commit
to mymeiyi/doris
that referenced
this pull request
Sep 5, 2026
…l cumulative compaction (apache#65915) When enable parallel cumulative compaction, the cu compaction with higher version range may finish before the cu compaction with lower range. For example, cu1: v10-v20, cu2: v21-v30. The cu2 may finish before cu1. When cu2 finish, it may set cu point to v31, does not consider if cu1 finish or fail. This pr correct the cu point advancement. related to apache#64619
mymeiyi
added a commit
to mymeiyi/doris
that referenced
this pull request
Sep 7, 2026
…l cumulative compaction (apache#65915) When enable parallel cumulative compaction, the cu compaction with higher version range may finish before the cu compaction with lower range. For example, cu1: v10-v20, cu2: v21-v30. The cu2 may finish before cu1. When cu2 finish, it may set cu point to v31, does not consider if cu1 finish or fail. This pr correct the cu point advancement. related to apache#64619
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
BE config enable_parallel_cumu_compaction to true.
On the meta-service, start_compaction_job only rejected a new job when its type strictly equalled an in-flight job's type. This left two races:
Fix
Behaviour matrix (new -> active, OK = accept, BUSY = JOB_TABLET_BUSY) ---------------------------------------------------------------------
before after
EMPTY_CUMU vs CUMU OK (race) BUSY
CUMU vs EMPTY_CUMU OK BUSY
BASE vs CUMU overlap OK (race) BUSY
CUMU vs BASE overlap OK (race) BUSY
BASE vs CUMU disjoint OK OK (unchanged)
same-type / FULL / STOP_TOKEN / idempotent same-id : unchanged
Tests
The cluster log:
The version (2-42431) generated by bc conficts with the version (42326-42474) gegerated by cc.
The compaction info:
{ "rowsets": [ "[0-1] 0 DATA NONOVERLAPPING 0200000000000000ffffffffffea4868ffffffffffffffe8 0", "[2-42431] 24 DATA NONOVERLAPPING 020000000008d8262c4a4e54ca9c6c96f347dc1417ad5db8 23.45 GB", "[42326-42474] 1 DATA NONOVERLAPPING 020000000008d8252c4a4e54ca9c6c96f347dc1417ad5db8 43.06 MB", "[42475-42475] 0 DELETE OVERLAP_UNKNOWN 020000000008d8242c4a4e54ca9c6c96f347dc1417ad5db8 0" ], "missing_rowsets": [ "[42432-42325]" ] }What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)