Repository navigation
[feat](row binlog)：support cloud binlog and independent binlog tablet - #65810
Merged
Merged
Conversation
Userwhite
requested review from
924060929,
Gabriel39,
airborne12,
csun5285,
deardeng,
eldenmoon,
englefly,
gavinchou,
liaoxin01,
luwei16,
morningman,
morrySnow,
mymeiyi,
starocean999 and
yiguolei
as code owners
July 20, 2026 07:54
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Userwhite
changed the title
Userwhite
force-pushed
the
support_cloud_binlog
branch
from
July 20, 2026 11:52
185bbfe to
a42d600
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29573 ms |
Contributor
TPC-DS: Total hot run time: 176800 ms |
Contributor
ClickBench: Total hot run time: 25.03 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29328 ms |
Contributor
TPC-DS: Total hot run time: 178121 ms |
Contributor
ClickBench: Total hot run time: 25.01 s |
5 of 16 tasks
gavinchou
reviewed
Jul 22, 2026
gavinchou
reviewed
Jul 22, 2026
AntiTopQuark
pushed a commit
to AntiTopQuark/doris
that referenced
this pull request
Aug 12, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: RowBinlogTabletSchedulerTest passed a cast null Tag through Deencapsulation. The reflection helper rejects untyped null arguments before invoking the scheduler, so the test observed IllegalArgumentException instead of the expected SchedException. Use the helper's Class token convention to resolve the Tag parameter while still passing null to the colocate scheduling path. ### Release note None ### Check List (For Author) - Test: Not run per request; only static source review and git diff checks were completed - Behavior changed: No. This only fixes a test reflection argument - Does this need documentation: No
Contributor
Author
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…apache#65810) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: This PR implements Row Binlog based on an independent row-binlog tablet model for both non-cloud mode and cloud mode. Before this change, Row Binlog data was tightly coupled with the base tablet write/compaction path, and cloud mode did not have a complete end- to-end implementation. This made it difficult to keep Row Binlog behavior consistent across storage-compute integrated and storage-compute separated deployments, especially for MOW tables, historical values, partial update/upsert publish conflicts, and compaction. The new implementation introduces row-binlog companion tablets and wires them through FE metadata, BE write/publish/compaction/query paths, and Cloud Meta Service. In non-cloud mode, Row Binlog write and compaction are refactored around the independent tablet model. In cloud mode, the same model is extended with cloud-specific rowset commit, publish, delete-bitmap, metadata, and compaction handling. Main changes: 1. Independent Row Binlog tablet model - Store Row Binlog data in companion row-binlog tablets instead of coupling it directly with base tablet rowsets. - Track the relationship between base tablets and row-binlog tablets through aligned tablet ids. - Represent Row Binlog indexes with dedicated materialized-index states. - Keep Row Binlog schema aligned with master: `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`. 2. Non-cloud write, publish, and compaction path - Support writing Row Binlog data through independent tablets in non- cloud mode. - Propagate Row Binlog tablet information through FE sink planning and BE load/write context. - Refactor Row Binlog compaction to reuse cumulative-compaction style logic. - Add `CumulativeCompactionBinlogPolicy` for non-cloud Row Binlog compaction. - Preserve MOW historical-value behavior, partial update/upsert semantics, and delete-bitmap correctness. 3. Cloud write, publish, and metadata path - Add cloud Row Binlog companion tablet creation and metadata flow. - Add paired base-rowset + row-binlog-rowset commit through Meta Service. - Support Cloud publish conflict handling for partial update and upsert. - Make Cloud delete-bitmap cache, rowset builder/writer, tablet manager, and transaction paths aware of Row Binlog rowsets. - Persist Row Binlog tablet statistics and compaction metadata in Cloud Meta Service. 4. Cloud Row Binlog compaction - Add `CloudBinlogCumulativeCompactionPolicy`. - Keep binlog-specific candidate selection separate from normal compaction paths. - Add `filter_new_visible_rowsets` to avoid compacting newly visible rowsets too early unless backlog requires it. - Fix cloud candidate selection so cloud rowsets are not incorrectly filtered as non-local rowsets. 5. Query, schema change, restore, and observability - Update Row Binlog scan/read paths to use TSO/LSN correctly. - Support Row Binlog behavior across schema change, restore, tablet report, SHOW/proc paths, and compaction actions. - Expose Row Binlog tablet/index information where needed for debugging and operation. 6. Tests - Add/update BE unit tests for non-cloud and cloud Row Binlog write, rowset builder/writer, transaction, and compaction policy behavior. - Add/update regression tests for Row Binlog basic behavior, compaction, schema change, multi-segment reads, Cloud compaction, and Cloud publish conflict. - Update regression ordering to use `__DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__` where result order depends on commit/visible TSO semantics.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…sion (apache#66553) Issue Number: close apache#65951 Related PR: apache#65810 Problem Summary: Row Binlog makes normal non-key AFTER columns nullable even when their source columns are NOT NULL. `OlapBlockDataConvertor` derives the null map from the runtime source column, so a non-nullable complex source provides no top-level null map. Nullable ARRAY, MAP, and STRUCT writers can then leave their physical null child empty and abort during segment finalization. This PR keeps the fix local to Row Binlog: - Wrap normal non-key AFTER source columns as `ColumnNullable<original>` with an all-zero null map before conversion. - Keep key columns unchanged, preserve original source columns in `full_block`, and do not mutate the shared input Block. - For partial-update missing AFTER columns, wrap each column filled from the historical Rowset before conversion and append. - Restore the original generic `ColumnWriter::append()` behavior and remove the implicit null-map state, avoiding behavior changes in Scalar, VARIANT, and other common write paths. The test covers full updates, explicit partial updates, and partial updates whose NOT NULL MAP/STRUCT values are filled from a historical Rowset. It verifies logical readback and the physical top-level null-child row count. ### Release note Fix a BE crash when Row Binlog writes NOT NULL complex columns into nullable AFTER columns.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
…ache#65859) Issue Number: apache#65418 Related PR: apache#65810 Problem Summary: Table Stream already provides incremental consumption in storage-compute integrated deployments, where FE owns each partition offset. In Cloud mode, the target-table commit is finalized by MetaService, so keeping offsets in FE cannot atomically commit target data and advance Stream offsets. This PR adds the Cloud Table Stream control plane: - treats FE Catalog as the authority for Stream existence and binding, and stores only Latest and Versioned per-partition offsets in MetaService; - reuses the existing Index prepare, commit, Drop Index Log, Recycle Index, and InstanceRecycler lifecycle without creating Stream Mapping keys or physical tablets; - resolves one statement-level read snapshot, including clone-chain Offset inheritance; - performs expected-offset CAS and target-table commit in the same immediate FDB transaction, with no lazy-commit fallback; - keeps Create retries idempotent without overwriting offsets that consumption has already advanced; - carries affected Stream identities through DROP PARTITION so the partition recycler removes the corresponding Latest and Versioned offsets; - preserves the existing non-Cloud Table Stream path. This draft depends on apache#65810 for the Cloud Row Binlog data plane. Before Cloud E2E is enabled, the integration must confirm that every visible partition, including an empty partition and a newly added partition, has an authoritative `VersionPB.visible_tso` and empty-boundary TSO. The control plane intentionally fails when that metadata is missing instead of treating metadata loss as an empty partition. Rolling upgrade rule: upgrade all MetaService nodes before enabling the FE Cloud Table Stream path. Once enabled, MetaService rollback requires stopping Stream consumption first because an older MetaService does not understand Stream offset updates. Add the storage-compute separated Table Stream control plane. End-to-end availability depends on the Cloud Row Binlog data plane in apache#65810. - Test - [x] Regression test - `table_stream_p0` and `row_binlog_p0`: 15 suites passed on the branch baseline. - [x] Unit Test - Cloud MetaService Create, Query, Update, key, clone-chain, recycler, checker, and operation-log tests passed. - FE Cloud catalog and Table Stream manager tests: 8 tests passed after the Offset-only metadata change. - The earlier full FE Table Stream, transaction, drop, planner, and MV rewrite baseline passed. - [x] Manual test - `./build.sh --cloud` passed. - `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe` passed with Checkstyle. - Cloud C++ formatting passed, and changed-line clang-tidy passed for all modified production C++ files. The existing Cloud HTTP test target still cannot be analyzed because its compile command exposes pre-existing private-API diagnostics. - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Cloud Table Stream uses MetaService-managed offsets, immediate atomic consumption commits, and FORCE-only drop semantics. Non-Cloud behavior is unchanged. - Does this need documentation? - [ ] No. - [x] Yes. The implementation and test designs have been reviewed; public user documentation should be published when apache#65810 completes the end-to-end feature.
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Aug 18, 2026
…RowBinlogSegmentWriter Last PR of the block-transform-chain stack. RowBinlogDeriveStage rebuilds the load block into a full-width block over the binlog schema -- key + AFTER values, optional __BEFORE__* values, and the TSO/LSN/op columns -- so the ordinary segment writers write it like any other block. build_transform_chain picks PlainRowBinlogDerive (no history probe) or MowRowBinlogDerive (partial update and/or BEFORE image) per flush, and keeps an empty chain for every non-direct write type, whose rows are already binlog shaped. RowBinlogSegmentWriter is deleted, and with it the dead VerticalRowBinlogRowsetWriter template, the retriever's clear() bridge and SegmentWriter::build_key_index's protected visibility. SegmentFlusher no longer special-cases binlog when it picks a writer. Three later fixes to the deleted writer are carried into the stage, since deleting the file would drop them silently: - apache#66046: the op revision from the old delete sign runs between the BEFORE fill and the op column fill, the order the writer used. - apache#65810: the cloud-mode rejection is gone, and the historical lookup uses source.base_tablet. The writer fell back to its own tablet when that was unset; the stage requires it instead, because on a binlog write the writer context's tablet is the binlog tablet, which holds no source rows. Every site that fills source.tablet_schema fills source.base_tablet too. - apache#66553: AFTER value columns are wrapped nullable before they land in the binlog block, which declares them nullable. Behavior change to ack: binlog segments are written by the ordinary segment writer, which is the vertical one under the default enable_vertical_segment_writer=true; the deleted writer was always horizontal. The derived rows are identical -- the plain, MoW and BEFORE binlog goldens recorded through the vertical writer are byte-identical to the baselines recorded from the legacy writer -- but the writer type and the physical page layout are visible to operational tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 of 2 tasks
gavinchou
pushed a commit
that referenced
this pull request
Aug 24, 2026
…66743) Related PR: #65810, #65859 Problem Summary: Cloud Row Binlog and Cloud Table Stream now have an integrated data and control plane, but the Cloud SQL path and several cross-module correctness boundaries did not have complete automated coverage. This PR adds layered coverage for the four Table Stream flows: - Create: batched Offset initialization, retries, partial-state cleanup, empty and newly added partitions. - Query: statement-level snapshots, initial/snapshot/reset reads, clone-chain inheritance, partial responses, and read-only behavior. - Update: Offset CAS, target-table atomicity, idempotent retry, multiple Streams, immediate commit, size limits, and source publish concurrency. - Delete: typed drop requests, partition identity propagation, operation-log recovery, clone-safe GC, paginated cleanup, and instance deletion. It also adds Cloud SQL suites for initial rows, repeated consumption, explicit partitions, multiple references, partition lifecycle, schema changes, metadata, and object lifecycle. A Docker specialty test verifies real FE Catalog and MetaChecker orphan-Offset detection. The expanded tests found and fix two operational validation issues: malformed versioned set_value input now returns INVALID_ARGUMENT instead of dereferencing a null protobuf, and check_instance now validates the typed RecycleIndex binding/state before suppressing a transient projection gap.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Aug 25, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Cloud Row Binlog currently does not refresh the implicit end TSO for direct table@incr scans. Keep the flexible partial-update Cloud coverage on base-table and raw Row Binlog results while skipping only the three direct @incr assertions until that generic FE issue is fixed. ### Release note None ### Check List (For Author) - Test: No local test run; this is a Cloud regression routing adjustment and buildall will validate it. - Behavior changed: No. Production code is unchanged. - Does this need documentation: No
hutiefang76
added a commit
to hutiefang76/doris
that referenced
this pull request
Aug 30, 2026
Issue Number: close apache#65383 Problem Summary: ALTER TABLE changes to binlog.format and binlog.need_historical_value must be routed through the binlog configuration path so immutable-setting validation reports the specific error. The production routing is now present on master via apache#65810, but that broader change did not add focused regression coverage for these properties. Add unit coverage for property classification and end-to-end ALTER TABLE error reporting for both immutable binlog settings. Test: JAVA_HOME=/Users/hutiefang/Library/Java/JavaVirtualMachines/ms-17.0.18/Contents/Home DORIS_THIRDPARTY=/Users/hutiefang/project/oss/doris/thirdparty bash run-fe-ut.sh --run org.apache.doris.alter.AlterOperationsTest,org.apache.doris.alter.SchemaChangeHandlerTest#testChangeBinlogFormatReportsBinlogError+testChangeBinlogNeedHistoricalValueReportsBinlogError Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
4 of 8 tasks
35 of 69 tasks
Yukang-Lian
added a commit
that referenced
this pull request
Sep 1, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: #65810 Problem Summary: Row Binlog supports fixed-column partial updates, but did not support flexible partial updates (`UPDATE_FLEXIBLE_COLUMNS`). Flexible updates carry a per-row skip bitmap and may merge rows before writing, so Row Binlog must reconstruct complete rows while keeping BEFORE/AFTER values, operations, row counts, and LSNs aligned. This PR: - Routes flexible partial updates through the existing `MowRowBinlogDeriveStage` transform chain. - Reuses the existing flexible-update aggregation, a read-only MOW key probe, and historical-row read plans to materialize complete AFTER rows and optional complete BEFORE rows. - Keeps per-row LSN and same-batch DELETE-to-INSERT state aligned through aggregation. For DELETE-to-INSERT, omitted AFTER cells use their default or null value, while a live historical row remains the UPDATE BEFORE image. - Preserves Base Rowset and Row Binlog `(segment_id, row_id)` alignment for sequence losers; the synchronized delete bitmap makes those rows logically invisible. - Uses a writer-private copy-on-write Block, so mutated columns detach without changing the Base Rowset writer's input. - Keeps the existing Row Binlog on-disk format, query protocol, and APPEND / UPDATE / DELETE semantics. The implementation is shared by local and Cloud storage. It does not change FE syntax, Table Stream metadata, Stream Offset handling, or protobuf definitions. Scope note: this PR does not lift the existing flexible partial-update multi-segment load restriction and does not add a zero-row Segment format.
This was referenced Sep 3, 2026
morningman
added a commit
to morningman/doris
that referenced
this pull request
Sep 3, 2026
### What problem does this PR solve? Issue Number: close apache#65388, related apache#65418 Related PR: apache#65810 Problem Summary: Review follow-up for this PR. Two problems were found in the first revision. 1. `visitAlterStreamComment` decoded the comment with `stripQuotes`, which only removes the outer delimiters. `ALTER STREAM s SET COMMENT 'a''b'` therefore stored `a''b` instead of `a'b`, and with the default sql mode `COMMENT '\n'` stored a backslash plus `n` instead of a newline. The value is exposed by `information_schema.table_streams.STREAM_COMMENT` and printed by `SHOW CREATE STREAM`, so the stored metadata was simply wrong. `visitCreateStream` had the other half of the problem: it decoded backslash escapes (`LogicalPlanBuilderAssistant.escapeBackSlash`) but did not collapse doubled quotes and ignored `NO_BACKSLASH_ESCAPES`, so the same literal produced different comments depending on whether it went through CREATE or ALTER. Both now use `SqlLiteralUtils.parseStringLiteral`, which collapses doubled quotes and honours the sql mode exactly like the lexer does (`NereidsParser` feeds `SqlModeHelper.hasNoBackSlashEscapes()` to the lexer). Not addressed here: `Env.addTableComment` wraps the comment in single quotes but only escapes double quotes (`SqlUtils.escapeQuota`), so a comment holding a single quote makes `SHOW CREATE` emit non-parsable DDL. That is pre-existing, shared by all 19 call sites of every table type, and is not stream specific. 2. The two new regression suites did not follow the repository test standards: names were indirected through `def`, deterministic results were asserted by hand instead of a generated `.out`, and both dropped their fixtures at the end. Fixed all three. While converting, the skip guard of `test_binlog_property_alter_exception` turned out to be dead: it compared the config name against `enable_feature_binlog`, but `SHOW FRONTEND CONFIG` reports the EXPERIMENTAL config as `experimental_enable_feature_binlog`, so the suite skipped itself unconditionally, in CI as well. It now uses the framework helper `getSyncer().checkEnableFeatureBinlog()`, and the suite actually runs. ### Release note None (the behaviour change of this PR is described in the first commit). ### Check List (For Author) - Test: - Unit Test: `AlterTableStreamCommentTest#testAlterStreamCommentStringLiteral` covers doubled single quotes, doubled double quotes, `\n`/`\t` and `NO_BACKSLASH_ESCAPES`. Both tests of the class pass locally (`Tests run: 2, Failures: 0, Errors: 0`). - Regression test: `test_table_stream_alter_comment` (the `.out` was generated with `-genOut` against a local cluster, then verified by a second run without it) and `test_binlog_property_alter_exception`, both pass locally and the six expected error messages were observed in the run log. - Behavior changed: Yes. A stream comment containing a doubled quote or a backslash escape is now stored decoded, and the sql mode is honoured. - Does this need documentation: No (the statement itself is documented by the first commit's doc PR). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xx7TgjXJCiChnzLYa6hgtL
morningman
added a commit
that referenced
this pull request
Sep 4, 2026
### What problem does this PR solve? Issue Number: close #65388, related #65418 Related PR: #65810 Problem Summary: **1. `ALTER STREAM ... SET COMMENT` was not supported (#65388)** A table stream can be created with a comment and the comment is fully wired up everywhere except for changing it: | step | before this PR | | --- | --- | | `CREATE STREAM s ON TABLE t COMMENT 'x'` | supported (`InternalCatalog#createTableStream`) | | persisted in the image | supported (`Table#comment`) | | `SHOW CREATE STREAM s` | prints the comment | | `information_schema.table_streams.STREAM_COMMENT` | exposes the comment | | changing the comment | **not possible** | There was no `ALTER STREAM` rule in `DorisParser.g4` at all — `STREAM` only appeared in `CREATE STREAM`, `DROP STREAM`, `SHOW STREAMS` and `SHOW CREATE STREAM` — so the statement failed at parser stage: ``` errCode = 2, detailMessage = no viable alternative at input 'ALTER STREAM'(line 1, pos 6) ``` `ALTER TABLE` is not an alternative either: `Alter#processAlterTable` rejects the `STREAM` table type with `Do not support alter STREAM table[...]`. This PR adds: ```sql ALTER STREAM <name> SET COMMENT 'new comment'; ALTER STREAM <name> MODIFY COMMENT 'new comment'; -- same thing ``` `MODIFY` is accepted alongside `SET` so the syntax stays consistent with `ALTER TABLE ... MODIFY COMMENT`, which is the existing Doris spelling for the same operation on a table. Implementation notes: - The comment of a stream lives in the `Table` metadata only, so `Alter#processAlterStreamComment` reuses `ModifyCommentOperationLog.forTable(...)` and the existing replay path `Alter#replayModifyComment`, which already resolves a generic `Table`. **No new edit log operation and no meta version bump.** - Cloud Meta Service only stores stream offsets and ids (`CloudInternalCatalog#beforeCreateTableStream` / `#afterCreateTableStream`), so no extra RPC is needed and the behaviour is the same in cloud mode. - `AlterStreamCommand` extends `AlterCommand`, which already provides `ForwardWithSync` and `StmtType.ALTER`. It carries an `AlterType` enum so that other `ALTER STREAM` clauses can be added later without reshaping the command. - Privilege required is `ALTER` on the stream, matching `ALTER TABLE`. Altering a non-stream table through `ALTER STREAM` reports `ERR_WRONG_OBJECT`, the same way `SHOW CREATE STREAM` does. - `Config.enable_table_stream` gates the operation, consistent with `CREATE STREAM` and `DROP STREAM`. - The comment literal is decoded with `SqlLiteralUtils.parseStringLiteral`, so a doubled quote collapses to one quote and backslash escapes follow the session sql mode, matching the lexer (`NereidsParser` drives the lexer with `SqlModeHelper.hasNoBackSlashEscapes()`). `CREATE STREAM ... COMMENT` was decoding the same literal differently -- it unescaped backslashes but never collapsed doubled quotes and ignored `NO_BACKSLASH_ESCAPES` -- so it was moved onto the same decoder, otherwise the comment stored by CREATE and by ALTER would differ for the same text. Not fixed here: `Env#addTableComment` quotes the value with single quotes while escaping only double quotes, so a comment holding a `'` makes `SHOW CREATE` emit non-parsable DDL. That is pre-existing, shared by all 19 call sites of every table type, and will be filed separately. **2. Regression coverage for immutable binlog properties (#65383)** `ALTER TABLE ... SET ("binlog.format" = ...)` on a ROW binlog table used to fail with a misleading light-schema-change error, because `AlterOperations#checkBinlogConfigChange` did not list `binlog.format` / `binlog.need_historical_value` and the statement was dispatched to the generic schema change path. That was fixed as a side effect of #65810 (`f745ddf9e22`), but no test locked the behaviour in. This PR adds `test_binlog_property_alter_exception.groovy` covering: | statement (on a `binlog.format = ROW` MOW table) | expected | | --- | --- | | `SET ("binlog.format" = "STATEMENT_AND_SNAPSHOT")` | `not support change binlog format from ROW to STATEMENT_AND_SNAPSHOT` | | `SET ("binlog.need_historical_value" = "false")` | `not support change binlog.need_historical_value from true to false` | | `SET ("binlog.enable" = "false")` | `can't disable binlog when format is [Row]` | | `SET ("binlog.format" = "ROW")` (same value) | accepted, no-op | | `SET ("binlog.ttl_seconds" = "7200")` | accepted | | `SET ("binlog.format" = "ROW")` on a table without binlog | `not support change binlog format from STATEMENT_AND_SNAPSHOT to ROW` |
morningman
added a commit
that referenced
this pull request
Oct 19, 2026
…licy (#69240) ### What problem does this PR solve? Issue Number: None Related PR: #68033, #65810 Problem Summary: **In short.** In local (non-cloud) mode, altering a compaction setting of a table with `binlog.format = 'ROW'` also reaches its hidden row binlog tablets. A new `compaction_policy` moves them off the `binlog` policy they must use. A `time_series_compaction_*` change on a `time_series` table fails halfway. #68033 fixed the same thing for cloud. This PR applies the same fix to the local path: the FE no longer sends compaction settings to row binlog tablets. As a second guard, the BE now refuses to change a row binlog tablet's policy. **Background.** - A table with `binlog.format = 'ROW'` has a hidden row binlog index. Each data tablet has a row binlog tablet next to it, which stores the change events. - Row binlog tablets are created with the `binlog` compaction policy (`BinlogCumulativeCompactionPolicy`). It is a tiered policy written for change events; for example, its quick merge links the segments of several rowsets into one `OVERLAPPING` rowset instead of rewriting them. `CreateReplicaTask` sets it when it creates a row binlog tablet. - `ALTER TABLE ... SET (<compaction property>)` sends the new value to the tablets through `UpdateTabletMetaInfoTask`. #68033 made the cloud handler (`CloudSchemaChangeHandler`) leave out row binlog tablets for `compaction_policy`. The local handler (`SchemaChangeHandler.updatePartitionProperties`) still uses `getMaterializedIndices(VISIBLE, true)`, so the row binlog index is included. **The problem, and what it cost.** Both cases are reproduced on master in local mode: | Statement on a table with `binlog.format = 'ROW'` | What happens on master | |---|---| | `ALTER TABLE t SET ("compaction_policy" = "time_series")` | Succeeds, and every row binlog tablet meta now says `time_series`. From the next compaction on, those tablets are compacted by the time series policy instead of the binlog policy. | | `ALTER TABLE t SET ("time_series_compaction_goal_size_mbytes" = "2048")` on a `time_series` table | Fails: `Failed to update partition[t]. tablet meta. Error: ...[INVALID_ARGUMENT]only time series compaction policy support time series config]. This operation maybe partial successfully, You should retry until success.` The BE rejects time series settings for tablets that are not on the time series policy, and the row binlog tablets are on `binlog`. The data tablets are already updated by then. Retrying fails the same way. | **How this PR fixes it.** 1. FE, `SchemaChangeHandler.updatePartitionProperties`: when the update carries a compaction policy or a time series setting, the row binlog index is left out. This is the same rule #68033 uses for cloud. - `ALTER TABLE ... SET` changes one table property at a time (`ModifyTablePropertiesOp.validate`), so such an update has nothing else for the row binlog tablets. - Every other tablet meta update (in memory, storage policy, binlog config, `disable_auto_compaction`, ...) still reaches them, as before. 2. BE, `update_tablet_meta_callback`: - It refuses to change the compaction policy of a row binlog tablet (`can not change compaction policy of row binlog tablet <id>`). - It no longer accepts `binlog` as a new policy for any tablet. #65810 added it to the accepted values, but no FE path sends it: the policy is only set when a row binlog tablet is created. - With these checks, an older FE, or a future caller, gets an error instead of silently moving a row binlog tablet off its policy. **Results.** | Statement (local mode, table with `binlog.format = 'ROW'`) | Before | After | |---|---|---| | `SET ("compaction_policy" = "time_series")` | Succeeds; row binlog tablets switch to `time_series` | Succeeds; data tablets switch, row binlog tablets stay on `binlog` | | `SET ("time_series_compaction_goal_size_mbytes" = "2048")` on a `time_series` table | Fails after updating the data tablets | Succeeds; data tablets get 2048, row binlog tablets are untouched | | The first statement from an FE without this fix, against a BE with it | Succeeds; row binlog tablets switch to `time_series` | Fails with `can not change compaction policy of row binlog tablet <id>`; row binlog tablets stay on `binlog` |
yiguolei
pushed a commit
that referenced
this pull request
Oct 19, 2026
…69238) ### What problem does this PR solve? Issue Number: None Related PR: #62058, #63643, #63850, #64133, #65810 Problem Summary: **In short.** Row binlog (`binlog.format = 'ROW'`) was reworked several times on master. The reworks left dead code behind, and they deleted thrift / protobuf fields without retiring their ids. This PR deletes the dead code and retires the ids: `reserved` in protobuf, a `// N: deprecated <name>` comment in thrift. After this, no later change can reuse an id that an older master build may have persisted or sent. Nothing that runs today changes. This is also groundwork for renaming row binlog to changelog: code deleted here does not have to be renamed there. **Background.** - #62058 and #63643 first kept row binlog inside the base tablet. The tablet meta carried its own binlog schema and rowset metas (`row_binlog_schema`, `row_binlog_rs_metas`, `row_binlog_schema_hash`). Delete bitmaps had an `is_binlog_delvec` flag. Clone and snapshot requests had a `copy_type`. Tablet reports carried `binlog_size` / `binlog_file_num`, which the FE kept on `Replica`. - #63850 added `binlog_timestamp_col_idx` / `binlog_lsn_col_idx` to the tablet schema. #64133 replaced them with `binlog_{tso,lsn,op}_col_idx` under new ids (41-43, and 42-44 in the cloud message). So the name `binlog_lsn_col_idx` is still in use, but under a new id. - #65810 moved row binlog into an independent companion tablet. Most of the per-tablet fields above were deleted then. #65810 also added a thrift `TTabletWithPartition.binlog_tablet_id`, which nothing uses. **The problem.** - Ten protobuf ids and six thrift ids were deleted without being retired. `gensrc/AGENTS.md` requires a deleted id to be retired so it is never reused. Protobuf is the risky case: these messages are persisted in tablet meta (local RocksDB, and the cloud meta-service). Suppose a later field took one of these ids with another type or meaning. A tablet meta written by an older master build could then be decoded into the new field. - Some code that nothing reaches is still there. FE `Replica` still carries `binlogSize` / `binlogFileNum` and writes them into the image, though nothing has set them since #65810. The thrift `TTabletWithPartition.binlog_tablet_id` was never used. The BE has unused constants, a method and two error codes. Without this PR, the changelog rename would have to rename all of it. **How this PR fixes it.** Dead code removed: | Item | Why it is dead | |---|---| | thrift `TTabletWithPartition.binlog_tablet_id` (3) | `TTabletWithPartition` only lives inside the BE (`VTabletWriter`); field 3 is never set or read. The load path carries the companion tablet id in the protobuf `PTabletWithPartition.binlog_tablet_id`, which stays. | | FE `Replica.binlogSize` / `binlogFileNum` (Gson `bs` / `bfn`) and their getters / setters | Their only writers, the tablet report fields, were removed in #65810; nothing reads them. Gson skips the two keys when it loads an image written by an older master. | | BE `kRowBinlogPrefix`, `FDRowBinlogSuffix`, `Tablet::enable_row_binlog()` | No references. | | BE error codes `BINLOG_COMPACTION_INVALID_PARAMETERS` (-2011), `BINLOG_COMPACTION_NO_SUITABLE_VERSION` (-2012) | No references. Removed the same way -2007 was, without a placeholder. | Retired ids: | File | Message | Ids | Old fields | |---|---|---|---| | olap_file.proto | `TabletMetaPB` | 42, 43, 44 | `row_binlog_schema`, `row_binlog_rs_metas`, `row_binlog_schema_hash` | | olap_file.proto | `TabletMetaCloudPB` | 43, 44 | `row_binlog_rs_metas`, `row_binlog_schema_hash` | | olap_file.proto | `TabletSchemaPB` / `TabletSchemaCloudPB` | 37, 38 / 38, 39 | `binlog_timestamp_col_idx`, `binlog_lsn_col_idx` (the old ids only; see Background) | | olap_file.proto | `DeleteBitmapPB` | 5 | `is_binlog_delvec` | | AgentService.thrift | `TCloneReq` / `TSnapshotRequest` | 14 / 15 | `copy_type` | | BackendService.thrift | `TTabletStat` | 12, 13 | `binlog_size`, `binlog_file_num` | | MasterService.thrift | `TTabletInfo` | 26, 27 | `binlog_size`, `binlog_file_num` | | PaloInternalService.thrift | `TTabletWithPartition` | 3 | `binlog_tablet_id` | None of these ids is used on branch-4.2, branch-4.1, branch-4.0 or branch-3.1 (checked with `git show` on each branch), so retiring them agrees with every release. `TabletMetaCloudPB` 42 is not in the table: it already means something else on master (it was `row_binlog_schema`, it is now `binlog_tablet_id`). This PR cannot undo that. Not included: `TTabletSchema.binlog_{tso,lsn,op}_idx` (28-30). The FE stopped sending them in #65810, but BE unit tests still set them when they create a companion tablet. Whether production needs them is a separate question, left to a follow-up. **Results.** - No behavior change. The one thrift field deleted here was never set, the other thrift changes are comments, and `reserved` only makes protoc reject a future reuse of the id. - A peer built from an older master that still sends one of the deleted thrift fields has it skipped as an unknown field, as it already is today. An FE image that still has `bs` / `bfn` loads as before. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Build: `build.sh --fe --be` (BE Release; FE with checkstyle). - FE unit tests (`run-fe-ut.sh`): `ReplicaTest` (5), `TabletTest` (11), `OlapTableTest` (18), all pass. - No BE unit test: nothing under `be/test` references the removed symbols (`git grep`). - Cloud was not built locally. It only sees the new `reserved` lines, which do not change the generated code. - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 19, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: A row binlog tablet has three hidden columns, `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__` and `__DORIS_BINLOG_OP__`. `TabletSchema` keeps their positions in `_binlog_{tso,lsn,op}_col_idx`. The positions came from thrift `TTabletSchema` 28-30 when a tablet was created, and were persisted in `TabletSchemaPB` 41-43 (`TabletSchemaCloudPB` 42-44). Since apache#65810 the FE no longer sets `TTabletSchema` 28-30. Every row binlog tablet created since then has -1 in its tablet meta schema (and in the meta service schema entry), and so do the schemas loaded from it: rowsets synced from the meta service, compaction outputs. Nothing reads these -1 today. Every reader gets a schema whose indexes are derived from the column names: `build_current_tablet_schema` for loads, `append_column` for scans and the publish-phase writer. But: 1. Any new reader of the tablet schema would get -1. 2. If a load ever came with an index schema without column unique ids, `RowsetBuilder` would fall back to copying the tablet schema, and row binlog derivation would hit `CHECK(idx >= 0)`. 3. BE unit tests set 28-30 and the PB fields by hand, so they exercise a path that production never takes. This PR makes `TabletSchema::init_from_pb` derive the three indexes from the column names, as `append_column` and `build_current_tablet_schema` already do. The stored values are then not needed: - thrift `TTabletSchema` 28-30 are retired (commented out); - proto `TabletSchemaPB` 41-43 and `TabletSchemaCloudPB` 42-44 are reserved; `TabletSchema::to_schema_pb`, `TabletMeta` and the cloud `pb_convert` stop writing and copying them; - BE unit tests stop setting them. A new `TabletSchemaTest` case checks the indexes after `init_from_pb` and after a round trip through the PB. Compatibility: none of these ids exists on branch-4.2, branch-4.1, branch-4.0 or branch-3.1. Meta written by an older master still loads: protobuf keeps the reserved ids as unknown fields, and the indexes are derived anyway. An older master BE that reads meta written by this one gets -1, which is what it already has for every row binlog tablet created since apache#65810. ### Release note None ### Check List (For Author) - Test: - Build: `build.sh --fe --be` (BE Release; FE with checkstyle). - Regression test: `row_binlog_p0` (23 suites) and `table_stream_p0` (16 suites) pass. - Manual test: create a row binlog table and load it; the row binlog tablet meta no longer carries the three fields. Restart the cluster, load again, run a binlog compaction ([0-1]...[5-5] into [0-5]) and read through `binlog()`: every event comes back. - Unit Test: BE, 191 tests in 19 suites, all pass: the new `TabletSchemaTest.test_binlog_col_idx_from_pb_column_names`, and every suite that builds a row binlog schema or touches the changed code (`TabletSchemaTest`, `TabletMetaTest`, `PbConvert`, `RowBinlogDeriveTest`, `RowBinlogVmergeCompactionTest`, `SegmentFlusher(Transform)FormatTest`, `(Cloud)GroupRowset*Test`, `TestEngineStorageMigrationTask`, the merge-on-write transform tests, ...). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman
added a commit
that referenced
this pull request
Oct 19, 2026
…#69242) ### What problem does this PR solve? Issue Number: None Related PR: #65810, #69238 Problem Summary: **In short.** Row binlog tablets keep the positions of their three hidden columns (`__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`) in `TabletSchema`. Since #65810 the FE no longer sends them, so every row binlog tablet created since then has -1 in its tablet meta. Nothing reads that -1 today, but any new reader would. Only the BE unit tests still fill these fields, and that takes them down a path production never runs. This PR makes the BE derive the three positions from the column names whenever a schema is loaded, and deletes the thrift and protobuf fields that used to carry them. It also removes the `reserved` lines and `// N: deprecated` comments that #69238 added for other deleted row binlog fields: none of these ids was ever in a release, and only ids that shipped in a release need to stay put. Nothing that runs today changes. **Background.** - A row binlog tablet stores change events. Besides the user columns, its schema has three hidden columns, `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__` and `__DORIS_BINLOG_OP__`. `TabletSchema` keeps their positions in `_binlog_{tso,lsn,op}_col_idx`. - The positions came from thrift `TTabletSchema` 28-30 when a tablet was created, and were persisted in `TabletSchemaPB` 41-43 (`TabletSchemaCloudPB` 42-44). - Schemas built column by column already derive them from the names: `TabletSchema::append_column` and `build_current_tablet_schema`. Only `TabletSchema::init_from_pb`, which loads a schema from its PB, took the stored values. **The problem.** - #65810 stopped the FE from setting `TTabletSchema` 28-30 (`CreateReplicaTask`, and `CloudInternalCatalog` for cloud). Every row binlog tablet created since then has -1 in: - its tablet meta schema; - the meta service schema entry, in cloud mode; - every schema loaded from those, such as cloud rowsets synced from the meta service and compaction outputs. - Nothing reads these -1 today. Every reader gets a schema whose positions are derived from the names: loads (`build_current_tablet_schema`), scans and the publish-phase writer (`append_column`), and the rowset written by the same load. Still: 1. Any new code that reads the positions from the tablet schema gets -1. 2. If a load ever came with an index schema without column unique ids, `RowsetBuilder` would fall back to copying the tablet schema, and row binlog derivation would hit `CHECK(idx >= 0)`. 3. The BE unit tests (`create_row_binlog_tablet_schema` and others) still set 28-30 or the PB fields by hand, so they exercise a path production never takes. **How this PR fixes it.** 1. `TabletSchema::init_from_pb` derives the three positions with `field_index(<column name>)`, as `append_column` does. A schema now has the same positions whether it was built column by column or loaded from a PB. 2. The stored values are no longer needed, so the fields are deleted. They never shipped in a release, so their ids are not reserved: | File | Message | Ids | Deleted fields | |---|---|---|---| | AgentService.thrift | `TTabletSchema` | 28, 29, 30 | `binlog_tso_idx`, `binlog_lsn_idx`, `binlog_op_idx` | | olap_file.proto | `TabletSchemaPB` | 41, 42, 43 | `binlog_tso_col_idx`, `binlog_lsn_col_idx`, `binlog_op_col_idx` | | olap_file.proto | `TabletSchemaCloudPB` | 42, 43, 44 | same | `TabletSchema::to_schema_pb`, `TabletMeta` and the cloud `pb_convert` stop writing and copying them. 3. #69238 retired the row binlog fields that earlier reworks had deleted, with `reserved` in protobuf and a `// N: deprecated <name>` comment in thrift. For the same reason, a separate commit removes those markers. A `reserved` line or a comment generates no code, so nothing else changes: | File | Message | Ids | Old fields | |---|---|---|---| | olap_file.proto | `TabletMetaPB` | 42, 43, 44 | `row_binlog_schema`, `row_binlog_rs_metas`, `row_binlog_schema_hash` | | olap_file.proto | `TabletMetaCloudPB` | 43, 44 | `row_binlog_rs_metas`, `row_binlog_schema_hash` | | olap_file.proto | `TabletSchemaPB` / `TabletSchemaCloudPB` | 37, 38 / 38, 39 | `binlog_timestamp_col_idx`, `binlog_lsn_col_idx` | | olap_file.proto | `DeleteBitmapPB` | 5 | `is_binlog_delvec` | | AgentService.thrift | `TCloneReq` / `TSnapshotRequest` | 14 / 15 | `copy_type` | | BackendService.thrift | `TTabletStat` | 12, 13 | `binlog_size`, `binlog_file_num` | | MasterService.thrift | `TTabletInfo` | 26, 27 | `binlog_size`, `binlog_file_num` | | PaloInternalService.thrift | `TTabletWithPartition` | 3 | `binlog_tablet_id` | 4. The BE unit tests stop setting them. A new `TabletSchemaTest.test_binlog_col_idx_from_pb_column_names` checks the positions: - right after `init_from_pb`; - after a round trip through the PB; - on a schema without these columns, where they stay -1. **Compatibility.** - **Releases.** None of the ids deleted or un-reserved here exists on branch-3.1, branch-4.0, branch-4.1 or branch-4.2 (checked with `git show` on each branch). No release writes or sends them, so an upgrade from a release never meets them. This PR reuses no id. - **Meta written by an older master build** still loads. Protobuf keeps these ids as unknown fields, thrift skips them, and the positions are derived from the column names anyway. - **An older master BE that reads meta written by this one** gets -1 for the positions. In tablet meta schemas that is what it already reads for every row binlog tablet created since #65810. The rowsets written here are the difference: their schemas used to carry the positions. Of the older code that reads these positions, only one place can see a rowset loaded back from its PB: the publish-phase conflict rewrite of a merge-on-write load with row binlog, when a local-mode BE restarts with that load committed but not yet published (`BaseTablet` reads `binlog_lsn_col_idx()` of the attached row binlog rowset). Before rolling a local-mode cluster back to an older master build, let pending loads publish. - **The ids are no longer reserved.** A later change may put a new field on one of them. A cluster that ran a master build keeps them in its persisted meta, for example `TabletSchemaPB` 41-43 (`TabletSchemaCloudPB` 42-44) in every tablet and rowset schema it wrote, -1 for ordinary tables. Such a cluster has to rebuild its tables before it upgrades to a build that reuses one of these ids with another meaning. Releases are not affected. - With the positions derived, `is_tso_enabled()` is now also true on a row binlog tablet's own schema. Two callers read it from the tablet schema, and neither changes behavior: - `CloudTablet::try_make_committed_rs_visible` returned without doing anything for a row binlog tablet before, and it still does. Row binlog tablets are not merge-on-write and never put rowsets into the committed rowset manager. - The cloud delete task never reaches a row binlog tablet, because predicate DELETE is rejected on these tables.
5 of 16 tasks
morningman
added a commit
that referenced
this pull request
Oct 21, 2026
…m CCR binlog (#69311) ### What problem does this PR solve? Issue Number: None Related PR: #69238, #69240, #69242 Problem Summary: **In short.** Row binlog (`binlog.format = 'ROW'`), the change stream that Table Stream and incremental materialized views (IVM) read, is renamed to **changelog**, and it gets its own configuration instead of riding on CCR's `BinlogConfig`. "Binlog" names a change stream in one fixed format; changelog is the general notion, and a binlog can later be built on top of it. The feature exists only on master and is documented as experimental, so this is a hard cut: there is no code that reads data written under the old names. Instead, guardrails make leftovers from an earlier master build fail with a clear message instead of crashing a process or being silently misread. CCR binlog and MySQL binlog CDC do not change. **Background.** - A row binlog table writes every change of a base tablet into a companion tablet. The companion schema carries the key, the after values, three hidden columns `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__`, `__DORIS_BINLOG_OP__`, and, with `binlog.need_historical_value`, a before-image column `__BEFORE__<col>__` per value column. The changes are read with the `binlog(...)` table function, `@incr`, Table Stream and IVM refresh. - It was switched on with CCR's properties: `binlog.enable = true` plus `binlog.format = ROW`. It shared `binlog.enable` and `binlog.ttl_seconds` with CCR, and `enable_feature_binlog` gated CCR, row binlog table creation, TSO allocation, Table Stream INSERT and TTL cleanup alike. - None of this is on a release branch (3.1 / 4.0 / 4.1 / 4.2). The documentation marks it experimental, and `mtmv/ivm/AGENTS.md` allows incompatible changes before 2027-01. Master already made incompatible changes to it in #64133, #65810 and #66889. **The problem, and what it cost.** 1. **One word for two features.** CCR binlog (cross-cluster replication) and row binlog share property names, a switch, a configuration object and most code names, so every reader has to work out which one a `binlog` means. 2. **Row binlog leaks into every table.** Every OLAP table created on master persists `binlog.format` and `binlog.need_historical_value` and prints them in SHOW CREATE TABLE, CCR tables included; a 4.x CCR downstream rejects that DDL as unknown properties. 3. **The shared keys interact badly.** - `ALTER DATABASE ... SET ("binlog.enable" = "true")` on a database whose default is ROW validates the tables with the CCR rules, but what takes effect is the ROW default. - A table with `binlog.format = ROW` and `binlog.enable = false` can never be enabled: the error says "can't disable binlog when format is [Row]". 4. **Before-image columns clash with user columns** (reproduced on master before this change). `__BEFORE__<col>__` is not under the reserved `__DORIS_` prefix, and schema change matched it with `contains("__BEFORE__")`: | User column | What happened | |---|---| | `__BEFORE__v1__` next to `v1` | the companion index had two columns named `__BEFORE__v1__`; `@incr` DETAIL returned after values in the UPDATE_BEFORE rows; the column could not be named in `binlog()` ("ambiguous"); as a key column every `@incr` read failed; after `DROP COLUMN v1` every load failed | | `x__BEFORE__y` as the last value column | after `ADD COLUMN v2` the companion index put `v2` before it, so the two columns swapped values in every row written afterwards, without any error; with different types every later load failed with a bad cast | 5. FE and BE had two families of change codes with the same names and different values (`DELETE` is 1 in FE and 2 in BE). **How this PR fixes it.** 1. **New names.** User-visible ones: | Old | New | |---|---| | `"binlog.enable" = "true"` + `"binlog.format" = "ROW"` | `"changelog.enable" = "true"` | | `binlog.ttl_seconds` (on a ROW table or database), `binlog.need_historical_value` | `changelog.ttl_seconds`, `changelog.need_historical_value` | | internal `__binlog_config_version` | `__changelog_config_version` (no longer shown in `information_schema.table_properties`) | | `binlog(...)` table function | `changelog(...)` | | `__DORIS_BINLOG_{TSO,LSN,OP}__` | `__DORIS_CHANGELOG_{TSO,LSN,OP}__` | | `__BEFORE__<col>__` | `__DORIS_CHANGELOG_BEFORE_<col>__` | | companion index `binlog<Row>(t)` | `changelog(t)` | | SHOW DATA `BinlogSize` / `LocalBinlogSize`, SHOW PARTITIONS `BinlogSize`, SHOW TABLET(S) / PROC `IsRowBinlog`, `RowBinlog{Mismatch,Redundant}{Num,Tablets}`, `RowBinlogRepairReason` | `ChangelogSize`, `LocalChangelogSize`, `IsChangelog`, `Changelog...` | | IVM fallback reasons `BINLOG_BROKEN`, `BINLOG_NOT_ENABLED` | `CHANGELOG_BROKEN`, `CHANGELOG_NOT_ENABLED` | | FE `tablet_binlog_missing_timeout_second`, `tablet_binlog_missing_max_times` | `tablet_changelog_missing_*` | | BE `binlog_compaction_*` (6), `max_binlog_compaction_threads`, cloud `binlog_compaction_thread_num_factor` | `changelog_compaction_*`, `max_changelog_compaction_threads`, `changelog_compaction_thread_num_factor` | | metrics `compaction_task_state_total{type="binlog"}`, `tablet_binlog_max_compaction_score`; bvars `row_binlog_ttl_*`; thread pools | `type="changelog"`, `tablet_changelog_max_compaction_score`, `changelog_ttl_*`, `Changelog...` pools | | HTTP `compact_type=row_binlog_ttl`; compaction policy `"binlog"`; load profile `RowsetBuilder {id} binlog<row>` | `changelog_ttl`; `"changelog"`; `RowsetBuilder {id} changelog` | CCR keeps every name: `binlog.enable / ttl_seconds / max_bytes / max_history_nums`, `SHOW PROC '/binlog'`, getBinlog / ingest_binlog, `BinlogManager`. 2. **Separate configuration.** - New `ChangelogConfig` (`enable`, `ttlSeconds`, `needHistoricalValue`, `configVersion`), stored as `changelog.*` keys in the table / database property map. `BinlogConfig` is back to the four CCR fields, so plain and CCR tables persist and print only the four `binlog.*` keys, as on 4.x. - The wire format does not change. `TabletBinlogConfig.of` is now the only place that builds the `TBinlogConfig` / `BinlogConfigPB` sent to BE and the meta service: a changelog table still sends `enable = true`, `binlog_format = 1` (now named `CHANGELOG`), its TTL, `need_historical_value` and `config_version`; plain and CCR tables send exactly what they sent before. The enum mapping is explicit instead of `valueOf(name())`. - The TTL generation protocol (short TTL first, published after every tablet accepted it) moves unchanged to the changelog configuration. - `changelog.enable` and `changelog.need_historical_value` can only be set at CREATE; `changelog.ttl_seconds` can be altered. `binlog.format` and `binlog.need_historical_value` are rejected in every DDL, naming their replacement. - In a database whose default is `"changelog.enable" = "true"`, SHOW CREATE TABLE of a table without changelog prints `"changelog.enable" = "false"` (output only), so CREATE TABLE LIKE and re-running the DDL keep it out of the default. Temporary tables never have changelog: an explicit `"changelog.enable" = "true"` fails, a database default is ignored. - CCR and changelog stay mutually exclusive, now by explicit checks in CREATE / ALTER DATABASE, CREATE / ALTER TABLE and RECOVER TABLE: the CCR full sync relies on backup / restore, which changelog tables do not support, and one tablet carries only one `BinlogConfigPB`. - New switch `enable_feature_changelog` (FE and BE) gates changelog tables, TSO allocation, Table Stream INSERT, TTL cleanup and the changelog part of SHOW CREATE. `enable_feature_binlog` is back to CCR only. 3. **IDL names** of the row-only elements are renamed; ids, enum values and types are unchanged, and shared names (`TBinlogConfig`, `BinlogConfigPB`, `binlog_config`, `binlog_format`) stay: | File | Old (id) | New | |---|---|---| | AgentService.thrift | `TTabletRole.TABLET_ROLE_ROW_BINLOG` (1), `TBinlogFormat.ROW` (1) | `TABLET_ROLE_CHANGELOG`, `CHANGELOG` | | Descriptors.thrift | `TOlapTableIndexSchema.row_binlog_id` (7), `TOlapTableSchemaParam.row_binlog_index_schemas` (17) | `changelog_index_id`, `changelog_index_schemas` | | HeartbeatService.thrift | `TMasterInfo.row_binlog_ttl_reference_tso` (15) | `changelog_ttl_reference_tso` | | PlanNodes.thrift | `TBinlogScanType`, `TPaloScanRange.binlog_scan_type` (12), `TOlapScanNode.read_row_binlog` (26) | `TChangelogScanType`, `changelog_scan_type`, `read_changelog` | | olap_file.proto | `RowsetMeta(Cloud)PB.is_row_binlog` (58), `TabletMeta(Cloud)PB.binlog_tablet_id` (37 / 42), `TabletRolePB.TABLET_ROLE_ROW_BINLOG` (1), `BinlogFormatPB.ROW` (1) | `is_changelog`, `changelog_tablet_id`, `TABLET_ROLE_CHANGELOG`, `CHANGELOG` | | descriptors.proto | `POlapTableIndexSchema.row_binlog_id` (6), `POlapTableSchemaParam.row_binlog_index_schemas` (18) | `changelog_index_id`, `changelog_index_schemas` | | internal_service.proto | `PTabletWithPartition.binlog_tablet_id` (3), `PSyncTabletMetaRequest.discover_row_binlog_ttl` (2) | `changelog_tablet_id`, `discover_changelog_ttl` | | cloud.proto | `TabletCompactionJobPB.row_binlog_ttl_seconds` (31) | `changelog_ttl_seconds` | 4. **Guardrails for leftovers of an earlier master build.** - FE, on image load, edit log replay and backup restore: plain and CCR tables and databases drop the three old keys and behave exactly as before. A table that still says `binlog.format = ROW` with `binlog.enable = true` cannot be used except by DROP; queries (including `@incr`, `changelog()` and streams), every load path and DDL fail with `Table <db>.<t> was created with "binlog.format" = "ROW", which is no longer supported; drop the table and create it again with "changelog.enable" = "true"`. Background daemons skip it instead of throwing or rebuilding its companion tablets. A database whose default was ROW gets `binlog.enable = false` back, with a warning, so it does not turn into a CCR database. - BE: publishing a load that an older BE committed on a changelog rowset without the new hidden columns fails with an error instead of reading column -1 (which crashed the BE on every publish retry). A changelog tablet uses the changelog compaction policy by its role, whatever policy string its meta carries; an unknown policy string falls back to size_based with a warning (looking it up used to throw and abort the BE on every restart); the derive stage returns an error instead of a `CHECK` when the hidden columns are missing; a changelog read or compaction without the TSO column fails instead of writing NULL TSO. 5. **Before-image columns** are `__DORIS_CHANGELOG_BEFORE_<col>__`. A user cannot create a column under `__DORIS_`, every check uses this prefix, and `IvmUtil.isCommonHiddenSlot` excludes it (a user column ending in `_COL` would otherwise look like a common hidden slot). 6. **Change codes** have two name families: FE `CHANGE_TYPE_*` (folded change types) and BE `CHANGELOG_OP_*` (persisted op values). Values do not change. 7. Every persisted or externally hard-coded name has an `// ATTN(changelog-rename): formerly "<old>"` line at its definition, so a later compatibility layer knows what to map. **Results.** | Scenario | Before | After | |---|---|---| | SHOW CREATE TABLE of a plain or CCR table | prints `binlog.format`, and `binlog.need_historical_value` on ROW tables | prints the four CCR `binlog.*` keys only (plus `"changelog.enable" = "false"` in a database whose default enables changelog) | | `ALTER DATABASE ... SET ("binlog.enable"="true")` on a database whose tables use changelog | CCR checks, ROW default applied | rejected: `Can not enable binlog in database <db> because table <t> enables changelog` | | user column `__BEFORE__v1__` / `x__BEFORE__y` with before images | wrong or swapped values, failing loads | ordinary user columns; values correct (`test_changelog_before_column_name`) | | FE and BE of this PR started on metadata written by an earlier master build (9 ROW tables) | — | both start; one WARN per old table; SHOW / DESC work; SELECT, INSERT, UPDATE, DELETE, ALTER, TRUNCATE, `@incr`, `changelog()`, CREATE TABLE LIKE and CREATE STREAM fail with the message above; DROP works and removes base and companion tablets; no repeating errors afterwards | | BE restarted on old changelog tablets with policy `"binlog"` | looking the policy up throws `std::out_of_range` (local mode) | the tablets load with role changelog and policy `"changelog"` | **Upgrade from an earlier master build** (none of this applies to a release): - Before upgrading, drop every table created with `"binlog.format" = "ROW"` together with its Table Streams and incremental MVs, and set `"binlog.enable" = "false"` on databases whose default was ROW. Recreate them with `"changelog.enable" = "true"` afterwards. Do not create or alter changelog tables while FEs of both versions run; downgrade is not supported. - Add `enable_feature_changelog = true` to fe.conf and be.conf; `enable_feature_binlog` no longer enables changelog or TSO. - Renamed FE / BE configs are not aliased: old names in conf files are ignored. Update dashboards that use the renamed metrics, bvars, thread-pool labels and SHOW columns. - SQL and views that reference `binlog(...)`, the hidden columns or the before columns must use the new names. Before-image columns now carry the reserved `__DORIS_` prefix, so `CREATE VIEW` / CTAS / MTMV `AS SELECT * FROM changelog(...)` on a table with `changelog.need_historical_value` has to list its columns (as it already had to for the TSO / LSN / OP columns). Flink Doris Connector incremental reads need a connector release that knows the new hidden column names.
3 of 16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
This PR implements Row Binlog based on an independent row-binlog tablet
model for both non-cloud mode and cloud mode.
Before this change, Row Binlog data was tightly coupled with the base
tablet write/compaction path, and cloud mode did not have a complete end-
to-end implementation. This made it difficult to keep Row Binlog behavior
consistent across storage-compute integrated and storage-compute
separated deployments, especially for MOW tables, historical values,
partial update/upsert publish conflicts, and compaction.
The new implementation introduces row-binlog companion tablets and wires
them through FE metadata, BE write/publish/compaction/query paths, and
Cloud Meta Service. In non-cloud mode, Row Binlog write and compaction
are refactored around the independent tablet model. In cloud mode, the
same model is extended with cloud-specific rowset commit, publish,
delete-bitmap, metadata, and compaction handling.
Main changes:
coupling it directly with base tablet rowsets.
through aligned tablet ids.
__DORIS_BINLOG_TSO__,__DORIS_BINLOG_LSN__,__DORIS_BINLOG_OP__.cloud mode.
load/write context.
logic.
CumulativeCompactionBinlogPolicyfor non-cloud Row Binlogcompaction.
semantics, and delete-bitmap correctness.
and transaction paths aware of Row Binlog rowsets.
Meta Service.
CloudBinlogCumulativeCompactionPolicy.compaction paths.
filter_new_visible_rowsetsto avoid compacting newly visiblerowsets too early unless backlog requires it.
filtered as non-local rowsets.
report, SHOW/proc paths, and compaction actions.
and operation.
rowset builder/writer, transaction, and compaction policy behavior.
schema change, multi-segment reads, Cloud compaction, and Cloud publish
conflict.
__DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__where result order depends on commit/visible TSOsemantics.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)