[Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418

Description

This issue tracks the end-to-end incremental computation capability in Apache Doris.

The implementation is being built in three layers:

OLAP table DML
    -> row binlog: durable row-level changes, before images, operation type, LSN/TSO
    -> Table Stream: change-query semantics, per-partition consumption offsets, snapshots and reset
    -> MTMV incremental maintenance: derive and transactionally apply only the affected result changes

This tracker replaces the stale umbrella issue #57921 and incorporates the row-binlog foundation originally tracked by #61956.

Goals

• Provide a durable row-level change log for Doris internal tables.
• Expose deterministic incremental consumption through Table Stream.
• Provide snapshot reads aligned with stream consumption points for joins and recovery.
• Integrate Table Stream with MTMV planning and refresh execution for row-level incremental maintenance.
• Preserve correctness across replicas, compaction, schema changes, concurrent consumers, failures, retries, and restart/replay.
• Add complete unit, regression, compatibility, and failure-recovery coverage.
• Document supported table models, SQL syntax, semantics, limitations, and operational guidance.

Scope boundary

This tracker covers the internal row-binlog and Table Stream stack used by incremental computation, plus its MTMV integration. External CDC Streaming Jobs and general MTMV partition refresh or query-rewrite changes are out of scope unless they directly modify this stack.

FE-side (CCR) binlog is listed under "Adjacent work" below: it is a separate feature, but several correctness issues tracked here live in that code and are reported against the same stack.

The repository review has been refreshed through 2026-09-01. PR #61382 predates the original review window but is included because it is the Table Stream metadata and DDL foundation.

Status summary (2026-09-01)

Progress

Row binlog foundation

Merged — core path:

• [Feature] add binlog meta module (row type) (1/3) #62058 — define row-binlog metadata and schema.
• [Feature] support binlog<row> read and write (row type) (2/3) #63110 — implement row-binlog read and write paths.
• [Feature] support binlog replica schedule and compaction (row type) (3/3) #63643 — implement replica clone, scheduling, and compaction.
• [Enhancement] (binlog<row>) move write lsn on sink, ensure lsn same on all replica #64133 — allocate LSNs at the sink and keep LSN/TSO semantics consistent across replicas.
• [feat](row binlog)：support cloud binlog and independent binlog tablet #65810 — support cloud binlog and an independent binlog tablet.
• [refactor](storage) derive row binlog in the transform chain; delete RowBinlogSegmentWriter #66883 — derive row binlog in the segment-writer transform chain and delete RowBinlogSegmentWriter. Supersedes [refactor](storage) Add a block transform chain before segment writers #64674.
• [feature](binlog) Add per-row LSN column for detail (DUP) tables to support IVM #66889 — add a per-row LSN column for detail (DUP) tables, required by MTMV incremental maintenance.

Merged — fixes:

• [fix](core): fix cumu policy npe when do binlog schedule #64249, [fix](binlog compaction): cumu policy may be null #65032 — cumulative policy NPE during binlog scheduling and binlog compaction.
• [fix](binlog): fix GroupRowsetWriterTest coredump due to incomplete binlog meta setting #64545, [fix](fe) Fix row binlog schema change test metadata #65663, [fix](be) Fix row binlog writer test metadata #65686 — incomplete binlog metadata in the grouped rowset writer and its tests.
• [fix](binlog) Support hidden key columns in row binlog #65076 — include hidden key columns while excluding hidden non-key internal columns.
• [fix](binlog) Align row binlog after schema with before values #65438 — align the row-binlog AFTER schema with before values.
• [fix](binlog) Report missing row binlog for INCR queries #65481 — report missing row binlog for INCR queries.
• [fix](row binlog): op should be insert but update when insert after delete #66046 — emit INSERT rather than UPDATE when an insert follows a delete.
• [fix](ccr) Disallow Row Binlog on MOW tables with cluster keys #66093 — disallow Row Binlog on MOW tables with cluster keys.
• [fix](binlog) Fix BE crash when APPEND_ONLY row-binlog scan omits key columns #66432 — fix the BE crash when an APPEND_ONLY row-binlog scan omits key columns.
• [fix](binlog) wrap Row Binlog AFTER columns as nullable before conversion #66553 — wrap Row Binlog AFTER columns as nullable before conversion.
• [fix](be) Support complex array_agg state serialization #66601 — support complex array_agg state serialization.
• [fix](binlog) Avoid mutating source columns during OLAP conversion #66648 — avoid mutating shared source columns during concurrent Row Binlog segment flushes.
• [fix](binlog) Fix row binlog routing for auto partitions #66665 — fix row-binlog routing for auto partitions.
• [fix](binlog) Allow key-only partial updates with row binlog #66840 — allow key-only partial updates with row binlog.

Open:

• [feature](binlog) Keep row binlog tablets colocated with base tablets #66191 — keep row binlog tablets colocated with base tablets. Approved by five reviewers but not merged; [Row Binlog] Replica repair and balancing can break base/companion colocation and block writes #67092 and [Row Binlog] Publish writes to the old shutdown companion tablet after migration #67106 are correctness issues on this path.
• [Enhancement](row binlog) Support flexible partial updates #66899 vs [fix](row binlog): reject flexible update on row binlog table #66764 — flexible partial update. [Enhancement](row binlog) Support flexible partial updates #66899 adds support, [fix](row binlog): reject flexible update on row binlog table #66764 rejects it at DDL time. These are mutually exclusive and a decision is needed before [Bug](High) Flexible Partial Update fails with ROW binlog due to illegal partial update block columns #65866 can be closed.
• [refactor](binlog) Resolve before columns by unique ID #67050 — resolve before columns by unique ID.
• [improvement](ddl) Restrict state types to aggregate tables #66963 — restrict state types to aggregate tables (HLL/QuantileState path behind [Bug](High) ROW binlog rejects valid HLL writes with a misleading QuantileState nullable error #65865).
• [fix](binlog) Fix missing binlog column index when converting TTabletSchema to TabletSchemaPB #64484, [fix](binlog) Fix nullable type mismatch for timestamp column in _fill_binlog_columns #64495 — missing binlog column index, and timestamp nullable mismatch. Open since mid-June; need confirmation of whether [refactor](storage) derive row binlog in the transform chain; delete RowBinlogSegmentWriter #66883 already covers them.
• [Bug](High) Enabling binlog reduces import throughput by about 65% #65947 — enabling binlog reduces import throughput by about 65%. The only attempted fix ([fix](load) Fix row binlog memtable flush backpressure #66894) was closed; no owner and no current fix. This blocks enabling the feature by default.

Table Stream foundation

Merged — core path:

• [feat](dynamic table) support stream part 1: stream meta data & basic DDL #61382 — add stream metadata, DDL, and information-schema visibility.
• [feature](dynamic table) support stream part 2: stream query & comsuption infrastructure #62453 — add stream query and transactional consumption infrastructure.
• [feat](dynamic table) support table stream part 3: docking olap table stream with binlog #63850 — connect OLAP Table Stream to row binlog and add DETAIL, APPEND_ONLY, and MIN_DELTA scan modes.
• [feat](dynamic table) support table stream part 4: support snapshot read #64776 — add stream snapshot/reset reads, table incremental reads, and time-travel snapshot support.
• [feature](tablet stream) Support Cloud Table Stream control plane #65859 — add the Cloud Table Stream control plane.
• [test](binlog stream) Add comprehensive Cloud Table Stream coverage #66743 — add comprehensive Cloud Table Stream coverage.

Merged — fixes:

• [fix](fe) Fix stream scan normalization and snapshot/reset pruning #65468, [fix](stream) Fix stream scan partition prune state propagation #65657 — stream scan normalization, snapshot/reset pruning, and partition-prune state propagation.
• [improvement] (table stream) refactor misleading name StreamTableInfo #65636 — rename the misleading StreamTableInfo.
• [fix](table stream) min delta delete op should return first op's befo… #65826 — MIN_DELTA delete op returns the first op's before image.
• [fix](table stream) force binlog tso pred pushdown and forbid other pred pushdown #65973 — force binlog TSO predicate pushdown and forbid other pushdowns.
• [fix](table stream) Fix multi-segment row-binlog reads for MoW tables #66338 — fix multi-segment row-binlog reads for MoW tables.
• [fix](binlog) Parse table stream properties before base validation #66683 — parse stream properties before base-table validation so valid APPEND_ONLY and DETAIL streams are not rejected.
• [refactor](stream) Localize Cloud Table Stream rewrite state handling #66739, [fix](test) use Cloud Table Stream read-state hook #67079 — localize Cloud Table Stream rewrite state handling and its test hook.
• [fix](binlog) Commit table stream offsets for empty inserts #67126 — commit table stream offsets for empty inserts.
• [fix](table stream) remove base table cache & use dynamic schema #67173 — remove the base-table cache and use a dynamic schema.

Open:

• [draft](test) Add Table Stream Exception cases #65274 — expand Table Stream negative and exception-path regression coverage. Draft since 2026-07-06; without it the negative paths have no regression protection.
• [fix](stream) Mark streams stale when base tables are dropped #66287 and [test](stream) Add regression test for dropped stream base table #65698 — both mark a stream stale/unavailable after its base table is dropped ([Bug](High) table_streams still shows dropped base table name after base table is dropped #65389). Duplicate effort; one needs to be picked. Note: [fix](stream) Mark streams stale when base tables are dropped #66287 was previously checked off in [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265 but is still a draft.
• [fix](be) Fix table stream change type predicate pushdown #65959 — fix table stream change-type predicate pushdown.
• [fix](binlog) Derive table stream schema from base table #67130 — derive table stream schema from the base table ([Table Stream] Stream metadata retains a dropped column after light DROP COLUMN #67109).
• [fix](cloud) Wait for running transactions before incremental reads #67181 — wait for running transactions before incremental reads in cloud mode.
• [Bug](Medium) ALTER STREAM SET COMMENT is not supported for table stream #65388, [Bug](Low) CANCEL ALTER STREAM is not supported by parser #65390 — ALTER STREAM SET COMMENT and CANCEL ALTER STREAM are not supported. No owner.

MTMV incremental maintenance

Corrected. The previous revision of this tracker stated that no MTMV implementation PR had been identified. That is no longer accurate:

• [Feature] Support incremental view maintenance (IVM) for materialized tables (MTMV) #66719 is the focused tracker for this layer.
• [feature](ivm) Support incremental view maintenance (IVM) for materialized tables (MTMV) #62606 is the implementation PR. It has been open since 2026-04-20 and is still a draft: +47922 / -889 across 343 files and 273 commits, with no review and no approval. [draft](ivm) for run pipeline test #66266 and [draft](ivm) for run the test pipeline #61496 are pipeline-test drafts from the same branch.

Per #66719, the branch already covers projection, filter, aggregate (COUNT/SUM/AVG/MIN/MAX, bitmap aggregates, expression arguments, bare GROUP BY), INNER/CROSS/LEFT OUTER/FULL OUTER joins, UNION ALL, subquery aliases, REFRESH MATERIALIZED VIEW ... INCREMENTAL/PARTITIONS, dry-run, automatic full-refresh fallback, chained IVM MTMV, ivm_use_full_keys, and regression tests under mtmv_p0/ivm.

The blocker is not implementation, it is reviewability and landing. A single PR of this size cannot be reviewed effectively, and this layer currently has no assigned reviewer.

• Split [feature](ivm) Support incremental view maintenance (IVM) for materialized tables (MTMV) #62606 into independently mergeable PRs (delta plan derivation, per-operator incremental semantics, MTMV lifecycle integration, atomic result-apply plus offset advancement, fallback and observability, tests).
• Assign FE and storage reviewers for the split PRs.
• Define supported MTMV query shapes and eligibility/fallback rules.
• Derive delta plans from Table Streams, including snapshot sides for joins.
• Define incremental semantics for projection, filter, join, union, aggregation, and supported expressions.
• Integrate stream creation and lifecycle with MTMV creation, alteration, pause/resume, and drop.
• Execute result changes and stream-offset advancement atomically.
• Support full-refresh bootstrap/reset and fallback when incremental maintenance is not valid.
• Define retry, idempotency, concurrent refresh, failover, and EditLog replay behavior.
• Add observability for refresh mode, consumed offsets, lag, fallback reason, and failures.
• Add FE and BE unit tests plus end-to-end regression tests for correctness and recovery.
• Add user and operator documentation.

Adjacent work

• [Feature] Add a global monotonically increasing timestamp service (TSO) for incremental computation in Doris #61198 — global monotonically increasing timestamp service (TSO). Closed.
• [Feature](binlog): keep row binlog tablet locality during scheduling #65978 / [feature](binlog) Keep row binlog tablets colocated with base tablets #66191 — row binlog tablet locality during scheduling.
• [fix](fe) Preserve config for replace table binlogs #66930 — preserve binlog config across REPLACE TABLE ([Bug] REPLACE TABLE may skip binlog on master FE when only table-level binlog is enabled #66571, [CCR][FE Binlog] REPLACE TABLE swap=false checks the old table ID and can lose events #67105).
• [test](fe) Cover binlog config validation routing #65484 — cover binlog config validation routing ([Bug](Low) ALTER TABLE changing ROW binlog.format returns wrong error message #65383).
• [feature](single replica ingest) support single replica ingest binlog #67156 — single replica ingest binlog.
• [Bug] Snapshot creation fails with hard link NOT_FOUND error when binlog enabled during CLONE task #63134 — snapshot creation fails with a hard-link NOT_FOUND error when binlog is enabled during a CLONE task.

Bug and correctness backlog

#65265 is the consolidated bug and test backlog. Current state:

• Bug list: 7 of 17 closed.
• Correctness issues: 3 of 22 closed ([stream] Empty INSERT SELECT does not advance table stream offset #66273, [fix](table stream) Fix multi-segment row-binlog reads for MoW tables #66338, [Table Stream] Recreating a base table with the same name can make planner locks and scan targets disagree #67095). [fix](stream) Mark streams stale when base tables are dropped #66287 is checked off there but is still a draft PR.
• 17 correctness issues have no linked PR and no assignee.

The unowned correctness issues (#67091–#67094, #67096–#67108) were filed together on 2026-08-25 and are design-level defects rather than isolated bugs: BE restart recovery, replica repair and balancing, TSO boundary semantics, cloud publish/commit failure handling, leader/follower metadata divergence, and lock/scan target disagreement after RENAME or REPLACE TABLE. They need owners before they can be scheduled.

Correctness and compatibility checklist

• DUP and supported UNIQUE KEY MOW table semantics are documented and tested.
• Full update, partial update, delete, sequence column, hidden key, and schema-change paths are covered.
• LSN/TSO ordering is deterministic across replicas and remains valid after clone and compaction.
• Concurrent stream consumption either commits data and offsets together or fails without advancing offsets.
• Snapshot, reset, and incremental reads agree at partition boundaries and after restart.
• Binlog retention cannot silently invalidate an active stream or MTMV; stale state is observable and recoverable.
• Unsupported plans fail clearly or use a documented full-refresh fallback.
• Cloud-mode support or exclusion is explicit.
• Upgrade, downgrade, rolling restart, backup/restore, and replication behavior is defined before general availability.
• Enabling binlog does not regress import throughput to an unacceptable degree ([Bug](High) Enabling binlog reduces import throughput by about 65% #65947).

Overlapping or superseded PRs

• [fix](be) Fix binlog compaction success log #64137 — closed; its compaction-policy crash path is covered by [fix](core): fix cumu policy npe when do binlog schedule #64249.
• [fix](binlog) Fix missing binlog column index when converting TTabletSchema to TabletSchemaPB #64484 and [fix](binlog) Fix nullable type mismatch for timestamp column in _fill_binlog_columns #64495 — overlap with the metadata and nullable-column fixes merged in [fix](binlog): fix GroupRowsetWriterTest coredump due to incomplete binlog meta setting #64545; needs confirmation against [refactor](storage) derive row binlog in the transform chain; delete RowBinlogSegmentWriter #66883.
• [refactor](storage) Add a block transform chain before segment writers #64674 — closed; superseded by [refactor](storage) derive row binlog in the transform chain; delete RowBinlogSegmentWriter #66883.
• [fix](binlog) Support nullable HLL conversion for row binlog #66664 — closed; the HLL/state-type path is now handled by [improvement](ddl) Restrict state types to aggregate tables #66963.
• [fix](load) Fix row binlog memtable flush backpressure #66894 — closed; [Bug](High) Enabling binlog reduces import throughput by about 65% #65947 remains unresolved.

Exit criteria

• Row binlog and Table Stream semantics are stable and documented.
• At least one supported MTMV query class refreshes incrementally end to end with atomic offset advancement.
• Correctness, failure recovery, replay, compaction, and schema evolution tests pass in the supported deployment modes.
• Unsupported cases have explicit diagnostics and a documented fallback.
• Remaining limitations and follow-up work are tracked by linked issues or PRs.