Repository navigation
[Bug] DETAIL reads on MoW tables without changelog.need_historical_value emit UPDATE_BEFORE rows that carry the new values #70278
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
master @ 72e9189 (2026-11-03). The code paths below are unchanged on master as of 2026-11-05.
What's Wrong?
On a Unique Key Merge-on-Write table with changelog.enable = true and changelog.need_historical_value = false, DETAIL reads are accepted but return UPDATE_BEFORE rows that carry the new values:
@incr(... 'incrementType' = 'DETAIL')is accepted. The binder rejects onlyMIN_DELTAwhen historical values are disabled (BindRelation.java#L778-L793).CREATE STREAM ... "type" = "detail"is accepted.checkAsTableStreamBaseTable()checks onlyMIN_DELTA(OlapTable.java#L4351-L4372); test_create_append_only_stream_without_historical_value.groovy#L49-L63 creates such a stream.- Partial-column updates, including
UPDATEstatements rewritten into partial updates, still recordop = UPDATEwith the full after image. - The DETAIL reader splits every
UPDATEintoUPDATE_BEFOREandUPDATE_AFTER, and readsUPDATE_BEFOREthrough the before-column mapping (block_reader.cpp#L395-L414). Without before columns,_init_before_column_ordinals()maps every column to itself (schema.cpp#L197-L216), soUPDATE_BEFOREcarries the after values.DELETErows are read the same way, so they carry the after image of the delete (defaults or NULL for non-key columns) instead of the deleted row.
Consumers cannot tell these rows from real before images. For example, a Flink Doris Connector job with source.changelog.increment-type = 'detail' receives -U(new) followed by +U(new); any downstream operator that retracts on -U, such as an aggregation, retracts the wrong value. The incremental overview page states that UPDATE_BEFORE / DELETE in min_delta and detail need before images, so the base table must be MoW with changelog.need_historical_value enabled.
What You Expected?
Either reject DETAIL reads and detail streams on MoW tables without historical values, as is already done for MIN_DELTA, or never emit rows that look like before images when none were recorded (for example, emit only UPDATE_AFTER).
How to Reproduce?
-- fe.conf: enable_feature_changelog = true
CREATE TABLE t2 (k INT, v1 INT, v2 INT, v3 INT)
UNIQUE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"enable_unique_key_merge_on_write" = "true",
"changelog.enable" = "true"
-- changelog.need_historical_value is left at false
);
INSERT INTO t2 VALUES (1, 10, 10, 10);
SET enable_unique_key_partial_update = true;
INSERT INTO t2 (k, v1) VALUES (1, 20); -- recorded as UPDATE with the full after image
SET enable_unique_key_partial_update = false;
SELECT __DORIS_CHANGELOG_OP__, k, v1, v2, v3
FROM t2@incr('startTimestamp' = '<a time before the first INSERT>', 'incrementType' = 'DETAIL')
ORDER BY __DORIS_CHANGELOG_TSO__, __DORIS_CHANGELOG_LSN__, __DORIS_CHANGELOG_OP__;
-- suspected: the UPDATE_BEFORE row (op = 2) shows v1 = 20, identical to UPDATE_AFTER (op = 3)Anything Else?
Found by code reading while documenting the changelog and Table Stream features; the reproduction above has not been run yet.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Nov 5, 2026
Breakwater-GitHub-Analysis-Slot: slot_37086109f6df
Assessment: The reported partial-update failure is supported by the source at
72e91896bbdd8760f8bd4e488e5672406b40ce52. This is a high-confidence correctness issue in the MoW DETAIL before-image contract, with potential silent errors for consumers that retract old values. This investigation did not execute the SQL reproduction or a Flink job, so the displayed result and downstream impact remain predictions from the verified code path. Scope is the cited revision; affected release versions have not been established. The issue currently has no labels, assignee, or milestone; suggested routing is changelog/Table Stream maintainers, with correctness priority.Verified code path
BindRelation.checkChangeScanCondition()andOlapTable.checkAsTableStreamBaseTable(). DETAIL passes these checks without history.generateTableChangelogSchema()creates value-column before images only when historical values are enabled.changelog_needs_historical_lookup()still selects historical lookup for direct partial updates. For an existing live key,retrieve_historical_row()records UPDATE. The fixed-partial-update derivation reconstructs the complete after image, whileemit_changelog_block()persists before values only ifwrite_beforeis true. SQL UPDATE can enter this path when its partial-update rewrite conditions hold; this is not a claim about every UPDATE statement._append_change_row()copies through that mapping;_init_before_column_ordinals()falls back to the same column when its before column is absent. Thus, for the supplied partial update, the code predicts both emitted update rows contain(1,20,10,10), instead of a before row(1,10,10,10). For DELETE, the same mechanism returns the stored delete payload; exact non-key values depend on the deletion path and need a separate reproduction.Coverage gap and recommended direction
Existing tests explicitly accept DETAIL stream creation without history. The no-history incremental-read test uses a full-row upsert and expects INSERT output. That exercises a different write path and does not validate partial-update before images. These are inspected test expectations, not tests run during this investigation.
The least ambiguous fix direction is to reject MoW DETAIL without historical values at both direct
@incranalysis and stream creation, and cover reads through already-created detail streams. Preserve Duplicate Key append semantics and APPEND_ONLY behavior. Maintainers should explicitly resolve the existing acceptance tests and document the resulting contract. An alternative after-image-only mode needs a defined consumer contract, including DELETE handling; merely suppressing UPDATE_BEFORE does not provide old values for retractions. Do not remove the schema's identity fallback globally: unchanged key and metadata columns legitimately use it.Evidence still needed / next steps
op=2andop=3rows both havev1=20. A corresponding table created with history enabled should instead yieldv1=10andv1=20.show_initial_rows=false. Add separate cases for SQL UPDATE that takes the partial-update rewrite, flexible partial updates, and DELETE against an existing row. Keep full-row upsert and Duplicate Key controls to avoid conflating their semantics.Until the contract is fixed, avoid using no-history MoW DETAIL as a source of old-row retractions. A replacement table intended for that use should be created with
changelog.need_historical_value=trueand given a fresh, consistent consumer baseline. At this revision,ChangelogConfig.alterFromProperties()rejects changing this setting, so enabling it in place is not an available workaround; previously unrecorded before images cannot be recovered from these changelog rows. Switching to MIN_DELTA on the same no-history table is also rejected.