Repository navigation
[fix](fe) Fix stream scan normalization and snapshot/reset pruning - #65468
Merged
Merged
Conversation
…and normalize after CTE inline Two fixes for LogicalOlapTableStreamScan handling in Nereids planner: 1. Override withVirtualColumns to return LogicalOlapTableStreamScan instead of LogicalOlapScan, preventing type downgrade during LogicalPlanDeepCopier.deepCopy used by CTEInline. 2. Move NormalizeOlapTableStreamScan execution to after CTE inline so StreamScans exposed by CTE expansion get normalized. Previously it ran inside notTraverseChildrenOf(CTEAnchor) which blocked traversal into CTE producers. This fixes CTE+stream hidden column queries and IVM incremental refresh failures caused by un-normalized stream scans.
Issue Number: close apache#65339 Related PR: Problem Summary: Normalize stream scan rules were executed in a late CTE-related rewrite topic, which could be skipped in trees with CTE-related traversal constraints, causing delta/rewrite paths to miss synthetic change_type/version project aliases and fail when buildDmlFactorExpr accessed hidden stream columns. This moves into the early table/physical optimization topic after partition pruning and removes the separate late topic from , and updates regression baseline for to the fixed incremental result behavior. None - Test: Regression test / Unit Test / Manual test / No need to test (with reason) - test_ivm_basic_mtmv (baseline .out updated) - Behavior changed: Yes (CTE stream-scan normalization now runs in the proper rewrite phase) - Does this need documentation: No
yujun777
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
July 10, 2026 10:44
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yujun777
changed the title
This was referenced Jul 10, 2026
Contributor
Author
|
run buildall |
yujun777
changed the title
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29280 ms |
Contributor
TPC-DS: Total hot run time: 180182 ms |
Contributor
ClickBench: Total hot run time: 25.04 s |
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 10, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 10, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Add a regression case for min_delta stream queries that project the stream change type column through a CTE and reuse that CTE twice in a self-join. The case verifies the output through an auto-generated .out file and keeps the existing stream regression coverage intact.
### Release note
None
### Check List (For Author)
- Test: Regression test
- ./run-regression-test.sh --run -d table_stream_p0 -s test_min_delta_stream
- Behavior changed: No
- Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Stream scan planning had split state for snapshot and reset, while partition pruning still depended on table-specific non-empty checks that did not receive stream read semantics. This made the stream planning path harder to reason about and caused snapshot/reset handling to diverge from incremental reads. This change replaces the dual boolean state on logical stream scan with a single StreamReadMode enum, threads the optional read mode through LogicalCatalogRelation into selectNonEmptyPartitionIds, and lets the stream wrapper choose incremental pruning from stream.hasData while delegating snapshot/reset pruning to the base table path. The result keeps the read-mode semantics continuous from binding through rewrite and pruning, while preserving the convenience isSnapshot/isReset/isIncremental helpers on the scan node.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.ExplainTableStreamPlanTest
- Behavior changed: Yes (stream scan snapshot/reset partition pruning now follows explicit read-mode plumbing instead of split boolean flags)
- Does this need documentation: No
yujun777
changed the title
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#65468 Problem Summary: This follow-up keeps the stream scan helper implementation order aligned with the surrounding builder methods by moving after in . The change is code organization only and does not alter semantics. ### Release note None ### Check List (For Author) - Test: No need to test (with reason) - Method order only, no logic change - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29961 ms |
Contributor
TPC-DS: Total hot run time: 180801 ms |
Contributor
ClickBench: Total hot run time: 24.94 s |
Contributor
TPC-H: Total hot run time: 29750 ms |
Contributor
TPC-DS: Total hot run time: 179803 ms |
Contributor
ClickBench: Total hot run time: 25.21 s |
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review complete for PR 65468 at 32a07b7.
I did not find a new blocking issue to raise. The existing review threads about aliased stream scans preserving stream identity/read mode and MV pre-rewrite temp plans normalizing stream scans were treated as already-known context and appear addressed in the current head, so I did not re-submit them.
Checkpoint conclusions:
- Goal and tests: the PR propagates stream read mode into pruning/normalization and moves stream normalization into the relevant rewrite batches. Added coverage includes aliased snapshot/reset FE tests, direct tmpPlanForMvRewrite normalization coverage, and a table-stream CTE reuse regression.
- Scope: the changes are focused on table-stream read-mode representation, empty-partition pruning semantics, rewrite ordering, and targeted tests.
- Concurrency/lifecycle/config/persistence: I did not find new concurrent state, lifecycle ownership, dynamic config, persistence format, or FE/BE protocol concerns in this change.
- Parallel paths: normal Nereids rewriting, MV pre-rewrite CTE-child rewriting, MV partition compensation, and old planner OLAP partition filtering were checked. Non-stream paths keep Optional.empty() semantics.
- Data correctness: incremental stream pruning still uses stream delta availability; snapshot/reset pruning delegates to the base table and is narrowed later by the stream snapshot/reset logic, which avoids the previously risky empty-pruning behavior.
- Performance/observability: no new material hot-path cost or observability gap stood out from the changed code.
No additional user-provided focus was specified. Three review subagents completed the normal and risk-focused passes and all returned NO_NEW_VALUABLE_FINDINGS. I did not run builds or tests because this review environment is read-only per the prompt; additionally, .worktree_initialized and thirdparty/installed/bin/protoc are absent in this checkout.
morrySnow
approved these changes
Jul 14, 2026
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 14, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
3 tasks done
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 15, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 15, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 20, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 23, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 30, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 31, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Jul 31, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 4, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 10, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 17, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 17, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 19, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 25, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Aug 27, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Sep 1, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
35 of 69 tasks
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Sep 2, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Sep 3, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Sep 3, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
yujun777
added a commit
to yujun777/doris
that referenced
this pull request
Sep 3, 2026
### What problem does this PR solve? Issue Number: close apache#65469 Related PR: apache#65468 Problem Summary: Several IVM aggregate regression cases used `REFRESH ... COMPLETE` before validating later incremental behavior. COMPLETE refresh does not advance the stream consumption offset, so the following `REFRESH ... INCREMENTAL` could re-consume historical changes and produce results that no longer matched the intended test point. This change updates the affected aggregate suites to establish the baseline with an initial incremental refresh, keeps the later COMPLETE refreshes that compare against snapshot ground truth, and regenerates the `.out` files so the expected results match the corrected stream-offset behavior. ### Release note None ### Check List (For Author) - Test: Regression test - `./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_agg_2,test_ivm_agg_3,test_ivm_agg_4,test_ivm_agg_6 -forceGenOut --no-parallel` - Behavior changed: Yes (the affected regression cases now validate incremental results after establishing stream offsets with incremental refresh) - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #65469
Related PR:
Problem Summary: This PR fixes two FE issues in the stream scan path.
First, when a stream scan appears inside a CTE,
NormalizeOlapTableStreamScanmay not be applied on the reused CTE branch, so the stream virtual columns are not lowered consistently and later rewrite steps see the wrong scan shape.Second,
PruneEmptyPartitionusesselectNonEmptyPartitionIdsbefore stream scan normalization. Forstream@snapshotandstream@reset, the old stream-wrapper pruning logic still used incremental stream visibility semantics, so a stream with no new delta could be pruned as empty even though snapshot/reset should still read snapshot data.This change keeps stream read mode as explicit logical scan state, threads that mode through
LogicalCatalogRelationinto partition pruning, preserves stream scan helper predicates on the logical node, and lets snapshot/reset pruning follow the base table path while incremental reads still use stream delta visibility. It also keeps stream scan normalization behavior consistent when the scan is referenced through a CTE.As a follow-up for the alias copy path, this PR also fills the remaining
LogicalOlapTableStreamScanrebuild methods inherited fromLogicalOlapScan, so alias / virtual-column / topn rewrite paths do not accidentally rebuild a stream scan as a plainLogicalOlapScanand skip stream normalization.Release note
None
Check List (For Author)
./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.ExplainTableStreamPlanTeststream@snapshot/stream@resetare no longer pruned by incremental stream visibility, and aliased stream scans preserve stream normalization state)