Repository navigation
fix: restrict reverse cast constraint propagation to safe conversions - #25531
Merged
jayzhan211 merged 1 commit intoSep 20, 2026
Merged
Conversation
github-actions
Bot
added
physical-expr
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25531 +/- ##
========================================
Coverage 82.38% 82.38%
========================================
Files 1138 1138
Lines 434328 434582 +254
Branches 434328 434582 +254
========================================
+ Hits 357824 358040 +216
- Misses 54872 54882 +10
- Partials 21632 21660 +28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jayzhan211
approved these changes
Sep 20, 2026
jayzhan211
left a comment
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @haohuaijin , LGTM!
Contributor
Author
|
Thanks for your reviews @jayzhan211 |
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
…apache#25531) ## Which issue does this PR close? Closes apache#25523. Related to apache#25407. ## Rationale for this change A filter on a cast can incorrectly make its input appear constant and remove a required sort. For example, `CAST(x AS INT) = 0` matches both -0.5 and 0.5, but casting the singleton result interval back to DOUBLE yields [0.0, 0.0]. The optimizer can then return the wrong order for `ORDER BY x DESC`. ## What changes are included in this PR? Gate reverse Cast constraint propagation through a common allowlist for all types. Preserve propagation for conversions recognized by `check_bigger_cast`, integer-to-integer casts, and Float32-to-Float64 casts. For other conversions, keep the existing input range instead of treating a cast back as an inverse. The conservative allowlist may reduce range refinement and pruning for safe conversions that are not yet recognized. Runtime CAST behavior and forward interval evaluation are unchanged. ## What is the testing strategy for this PR? - Extend existing `cast.slt` with float-to-integer, integer-to-Float32, timestamp-to-date, and string-to-integer cases. The first three reproduce incorrect sort elimination before the corresponding guards; the string case checks compatibility. - Add a table-driven test for permitted and rejected reverse propagation paths. - Check Float32 widening against actual Cast results at NaN, infinity, signed zero, subnormal, and rounding boundaries. - Passed 24 Cast unit tests (one existing test ignored), 46 filter tests, and four relevant SLT files. ## Are there any user-facing changes? Queries retain required sorting when cast constraints cannot safely determine the input range. No public API changes.
pepijnve
pushed a commit
to pepijnve/datafusion
that referenced
this pull request
Oct 23, 2026
…che#25790) ## Which issue does this PR close? - Closes apache#25613. ## Rationale for this change Decimal columns were excluded from interval analysis, so any filter touching one fell back to the default 20% selectivity. For the Q6-style predicate in the issue, that meant estimating ~1.2M rows when only ~114K match. ## What changes are included in this PR? - `FilterExec` statistics use a new `check_statistics_support`, which also accepts Decimal32/64/128/256. - `next_value` / `prev_value` handle decimals, so strict comparisons get tight bounds (`x < 24.00` becomes `x <= 23.99`). At the limit of a decimal's precision the bound becomes unbounded instead of overflowing. - `get_extreme_value!` supports Decimal32/64, which previously hit `unreachable!()`. - `CastExpr` now propagates constraints back through lossless decimal widening casts. Without this, a literal with a different scale than the column (e.g. `a >= 0.055` on `Decimal(15, 2)`, coerced to `Decimal128(30, 15)`) left the column range untouched and the estimate at 100%. - Decimal arithmetic (`+ - * / %`) is not analyzed: it changes the precision and scale of its operands, so those predicates keep the default selectivity. `check_support` and `is_datatype_supported` are unchanged, so symmetric hash join pruning behaves as before. Decimal support there could be a follow-up. ## What is the testing strategy for this PR? - Unit tests for decimal next/prev values across all four widths, including precision boundaries like 9.99 / -9.99 for `Decimal(3, 2)`, plus strict comparison bounds and Decimal32/64 overflow handling. - `test_filter_statistics_decimal_expr` and `test_filter_statistics_decimal_mixed_scale_expr` check row estimates and tightened min/max for same-scale and mixed-scale predicates. The mixed-scale test mirrors the plan the SQL planner produces. - `test_filter_statistics_decimal_arithmetic_expr` checks that decimal multiplication and subtraction predicates use the default selectivity and keep the column's own type. - `filter_statistics_decimal.slt` runs Q6-style ranges with fractional literals on a parquet table with min/max stats, checking EXPLAIN estimates and query results, including a mixed-scale literal and decimal arithmetic. - New cases in `test_cast_constraint_propagation`, plus boundary checks that propagating through a decimal widening cast never excludes a valid input, covering cross-width casts, a negative scale and the precision limits. - `test_custom_filter_selectivity` relied on decimals being unsupported, so I switched it to `Utf8`. - No existing sqllogictest or TPC-H/TPC-DS plan outputs changed. Reproducer from the issue (TPC-H SF1, current main): | SELECT | Before | After | Actual | | ------------------------- | --------: | ------: | ------: | | Date range | 869,535 | 869,535 | 909,455 | | Date + Decimal conditions | 1,200,243 | 111,291 | 114,160 | Worth a closer look in review: - The cast change extends the allowlist added in apache#25531. Decimal widening is lossless and order-preserving, and casting bounds back rounds to a neighboring value, which only widens the range. That reasoning is what makes it safe, so I'd appreciate a second opinion on it. - A decimal column with no min/max stats now gets selectivity 1.0 instead of the default, which matches how integer columns already behave. ## Are there any user-facing changes? Better row estimates for filters on decimal columns, which may change some plan choices. One new public function, `check_statistics_support`; no breaking changes.
discord9
added a commit
to GreptimeTeam/datafusion
that referenced
this pull request
Oct 28, 2026
Adapt apache#25531 (4e603e1) to this fork. CastExpr::propagate_constraints cast the output interval straight back to the input type. A many-to-one cast cannot be inverted that way: the filter `CAST(k AS BIGINT) = 1` yields the singleton output interval [1, 1] but also matches k = 1.5, so the input interval incorrectly became [1.0, 1.0]. FilterExec then derived a singleton range for k, the equivalence classes marked k constant, and the requested ORDER BY k was removed, letting rows come back in physical scan order. Gate reverse propagation behind the same safe-conversion allowlist as upstream: check_bigger_cast, integer-to-integer, and Float32 to Float64. Unrecognized conversions return Ok(Some(vec![])), which per the PhysicalExpr contract leaves the child interval unchanged. The fork's check_bigger_cast does not list the Int32/Date32 and Int64/Date64 identity pairs, so they are matched explicitly here instead of broadening the shared helper, which also drives ordering guarantees. Adds the upstream table-driven propagation test, a Float32 -> Int64 range-preservation test, the upstream Float32 widening boundary test, and cast.slt regressions for ASC/DESC under one and four target partitions. Signed-off-by: discord9 <discord9@163.com>
WenyXu
pushed a commit
to WenyXu/greptimedb
that referenced
this pull request
Oct 28, 2026
Update the pinned GreptimeTeam/datafusion revision from e3221b8 to cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger regression test). CastExpr::propagate_constraints used to cast a filter's output interval straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so FilterExec advertised k as constant, the equivalence classes dropped the ordering required by ORDER BY k, and rows returned in scan order. The sqlness case keeps every GreptimeTeam#9600 assertion and the explicit ORDER BY, and adds ascending and descending checks for the partitioned table, the control table and a table whose matching rows are stored in neither ascending nor descending k order. Cargo.lock only rewrites the 35 datafusion source lines. Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit fe637b0) Signed-off-by: WenyXu <wenymedia@gmail.com>
WenyXu
added a commit
to GreptimeTeam/greptimedb
that referenced
this pull request
Oct 28, 2026
* fix(hdfs): clean up temporary file when an atomic write fails (#9483) * fix(hdfs): clean up temporary file when an atomic write fails Signed-off-by: fengyue-xve <1759519137@qq.com> * style: apply rustfmt to hdfs layer test Signed-off-by: fengyue-xve <1759519137@qq.com> * fix(hdfs): report earlier write failures and log cleanup failures - Later `write`/`close` calls on a writer whose failed write was cleaned up now mention the original failure instead of the bare "unavailable after close or abort" message. - `abort_and_delete` logs abort and deletion failures with the temporary path instead of discarding them; the original write error is still returned to the caller. - The temporary-file cleanup test now lists the directory holding the temporary file, so it fails if the cleanup is reverted. Signed-off-by: 风月 <1759519137@qq.com> Signed-off-by: fengyue-xve <1759519137@qq.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> Signed-off-by: 风月 <1759519137@qq.com> (cherry picked from commit a35661d) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(jaeger): return the most recent traces when a search hits its limit (#9595) * fix(jaeger): return the most recent traces when a search hits its limit Trace search applied LIMIT to an unordered DISTINCT trace_id, so a search matching more traces than the limit returned an arbitrary subset that changed between identical requests. Order traces by their latest matching span before applying the limit. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(jaeger): name the trace dataframe builder with a verb and document execution Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit cf16c73) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(pprof): initialize framehop before signal sampling (#9612) fix(pprof): pin upstream framehop initialization fix Pin pprof workspace-wide to 01cff82d containing tikv/pprof-rs#282. Initialize framehop before sampling instead of allocating and discovering shared libraries in the first SIGPROF handler. Regenerate the lockfile without unrelated dependency changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit 0dc8177) Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: bump rustls to 0.23.46 (#9614) rustls 0.23.46 fixes rustls/rustls#3301: on TLS 1.2 with AES-GCM, reaching the 2^24 record limit dropped a fragment while reporting the write as complete, then kept sending records after close_notify. This stalled Kafka WAL produce requests until the request timeout. (cherry picked from commit 33879b0) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(prom): return empty remote read results for missing metrics (#9613) * fix: return empty remote read results for missing metrics Signed-off-by: fys <fengys1996@gmail.com> * test(prom): cover empty reads for missing databases Signed-off-by: fys <fengys1996@gmail.com> * test(prom): simplify remote read regression cases Signed-off-by: fys <fengys1996@gmail.com> * fix(prom): log missing tables during remote read Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> (cherry picked from commit cf0f6c3) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): compare release validation against the previous stable version (#9636) Ancestry-based tag selection skips stable tags that live only on older release branches. Signed-off-by: paomian <xpaomian@gmail.com> (cherry picked from commit 3de8610) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): keep processing bulk requests after one fails to encode (#9638) * fix(mito2): keep processing bulk requests after one fails to encode When `push_bulk` fails to build the WAL entry of a bulk request, it answers that request with the error. `process_bulk_requests` then returned from the loop, so the remaining requests of the drained batch were dropped without an answer and their callers only saw a receive error. Skip the failed request and continue with the rest of the batch instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): keep bulk install barriers independent per target The bulk install test barrier was a single process-wide slot, so tests that arm it while running in the same process replaced each other's barrier. Keep one barrier per armed target instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> (cherry picked from commit 88bdab0) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(promql): honor @ modifier on subqueries (#9446) * fix(promql): honor @ modifier on subqueries `prom_subquery_expr_to_plan` never read the `at` field of a subquery, so `sum_over_time(m[20s:10s] @ 60)` was evaluated like `m[20s:10s]` and followed the outer evaluation grid. Handle `@` the way Prometheus does: fold the anchor into the subquery offset (`offset + (eval_start - anchor)`, as `setOffsetForAtModifier` does), evaluate the subquery once at the start of the evaluation, and replay the folded window over the outer grid, reusing the anchored range selector machinery. A call whose range argument is an anchored subquery is promoted as step-invariant, like one over an anchored range selector. `@ start()`/`@ end()` and `offset` combine as for selectors, and an unrepresentable anchor is rejected with the existing error. Closes #9411 Signed-off-by: Aarav <aaravsjadav@gmail.com> * test(promql): add sqlness cases for @ on subqueries Cover the #9411 reproducer end to end: `@ <ts>`, `@ start()`/`@ end()`, `@` combined with `offset` in either order, an instant query away from the anchor, `*_over_time` functions and `rate` over an anchored subquery, a computed inner expression, and an aggregation above the anchored call. Two series per table so a dropped or merged series shows up as a wrong row. Refs #9411 Signed-off-by: Aarav <aaravsjadav@gmail.com> --------- Signed-off-by: Aarav <aaravsjadav@gmail.com> (cherry picked from commit 095c288) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): validate bulk insert batches against the region schema (#9698) * fix(mito2): validate bulk insert batches against the region schema Bulk inserts were only checked for the time index column, unlike row writes, which WriteRequest::check_schema checks against the region schema. A batch with columns of another type, nulls in the time index or in non-nullable columns, duplicate or unknown columns reached the region worker: a time index that is not a timestamp or holds only nulls made the worker fail, and other mismatches failed after the WAL append or were stored as written, depending on the SST format. Move the column checks of check_schema into a core shared by row writes and bulk inserts, and check bulk insert batches at the same points as rows: when the engine builds the worker request, and again in the worker if the region schema changed since. check_schema keeps its behavior. Bulk insert columns must have the column's Arrow type, except that tags may use any string encoding, as the memtables already convert them. Sparse batches must carry one encoded primary key column of binary without nulls. A batch that misses columns gets them filled by the worker even if it claims to match the current schema. The metric engine checks logical bulk inserts against the logical region before rewriting them into the physical region, as the rewrite copies columns without checking them and would encode a duplicated tag into the primary key twice. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): report an error when a write is interrupted by a panic WriteNotify answers its waiter when it is dropped and reported success unless an error had been set, also when it was dropped while unwinding from a panic in the middle of the write. Report an error in that case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): tolerate a poisoned series lock when dropping a series map SeriesMap::drop reads every series to update the memtable metrics and unwrapped the lock, so dropping the map while unwinding from a panic that poisoned a series lock panicked again. Read through the poisoned lock instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> (cherry picked from commit 791f429) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence writes before a graceful downgrade (#9716) * test(mito2): cover graceful role changes racing with writes A graceful role change flips the region state in a background task while the worker keeps handling writes. Add bulk and row write tests that check a write handled after the role change request is either rejected or covered by the returned last_entry_id. Add tests for DowngradingLeader while an ALTER keeps the region in Altering, alone and in the downgrade sequence the datanode runs (DowngradingLeader, Flush, Follower) with a migration candidate on its own local WAL. Add a test that the downgrade sequence covers a write in flight when a lease renewal sets Downgrading. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fence writes before a graceful downgrade set_region_role_state_gracefully hands the role change to a background task that waits for the manifest lock. The worker keeps handling writes until the task flips the state, so a write could be acknowledged with an entry id above the returned last_entry_id. Set Downgrading in the worker before spawning the task for Follower and DowngradingLeader requests, as lease renewal already does, so every write the worker handles afterwards is rejected. DowngradingLeader also did nothing and returned success when the region was in another leader state such as Altering. The region became writable again when the operation finished, and the datanode then flushed and turned it into a follower in one step. Return a retryable error instead, so metasrv retries the downgrade. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): assert the retryable downgrade error during ALTER Check that DowngradingLeader during an ALTER returns a retryable error, and release the paused ALTER on that path. The local WAL test now retries the downgrade after the ALTER, as metasrv does. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): write before the retried downgrade in the local WAL test Rows written before an ALTER were already flushed, so the candidate check did not depend on the downgrade's final flush. Write rows after the ALTER and before the downgrade is retried, which only that flush can carry to the candidate. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> (cherry picked from commit 660c4ef) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): validate persisted partition bounds before altering regions (#9487) * fix(meta): validate persisted partition bounds before altering regions A partition bound is persisted with the type of the column at the time it was written. When an ALTER TABLE changes the type of a column referenced by a partition expression (widening the time index unit is the common case), each region rewrites the bound literal to the new type while building its scan predicate. A millisecond bound like year 3000 fits milliseconds but overflows nanoseconds, so every scan of the region fails once the table schema is updated -- and some regions may already be altered when the failure surfaces. Validate every region's persisted expression on prepare, before any region is modified: parse the persisted JSON (mirrored types, since common-meta cannot depend on the partition crate which depends on it), replay the same value -> ScalarValue conversion and checked cast the region performs, and reject the ALTER naming the offending region when a bound cannot be represented with the new column type. Incompatible changes are rejected atomically; compatible widening proceeds unchanged. Adds meta-level tests (multi-region rejection naming the second region, compatible widening completing) and an end-to-end sqlness case. Fixes #9454 Signed-off-by: fengyue-xve <1759519137@qq.com> * fix: access ColumnSchema fields directly and use plain format args ColumnSchema exposes name and data_type as public fields, not methods, and the telemetry macros only accept a format string after the error. Signed-off-by: fengyue-xve <1759519137@qq.com> * test: access ColumnSchema data_type field in alter table tests Same fix as the previous commit: data_type is a public field on ColumnSchema, not a method. Signed-off-by: fengyue-xve <1759519137@qq.com> * test: drop borrows now that data_type is read as a field Signed-off-by: fengyue-xve <1759519137@qq.com> * test: drop useless format! in nested expression test Signed-off-by: fengyue-xve <1759519137@qq.com> * test: cover the partition tail in change_col_type sqlness case Signed-off-by: fengyue-xve <1759519137@qq.com> * test: drop spurious blank line in change_col_type result Signed-off-by: fengyue-xve <1759519137@qq.com> * fix(meta): compare partition bounds with the scan comparison coercion The preflight validator cast the persisted bound directly to the altered column type, which rejected ALTERs the scan itself accepts: the region coerces both the column and the bound to the common comparison type (ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field column with an out-of-range bound still scans. Compare through comparison_coercion(target_type, bound_type) instead and only require the bound to be castable to the common type. Also add a round-trip test in the partition crate that serializes real PartitionExprs and runs them through the validator, so drift between the Persisted* mirror and the partition types fails CI, and fix the error message to match the comparison semantics. Signed-off-by: fengyue-xve <1759519137@qq.com> * test: update the sqlness fixture for the new incompatibility message The error message now reflects the comparison semantics: the persisted bound can no longer be compared with the new column type. Signed-off-by: fengyue-xve <1759519137@qq.com> * build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock file was not refreshed, so cargo check --locked refused to run on CI. Signed-off-by: fengyue-xve <1759519137@qq.com> * fix(meta): reject every non-widening type change of a partitioned column The region routes rows on write through `MultiDimPartitionRule::find_region`, which compares the persisted bound against row values directly: `Value::cmp` panics on mismatched variants and equality misroutes silently. After a column partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT compares `Value::Int32` rows against the persisted `Value::Int64` bound and panics in the frontend, even though the scan-side coercion this check mirrors still works. Reject any type change of a column referenced by a persisted partition expression unless it is a timestamp unit widening, which keeps the bound and the rows comparable because `Timestamp` comparisons convert units first; the widening keeps the scan cast check unchanged. Signed-off-by: fengyue-xve <1759519137@qq.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> (cherry picked from commit 1e29e8d) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697) * fix(partition): route NULLs like row inserts and accept signed-minimum bounds Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep equality two-valued for NULL rows and accept unsigned -0 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): treat NULL masks as false in batch routing and keep comparisons bare Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(compat): update partition filter in analyze verbose expected output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit da88ff8) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): re-check a filled write as sent after a schema change (#9701) * fix(mito2): re-check a filled write as sent after a schema change The engine fills the columns a write leaves out with their defaults when it accepts the write, appending them to the rows without adding them to `name_to_index` and `has_null`. If the region schema changes before the worker handles the write, the worker checks the request again and `check_schema` looks up a filled column in `name_to_index`, which panics and stops the region worker. If the filled column was dropped, the write fails with an unknown column that the client never sent. Remove the filled columns before checking, so the request is checked as sent and filled again for the current schema. Nothing is filled yet when the engine checks the request, so that check does no extra work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): rename put_request helper to build_put_request Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> (cherry picked from commit 93c3b11) Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): gather last rows with one take per input batch (#9737) * perf(mito2): gather last rows with one take per input batch FlatLastTimestampSelector buffered a single-row slice per series and concatenated up to 8192 of them. Concatenating slices with dictionary columns merges the dictionaries once per slice, which made high-cardinality instant PromQL queries using the LastRow scan path (#9034) slower. Select the rows of series that end inside a batch with one take instead. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(mito2): cover duplicate last timestamps in the take path Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): group last rows by primary key bytes Flat batches may assign different dictionary keys to the same primary key, for example when binary keys are wrapped with identity keys. Merge adjacent ranges with equal key bytes before selecting last rows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit 4645ab9) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): apply SST internal column encodings to bulk-encoded parts (#9735) * fix(mito2): apply SST internal column encodings to bulk-encoded parts Bulk memtable parts are encoded to parquet and flushed as SSTs as-is, but the bulk encoder did not apply the time index / __sequence DELTA_BINARY_PACKED encoding (with dictionary disabled) and the uncompressed op_type setting that the regular SST writer uses. Share these settings through one helper. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): drop the answered bulk encoder compression TODO Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit 977e76d) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: avoid exact scan row counts for SSTs with unknown counts (#9759) fix: avoid exact scan row counts for legacy SSTs Treat selected SSTs with unknown row counts as non-exact before checking region partition visibility. Preserve exact statistics for known counts and cover legacy metadata mixed with known-count files. Fixes #9718 Signed-off-by: Aumnertic <hznuzc@gmail.com> (cherry picked from commit 6f53d57) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: avoid duplicate keys in update flow metadata txn (#9816) CREATE OR REPLACE FLOW deleted and then put the same flow route, flownode flow and table flow keys in a single txn when the replacement kept the partitions, flownodes or source tables, which etcd rejects with "duplicate key given in txn request". Only delete the old keys that are not put again. Closes #9810. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit a1b2362) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): increase release integration test stack size (#9817) Signed-off-by: evenyag <realevenyag@gmail.com> (cherry picked from commit e6fa1a0) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(datanode): release region map guard before awaiting role changes (#9824) Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit 0dc649e) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860) * fix(ci): increase Windows nightly test stack size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(ci): align optional macOS artifact test stack size Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> (cherry picked from commit 8b1d683) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): keep SSTs when a manifest save ends with RateLimited or PermissionDenied (#9898) * fix(mito2): keep SSTs when a manifest save ends with RateLimited The object store's retry layer retries temporary errors and returns the last attempt's error. A manifest delta PUT can be stored by the server while the client sees a timeout, reset or 5xx, and the retries that follow can be throttled, so the save ends with RateLimited even though the delta exists. may_have_persisted_manifest_update treated RateLimited as proof that nothing was written, so flush and compaction deleted the SSTs the stored delta references. A later open or follower sync replays the delta and reads fail on the missing files. Treat RateLimited like other ambiguous errors: the SSTs are kept and become orphans when the save really failed. Fixes #9897 Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep SSTs when a manifest save ends with PermissionDenied The retry layer returns the last attempt's error, so a 403 from a revoked key, a changed bucket policy or clock skew can follow a delta PUT that the server already stored. Like RateLimited, PermissionDenied does not prove that nothing was written, so it no longer lets flush or compaction delete the SSTs; they become orphans when the save really failed. The compaction test that rejects a manifest write before persistence now injects Unsupported, which is raised before any request is sent. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> (cherry picked from commit 1cb01d0) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: upload staging manifests in chunks (#9919) Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit f2be11a) Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: stream and prune SST manifest scans (#9931) * refactor: add reusable metadata pruning and projected SST batches Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): stream and prune manifest SST metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(datanode): stream inspection queries through reusable providers Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(catalog): push manifest filters through inspection queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: align manifest column constants with module conventions Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: reduce manifest inspection allocations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: narrow manifest inspection API visibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: retain residual filters for manifest inspection Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit c96cdd0) Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: improve SQL analyze scan diagnostics (#9565) * feat: improve SQL analyze scan diagnostics Signed-off-by: evenyag <realevenyag@gmail.com> * test: redact dynamic filters in sqlness results Signed-off-by: evenyag <realevenyag@gmail.com> * test: stabilize analyze snapshots for merge queue Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove obsolete scan_details assertion Signed-off-by: evenyag <realevenyag@gmail.com> * test: refresh explain snapshots after partition filter changes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> (cherry picked from commit 3e2751d) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(query): avoid lossy casts in partition pruning (#9600) * fix(query): keep lossy casts out of partition pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): align cast pruning expectations with conservative fallback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): record lossy cast partition pruning regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain safe pruning conjuncts beside unsupported casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh cast pruning plans after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit df8f684) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601) * fix(query): skip LastRow pushdown for IGNORE NULLS Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep IGNORE NULLS aggregates above remote scans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): record IGNORE NULLS results before and after flush Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): share IGNORE NULLS coverage across deployments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh last-value plans after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit 0dab3ae) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): share peer discovery with plugin hooks (#9977) Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit 9f0e58e) Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: adapt release backport test dependencies Copy the writer observation test utility from #9505 (18726c7) for the #9919 staging regression, initialize the release-only catchup location hint, and remove main-only unused imports. Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: bump version to 1.3.0 Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(query): pin DataFusion cast constraint propagation fix Update the pinned GreptimeTeam/datafusion revision from e3221b8 to cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger regression test). CastExpr::propagate_constraints used to cast a filter's output interval straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so FilterExec advertised k as constant, the equivalence classes dropped the ordering required by ORDER BY k, and rows returned in scan order. The sqlness case keeps every #9600 assertion and the explicit ORDER BY, and adds ascending and descending checks for the partitioned table, the control table and a table whose matching rows are stored in neither ascending nor descending k order. Cargo.lock only rewrites the 35 datafusion source lines. Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit fe637b0) Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: fengyue-xve <1759519137@qq.com> Signed-off-by: 风月 <1759519137@qq.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: paomian <xpaomian@gmail.com> Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: Aarav <aaravsjadav@gmail.com> Signed-off-by: Aumnertic <hznuzc@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> Co-authored-by: fengyue-xve <1759519137@qq.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: localhost <xpaomian@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me> Co-authored-by: DeviousCardi <115358213+DeviousCardi@users.noreply.github.com> Co-authored-by: Aum <hznuzc@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: discord9 <discord9@163.com>
WenyXu
pushed a commit
to WenyXu/greptimedb
that referenced
this pull request
Oct 28, 2026
…imeTeam#10026) fix(query): pin DataFusion cast constraint propagation fix Update the pinned GreptimeTeam/datafusion revision from e3221b8 to cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger regression test). CastExpr::propagate_constraints used to cast a filter's output interval straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so FilterExec advertised k as constant, the equivalence classes dropped the ordering required by ORDER BY k, and rows returned in scan order. The sqlness case keeps every GreptimeTeam#9600 assertion and the explicit ORDER BY, and adds ascending and descending checks for the partitioned table, the control table and a table whose matching rows are stored in neither ascending nor descending k order. Cargo.lock only rewrites the 35 datafusion source lines. Signed-off-by: discord9 <discord9@163.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #25523.
Related to #25407.
Rationale for this change
A filter on a cast can incorrectly make its input appear constant and remove a required sort. For example,
CAST(x AS INT) = 0matches both -0.5 and 0.5, but casting the singleton result interval back to DOUBLE yields [0.0, 0.0]. The optimizer can then return the wrong order forORDER BY x DESC.What changes are included in this PR?
Gate reverse Cast constraint propagation through a common allowlist for all types. Preserve propagation for conversions recognized by
check_bigger_cast, integer-to-integer casts, and Float32-to-Float64 casts. For other conversions, keep the existing input range instead of treating a cast back as an inverse.The conservative allowlist may reduce range refinement and pruning for safe conversions that are not yet recognized. Runtime CAST behavior and forward interval evaluation are unchanged.
What is the testing strategy for this PR?
cast.sltwith float-to-integer, integer-to-Float32, timestamp-to-date, and string-to-integer cases. The first three reproduce incorrect sort elimination before the corresponding guards; the string case checks compatibility.Are there any user-facing changes?
Queries retain required sorting when cast constraints cannot safely determine the input range. No public API changes.