Repository navigation
[common] Skip transform predicates when extracting system table filters - #11004
chenjian2664 wants to merge 6 commits into
Conversation
LeafPredicateExtractor threw UnsupportedOperationException for a leaf whose transform is not a plain field reference, such as UPPER(tag_name). System tables call it while pushing down filters, so a query like SELECT * FROM t$tags WHERE upper(tag_name) = 'X' failed instead of returning the matching rows. Such a leaf now contributes no field constraint, and the predicate is still evaluated on the rows.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS.
Skipping unsupported extraction fixes the tag-query failure, and all 30 selected Tags/Branches/Snapshots/Files tests pass under JDK 8 with normal Maven checks. An additional real-table Java SDK regression below returns files from an excluded partition even with executeFilter(). The equivalent direct-field filter passes; using the original extractor instead produces the previous UnsupportedOperationException.
The files and file key ranges tables dropped every conjunct on
partition, bucket or level from the row filter, assuming the scan had
applied it. The scan only pushes top-level leaves on the plain field,
so a predicate such as UPPER(partition) = '{2, 20}' was applied nowhere
and the read returned files from every partition. Only conjuncts the
scan pushes are now dropped; the rest are evaluated on the rows.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Re-reviewed the revision. The original $files issue is fixed, including an SDK executeFilter() regression with a transform inside OR; the $file_key_ranges regression also passes. All 43 selected system-table tests pass with normal Maven checks under JDK 8. One remaining caller of the shared extractor still needs its residual preserved, as reproduced below.
BucketsRead ignored its filter and relied on the scan, which only
pushes plain partition and bucket leaves. A predicate such as
UPPER(partition) = '{2}' was therefore applied nowhere and the read
returned buckets from every partition. The read now keeps the filter
and evaluates it on the rows before the output projection.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
The updated buckets, files, file-key-ranges and tags tests passed. In the combined eight system-table suites, 63 existing tests passed and the added SDK reproduction failed.
The snapshots, schemas, manifests and consumers tables use the filter only to narrow the IDs they load, so a transform leaf that LeafPredicateExtractor now skips was applied nowhere. A read filtered by UPPER(consumer_id) = 'ID1' returned every consumer. These reads now evaluate the conjuncts that contain a transform on the rows, while plain conjuncts keep their existing ID pushdown.
zhangyue19921010
left a comment
zhangyue19921010
left a comment
There was a problem hiding this comment.
Reviewed the updated residual handling and all extractor callers. With normal Checkstyle, Spotless and Enforcer checks on JDK 11, 71 existing tests across nine system-table suites passed. An added SDK executeFilter regression fails as described below.
| public static Predicate transformConjuncts(@Nullable Predicate predicate) { | ||
| List<Predicate> conjuncts = | ||
| PredicateBuilder.splitAnd(predicate).stream() | ||
| .filter(LeafPredicateExtractor::containsTransform) |
There was a problem hiding this comment.
[P2] Preserve unconsumed sibling conjuncts with the transform residual
On 07a3f3d, a real $snapshots SDK read with LOWER(commit_kind) = 'append' AND schema_id = 999, including newRead().executeFilter(), returns both append snapshots with schema ID 0 instead of no rows. The reader only pushes snapshot_id; this filter drops the plain schema_id conjunct and evaluates only LOWER(commit_kind).
The compound query previously failed on its transform, so accepting it while silently losing a sibling changes the complete-filter behavior. Could these readers retain all conjuncts their ID optimization does not actually consume, while preserving the existing Java String IN path? The same structure exists in the consumers, schemas and manifests readers. I added SnapshotsTableTest#testTransformWithUnconsumedConjunctExecuteFilter: 71 existing tests passed and this additional regression failed.
The snapshots, schemas, manifests and consumers reads evaluated only the conjuncts containing a transform, so a plain sibling that the ID pushdown does not consume was dropped. LOWER(commit_kind) = 'append' AND schema_id = 999 returned every append snapshot. When a filter contains a transform, these reads now evaluate the complete filter on the rows. Filters without a transform keep their existing ID pushdown.
Purpose
Filtering a system table on a function of a column fails with
UnsupportedOperationException, for exampleSELECT * FROM t$tags WHERE upper(tag_name) = 'X'in Spark. Spark pushesUPPER,SUBSTRING,CASTand similar predicates down as Paimon transforms, andLeafPredicateExtractorthrows for any leaf that is not a plain field. All system tables that push down filters are affected, including$files,$snapshots,$tagsand$branches.Such a leaf now adds no field constraint, so pushdown skips it and the predicate is still evaluated on the rows.
Tests