Repository navigation
[fix](paimon) Keep the nested-key of a nested_update column through nested pruning - #68835
Open
zhang-arvin wants to merge 1 commit into
Open
zhang-arvin wants to merge 1 commit into
zhang-arvin wants to merge 1 commit into
Conversation
…ested pruning A primary-key Paimon table can bind an ARRAY<ROW> column to the nested_update aggregate function (fields.<col>.aggregate-function = nested_update). Its merge engine builds a key projection against the DECLARED element row type and reads the nested-key field by its original position while merging sorted runs. Doris' nested-column pruning rewrites the read type to only the sub-fields a query touches and pushes it down with ReadBuilder.withReadType. When that drops the field named by fields.<col>.nested-key, the merge engine still projects that position out of the narrower rows and reads whatever now sits there — a ClassCastException (e.g. HeapBytesVector as LongColumnVector) that only appears for splits that need merge-on-read, so it is intermittent and vanishes once the files are compacted. Setting enable_prune_nested_column = false masks it. Resolve the table's nested_update columns from its merged options and, for such a column, keep the whole element row instead of the pruned shape. The query still reads only the sub-fields it asked for; the extra fields exist so the merge engine keeps working. Non-nested_update ARRAY<ROW> columns keep pruning. Fixes apache#68802
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Fixes #68802.
Reading a Paimon primary-key table whose
ARRAY<ROW>column uses thenested_updateaggregate function fails through the JNI scanner with aClassCastException(e.g.HeapBytesVector cannot be cast to LongColumnVector) on splits that need merge-on-read. It only reproduces for such splits, so it is intermittent and disappears once files are compacted;SET enable_prune_nested_column = falsemasks it.Root cause: paimon's merge engine (
FieldNestedUpdateAgg) builds its key projection against the column's declared element row type and reads thenested-keyfield by its original position. Doris' nested-column pruning narrows the read type to the sub-fields the query touches and pushes it down withReadBuilder.withReadType; when the nested-key (the first element field) is dropped, the merge engine still projects that position out of the narrower rows and reads whatever now sits there.What changed
PaimonNestedUpdateColumns: resolves, from the table's merged options, every column withfields.<col>.aggregate-function = nested_updatethat also declares afields.<col>.nested-key.PaimonJniScanner.initReadernow keeps the whole element row (instead of the pruned shape) for such a column, so the merge engine's positional projection stays valid. The query still reads only the fields it asked for; the extra fields exist only to satisfy the merge engine.nested_updateARRAY<ROW>columns keep ordinary nested-column pruning.Tests
PaimonReadTypeProjectionTest: anested_updatecolumn keeps its whole element row when pruned; a plainARRAY<ROW>column still narrows.PaimonNestedUpdateColumnsTest: option resolution (single/multi field nested-key, case-insensitivity, anested_updatecolumn without a nested-key, non-primary-key tables).mvn -pl be-java-extensions/paimon-scanner -am test: 117 tests pass, checkstyle clean.