Skip to content

[fix](paimon) Keep the nested-key of a nested_update column through nested pruning - #68835

Open
zhang-arvin wants to merge 1 commit into
apache:masterfrom
zhang-arvin:fix/68802-paimon-nested-update-nested-key
Open

zhang-arvin wants to merge 1 commit into
apache:masterfrom
zhang-arvin:fix/68802-paimon-nested-update-nested-key

Conversation

@zhang-arvin

Copy link
Copy Markdown

What problem does this PR solve?

Fixes #68802.

Reading a Paimon primary-key table whose ARRAY<ROW> column uses the nested_update aggregate function fails through the JNI scanner with a ClassCastException (e.g. HeapBytesVector cannot be cast to LongColumnVector) on splits that need merge-on-read. It only reproduces for such splits, so it is intermittent and disappears once files are compacted; SET enable_prune_nested_column = false masks it.

Root cause: paimon's merge engine (FieldNestedUpdateAgg) builds its key projection against the column's declared element row type and reads the nested-key field by its original position. Doris' nested-column pruning narrows the read type to the sub-fields the query touches and pushes it down with ReadBuilder.withReadType; when the nested-key (the first element field) is dropped, the merge engine still projects that position out of the narrower rows and reads whatever now sits there.

What changed

  • New PaimonNestedUpdateColumns: resolves, from the table's merged options, every column with fields.<col>.aggregate-function = nested_update that also declares a fields.<col>.nested-key.
  • PaimonJniScanner.initReader now keeps the whole element row (instead of the pruned shape) for such a column, so the merge engine's positional projection stays valid. The query still reads only the fields it asked for; the extra fields exist only to satisfy the merge engine.
  • Non-nested_update ARRAY<ROW> columns keep ordinary nested-column pruning.

Tests

  • PaimonReadTypeProjectionTest: a nested_update column keeps its whole element row when pruned; a plain ARRAY<ROW> column still narrows.
  • PaimonNestedUpdateColumnsTest: option resolution (single/multi field nested-key, case-insensitivity, a nested_update column without a nested-key, non-primary-key tables).
  • Verified the new projection test fails without the fix.

mvn -pl be-java-extensions/paimon-scanner -am test: 117 tests pass, checkstyle clean.

…ested pruning

A primary-key Paimon table can bind an ARRAY<ROW> column to the nested_update
aggregate function (fields.<col>.aggregate-function = nested_update). Its merge
engine builds a key projection against the DECLARED element row type and reads
the nested-key field by its original position while merging sorted runs.

Doris' nested-column pruning rewrites the read type to only the sub-fields a
query touches and pushes it down with ReadBuilder.withReadType. When that drops
the field named by fields.<col>.nested-key, the merge engine still projects that
position out of the narrower rows and reads whatever now sits there — a
ClassCastException (e.g. HeapBytesVector as LongColumnVector) that only appears
for splits that need merge-on-read, so it is intermittent and vanishes once the
files are compacted. Setting enable_prune_nested_column = false masks it.

Resolve the table's nested_update columns from its merged options and, for such
a column, keep the whole element row instead of the pruned shape. The query
still reads only the sub-fields it asked for; the extra fields exist so the
merge engine keeps working. Non-nested_update ARRAY<ROW> columns keep pruning.

Fixes apache#68802
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Paimon JNI read fails with ClassCastException when nested column pruning drops the nested-key of a nested_update ARRAY<ROW> column

2 participants