Repository navigation
[Bug] Paimon JNI read fails with ClassCastException when nested column pruning drops the nested-key of a nested_update ARRAY<ROW> column #68802
Description
Search before asking
- I had searched in the issues and found no similar issues. (Searched
paimon nested_update,paimon ClassCastException struct,prune nested column paimon jni. Related but different: [feat](fluss) Support reading Apache Fluss tables through a fluss catalog #66399, branch-4.2 [improvement](meta path) Port nested column pruning / meta path series #68214, [fix](scan) Read all nested access paths before evaluating TopN filters on pruned STRUCT columns #68110.)
Version
doris-4.1.4-rc04-ad35a140c7f (FE + BE). Paimon external catalog (filesystem catalog on S3/MinIO). Table written by Flink 1.20.5 + Paimon 1.3.1.
What's Wrong?
Reading a Paimon primary-key, partial-update table whose ARRAY<ROW<...>> column uses the nested_update aggregate function fails through the JNI scanner with a ClassCastException when nested column pruning drops the struct field that is the nested-key.
It only fails for splits that need merge-on-read (several sorted runs for the key), so it is intermittent: the same query succeeds again once the files are compacted.
java.lang.ClassCastException: class org.apache.paimon.data.columnar.heap.HeapBytesVector cannot be cast to class org.apache.paimon.data.columnar.LongColumnVector
at org.apache.paimon.data.columnar.VectorizedColumnBatch.getLong(VectorizedColumnBatch.java:85)
at org.apache.paimon.data.columnar.ColumnarRow.getLong(ColumnarRow.java:121)
at Projection$5236.apply(Unknown Source)
at org.apache.paimon.mergetree.compact.aggregate.FieldNestedUpdateAgg.agg(FieldNestedUpdateAgg.java:96)
at org.apache.paimon.mergetree.compact.PartialUpdateMergeFunction.updateWithSequenceGroup(PartialUpdateMergeFunction.java:241)
at org.apache.paimon.mergetree.compact.PartialUpdateMergeFunction.add(PartialUpdateMergeFunction.java:171)
at org.apache.paimon.mergetree.compact.ReducerMergeFunctionWrapper.merge(ReducerMergeFunctionWrapper.java:66)
at org.apache.paimon.mergetree.compact.ReducerMergeFunctionWrapper.add(ReducerMergeFunctionWrapper.java:61)
at org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree$SortMergeIterator.merge(SortMergeReaderWithLoserTree.java:109)
at org.apache.paimon.mergetree.compact.SortMergeReaderWithLoserTree$SortMergeIterator.next(SortMergeReaderWithLoserTree.java:97)
at org.apache.paimon.mergetree.DropDeleteReader$1.next(DropDeleteReader.java:54)
Seen through the MySQL client as [JNI_ERROR]RuntimeException: java.io.IOException: java.lang.ClassCastException: .... Depending on which fields are kept, the message varies, e.g. ParquetTimestampVector cannot be cast to IntColumnVector, HeapBytesVector cannot be cast to IntColumnVector.
Result matrix (same table, single primary key, EXPLAIN shows paimonNativeReadSplits=0/1, i.e. JNI):
fields read from ROW<ext_id BIGINT, oligo_tag STRING, scheduling_date TIMESTAMP(0)> (nested-key = ext_id) |
result |
|---|---|
ext_id |
OK |
ext_id, oligo_tag |
OK |
ext_id, scheduling_date |
OK |
oligo_tag |
ClassCastException |
scheduling_date |
ClassCastException |
oligo_tag, scheduling_date |
ClassCastException |
The same pattern holds for every other nested_update column in the table (ROW<oi_id INT, ...8 fields>, ROW<oi_id INT, ..., n1 TIMESTAMP_LTZ, ...>, ROW<wo_order_no INT, order_date TIMESTAMP(0)>): any projection that keeps the first field (the nested-key) works; any projection without it fails.
With SET enable_prune_nested_column = false; (session variable experimental_enable_prune_nested_column, default true) every query above returns correct results.
Analysis (please correct me)
The stack trace suggests the pruned nested read type is handed to Paimon's merge-on-read path. FieldNestedUpdateAgg extracts the nested key with a projection built from the full row type (key at position 0), but the rows it receives are already pruned, so position 0 is now oligo_tag / scheduling_date and getLong fails.
So either Doris should not prune nested fields that the merge engine needs (the nested-key of nested_update, and possibly sequence/aggregation inputs) for splits that are not raw-convertible, or Paimon should widen the read type internally. I'm filing here first because Doris decides the read type; happy to cross-post to apache/paimon if you think the fix belongs there.
What You Expected?
Correct values, same as with enable_prune_nested_column = false, or the planner keeping the fields required by the merge engine.
How to Reproduce?
- In Flink (1.20 + Paimon 1.3.1), create a partial-update table with a
nested_updatearray-of-row column:
CREATE TABLE t (
pk INT NOT NULL,
exts ARRAY<ROW<ext_id BIGINT, oligo_tag STRING, scheduling_date TIMESTAMP(0)>>,
exts_seq BIGINT,
PRIMARY KEY (pk) NOT ENFORCED
) WITH (
'merge-engine' = 'partial-update',
'bucket' = '1',
'fields.exts_seq.sequence-group' = 'exts',
'fields.exts.aggregate-function' = 'nested_update',
'fields.exts.nested-key' = 'ext_id'
);- Write the same
pkin two or more commits (e.g. twoINSERTs with differentext_id), and do not let full compaction run, so that the key has several sorted runs. - In Doris (Paimon catalog), with default session variables:
SELECT struct_element(e, 'oligo_tag')
FROM paimon_catalog.db.t w LATERAL VIEW explode_outer(w.exts) x AS e
WHERE w.pk = 1; -- ClassCastException
SELECT struct_element(e, 'ext_id'), struct_element(e, 'oligo_tag')
FROM paimon_catalog.db.t w LATERAL VIEW explode_outer(w.exts) x AS e
WHERE w.pk = 1; -- OKNote: we observed this on a production-like table that is continuously written by a Flink streaming job; on a small, already-compacted table with the same schema the failing query succeeds (consistent with the merge-on-read explanation). The steps above are derived from that observation and the stack trace.
Anything Else?
A second, separate observation (not a bug, mentioned only for context): a query that explodes several such array columns over the whole table fails with [JNI_ERROR]OutOfMemoryError: Java heap space under the default -Xmx1024m JNI heap; that one is a sizing issue on our side.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Oct 9, 2026
Breakwater-GitHub-Analysis-Slot: slot_07af1467818c
Initial assessment: The reported projection/merge-on-read explanation is strongly supported by the source at the exact Doris commit
ad35a140c7fd0b842f18c23300bac581f7d04326. This should be investigated as a Paimon JNI nested-projection correctness issue. It is currently open and unlabelled; Paimon/external-catalog maintainers are the appropriate first owners. I have verified the code mechanism, but have not run the Flink → Paimon → Doris reproducer, so the deployed failure and fix remain to be confirmed experimentally.Verified source evidence
PaimonReadTypeProjectionrebuilds ROW fields from the requested children and recursively applies that shape through ARRAY. It preserves field IDs, but does not inspect aggregation options or retain anested-keyautomatically.PaimonJniScanner.initReader()passes this shape towithReadType()before creating the reader.MergeFileSplitRead.withReadType()applies the merge factory's read-type adjustment and uses the resulting type for file reading and merging.PartialUpdateMergeFunction.Factoryremaps aggregators at the top level while retaining suppliers created from the original field types. Its adjustment adds missing sequence-group fields, but does not restore nested aggregation inputs.FieldNestedUpdateAggbuilds its key projection from that original element ROW type. During aggregation it applies the projection to the incoming rows. For your schema, droppingext_idmakesoligo_tagoccupy position 0, while the generated key accessor still expects BIGINT there. Paimon's columnargetLong()casts the vector at that position toLongColumnVector, explaining the reported exception.This also explains why the first input can succeed and the error appears when arrays actually need aggregation. The reported compaction dependence is consistent with this mechanism, although JNI selection alone does not prove that a split requires merging. Keeping the key at its original position explains your successful matrix entries; it is not a general guarantee for other schemas.
The relevant Doris integration change is #66573. Its nested-pruning regression fixture is append-only, so it does not exercise this merge dependency. The references listed in the issue do not establish a fix for this path.
Version detail and evidence still needed
The exact Doris commit declares Paimon 1.4.2 in
fe/pom.xml. The Flink writer's 1.3.1 version does not identify the JNI reader version. Please provide:Workaround and next steps
Use the already validated session workaround:
At this commit the variable is experimental and defaults to true. Adding the nested key to SELECT is a schema-specific workaround; compaction does not provide a durable remedy under continuing writes.
For maintainers, first reproduce against the deployed reader version on a fixed snapshot. A conservative Doris mitigation is to prevent nested pruning of affected merge-dependent columns while retaining top-level projection. A Paimon fix could retain the required nested schema before merging and project it afterwards. Both require preserving field order, types, and the JNI output schema: simply widening the scanner read type is insufficient because Doris decodes using its requested shape. Paimon's existing outer projection is top-level only, so nested output projection also needs explicit handling.
Regression coverage should compare actual values with pruning disabled, using overlapping runs and compacted controls, omitted/included keys, non-first/composite keys, and same-typed fields that could hide an ordinal mismatch without throwing. Verify sequence-group ordering and duplicate nested-key updates too. The separate Java heap exception should be investigated independently; it does not explain this typed-accessor failure.
zhang-arvin commentedon Oct 9, 2026
@zhang-arvin here — I'd like to take this.
Plan: the Paimon JNI reader drops the nested-key of a
nested_updateARRAY<ROW>column during nested-column pruning, producing aClassCastExceptionon read. Fix the JNI projection so it keeps the nested-key column required bynested_update, and add a regression test with a nested_update array column under projection pruning.Please assign this to me if convenient.