[vinodkc]

What changes were proposed in this pull request?

This PR fixes two bugs in the vectorized Parquet reader that are hit when, within one batch, a dictionary-encoded data page follows a PLAIN-encoded data page of the same column chunk.

1. Wrong results from lazy dictionary decoding (VectorizedColumnReader.readBatch). After reading a dictionary-encoded page, the reader attaches the dictionary to the column vector (lazy decoding) only if no non-dictionary values were added to the batch yet. Since SPARK-34859 this was checked with startRowId == pageFirstRowIndex, but ParquetReadState.resetForNewPage sets rowId to pageFirstRowIndex, so the check is always true on the first read of a page. The dictionary was then attached over values that a PLAIN page had already written to the batch, and those rows were decoded from dictionary id slots that were never written. This PR restores the check to startOffset == 0 (no values in the batch yet), so such pages take the existing eager decodeDictionaryIds path. This affects flat and nested columns.

   startOffset == 0 was the check before SPARK-34859, and it matches that PR's intent: when asked in review why readState.rowId == pageFirstRowIndex is a condition for lazy decoding, the answer was that it is "just to check whether we are at the first iteration of the loop". Rows skipped by column index filtering are not written to the batch, so they don't change startOffset.

2. ArrayIndexOutOfBoundsException for repeated columns (VectorizedRleValuesReader.readValues). Values are written at state.valueOffset, but values / nulls were reserved with reserveAdditional(total), i.e. relative to their own elementsAppended. For a dictionary-encoded page, values is the dictionary ids vector, whose elementsAppended does not count the values a PLAIN page already wrote to the batch, so the reservation falls short and putInts overflows. This PR reserves state.valueOffset + total instead. In all the other cases the two are equal, so the behavior there is unchanged.

   With off-heap column vectors (spark.sql.columnVector.offheap.enabled=true), OffHeapColumnVector.putInts does not check bounds, so this overflow writes past the allocated native memory instead of throwing.

Why are the changes needed?

The Parquet format allows a column chunk to mix dictionary-encoded and PLAIN data pages in any order. parquet-mr only falls back from dictionary to PLAIN for good, so files written by Spark do not hit this, but other writers do. We hit this with files written by Snowflake Openflow, where a list element column of a shredded variant has the pages RLE_DICTIONARY -> PLAIN -> RLE_DICTIONARY. Reading such files:

• silently returns wrong values, e.g. sum(aggregate(tags, 0L, (a, x) -> a + x)) gives 31228268 instead of 31232468, or
• fails with java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 30504 out of bounds for int[3100] at OnHeapColumnVector.putInts via VectorizedRleValuesReader.readIntegersRepeated.

Setting spark.sql.parquet.enableNestedColumnVectorizedReader=false avoids it only for scans that read nested columns; scans of flat columns stay vectorized and can still return wrong results.

The wrong results (1.) exist since Spark 3.2.0 (SPARK-34859). The overflow (2.) can only happen with the nested column vectorized reader, so since Spark 3.3.0.

Does this PR introduce any user-facing change?

Yes, it is a bug fix. Reading Parquet files in which a dictionary-encoded page follows a PLAIN page in the same batch now returns correct results instead of wrong values or an ArrayIndexOutOfBoundsException.

How was this patch tested?

Added tests to ParquetVectorizedSuite that build column chunks with a given sequence of PLAIN and dictionary-encoded pages and read them with batch sizes that span the pages, with on-heap and off-heap vectors:

• primitive type, PLAIN page followed by a dictionary page (wrong results before the fix)
• nested type, PLAIN page followed by a dictionary page, one element per row (wrong results before the fix)
• nested type, PLAIN page followed by a dictionary page, more values than rows (ArrayIndexOutOfBoundsException before the fix)
• nested string array with dictionary, PLAIN and dictionary pages, mirroring the files above (wrong results before the fix)
• flat and nested columns with column index row ranges that skip the first rows of both pages, so that batches start in the middle of the dictionary page (wrong results before the fix)

checkAnswer in the suite now also asserts that the reader returns exactly the expected number of rows; before, a reader that stopped early would pass.

Also ran ParquetVectorizedSuite, ParquetEncodingSuite, ParquetIOSuite, ParquetColumnIndexSuite, ParquetTypeWideningSuite, ParquetV1QuerySuite, ParquetV2QuerySuite, ParquetFileFormatV1Suite, ParquetFileFormatV2Suite, ParquetV1SchemaPruningSuite and ParquetV2SchemaPruningSuite.

Was this patch authored or co-authored using generative AI tooling?

Yes, Generated-by: Claude Code (Claude Opus 5.5)

[HyukjinKwon]

Thanks, I went through it. The approach looks right to me, and both changes go after the actual root causes:

1. startOffset == 0 is the condition from before SPARK-34859 (a5c8866), and it matches the PLAIN branch, which already uses valueOffset != 0 to mean "this batch already has values". startRowId == pageFirstRowIndex is always true on the first read of a page, so it never guarded the PLAIN -> dictionary case.
2. Reserving from state.valueOffset is correct because that's where the values are written. The dictionary ids vector's elementsAppended doesn't count values from PLAIN pages.

Please keep both in one PR. I ran the new tests with each fix reverted:

• with neither: all 5 fail; "more values than rows" throws the same ArrayIndexOutOfBoundsException as in the PR description (arraycopy: last destination index 15 out of bounds for int[10])
• with only the reserve fix: all 5 return wrong values, so the exception turns into silently wrong results
• with only startOffset == 0: the two multi-value nested tests still hit the ArrayIndexOutOfBoundsException

With both, the whole ParquetVectorizedSuite passes.

Next: please file a JIRA (Bug, SQL, correctness label), put the ID in the PR title and the new test names, and drop WIP. branch-3.5/4.0/4.1/4.2/4.x all have both lines, so this should be backported. It cherry-picks cleanly to branch-4.x, while 4.2 and older need small manual backports (context drift from SPARK-59854).

[spark-test]

The AI review requested by @HyukjinKwon did not finish. A committer can add the ai-review-requested label again to retry.