Repository navigation
Conversation
parquet-avro keeps the DECIMAL annotation only on BINARY and FIXED_LEN_BYTE_ARRAY columns, so a decimal stored as INT32 or INT64 surfaced to the Avro reader as a plain int or long and Hudi then promoted the unscaled integer to the table's decimal as if it were the value (17.00 read as 1700.00). The Hudi schema converter now maps such columns to a fixed decimal, the requested projection is re-typed to the file's physical type, and the parent materializer's root converter is wrapped so the values are re-encoded as the Avro decimal of the read schema. The Hive reader goes through the same read support. A table schema that typed such a column as int or long keeps reading the unscaled value.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #20387 +/- ##
============================================
- Coverage 80.75% 80.74% -0.01%
- Complexity 35317 35369 +52
============================================
Files 2562 2563 +1
Lines 144143 144329 +186
Branches 17560 17602 +42
============================================
+ Hits 116401 116545 +144
- Misses 19768 19790 +22
- Partials 7974 7994 +20
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
closes #20386
The Avro-based parquet reader returns wrong values for decimal columns stored as INT32 or INT64, the encoding Spark, Iceberg, Trino, DuckDB and parquet-rs use for precision up to 18. parquet-avro only keeps the DECIMAL annotation on BINARY and FIXED_LEN_BYTE_ARRAY, so the column surfaces as a plain
long, and Hudi then promotes the unscaled integer to the table's decimal as if it were the value: 17.00 reads as 1700.00 with no error. The Hive reader in hudi-hadoop-mr shares the problem since it derives its row schema the same way.Summary and Changelog
Integer-backed decimals now read correctly through
HoodieAvroParquetReaderand the Hive reader, and a shape the reader cannot handle fails instead of returning wrong numbers.AvroSchemaConverterWithTimestampNTZmaps an INT32/INT64 DECIMAL column to a fixed Avro decimal of the minimal width for its precision, the type Spark derives for the same column;HoodieSchema.minBytesForDecimalPrecisionis the shared width helper.SchemaRepair.alignDecimalPhysicalTypesre-types decimal leaves of the requested projection to the file's INT32/INT64 physical type so the projection matches the column chunks.HoodieAvroReadSupportapplies that alignment ininit, supplies the Hudi-converted schema as the read schema when neither the caller nor the footer provides one, and wraps the root converter of the parent's materializer inIntegerDecimalRecordConverter, whoseIntegerDecimalConverterturns plain and dictionary-encoded int/long values into the fixed or bytes decimal of the read schema, including inside nested records. Such columns inside arrays or maps throwUnsupportedOperationExceptionrather than read wrong. The wrapper keeps the parent's converters for every other field, and the parent materializer is obtained fromAvroReadSupportitself (through a super-call method handle on parquet 1.14+, whose record reader dispatches throughParquetConfiguration), so no parquet-avro constructor is named: hudi-hadoop-common compiles against parquet 1.12.3, 1.13.1, 1.15.2 and 1.17.0.HoodieAvroParquetReaderreads throughHoodieAvroReadSupportinstead of plainAvroReadSupport.HoodieAvroUtilsandHoodieArrayWritableSchemaUtilsrewrite a decimal to a plain int/long target that way.Tests:
TestHoodieAvroParquetReader(both modules) read INT32/INT64 decimal files written with parquet-mr's example writer against fixed, bytes, file-derived and reordered reader schemas, with and without dictionary encoding, plus nulls, a nested record, a wider and a narrower scale, and the array rejection. Without the fix they fail withCannot encode decimal with precision 11 as max precision 9, aLongWritablewhere aHiveDecimalWritableis expected, andcannot support rewrite value for schema type ... decimal ... since the old schema type is: "int". The twoTestAvroSchemaConvertercases that pinned the lossy int/long conversion now expect the decimal.Impact
Reader-only. Base files with FIXED_LEN_BYTE_ARRAY decimals, which is everything Hudi writes today, take the same path as before. Two visible changes for files with integer-backed decimals: schema inference from such a file now yields a decimal instead of int/long, and a projection containing such a column is read with parquet-avro's regular record converter rather than its compat converter (strings arrive as
Utf8instead ofStringon paths that default to compat, such as Hive).ParquetUtils.readAvroRecordsstill uses the stock parquet-avro reader and is left for a follow-up.Risk Level
low. The new converter only engages when the requested projection contains an INT32/INT64 decimal column. The Hive reader additionally picks up
HoodieAvroReadSupport's legacy list and map handling; the hudi-hadoop-common and hudi-hadoop-mr suites were run against the change.Documentation Update
none
Contributor's checklist