Repository navigation
Conversation
… 2.0
Paimon 1.3+ is binary incompatible with code compiled against 1.2
(DataFileMeta became an interface and DataFileMeta.forAppend gained
parameters), so a module compiled against 1.2 fails at planning time with
IncompatibleClassChangeError/NoSuchMethodError on Paimon 2.0. Such
LinkageErrors are not caught by the conversion fallback.
- add a paimon-2.0 profile compiling the module against Paimon 2.0.0, using
the Scala-suffixed paimon-spark artifact introduced in 1.3
- support Paimon version specific sources (src/{main,test}/paimon-<ver>/scala)
and include them in spotless/scalastyle and dev/reformat
- expose the profile through auron-build.sh and the Paimon CI matrix
…th Paimon 1.4+ apache/paimon#7098 (Paimon 1.4) changed the __paimon_file_path metadata column from path.toUri.toString to path.toString. They differ for paths with characters that need escaping (e.g. a '%' in a partition value), so the native scan returned a different value than the Paimon reader on Paimon 2.0. Follow the Paimon version found on the classpath.
Execute DataSourceV2 writes (spark.paimon.write.use-v2-write=true) into Paimon bucket-unaware append-only parquet tables with a native sink. Requires the paimon-2.0 profile; disabled by default (spark.auron.enable.paimon.write). - native PaimonSinkExec: one open file per partition, size based rolling at target-file-size, bounded open files / row group memory, Paimon field ids as parquet field ids - data file naming, column statistics and commit stay in Paimon: paths from FileStorePathFactory, stats from Paimon's SimpleStatsExtractor, commit via Paimon's BatchWrite (overwrite semantics unchanged) - unsupported tables (primary key, fixed/dynamic bucket, deletion vectors, row tracking, data evolution, clustering, file index, external paths, merge-schema, snapshot sequence init, non-parquet, nested/timestamp types, row level operations) fall back to the Paimon writer
…mon write INSERT ... SELECT into a wide table typically pads most columns with constants. When such a projection sits on top of a source that cannot be read natively, converting the projection output to Arrow carries every padded column (201 instead of 18 on a production job), which cancelled the gain of the native writer. Convert the projection's child instead and run the projection natively; fall back to converting the whole input when the projection cannot be executed natively. Controlled by spark.auron.paimon.write.narrowConvert (default true).
github-actions
Bot
added
documentation
Unsupported Paimon writes were rejected in isSupported() with a WARN log, so the plan only showed 'not marked, default to NeverConvert'. Follow the Iceberg and Hudi scans instead: claim every Paimon V2 write, reject unsupported tables in convert() with an AssertionError, and let Auron record the reason in auron.never.convert.reason. The fallback tests now assert the recorded reason.
Unsupported Paimon writes were rejected in isSupported() with a WARN log, so the plan only showed 'not marked, default to NeverConvert'. Follow the Iceberg and Hudi scans instead: claim every Paimon V2 write, reject unsupported tables in convert() with an AssertionError, and let Auron record the reason in auron.never.convert.reason. The fallback tests now assert the recorded reason.
This comment was marked as resolved.
This comment was marked as resolved.
Contributor
Author
|
Update: this is done. I merged the latest master into #2637, #2639 and #2640, and adapted Sorry, I should have posted a follow-up here instead of just hiding the comment above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #2636 (follow-up to #2639)
Rationale for this change
INSERT INTO wide_table SELECT ...usually pads most target columns with constants or NULLs.When the source cannot be read natively, the padding
Projectruns in the JVM, and therow→Arrow conversion in front of the native writer has to carry every padded column. In our
job that is 201 columns, while only 18 are actually read. This conversion took about 36% of
task time and cancelled the gain of the native writer.
What changes are included in this PR?
NativePaimonWriteExec: if its input is not native and is aProjectExec(possibly wrapped by AQE /
WholeStageCodegenExec/InputAdapter) that outputs morecolumns than it reads, convert the project's child to native and run the project natively.
If the project cannot be converted, the whole input is converted as before.
spark.auron.paimon.write.narrowConvert, default true. Setting it to falserestores the previous behaviour.
Are there any user-facing changes?
The new config above. Results are unchanged; only where the row→columnar conversion happens
moves.
How was this patch tested?
AuronPaimonWriteSuite(11 tests in total with this PR): a padding projection over a non-native source. Itasserts that the narrow path was taken (the converted schema has only the read columns) and
compares the result with Paimon's writer.
16.54 task-hours. Output is identical: row count, all-column hash, key distinct count. In the
event log the RDD chain is
ConvertToNative → Project → PaimonSink.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Hermes Agent 0.22.2 (Claude Opus 5.5)