Repository navigation
[AURON #2624] Support Spark's array_except natively - #2648
Open
zoddtheimmortal wants to merge 4 commits into
Open
zoddtheimmortal wants to merge 4 commits into
zoddtheimmortal wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block approval of the native array_except implementation.
No open issues 🎉
What changed in this PR
Adds native execution of Spark’s array_except in Auron, reducing fallback for supported atomic element types while preserving Spark’s ordering, deduplication, null, and floating-point semantics.
Changes:
- Connects Spark expression conversion to the new native function, retaining fallback for unsupported types and Spark 3.0 floating-point arrays.
- Implements hash-based array subtraction and shares type checks and NaN canonicalization with
array_distinct. - Adds Rust unit tests and Spark result, native-plan, and fallback checks.
| File | Description |
|---|---|
| spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala | Adds ArrayExcept conversion and shared set-array type checks. |
| spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronFunctionSuite.scala | Tests Spark-compatible results, native execution, and fallback behavior. |
| native-engine/datafusion-ext-functions/src/spark_array.rs | Implements native array subtraction, extracts NaN handling, and adds unit tests. |
| native-engine/datafusion-ext-functions/src/lib.rs | Registers Spark_ArrayExcept in the native function dispatcher. |
Review effort: Balanced
1 of 2 tasks
Copilot stopped reviewing on behalf of
zoddtheimmortal due to an error
November 8, 2026 09:57
Contributor
There was a problem hiding this comment.
Note
Copilot was unable to run its full agentic suite in this review.
Copilot review overview
4 open issues
array_distinct/union/except) only canonicalizes NaNs but never… · Newseenis repeatedly repopulated from the right-side elements but is… · Newseenis repeatedly repopulated from the right-side elements but is… · New
Review effort: Lite
Comment on lines
+41
to
+57
| // Spark canonicalizes NaNs but keeps signed zeros distinct before Spark 4.2's | ||
| // normalization. | ||
| fn canonicalize_nans(values: &ArrayRef) -> Result<ArrayRef> { | ||
| Ok(match values.data_type() { | ||
| DataType::Float32 => Arc::new(Float32Array::from_iter( | ||
| downcast_any!(values, Float32Array)? | ||
| .iter() | ||
| .map(|v| v.map(|v| if v.is_nan() { f32::NAN } else { v })), | ||
| )), | ||
| DataType::Float64 => Arc::new(Float64Array::from_iter( | ||
| downcast_any!(values, Float64Array)? | ||
| .iter() | ||
| .map(|v| v.map(|v| if v.is_nan() { f64::NAN } else { v })), | ||
| )), | ||
| _ => Arc::clone(values), | ||
| }) | ||
| } |
| let mut offsets = Vec::with_capacity(num_rows + 1); | ||
| let mut valids = Vec::with_capacity(num_rows); | ||
| // Seeded with the right elements, so each left element needs one lookup. | ||
| let mut seen = HashSet::new(); |
| if valid { | ||
| seen.clear(); | ||
| let right_start = right.value_offsets()[r] as usize - right_base; | ||
| let right_end = right.value_offsets()[r + 1] as usize - right_base; |
Comment on lines
+120
to
+123
| private def isSupportedSetArrayType(dataType: DataType): Boolean = dataType match { | ||
| // Spark 3.0 has different NaN deduplication behavior. | ||
| case ArrayType(FloatType | DoubleType, _) => | ||
| SparkVersionUtil.SPARK_RUNTIME_VERSION >= "3.1" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #2624
Rationale for this change
Spark's
array_exceptcurrently falls back to Spark.What changes are included in this PR?
ArrayExceptto a new nativeSpark_ArrayExceptfunction.-0.0and0.0stay distinct; Spark 4.2 normalizes them before this function.ArrayDistinctintoisSupportedSetArrayType, shared by both functions.array_distinct's NaN canonicalization intocanonicalize_nans, shared by both functions.Are there any user-facing changes?
array_exceptcan execute natively for supported atomic element types. Nested element type keeps the fallback, and so does floating point arrays on Spark 3.0.How was this patch tested?
All passed.
AuronFunctionSuite: 97 succeeded, 0 failed on each Spark version.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5 on GitHub Copilot CLI 1.0.105
ASF guidance: https://www.apache.org/legal/generative-tooling.html