Skip to content

[SPARK-60484][SQL] Keep the sign of CAST(string AS INTERVAL SECOND) for values between -1 and 0 - #59725

Open
ykisana wants to merge 1 commit into
apache:masterfrom
ykisana:SPARK-60484
Open

ykisana wants to merge 1 commit into
apache:masterfrom
ykisana:SPARK-60484

Conversation

@ykisana

@ykisana ykisana commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

IntervalUtils.parseSecondNano took the sign of the fractional part from the parsed whole seconds:

val seconds = parseSeconds(secondsStr)
Math.addExact(seconds, parseNanos(nanosStr, seconds < 0))

When casting a string to INTERVAL SECOND, a negative input such as -0.5 reaches this method as "-0.5". Its whole seconds parse to 0, so seconds < 0 is false and the fraction is added as a positive value. This PR takes the sign from the seconds string instead (secondsStr.startsWith("-")). For any input whose whole seconds are non-zero, the result is unchanged.

The other callers of parseSecondNano (toDTInterval for ... TO SECOND types) pass unsigned seconds and apply the sign outside, so they are not affected.

Why are the changes needed?

Every value between -1 and 0 seconds loses its sign when cast from a string:

SELECT CAST('-0.5' AS INTERVAL SECOND);
-- before: INTERVAL '00.5' SECOND
-- after:  INTERVAL '-00.5' SECOND

Spark formats such an interval as INTERVAL '-00.5' SECOND, and the JSON and CSV readers parse INTERVAL SECOND with the same code. As a result, writing -0.5 seconds to JSON or CSV and reading it back returns +0.5 seconds.

Does this PR introduce any user-facing change?

Yes, it fixes a bug. Casting a string to INTERVAL SECOND now keeps the sign of values between -1 and 0 seconds, as the INTERVAL '-0.5' SECOND literal already does. JSON and CSV round trips of such values are fixed as well. The bug reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.

How was this patch tested?

Added SPARK-60484: Cast string to INTERVAL SECOND keeps the sign of values in (-1, 0) to CastSuiteBase. It covers:

  • plain strings (-0.5, -00.5, -0.000001, -0.999999999, and 0.5 as a positive control);
  • the literal forms (INTERVAL '-0.5' SECOND, INTERVAL -'0.5' SECOND, INTERVAL -'-0.5' SECOND);
  • an interval -> string -> interval round trip.

Without the fix, the new test fails in all three suites with cast(-0.5 as interval second), actual: 500000, expected: -500000.

build/sbt 'catalyst/testOnly *CastWithAnsiOffSuite *CastWithAnsiOnSuite *TryCastSuite *IntervalUtilsSuite'

All 458 tests in these four suites pass with the fix. Scalastyle (build/sbt catalyst/scalastyle catalyst/Test/scalastyle) reports 0 errors.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)
Fix, tests, PR description, and comments by Claude.
All code manually reviewed by @ykisana.

…or values between -1 and 0

IntervalUtils.parseSecondNano took the sign of the fractional part from
the parsed whole seconds (`seconds < 0`). For an input such as "-0.5"
the whole seconds are 0, so the fraction was added as positive and
CAST('-0.5' AS INTERVAL SECOND) returned 0.5 seconds. Because Spark
formats such a value as INTERVAL '-00.5' SECOND and the JSON/CSV
readers use the same parser, round trips also flipped the sign.

Take the sign from the seconds string instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant