Repository navigation
[Bug] Arrow Flight SQL: year-zero DATE values are encoded one day late #67366
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
Apache Doris 4.1.3-rc02, commit 31263df4dc1d4d3a27517d264802cd4d6b92c874
Client: Python + ADBC Flight SQL driver (adbc_driver_flightsql), FE arrow_flight_sql_port = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.
What's Wrong?
DATE values in year zero are shifted by one day when read over Arrow Flight SQL. 0000-01-01 (as returned by JDBC) is rendered as 0000-01-02 in Arrow, and 0000-02-28 is rendered as 0000-02-29. Modern dates are not affected.
The Flight SQL schema is date32[day] (and list<date32[day]> for arrays). The raw day value sent for 0000-01-01 is -719527, which PyArrow renders as 0000-01-02. 2024-01-01 matches JDBC exactly.
What You Expected?
ADBC/Arrow and JDBC should return the same DATE value. 0000-01-01 and 0000-02-28 should not be shifted by the calendar conversion.
How to Reproduce?
- Run the query over MySQL/JDBC and record the result.
- Run the same query over the Python ADBC Flight SQL driver.
- Compare the year-zero dates with the modern date.
SELECT
CAST('0000-01-01' AS DATE) AS d1,
CAST('0000-02-28' AS DATE) AS d2,
CAST('2024-01-01' AS DATE) AS modern_date;
SELECT [CAST('0000-01-01' AS DATE)] AS date_array;Client side:
import adbc_driver_flightsql.dbapi as flight_sql
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT CAST('0000-01-01' AS DATE), CAST('0000-02-28' AS DATE), CAST('2024-01-01' AS DATE)")
print(cur.fetch_arrow_table().to_pylist())Anything Else?
The off-by-one looks like a proleptic-Gregorian vs. Julian calendar mismatch in the day-number computation used to build the Arrow date32 value; Arrow date32 is defined as days since the epoch in the proleptic Gregorian calendar.
Workaround: CAST the column to STRING in Doris and parse it on the client. This bypasses the Arrow DATE encoding but loses the native date type.
Verified by comparing the raw Arrow day values against the JDBC results directly.
Tracking issue: #65615
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Sep 1, 2026
Initial assessment: confirmed server-side BE serialization bug at commit
31263df4dc1d4d3a27517d264802cd4d6b92c874. The raw value reported by the client is exactly what the Doris code produces, so additional logs or a Doris profile are not required to establish the initial root cause.Verified facts
DATEV2to Arrowdate32inconvert_to_arrow_type, and the Flight result converter delegates the value encoding to the type SerDe (call site).DataTypeDateV2SerDe::write_column_to_arrowsendsDateV2Value::daynr() - 719528directly toDate32Builder.daynr()uses Doris's MySQL-compatiblecalc_daynr. That internal numbering is not identical to Arrow's proleptic-Gregorian day ordinal around the start of year zero. Doris also deliberately treats year zero as non-leap inis_leap, while proleptic Gregorian year zero is divisible by 400.daynr()date32date320000-01-010000-02-280000-03-012024-01-01This confirms the reported values and narrows the affected range to valid Doris dates from
0000-01-01through0000-02-28; this is not a blanket shift for every year-zero date. The issue's calendar-mismatch hypothesis is directionally correct, but the precise problem is reusing the Doris/MySQL internal ordinal without an explicit calendar-boundary conversion.ARRAY<DATE>is affected through the same code path:DataTypeArraySerDedelegates each element to the nested DATE SerDe. There is no separate array root cause.Important reverse-path consideration
The fix should cover Arrow-to-Doris conversion as well as Doris-to-Arrow conversion. The current DATE32 reader adds
719528and callsget_date_from_daynr(code). A correct Arrow value of-719528for0000-01-01becomes internal day number0, which that API rejects;-719470for0000-02-28becomes58and decodes one day early. Fixing only the Flight output expression would therefore leave asymmetric round trips.Missing information
Nothing is blocking initial confirmation. For a complete compatibility record, the exact
adbc_driver_flightsqland PyArrow versions would still be useful. It would also be helpful to record raw values for0000-01-02,0000-03-01, and0001-01-01, but the checked source already predicts that only the pre-March year-zero boundary needs adjustment.Recommended next steps
calc_daynr, changing the epoch constant globally, or applying an unconditional-1: those approaches would break0000-03-01and/or the many SQL date operations that rely on Doris's existing numbering.write_column_to_arrowandread_column_from_arrow, covering0000-01-01,0000-02-28,0000-03-01,0001-01-01,1969-12-31,1970-01-01, and a modern date. Include scalar, nullable, and nested array cases.date32results with the MySQL/JDBC control path.daynr() - 719528, and Parquet/Arrow input paths add the same threshold before callingget_date_from_daynr. This is code evidence of potential shared boundary behavior, but those formats have not been reproduced end to end here.branch-4.1backport because the supplied reproducer is from 4.1.3-rc02.The current
CAST(... AS STRING)workaround is valid until a native-type fix is available. The issue currently has no labels;kind/bugandarea/rpcwould be appropriate if those labels remain the repository convention.Breakwater-GitHub-Analysis-Slot: slot_eb5d685887c8
[fix](arrow) encode DATE in the proleptic Gregorian calendar at forma…
[fix](orc) decode ORC DATE statistics with the row decoder's calendar
[fix](arrow) Correct year-zero DATE conversion on branch-4.1 (#68649)
[fix](arrow) Correct year-zero DATE conversion on branch-4.1 (#68649)