Repository navigation
[Bug] Arrow Flight SQL: Iceberg BINARY column is declared as Arrow string but carries invalid UTF-8 #67371
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
Apache Doris 4.1.3-rc02, commit 31263df4dc1d4d3a27517d264802cd4d6b92c874
Client: Python + ADBC Flight SQL driver (adbc_driver_flightsql), FE arrow_flight_sql_port = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.
What's Wrong?
When an Iceberg catalog is configured with "enable.mapping.varbinary" = "false", Doris Flight SQL declares an arbitrary BINARY column as Arrow string, but the payload actually contains non-UTF-8 bytes.
PyArrow fails before returning any row:
UnicodeDecodeError (byte 0x84)
The Arrow schema for the query is id: int32, col1: string, col2: string. JDBC reads the same Iceberg rows without a problem, and with VARBINARY mapping enabled the ADBC binary read works.
What You Expected?
The Arrow type declared by the server and the encoding of the payload must agree. Arbitrary binary content should be sent as binary/large_binary, or, if it is sent as string, it must be a valid UTF-8 representation.
How to Reproduce?
- Bring up the Iceberg regression fixture.
- Create or use an Iceberg catalog with
"enable.mapping.varbinary" = "false". SWITCHto that catalog andUSE test_varbinary.- Run the query over Python ADBC, inspect the schema and call
to_pylist().
SWITCH test_iceberg_no_mapping;
USE test_varbinary;
SELECT * FROM test_ice_uuid_orc ORDER BY id;Client side:
import adbc_driver_flightsql.dbapi as flight_sql
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
cur.execute("SELECT * FROM test_iceberg_no_mapping.test_varbinary.test_ice_uuid_orc ORDER BY id")
table = cur.fetch_arrow_table()
print(table.schema) # id: int32, col1: string, col2: string
print(table.to_pylist()) # UnicodeDecodeError on byte 0x84Anything Else?
Without VARBINARY mapping, the Iceberg binary/uuid column falls back to the Doris STRING type, and the Arrow conversion follows that declared type rather than the real content. Since Arrow string is defined as UTF-8, any consumer that validates the encoding (PyArrow does) fails on the whole batch.
Workaround: set "enable.mapping.varbinary" = "true" on the Iceberg catalog so the column is returned through the Arrow binary type.
Found with the external_table_p0/iceberg/test_iceberg_varbinary fixture; four no-mapping cases are affected, while the mapping-enabled binary reads keep passing over ADBC.
Tracking issue: #65615
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Sep 1, 2026
Initial assessment
Confirmed as a valid server-side Arrow schema/payload mismatch in the reported
4.1.3-rc02commit (31263df4dc1d4d3a27517d264802cd4d6b92c874). This is not only a Python presentation issue: Arrowstringcarries UTF-8 semantics, while this path can place arbitrary Iceberg binary bytes in its value buffer. A validating client can therefore reject the entire batch before exposing rows.The issue is actionable as written. The supplied fixture inserts non-UTF-8 sequences including byte
0x84, so no BE log or profile is needed to establish the type-contract problem.Code evidence
IcebergUtils.icebergPrimitiveTypeToDorisType()maps IcebergUUIDandBINARYto DorisSTRINGwhenenable.mapping.varbinary=false(fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:691-714).BINARYbecomesTYPE_STRINGwhen mapping is disabled (be/src/format/orc/vorc_reader.cpp:1705-1738), and raw ParquetBYTE_ARRAY/UUID similarly becomeTYPE_STRING(be/src/format/parquet/schema_desc.cpp:251-258,323-329; the v2 reader has the same mapping).be/src/exec/operator/result_sink_operator.cpp:58-67andbe/src/format/arrow/arrow_row_batch.cpp:220-238). At that point the original Iceberg binary identity is no longer present; DorisSTRINGis unconditionally mapped to Arrowutf8, while DorisVARBINARYmaps to Arrowbinary(arrow_row_batch.cpp:88-96,176-178).DataTypeStringSerDe, which appends the stored bytes unchanged toStringBuilder(be/src/core/data_type_serde/data_type_string_serde.cpp:399-445). The Flight conversion path does not run Arrow full validation before returning the batch. This directly explains why schema inspection reportsstringand PyArrow later raisesUnicodeDecodeError.enable.mapping.varbinary=true, the FE/reader type remainsVARBINARY, so the existingVARBINARY -> arrow::binary()path is internally consistent. The documented workaround is therefore sound.The MySQL/JDBC control path succeeding does not contradict this diagnosis: that protocol can transport length-delimited byte sequences under Doris string metadata and does not impose Arrow's UTF-8 invariant.
Fix boundary and open design decision
Changing only the string byte-copy loop is insufficient. The Arrow type must be selected before the stream schema is published, so the result schema builder needs binary semantic information that is currently lost between Iceberg schema mapping and the result sink.
Two viable directions should be evaluated:
binary(or fixed-size binary for UUID/FIXED where appropriate) for direct binary-origin results. The existing string SerDe already supports writing a Doris string column into binary/fixed-size-binary Arrow builders, so payload conversion support largely exists.VARBINARYfor Flight SQL even when the catalog uses the legacy no-mapping mode.The second option is simpler but may make SQL/result metadata protocol-dependent. The first option must define propagation through aliases, projections, expressions, nested fields, and casts so a genuine Doris
STRINGis never silently reclassified as binary. Silently hex/base64-encoding only on Flight would avoid invalid UTF-8 but would change values relative to the SQL result and should not be done without an explicit compatibility decision.Missing information / triage notes
adbc_driver_flightsqland PyArrow versions would help document the client compatibility matrix, but they are not blocking for accepting the bug because the server-side schema invariant is independently visible in code.enable.mapping.varbinary=false: byte-preserving binary output or a textual representation. That decision determines which fix direction is compatible.Recommended next steps
test_iceberg_varbinaryORC and Parquet fixtures with mapping disabled. It should assert the Arrow field type, call a validating consumer such as Python ADBC/PyArrowto_pylist()(orvalidate(full=True)), and compare the exact returned bytes. Keep mapping-enabled cases as controls.STRINGstill produces Arrow UTF-8 and binary-origin fallback data produces Arrow binary, including nullable values and both local/remote Flight readers.No code was modified during this analysis, and the reported failure was not independently rerun against a live cluster; the root-cause judgment above is based on the exact reported commit and the checked-in fixture bytes.
Breakwater-GitHub-Analysis-Slot: slot_c8ad7b1e212c
shiliang commentedon Sep 10, 2026
I'd like to work on this