[Feature] Implement DataTypeObjectSerDe::read_column_from_arrow #42406

Activity

1. xx-derek

   changed the title [-][Feature] Implement `DataTypeObjectSerDe::write_column_to_arrow`[/-] [+][Feature] Implement `DataTypeObjectSerDe::read_column_to_arrow`[/+] on Oct 24, 2024
2. xx-derek

   changed the title [-][Feature] Implement `DataTypeObjectSerDe::read_column_to_arrow`[/-] [+][Feature] Implement `DataTypeObjectSerDe::read_column_from_arrow`[/+] on Oct 24, 2024
3. wyxxxcat commented on Oct 27, 2024

   

   wyxxxcat

   on Oct 27, 2024

   Collaborator

   More actions

   Can you give me a reproducible statement?
4. xx-derek commented on Nov 11, 2024

   

   xx-derek

   on Nov 11, 2024

   Author

   More actions

   Please refer to the following steps to reproduce.

   1. create test table

   CREATE TABLE `demo` (
     `id` int NULL,
     `value` variant NULL
   )
   DUPLICATE KEY(`id`)
   DISTRIBUTED BY RANDOM BUCKETS 16

   2. generate data in arrow stream format

   import numpy as np
   import pyarrow as pa
   
   ids = pa.array(np.arange(100))
   values = pa.array([f'{{\"a\": {i}}}' if i % 2 == 0 else f'{{\"a\": \"{i}\"}}' for i in range(100)])
   
   schema = pa.schema([
       pa.field('id', pa.int32()),
       pa.field('value', values.type)
   ])
   
   with pa.OSFile('payload.dat', 'wb') as sink:
       with pa.ipc.new_stream(sink, schema=schema) as writer:
           batch = pa.record_batch([ids, values], schema=schema)
           writer.write(batch)
   
   print('Done')

   Here I'm populating json strings to value field which is VARIANT in Doris table. The type for this field in arrow structure does not matter in this case.

   3. initiate stream load

   curl --location-trusted -u user:pass \
   -H "Expect:100-continue" \
   -H "format:arrow" \
   -T payload.dat \
   -XPUT http://host:port/api/db_name/demo/_stream_load

   The stream load operation will fail with error message "Failed to convert from arrow to block: [E3] read_column_from_arrow with type variant".
5. xx-derek commented on Nov 12, 2024

   

   xx-derek

   on Nov 12, 2024

   Author

   More actions

   I'm working on a patch. Can I take this?
6. github-actions commented on Sep 12, 2025

   

   github-actionsBot

   on Sep 12, 2025 – with GitHub Actions

   Contributor

   More actions

   We're closing this issue because it hasn't been updated in a while.
   If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!
7. github-actions

   added

   Stale

   on Sep 12, 2025
8. morningman

   closed this as completedon Sep 15, 2025