Skip to content

[client-v2] Enum16 folded to 8-bit #3342

Closed

Description

@chernser

Description

Original ClickHouse/clickhouse-js#1130

Activity

  1. added theissue type on Nov 13, 2026
  2. chernser commented on Nov 13, 2026

    @chernser
    ContributorAuthor

    @polyglotAI-bot please do preliminary investigation.

  3. polyglotAI-bot commented on Nov 13, 2026

    @polyglotAI-bot
    Collaborator

    Yes. client-v2 has the same defect for the spellings enum(...) and ENUM(...): the client always reads them as Enum8, so an ordinal outside -128..127 silently shifts the row. The canonical spelling Enum(...) does not fold: read and write fail with Unsupported data type: Enum. Only type strings written by the application are affected.

    Checked on main (f73e372bc) and ClickHouse 26.9.1.1629. The released v0.11.1 has the same code.

    Findings

    1. enum(...) / ENUM(...) → always Enum8 (silent wrong data on read). ClickHouseDataType.Enum8 declares the alias "ENUM" (clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java:61, since 2023-01), and ClickHouseDataType.of() falls back to the upper-case name (:493). The server types enum('a' = 1000) as Enum16. The reader reads 1 of 2 bytes, and the next column reads the high byte. 2 server rows give 3 client rows, without an error. When the total byte count is odd, the read ends with an EOFException.
      • Write: for ordinal 1000, the Int8 range check throws. When the written value is small but the column is Enum16 (mixed ordinals), the client writes 1 byte and the server rejects the insert (CANNOT_READ_ALL_DATA).
    2. Enum(...) → ClickHouseDataType.Enum, which has no width (ClickHouseDataType.java:60, case-sensitive). BinaryStreamReader.readValue (client-v2/.../internal/BinaryStreamReader.java:301) and SerializerUtils.serializeData (client-v2/.../internal/SerializerUtils.java:784) have no case for it. Both throw, also for Enum('a' = 1). This is not silent, but a bare Enum does not work in a schema written by hand.
    3. enum16(...) → Unknown data type: enum16. The server accepts it (Enum16('a' = 1000)). Enum8/Enum16 are case-sensitive in ClickHouseDataType and have no alias.

    Where it is reachable: only through ClickHouseColumn.of(name, type) in a TableSchema that the application builds, for example for Client.newBinaryFormatReader(response, schema) on RowBinary without a header, RowBinaryFormatWriter, or register(Class, schema). A schema from the server (RowBinaryWithNamesAndTypes / Native header, getTableSchema) always spells Enum8/Enum16, and these read correctly (rows Enum16('a' = 1000) below).

    Suggested fix (same rule as the server and ClickHouse/clickhouse-js#1131): in ClickHouseColumn.update() (clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseColumn.java:177-181), for a bare enum name (Enum / enum / ENUM), set the data type from the ordinals: Enum16 if a value is outside -128..127, else Enum8. Explicit Enum8/Enum16 stay as they are. As an option, accept enum8/enum16 without case sensitivity, as the server does. I do not expect a breaking change: each input that changes behavior throws or returns wrong data today.

    Open: the parser is in clickhouse-data, which jdbc-v2 and v1 share. I did not test those clients. #3304 and PR #3312 (enum member validation on insert) are a different problem.

    Output
    server 26.9.1.1629
    1) server type name | client-v2 ClickHouseColumn.of(...).getDataType()
      Enum('a' = 1)        server=Enum8('a' = 1)         client=Enum
      Enum('a' = 1000)     server=Enum16('a' = 1000)     client=Enum
      enum('a' = 1)        server=Enum8('a' = 1)         client=Enum8
      enum('a' = 1000)     server=Enum16('a' = 1000)     client=Enum8
      ENUM('a' = 1000)     server=Enum16('a' = 1000)     client=Enum8
      Enum8('a' = 1)       server=Enum8('a' = 1)         client=Enum8
      Enum16('a' = 1000)   server=Enum16('a' = 1000)     client=Enum16
      enum16('a' = 1000)   server=Enum16('a' = 1000)     client=error: Unknown data type: enum16
    2) RowBinary read, hand-written schema [e <type>, u UInt8], expect one row [a, 7]
      read  Enum('a' = 1)        x1 -> rows=0  then error: ClientException: Failed to read value for column e <- IllegalArgumentException: Unsupported data type: Enum
      read  Enum('a' = 1000)     x1 -> rows=0  then error: ClientException: Failed to read value for column e <- IllegalArgumentException: Unsupported data type: Enum
      read  enum('a' = 1)        x1 -> rows=1 [a, 7]
      read  enum('a' = 1000)     x1 -> rows=0  then error: ClientException: Reading  row 2 (time since last next call 1) <- IOException: Reading column 1 (`u`) in  row 2 (time since last next call 1) <- EOFException: End of stream reached before reading all data
      read  ENUM('a' = 1000)     x1 -> rows=0  then error: ClientException: Reading  row 2 (time since last next call 0) <- IOException: Reading column 1 (`u`) in  row 2 (time since last next call 0) <- EOFException: End of stream reached before reading all data
      read  Enum8('a' = 1)       x1 -> rows=1 [a, 7]
      read  Enum16('a' = 1000)   x1 -> rows=1 [a, 7]
      read  enum16('a' = 1000)   x1 -> rows=0  then error: IllegalArgumentException: Unknown data type: enum16
    2b) same read, 2 source rows (expect [a, 7], [a, 7])
      read  enum('a' = 1000)     x2 -> rows=3 [<unknown>, 3], [<unknown>, 232], [<unknown>, 7]
      read  Enum16('a' = 1000)   x2 -> rows=2 [a, 7], [a, 7]
    3) RowBinary write into (e Enum16('a' = 1000), u UInt8), expect stored a,7
      write Enum('a' = 1000)     -> error: UnsupportedOperationException: Unsupported data type: Enum
      write enum('a' = 1000)     -> error: IllegalArgumentException: int(1000) should be between -128 and 127 inclusive of both values
      write Enum16('a' = 1000)   -> bytes=[e8 03 07] stored a,7
    3b) RowBinary write 'a' into (e Enum16('a' = 1, 'b' = 1000), u UInt8), expect stored a,7
      write enum('a' = 1, 'b' = 1000) -> bytes=[01 07] server error: ServerException: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 1: (at row 1) : While executing WaitForAsyncInsert. (CANNOT_READ_ALL_DATA) (version 26.9.1.1629 (official build))
      write Enum16('a' = 1, 'b' = 1000) -> bytes=[01 00 07] stored a,7
    
    Reproduction program (java -cp client-v2.jar:<deps> Repro.java, after mvn -pl client-v2 -am install -DskipTests)
    import com.clickhouse.client.api.Client;
    import com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader;
    import com.clickhouse.client.api.data_formats.RowBinaryFormatWriter;
    import com.clickhouse.client.api.metadata.TableSchema;
    import com.clickhouse.client.api.query.QueryResponse;
    import com.clickhouse.client.api.query.QuerySettings;
    import com.clickhouse.data.ClickHouseColumn;
    import com.clickhouse.data.ClickHouseFormat;
    
    import java.io.ByteArrayInputStream;
    import java.io.ByteArrayOutputStream;
    import java.util.Arrays;
    import java.util.List;
    
    public class Repro {
        static Client client;
    
        static String sql(String q) {
            try {
                List<com.clickhouse.client.api.query.GenericRecord> rows = client.queryAll(q);
                StringBuilder sb = new StringBuilder();
                for (com.clickhouse.client.api.query.GenericRecord rec : rows) {
                    if (sb.length() > 0) sb.append(" | ");
                    for (int i = 1; i <= rec.getSchema().getColumns().size(); i++) {
                        sb.append(i > 1 ? "," : "").append(rec.getString(i));
                    }
                }
                return sb.toString();
            } catch (Exception e) {
                return "server error: " + rootMessage(e).split("\n")[0];
            }
        }
    
        static void exec(String q) throws Exception {
            client.query(q).get().close();
        }
    
        static String rootMessage(Throwable e) {
            Throwable c = e;
            StringBuilder sb = new StringBuilder();
            while (c != null) {
                if (sb.length() > 0) sb.append(" <- ");
                sb.append(c.getClass().getSimpleName()).append(": ").append(c.getMessage());
                c = c.getCause();
            }
            return sb.toString();
        }
    
        static String lit(String type) {
            return "'" + type.replace("'", "\\'") + "'";
        }
    
        static TableSchema schema(String... nameType) {
            ClickHouseColumn[] cols = new ClickHouseColumn[nameType.length / 2];
            for (int i = 0; i < cols.length; i++) {
                cols[i] = ClickHouseColumn.of(nameType[2 * i], nameType[2 * i + 1]);
            }
            return new TableSchema(Arrays.asList(cols));
        }
    
        static void read(String type, int ordinal, int sourceRows) {
            String q = "SELECT CAST(" + ordinal + ", " + lit(type) + ") AS e, toUInt8(7) AS u FROM numbers(" + sourceRows + ") FORMAT RowBinary";
            StringBuilder out = new StringBuilder();
            int rows = 0;
            try (QueryResponse r = client.query(q, new QuerySettings().setFormat(ClickHouseFormat.RowBinary)).get()) {
                ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(r, schema("e", type, "u", "UInt8"));
                while (reader.hasNext() && rows < 5) {
                    reader.next();
                    rows++;
                    out.append(rows > 1 ? ", " : "").append("[").append(reader.getString(1))
                            .append(", ").append(reader.getShort(2)).append("]");
                }
                System.out.printf("  read  %-20s x%d -> rows=%d %s%n", type, sourceRows, rows, out);
            } catch (Exception e) {
                System.out.printf("  read  %-20s x%d -> rows=%d %s then error: %s%n", type, sourceRows, rows, out, rootMessage(e));
            }
        }
    
        static void write(String clientType) throws Exception {
            write(clientType, "Enum16('a' = 1000)");
        }
    
        static void write(String clientType, String tableType) throws Exception {
            exec("DROP TABLE IF EXISTS t3342");
            exec("CREATE TABLE t3342 (e " + tableType + ", u UInt8) ENGINE = Memory");
            ByteArrayOutputStream buf = new ByteArrayOutputStream();
            try {
                RowBinaryFormatWriter w = new RowBinaryFormatWriter(buf, schema("e", clientType, "u", "UInt8"), ClickHouseFormat.RowBinary);
                w.setValue("e", "a");
                w.setValue("u", (short) 7);
                w.commitRow();
            } catch (Exception e) {
                System.out.printf("  write %-20s -> error: %s%n", clientType, rootMessage(e));
                return;
            }
            byte[] bytes = buf.toByteArray();
            StringBuilder hex = new StringBuilder();
            for (byte b : bytes) hex.append(String.format("%02x ", b));
            String result;
            try {
                client.insert("t3342", new ByteArrayInputStream(bytes), ClickHouseFormat.RowBinary).get().close();
                result = "stored " + sql("SELECT e, u FROM t3342").replace("\t", ",").replace("\n", " | ");
            } catch (Exception e) {
                result = "server error: " + rootMessage(e).split("\n")[0];
            }
            System.out.printf("  write %-20s -> bytes=[%s] %s%n", clientType, hex.toString().trim(), result);
        }
    
        public static void main(String[] args) throws Exception {
            client = new Client.Builder()
                    .addEndpoint("http://clickhouse:18123")
                    .setUsername("default")
                    .setPassword("test_default_password")
                    .build();
    
            System.out.println("server " + sql("SELECT version()"));
            List<String> types = Arrays.asList(
                    "Enum('a' = 1)", "Enum('a' = 1000)",
                    "enum('a' = 1)", "enum('a' = 1000)", "ENUM('a' = 1000)",
                    "Enum8('a' = 1)", "Enum16('a' = 1000)", "enum16('a' = 1000)");
    
            System.out.println("1) server type name | client-v2 ClickHouseColumn.of(...).getDataType()");
            for (String t : types) {
                String dt;
                try {
                    dt = ClickHouseColumn.of("e", t).getDataType().name();
                } catch (Exception e) {
                    dt = "error: " + e.getMessage();
                }
                String ordinal = t.contains("1000") ? "1000" : "1";
                System.out.printf("  %-20s server=%-22s client=%s%n", t,
                        sql("SELECT toTypeName(CAST(" + ordinal + ", " + lit(t) + "))"), dt);
            }
    
            System.out.println("2) RowBinary read, hand-written schema [e <type>, u UInt8], expect one row [a, 7]");
            for (String t : types) {
                read(t, t.contains("1000") ? 1000 : 1, 1);
            }
    
            System.out.println("2b) same read, 2 source rows (expect [a, 7], [a, 7])");
            for (String t : Arrays.asList("enum('a' = 1000)", "Enum16('a' = 1000)")) {
                read(t, 1000, 2);
            }
    
            System.out.println("3) RowBinary write into (e Enum16('a' = 1000), u UInt8), expect stored a,7");
            for (String t : Arrays.asList("Enum('a' = 1000)", "enum('a' = 1000)", "Enum16('a' = 1000)")) {
                write(t);
            }
            System.out.println("3b) RowBinary write 'a' into (e Enum16('a' = 1, 'b' = 1000), u UInt8), expect stored a,7");
            for (String t : Arrays.asList("enum('a' = 1, 'b' = 1000)", "Enum16('a' = 1, 'b' = 1000)")) {
                write(t, "Enum16('a' = 1, 'b' = 1000)");
            }
            exec("DROP TABLE IF EXISTS t3342");
            client.close();
        }
    }
  4. chernser commented on Nov 13, 2026

    @chernser
    ContributorAuthor

    @polyglotAI-bot
    Thanks! Prepare the fix that do what is suggested in your comment.
    Make sure having integration tests especially with dynamic and variant types. Add needed metadata to com.clickhouse.client.api.data_formats.internal.BinaryStreamReader.EnumValue so when we read it from stream we know size of 8 or 16

  5. added this to the 0.12.0 milestone on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions