ArrowMessage.frame() does not skip unknown FlightData fields #1337

ArrowMessage.frame(BufferAllocator, InputStream) (flight/flight-core/src/main/java/org/apache/arrow/flight/ArrowMessage.java) is a hand-written parser for FlightData. Its switch (tag) ends with:

default:
  // ignore unknown fields.

but it only consumes the tag; the field's payload is never skipped. The loop then reads the payload bytes as the next tag. Payload bytes 0x0A, 0x12 and 0x1A equal DESCRIPTOR_TAG, HEADER_TAG and APP_METADATA_TAG, so the payload of an unknown field can be misread as a descriptor, an IPC header or application metadata. Depending on how the lengths line up this raises a confusing error or silently replaces data. A known field number arriving with an unexpected wire type falls into the same default (the switch matches the whole tag).

Protobuf requires unknown fields to be skipped. The C++ fast path does it correctly (WireFormatLite::SkipField in cpp/src/arrow/flight/serialization_internal.cc in apache/arrow). apache/arrow#26885 (ARROW-10962) already aligned this parser with protobuf semantics for missing fields.

Reproduction: parse, through ArrowMessage.createMarshaller(allocator).parse(...), a frame holding a valid descriptor, header and app_metadata plus an unknown field, e.g. a varint field 4 with value 10, or a length-delimited field 5 whose payload is 0A 12 1A .... The result differs from, or fails to parse compared with, the same frame without the unknown field.

Impact: any peer sending extra fields can break Java receivers on this path, and it blocks adding fields to FlightData (for example a sequence number or a control message).

Version: main (20.0.0-SNAPSHOT), any platform.

🤖 This issue was drafted and posted by Claude Code on Laurent's behalf.