Repository navigation
Decode the whole file schema when a file opens - #11578
connortsui20 wants to merge 1 commit into
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
|
@claude review this, and also summarize the actual logical changes here because theres a bunch of refactoring in here as well |
|
Claude finished @connortsui20's task in 47s —— View job Review of #11578I read the whole diff and checked every caller of What actually changes in behaviourMost of the 228 changed lines are a refactor. Only two things change behaviour:
Everything else gives the same result as before:
ReviewThe change looks correct, and I found no bugs. Details:
Nits, all optional:
Overall it looks good to merge. |
Merging this PR will degrade performance by 0.82%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Summary
StructFields::fields()on malformed file: unwraps a fallible DType decode #8848A file whose schema has a struct field dtype that fails to decode, such as one with an unknown type tag, panics when it opens instead of returning an error. Struct field dtypes are decoded lazily, and
StructFields::fields()can't return an error, so it callsvortex_expecton the decode. While the file opens, the footer reads the file statistics against the schema, which reaches that accessor.Changes
The file reader now decodes the whole schema once when the file opens, before the statistics and the layout, and returns the first decode error. The alternative was to make the field accessors fallible, which changes an accessor with many callers and leaves each caller to handle an error that only a corrupt file can produce. Making only the statistics path fallible isn't enough either, since the footer drops statistics errors on purpose, so a broken schema would get through a successful open and fail later in a scan. The other users of
DType::from_flatbuffer(IPC, Python, and CUDA) stay lazy, and a dtype or footer that the caller supplies throughwith_dtypeorwith_footerstill skips the embedded schema.Lazy field dtypes exist for wide schemas (#1826), so the open time of files with 10,000
i64columns was measured before and after. A nested schema opened 0.31 ms slower without file statistics and 0.29 ms faster with them, and the flat schema showed no clear change. The full numbers are below. The eager decoder also reads straight from the footer bytes, so the copy of the dtype buffer is gone.The new tests open a file whose field tag was changed after it was written, once with file statistics and once without, and expect the decode error in both cases.
Wide-file open timings
Apple M4 Max, 128 GB RAM, macOS 27.0.1, rustc 1.100.0, bench profile. Each file has one row and 10,000 non-nullable
i64leaves. The flat schema has 10,000 root fields, and the nested schema has 100 root structs with 100 fields each. The timing usesVortexOpenOptions::open_bufferwith fresh options on every iteration and no cached footer. Each process warmed up with 40 opens per shape, then ran nine batches of 80 opens. Each cell gives the medians of two processes, run in before, after, after, before order. File generation, disk IO, and scans are outside the timed region.The flat file with statistics varied between the two runs after the change, so that row is inconclusive.