Skip to content

test(dist): cover streamed unpacking of large files - #5195

Open
anugrah55 wants to merge 1 commit into
rust-lang:mainfrom
anugrah55:streaming-unpack-test
Open

anugrah55 wants to merge 1 commit into
rust-lang:mainfrom
anugrah55:streaming-unpack-test

Conversation

@anugrah55

@anugrah55 anugrah55 commented Nov 6, 2026 •

Copy link
Copy Markdown

Closes #2754

Adds a unit test in src/dist/component/package.rs for the streaming unpack path. The test builds an in-memory tar with one small file and one file of IO_CHUNK_SIZE + 1 bytes, then unpacks it on both the immediate (1-thread) and threaded (2-thread) executors and checks both files byte for byte.

The large file uses a repeating i % 251 pattern rather than zeros, so the NUL-bytes bug from 98365c3 is caught if it comes back. The test uses IoThreadCount::UserSpecified because with Default(2), get_executor falls back to one thread when the RAM budget (32 MiB here) is below LOW_MEMORY_THRESHOLD (512 MiB).

Tested on macOS aarch64: the full test suite (150 unit + 543 integration) passed; clippy and xfmt are clean.

AI disclosure: Claude (AI) wrote the test code and drafted this description.

@rami3l rami3l left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

#[test]
fn unpack_small_and_streamed_files_immediate() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Test functions can actually return Result<> so there should be no use in unwrapping here.

/// Unpack one file below and one above `IO_CHUNK_SIZE`, so that both the
/// whole-file and the streaming paths are taken, and check their contents.
fn unpack_small_and_streamed_files(io_threads: usize) -> anyhow::Result<()> {
let small = b"small file contents".to_vec();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .to_vec() call here is redundant since later on this variable is used as &small.

Comment on lines +479 to +481
let large = (0..IO_CHUNK_SIZE + 1)
.map(|i| (i % 251) as u8)
.collect::<Vec<_>>();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Consider using array::from_fn() here so you can avoid making a Vec<>.

Comment on lines +477 to +478
// Not all zeros, since a broken stream used to produce NUL bytes
// (#2754), and with a period that doesn't divide the chunk size.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and with a period that doesn't divide the chunk size.

I'm not super sure what this means. Suggest enrich the comments a bit?


/// Unpack one file below and one above `IO_CHUNK_SIZE`, so that both the
/// whole-file and the streaming paths are taken, and check their contents.
fn unpack_small_and_streamed_files(io_threads: usize) -> anyhow::Result<()> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: As per our coding style, the helper function should be put below its callers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test needed - Verify streaming file installation

2 participants