Skip to content

Replace randomized smoke scripts with deterministic CLI integration tests - #529

Open
rye wants to merge 1 commit into
pr/clap-4.6from
pr/integration-tests
Open

rye wants to merge 1 commit into
pr/clap-4.6from
pr/integration-tests

Conversation

@rye

@rye rye commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Replaces the randomized shell smoke tests (they used $RANDOM, and CI never ran them) with deterministic Rust integration tests in tests/cli.rs, resolving #101 and #1.

  • assert_cmd + predicates, with a fixture (tests/support/) that counts invocations in a temp dir and succeeds on the Nth call, so tests are fast and deterministic.
  • Covers: no command; success on the first try; fail twice then succeed; -x limits; invalid -x; a child killed by a signal; arguments after the command passed through verbatim.
  • Snapshot files for --version and the no-command error; tests/README.md explains how to update them.
  • Characterizes a bug, fixed in the next PR: exhausting -x exits 0.

Verification: fmt, clippy, tests (default and all features), Rust 1.88; repeated runs showed no flakiness (about 0.2 s per run).

Closes #101
Closes #1


Stack (merge bottom-up; each PR targets the one below it):

@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.64%. Comparing base (fc65217) to head (74f0b56).

Additional details and impacted files
@@               Coverage Diff                @@
##           pr/clap-4.6     #529       +/-   ##
================================================
+ Coverage        83.09%   99.64%   +16.54%     
================================================
  Files                4        4               
  Lines              278      278               
================================================
+ Hits               231      277       +46     
+ Misses              47        1       -46     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A moderate active finding remains because the Unix execution test fixture hard-codes /bin/sh, which can break on systems without that path (e.g., Nix-style layouts), so the change needs a more portable shell resolution approach.

1 open issue
What changed in this PR

Replaces legacy randomized shell smoke tests with deterministic Rust CLI integration tests and supporting fixtures/snapshots so coverage runs reliably in CI.

Changes:

  • Add assert_cmd/predicates-based CLI integration tests covering core retry and argument behaviors.
  • Introduce a deterministic Unix fixture and snapshot files for stable output assertions.
  • Remove obsolete tests/smoke/* shell scripts and document snapshot updates.
File Description
tests/​support/​fixture.rs Adds a Rust test fixture that sets up a tempdir-backed invocation counter and helpers to run the CLI against the deterministic child program.
tests/​support/​child.sh Adds a deterministic Unix shell child script used by integration tests to simulate repeated failures, eventual success, and signal termination.
tests/​snapshots/​version.stdout Adds a snapshot file capturing the expected --version standard output.
tests/​snapshots/​no-command.stderr Adds a snapshot file capturing the expected error output when no command is provided.
tests/​smoke/​test-no-options.t.sh Removes a legacy randomized shell smoke test script that was not reliably exercised in CI.
tests/​smoke/​no-arguments.t.sh Removes a legacy shell smoke test script for the no-arguments case.
tests/​smoke/​commands/​test.sh Removes the legacy randomized shell test command helper used by the old smoke tests.
tests/​README.md Documents how the new CLI integration tests and snapshots work and how to update the snapshot expectations.
tests/​cli.rs Adds deterministic Rust integration tests for CLI behavior (no command, retries/limits, invalid flags, signal termination, and argument pass-through) using assert_cmd and snapshots.
Cargo.toml Updates dev-dependencies to include the crates needed for the new integration test suite.
Cargo.lock Updates the lockfile to reflect the newly added test-related dependencies.

Review effort: Balanced

Comment thread tests/support/fixture.rs
Comment on lines +25 to +33
pub fn command(&self, mut command: Command) -> Command {
command
.arg("/bin/sh")
.arg(self.path("child.sh"))
.arg(self.path("count"))
.arg(self.path("arguments"))
.arg(&self.mode);
command
}
@rye
rye added this pull request to stack #534 November 13, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants