[eserscor]

Description

Try four GoogleTest shards of the same onnxruntime_provider_test executable, without duplicating compilation or registering individual cases. onnxruntime_ENABLE_PROVIDER_TEST_SHARDING defaults OFF; only the existing Linux CPU x64 Debug ASan and arm64 Debug jobs opt in. Release, GPU, other platforms, other executables and standalone binary execution remain unchanged. This PR remains draft.

Each CTest shard reserves four processor slots, retains the existing three-hour timeout, and uses GTEST_TOTAL_SHARDS=4 / its own GTEST_SHARD_INDEX. Plugin-test environment setup is applied to every registered entry. The runner isolates working directories, fixed temporary filenames and shard-status files; it links existing fixture/model inputs and shared libraries. Each shard writes <binary>.<config>.shard_<index>.results.xml, matching the existing **/*.results.xml collector. Missing GoogleTest shard acknowledgment and zero selected cases fail explicitly; child-process failures propagate.

Default ORT intra-op/OpenMP threads are three and inter-op threads two. Inter-op must stay above one: otherwise per-session parallel execution silently falls back to sequential. Explicit per-test thread counts are not overridden. This bounds the usual pools, not all memory or explicitly created threads; Gather_overflow_check alone needs multiple GiB. Narrow filters can leave an empty shard and intentionally fail; use the standalone binary for individual-case debugging. Shared fixture inputs are linked, not enforced read-only.

Focused validation — passed in an isolated Python environment under Ubuntu 24.04 WSL:

.venv/bin/python -m unittest discover -s tools/ci_build -p test_run_provider_test_shard.py -v
# Passed: 6 tests (registration/properties, isolation, filter forwarding, stale reports,
# missing reports/acknowledgment, zero selected cases, failure propagation).
.venv/bin/python -m ruff check tools/ci_build/run_provider_test_shard.py tools/ci_build/test_run_provider_test_shard.py
# Passed, pinned Ruff 0.12.12.
.venv/bin/python -m ruff format --check tools/ci_build/run_provider_test_shard.py tools/ci_build/test_run_provider_test_shard.py
# Passed, 2 files already formatted.
ctest --test-dir build/gtest-sharding-validation/cmake --parallel 16 --output-on-failure -V
# Passed against a compiled GoogleTest 1.17 smoke harness using the production helper/runner.

The smoke harness's actual XML shows 16 filtered eligible cases split 4/4/4/4, no duplicates, and union equal to unsharded execution; the disabled case remains excluded. Concurrent fixtures deliberately use identical cwd/temp filenames. Injected test failure makes CTest fail and a zero-match invocation is rejected. Negative checks deliberately produce failures, not green zero-test results.

Full local provider comparison — passed, same CPU Release executable/source/options, x86_64 WSL with 24 available CPUs. The complete provider target was built successfully with:

.venv-linux/bin/python tools/ci_build/build.py --build_dir build/sharding --config Release --update --build --parallel 8 --target onnxruntime_provider_test --skip_submodule_sync --cmake_generator Ninja --cmake_extra_defines onnxruntime_GENERATE_TEST_REPORTS=ON

The isolated environment was subsequently moved/reinitialized at the standard ignored .venv path. CMake registration was checked with the option OFF and ON: one original provider entry versus four shared-binary entries, with all non-provider commands/properties unchanged. After enabling sharding, the successful unsharded comparison used the exact saved OFF registration's executable/XML arguments directly, rather than claiming another OFF CTest run. The sharded invocation was:

ctest --test-dir build/sharding/Release -C Release --parallel 16 --output-on-failure -R '^onnxruntime_provider_test_shard_'

Both runs used the same private LOCPATH containing en_US.UTF-8 and the executable directory as cwd. Initially, the WSL image lacked that locale: the first full OFF/ON runs each failed 19 StringNormalizer cases. Generating a private locale fixed this environment-only prerequisite; all 32 targeted StringNormalizer cases and both full-suite reruns then passed. No cases were removed or filtered out.

Local mode/shard	Selected cases	CTest duration	GoogleTest duration
Unsharded standalone	6,926	n/a	106.229s
Shard 0	1,732	11.78s	11.219s
Shard 1	1,732	50.17s	49.536s
Shard 2	1,731	33.69s	33.080s
Shard 3	1,731	11.93s	11.415s

Measured process wall time was 106.81s unsharded versus 51.22s sharded; CTest's four-shard real time was 50.97s. Both exited 0. Actual, case-sensitive XML comparison proves 6,926 unique selected identities, union exactly equal to baseline, no duplicates/missing/extra cases. The same 513 selected cases were skipped in both modes; five disabled cases remain excluded. Every shard is nonempty and acknowledged GoogleTest sharding. This is one local Release trial of the complete sharding/thread policy, not an isolated estimate of sharding's contribution or an ARM64/ASan speedup claim.

Sampled peak summed process-tree RSS was 4,451,532 KiB (~4.25 GiB) unsharded versus 4,606,340 KiB (~4.39 GiB) sharded; sampled peak process-tree threads were 34 versus 32. Samples are every 250ms and sum per-process RSS, double-counting shared pages; they are not unique-memory or guaranteed peak measurements. The successful full concurrent run found no fixture collisions, but does not prove all future/platform-specific fixtures are race-free.

Actual Linux CI — run 43475294984: both opted-in Debug jobs passed.

CI shard	x64 Debug ASan selected cases	x64 CTest duration	ARM64 Debug selected cases	ARM64 CTest duration
0	1,732	143.53s	1,756	877.85s
1	1,731	722.70s	1,756	507.89s
2	1,731	256.06s	1,756	507.14s
3	1,731	155.52s	1,756	381.89s
Total selected / whole CTest real time	6,925	734.40s	7,024	881.04s

x64 Debug ASan job finished successfully at 18:19:15Z; its test step was approximately 736s. ARM64 Debug job finished successfully at 18:31:11Z; its test step was 18:16:23Z–18:31:08Z (885s). Whole CTest times include the other test executables and are not sums of concurrent shard durations.

Verbose CI [ RUN ] identities prove 6,925 case-sensitively unique selected names on x64 and 7,024 on ARM64, no cross-shard duplicates. Each job logged four distinct reports at /onnxruntime_src/build/Debug/Debug/onnxruntime_provider_test.Debug.shard_<0..3>.results.xml; each successful runner parsed its XML, verified a nonzero selected count and acknowledged GoogleTest sharding. Each job's existing collector processed 15 result files overall. Raw CI XML was not available as a downloaded artifact: CI identity/disjointness evidence comes from verbose logs plus runner XML counts, while the full unsharded-versus-sharded XML union proof is local. Different build configurations need not select identical counts.

CTest initially scheduled three provider shards in both jobs; the fourth started when processor slots became available. PROCESSORS=4 cooperates with the existing executable-level scheduler, not a guarantee of four simultaneous starts. ASan shard 1 and ARM64 shard 0 dominate their respective tails despite nearly equal case counts. No CI peak memory/thread measurements or controlled same-head unsharded ARM64/ASan comparisons are available. These green trials do not establish an architecture-specific speedup or a strict process memory/thread cap.

No manual reruns, workflow dispatches, additional GPU workflow triggers or shared-image changes were made. Normal existing PR checks may run automatically; sharding is enabled only in the two CPU Debug jobs. Raw local XML/logs, CI logs and measurements are retained in session artifacts, not tracked source.

Motivation and Context

ctest --parallel 16 distributes executables, but cannot split cases within the long provider executable. In baseline run https://github.com/microsoft/onnxruntime/actions/runs/43458960142, ARM64 Debug provider tests took 2112.21s versus 340.30s for the core executable; x64 Debug ASan provider tests took 1241.21s. Release provider durations were much lower (ARM64 256.13s, x64 301.68s), so this experiment opts in only the two Debug variants. These historical observations are not an apples-to-apples CI speedup comparison.

Independent branch from main (6080d7c7cc), not stacked on the cache experiment or separate PackedAttention fix. No cache/vcpkg policy, workflow triggers, shared runner images, signing/package/deployment workflows or operator-test sources are changed. The app's kebab-case branch control could not express the requested date/slash naming convention; its automatically prefixed branch is eserscor-google-test-provider-sharding.

[Copilot]

🔵 Needs a closer look

Human review of full provider-suite CI results on x64 Debug ASan and arm64 Debug is needed to validate four-way concurrency, test isolation, and peak-memory/OOM safety.

No open issues 🎉

What changed in this PR

This PR introduces opt-in sharding of ONNX Runtime’s provider-test executable to trial shorter Linux CPU Debug CI runs without duplicating compilation. Sharding defaults to off; only x64 Debug ASan and arm64 Debug CI enable it.

Changes:

• Registers four CTest shards sharing one executable, with bounded default thread counts and the existing three-hour timeout.
• Adds an isolated shard runner with unique XML reports, shard acknowledgment checks, and empty-shard rejection.
• Adds runner and registration tests plus usage documentation. Full provider-suite performance validation remains pending.

File	Description
tools/​ci_build/​test_run_provider_test_shard.py	Tests isolation, failure handling, reports, and CTest registration.
tools/​ci_build/​run_provider_test_shard.py	Runs each shard with isolated writable state and validates execution.
cmake/​onnxruntime_unittests.cmake	Integrates optional sharding and applies plugin environments to every shard.
cmake/​onnxruntime_provider_test_shards.cmake	Defines four shard entries, resource settings, and unique report paths.
cmake/​CMakeLists.txt	Adds the default-off sharding option.
.github/​workflows/​linux_ci.yml	Enables sharding for the two Linux CPU Debug variants.
.github/​skills/​ort-test/​SKILL.md	Documents shard execution, resource limits, and debugging guidance.

_{Review effort: Balanced}