Skip to content

Commit 2076636

Browse files
committed
cherry-pick(#43920): docs(dotnet): document retrying failed tests
1 parent fb8e87f commit 2076636

1 file changed

Lines changed: 118 additions & 0 deletions

File tree

‎docs/src/test-runners-csharp.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,124 @@ CLI. See the following example:
294294
dotnet test -- Playwright.BrowserName=chromium Playwright.LaunchOptions.Headless=false Playwright.LaunchOptions.Channel=msedge
295295
```
296296

297+
## Retrying failed tests
298+
299+
Playwright for .NET does not retry failed tests itself, use the retries of your test runner. NUnit and MSTest have an attribute that retries a single test, and [Microsoft.Testing.Platform](#retrying-with-microsofttestingplatform) can retry all failed tests of a run. Every attempt gets a new [BrowserContext] and [Page].
300+
301+
<Tabs
302+
groupId="test-runners"
303+
defaultValue="mstest"
304+
values={[
305+
{label: 'MSTest', value: 'mstest'},
306+
{label: 'NUnit', value: 'nunit'},
307+
{label: 'xUnit', value: 'xunit'},
308+
{label: 'xUnit v3', value: 'xunit-v3'},
309+
]
310+
}>
311+
<TabItem value="nunit">
312+
313+
Add the [`Retry`](https://docs.nunit.org/articles/nunit/writing-tests/attributes/retry.html) attribute to a test method. Its argument is the total number of attempts, so `[Retry(3)]` retries a failed test up to two times.
314+
315+
By default NUnit only retries a test that failed an NUnit assertion. Playwright assertions throw `PlaywrightException` and actions that time out throw `TimeoutException`, so pass both as `RetryExceptions`. This property requires NUnit 4.5 or later.
316+
317+
```csharp
318+
using Microsoft.Playwright;
319+
using Microsoft.Playwright.NUnit;
320+
321+
namespace PlaywrightTests;
322+
323+
[Parallelizable(ParallelScope.Self)]
324+
[TestFixture]
325+
public class MyTest : PageTest
326+
{
327+
[Test]
328+
[Retry(3, RetryExceptions = [typeof(PlaywrightException), typeof(TimeoutException)])]
329+
public async Task TestWithRetries()
330+
{
331+
await Page.GotoAsync("https://playwright.dev");
332+
await Expect(Page.GetByRole(AriaRole.Link, new() { Name = "Get started" })).ToBeVisibleAsync();
333+
}
334+
}
335+
```
336+
337+
</TabItem>
338+
<TabItem value="mstest">
339+
340+
Add the [`Retry`](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-writing-tests-controlling-execution) attribute to a test method. It requires MSTest 3.8 or later. Its argument is the number of retries, so `[Retry(2)]` runs a failing test up to three times. MSTest retries after any failure, including a failed Playwright assertion and a timeout.
341+
342+
```csharp
343+
using System.Threading.Tasks;
344+
using Microsoft.Playwright;
345+
using Microsoft.Playwright.MSTest;
346+
using Microsoft.VisualStudio.TestTools.UnitTesting;
347+
348+
namespace PlaywrightTests;
349+
350+
[TestClass]
351+
public class ExampleTest : PageTest
352+
{
353+
[TestMethod]
354+
[Retry(2)]
355+
public async Task TestWithRetries()
356+
{
357+
await Page.GotoAsync("https://playwright.dev");
358+
await Expect(Page.GetByRole(AriaRole.Link, new() { Name = "Get started" })).ToBeVisibleAsync();
359+
}
360+
}
361+
```
362+
363+
</TabItem>
364+
<TabItem value="xunit">
365+
366+
xUnit v2 has no built-in way to retry failed tests. Retries need xUnit v3, which runs on [Microsoft.Testing.Platform](#retrying-with-microsofttestingplatform).
367+
368+
</TabItem>
369+
<TabItem value="xunit-v3">
370+
371+
xUnit v3 has no retry attribute. Retry failed tests with [Microsoft.Testing.Platform](#retrying-with-microsofttestingplatform).
372+
373+
</TabItem>
374+
</Tabs>
375+
376+
### Retrying with Microsoft.Testing.Platform
377+
378+
The [retry extension](https://learn.microsoft.com/en-us/dotnet/core/testing/microsoft-testing-platform-retry) of Microsoft.Testing.Platform reruns the failed tests of a run in a new process, without changes to the tests. It works with MSTest, NUnit and xUnit v3, but not with xUnit v2.
379+
380+
First, run your tests on Microsoft.Testing.Platform. xUnit v3 [supports it](https://xunit.net/docs/getting-started/v3/microsoft-testing-platform) out of the box. For MSTest, set `EnableMSTestRunner` in the project file, and for NUnit set `EnableNUnitRunner`:
381+
382+
```xml
383+
<PropertyGroup>
384+
<OutputType>Exe</OutputType>
385+
<EnableMSTestRunner>true</EnableMSTestRunner>
386+
</PropertyGroup>
387+
```
388+
389+
With the .NET 10 SDK, `dotnet test` also needs the following `global.json` file:
390+
391+
```json
392+
{
393+
"test": {
394+
"runner": "Microsoft.Testing.Platform"
395+
}
396+
}
397+
```
398+
399+
Then add the extension to the test project:
400+
401+
```bash
402+
dotnet add package Microsoft.Testing.Extensions.Retry
403+
```
404+
405+
:::note
406+
The major version of the package has to match the version of Microsoft.Testing.Platform that your test runner uses, otherwise no tests run. For example, xUnit v3 3.x needs version 1.x of the package, and xUnit v3 4.x needs version 2.x.
407+
:::
408+
409+
Now you can retry each failed test up to two times:
410+
411+
```bash
412+
dotnet test --retry-failed-tests 2
413+
```
414+
297415
## Using Verbose API Logs
298416

299417
When you have enabled the [verbose API log](./debug.md#verbose-api-logs), via the `DEBUG` environment variable, you will see the messages in the standard error stream. Within Visual Studio, that will be the `Tests` pane of the `Output` window. It will also be displayed in the `Test Log` for each test.

0 commit comments

Comments
 (0)