fix(appautomate): ask native runs for devices as App Automate lists them by Aaryan430 · Pull Request #510 · browserstack/mcp-server

Aaryan430 · 2026-10-22T18:31:01Z

runAppTestsOnBrowserStack checks the requested devices against App Automate's device list (validateAppAutomateDevices), but then ignores what that returns and builds the device strings from the raw input.

App Automate lists Android versions as 14.0. An Espresso run requested as Google Pixel 8 on Android 14, which is how agents ask and how the tool's own schema example writes it, goes out as Google Pixel 8-14. The build starts, its session never gets a device, and it ends in error about an hour later.

Seen in an eval run: an Espresso build asked for Google Pixel 8-14 sat queued and errored after 64 minutes. The same app and suite asked for Google Pixel 8-14.0 ran straight away.

Change

Espresso and XCUITest runs send the device name and version the validator resolved (Google Pixel 8-14.0, App Automate's own display name). A run with no devices still sends none.

Verification

Live, through this branch's build: an Espresso run requested with "osVersion": "14" went out as Google Pixel 8-14.0, and its session started within 10 seconds.
New tests run Espresso and XCUITest requests through the real validator with a mocked device list, and check the device strings the build is triggered with: 14 → 14.0, a lower-case device name → App Automate's display name, and no devices → none. npm run build (lint, format, 1355 tests, tsc) passes.

🤖 Generated with Claude Code

Changes after review (Shreyas)

The success text lists the devices the run uses. When a requested OS version isn't listed, it names the version the run uses instead.

Summary by CodeRabbit

Bug Fixes
- App Automate now uses canonical device names and OS-version formats for Android and iOS test requests. Requests without devices pass an empty device list.

runAppTestsOnBrowserStack validated its devices against App Automate's device list but then sent the device strings built from the raw input. App Automate lists Android versions as "14.0", so a run asked for "Google Pixel 8" on "14" went out as "Google Pixel 8-14": the build started, its session never got a device, and it errored about an hour later. The tool's own schema example uses "14". Espresso and XCUITest runs now send the validated name and version ("Google Pixel 8-14.0"). A run with no devices still sends none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-22T18:31:11Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 87879385-938f-4592-b768-5b57a4a72202

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 5805f36e-770f-400a-84cb-3a072e18fe4b

📥 Commits

Reviewing files that changed from the base of the PR and between 7777a94 and da9f51c.

📒 Files selected for processing: 2

src/tools/appautomate.ts
tests/tools/appautomate-native-devices.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

📜 Recent review details

🔇 Additional comments: 2

src/tools/appautomate.ts: 1

196-203: LGTM!

tests/tools/appautomate-native-devices.test.ts: 1

1-127: LGTM!

📝 Walkthrough

App Automate test builds now use device names and OS versions from validated device records. Tests cover canonical Android and iOS device strings and an empty device list.

Changes: Validated device lists

Layer / File(s)	Summary
Build requests use validated device records `src/tools/appautomate.ts`, `tests/tools/appautomate-native-devices.test.ts`	The Espresso and XCUITest build requests use device strings derived from validated records. Tests check canonical Android and iOS names and versions, plus an empty Espresso device list.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: tech-sushant

Merge Risk: ⚪ Minimal · up to da9f5

The native build paths use validated device names and OS versions, and no actionable merge risk remains after normal checks.

Pre-merge checks: 4

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: native App Automate runs now use the device names and versions returned by App Automate.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{A rabbit checks the device names,
Then hops through Android testing lanes.
iOS versions line up right,
Empty lists stay empty and light.
The build begins; the bunny beams.}

_{Comment @coderabbitai help to get the list of available commands.}

sarve-shreyas · 2026-10-23T11:12:37Z

Review: #510

Correct for the reported bug (Pixel 8-14 → Google Pixel 8-14.0). CI shows only CodeQL and CodeRabbit, so the test counts in the description aren't CI-verified.

Medium

appautomate.ts:199-203 with version-resolver.ts:65-77: when the requested OS version doesn't exist, the resolver silently uses the closest numeric version, and the success message (:249) doesn't mention it. A run on the wrong OS version is easy to miss. Include the resolved deviceStrings in the success text, and optionally fail on the fuzzy-fallback branch.

Low

osVersion: "latest" now resolves to the highest listed version (via parseFloat) instead of sending Pixel-latest. Untested; document it if intended.
Missing tests: unknown device/platform, multi-device ordering (arrays align by index), and the browserstackAppUrl / browserstackTestSuiteUrl branches.
The args.devices.length > 0 ? … : [] guard is needed, since validateAppAutomateDevices([]) returns a default entry. Worth keeping.

AI-assisted read-only review of the diff. Items marked (unverified) were not checked against the live APIs.

…e-device-versions

…run asks for The run-started answer names the devices the run asks App Automate for, and when a requested OS version is not listed and the resolver used the closest one, says so, so a run on another version than asked for is not missed. Non-numeric versions ("latest") are expected to resolve and are not flagged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…them A validated device's name is optional in its type; the devices note now says so, which fixes the build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Aaryan430 · 2026-10-23T11:38:01Z

Thanks @sarve-shreyas. Addressed the Medium in 61f02c6 and 2c5b5b6:

The success text now lists the devices the run uses (📱 Devices: …). When the requested OS version isn't listed, it says so directly: ⚠️ Google Pixel 8: version 15 is not listed on App Automate, so the run uses 14.0, the closest it lists. This applies to both Espresso and XCUITest. Tests are in tests/tools/appautomate-native-devices.test.ts.
I kept the closest-version fallback rather than failing. The run still starts, and the message makes the substitution visible.

latest → highest listed version is intended. The Low items are otherwise unchanged in this round.

Aaryan430 · 2026-10-23T12:07:28Z

This has been merged into #502 (f2fd8c4), which now ships it to main together with #509, #513 and #492. It merged cleanly, and the combined build passes (1400 tests).

Brings in #502 (Automate links, share links, large exports, one-page view, scan-run comparison, #510's device note, #513's scan settings, #492's Ask AI pointer). Conflicts resolved by keeping both sides: - fetchAccessibilityIssues resolves the source first, then compareWith goes to the scan-run comparison for a scan and to the app-build comparison for an app build; pageUrl and exportTaskId pass through for web reports, and the next-page hint keeps the pageUrl with the exact remaining count. - Report links: a share link is unwrapped first, then app and manual-report links, then Automate links. - The CSV parser pages through the shared pageOf/sortBySeverity; its local copy is dropped. - The run-started message carries both the device note and the accessibility guidance. - The ids handoff keeps main's Ask AI pointer within the 200-character budget; the app lists' searchCapability pointer moves to the buildId and appReportId argument descriptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Oct 22, 2026

feat(accessibility): route a11y checks, app build summary + compareWith, exact paging, reliable app build links #511

Merged

chore(accessibility): remove the accessibilityExpert tool #512

Merged

Aaryan430 and others added 3 commits October 23, 2026 16:59

Merge remote-tracking branch 'origin/main' into fix/appautomate-nativ…

0fa20be

…e-device-versions

fix(appautomate): type the resolved devices as the validator returns …

2c5b5b6

…them A validated device's name is optional in its type; the devices note now says so, which fixes the build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Oct 23, 2026

feat(accessibility): read an Automate build or session link's accessibility issues #502

Merged

feat(ask-ai): offer a11y as a product again #492

Merged

Aaryan430 mentioned this pull request Oct 23, 2026

feat(accessibility): choose a scan's WCAG version, best practices and needs review #513

Merged

gaurav-singh-9227 merged commit f2fd8c4 into main Oct 23, 2026
5 checks passed

Aaryan430 mentioned this pull request Oct 23, 2026

chore: bump version to 2.2.7 #518

Merged

Navigation Menu

Repository navigation

fix(appautomate): ask native runs for devices as App Automate lists them - #510

fix(appautomate): ask native runs for devices as App Automate lists them#510
gaurav-singh-9227 merged 4 commits into
mainfrom
fix/appautomate-native-device-versions

Aaryan430 commented Oct 22, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

Review skipped

Uh oh!

sarve-shreyas commented Oct 23, 2026

Uh oh!

Aaryan430 commented Oct 23, 2026

Uh oh!

Aaryan430 commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Aaryan430 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change

Verification

Changes after review (Shreyas)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

sarve-shreyas commented Oct 23, 2026

Uh oh!

Aaryan430 commented Oct 23, 2026

Uh oh!

Aaryan430 commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Aaryan430 commented Oct 22, 2026 •

edited

Loading

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading