Skip to content

test: fix vacuous built-ins SDK tests (#5662) - #5921

Merged
mmabrouk merged 3 commits into
Agenta-AI:release/v0.112.3from
panshularora:fix/5662-builtins-sdk-tests
Aug 20, 2026
Merged

mmabrouk merged 3 commits into
Agenta-AI:release/v0.112.3from
panshularora:fix/5662-builtins-sdk-tests

Conversation

@panshularora

@panshularora panshularora commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes two SDK unit tests that currently pass without exercising what they claim to check (follow-up from #5651 review / #5662).

1. Vacuous resolver test (test_a_bare_tool_name_string_is_ignored_too)

coerce_tool_configs([\"read\"]) returns a pydantic ToolConfigParseResult, not a sequence of tool configs. Passing the parse result into ToolResolver().resolve() meant iteration never saw the coerced BuiltinToolConfig, so the empty-spec assertion could pass for the wrong reason.

Change: assert the parse result contains a single BuiltinToolConfig, then pass parsed.tool_configs into resolve().

2. Substring-only JSON check (test_no_json_example_writes_a_builtin_tool_entry)

The old check looked for the literal substring "type": "builtin", so whitespace variants like "type":"builtin" could slip through.

Change: parse each fenced JSON block and recursively reject any object with type == "builtin".

Test plan

Closes #5662

@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Aug 11, 2026
@vercel

vercel Bot commented Aug 11, 2026

Copy link
Copy Markdown

@panshularora is attempting to deploy a commit to the agenta projects Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Aug 11, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Tests
    • Expanded validation for nested built-in tool entries in JSON examples, including fenced JSON with varying whitespace.
    • Strengthened coverage for built-in tool configuration resolution and confirmed these tools are ignored without producing tool specifications.

Walkthrough

The tests now parse fenced JSON to detect nested builtin tool entries and validate coerced builtin configurations before resolver execution.

Changes

Builtin tool validation

Layer / File(s) Summary
Recursive JSON builtin detection
sdks/python/oss/tests/pytest/unit/agents/test_agenta_builtins_reference_files.py
The test parses fenced JSON, accepts flexible whitespace, requires a JSON example, and recursively detects objects with type set to "builtin".
Coerced configuration resolver test
sdks/python/oss/tests/pytest/unit/agents/tools/test_resolver.py
The test validates the BuiltinToolConfig returned by coerce_tool_configs before passing configurations to ToolResolver.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

  • Issue 5662 — The changes address resolver coercion and recursive JSON validation for builtin tools.

Possibly related PRs

  • Agenta-AI/agenta#5680 — This PR modifies the same tests for builtin-tool detection and resolver coercion.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 60.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the fix to vacuous built-in SDK tests.
Description check ✅ Passed The description directly explains both test fixes and their validation changes.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d270915a-4c73-493d-8499-822c0a897b73

📥 Commits

Reviewing files that changed from the base of the PR and between 3db504c and b7c1e5b.

📒 Files selected for processing (2)
  • sdks/python/oss/tests/pytest/unit/agents/test_agenta_builtins_reference_files.py
  • sdks/python/oss/tests/pytest/unit/agents/tools/test_resolver.py

@mmabrouk

Copy link
Copy Markdown
Member

@all-contributors please add @panshularora for tests

@mmabrouk
mmabrouk changed the base branch from main to release/v0.112.3 August 20, 2026 15:35

@mmabrouk mmabrouk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @panshularora for the tests !

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 20, 2026
@mmabrouk
mmabrouk merged commit 523595d into Agenta-AI:release/v0.112.3 Aug 20, 2026
5 of 7 checks passed
@allcontributors

Copy link
Copy Markdown
Contributor

@mmabrouk

I've put up a pull request to add @panshularora! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer python Pull requests that update Python code size:XS This PR changes 0-9 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(test) Built-ins SDK tests: vacuous coerce_tool_configs resolver test and substring-only JSON example check

3 participants