[tino097]

https://linear.app/mindsdb/issue/ENG-3549

Why this matters

A Databricks connection can now ask Databricks Genie instead of running SQL from the scratchpad. Genie writes and runs the SQL inside Databricks under the service principal's Unity Catalog grants, and the agent gets a short answer plus the rows. Signing in uses a Databricks-managed service principal's client ID and secret, so it works for every desktop app user and on the self-hosted build, which has no browser sign-in. It ships to all desktop users once released. Building it also surfaced three bugs in the shared MCP client that hit HubSpot and Notion too: a 5 s HTTP timeout, a reconnect that crossed task scopes and could hang a turn for good, and a session that stayed dead after one failed reconnect.

What changed

• Add the Databricks workspace host check for Genie connections: a token or MCP call only ever goes to *.cloud.databricks.com, *.azuredatabricks.net or *.gcp.databricks.com, with no port, path, user info or IP literal.
• Leave hidden connection fields out of a published app's environment, so _client_secret never reaches a published app.
• Build the Genie One or per-space MCP address from a workspace host; a space id must be 32 lowercase hex.
• Mint a Genie access token from a service principal's client credentials: one POST to /oidc/v1/token, scope all-apis, no redirects, and no secret, token or body text in any error.
• Treat a rejected client on any 4xx as refused credentials, since OAuth servers answer 400 as well as 401.
• Discover Genie MCP connections with a token minted for the turn: genie-mcp joins mcp as an MCP method, the four genie_* tools are read-only, and genie_ask waits for the answer inside the call and returns only status, final answer, query item ids and the deep link.
• Hold the Genie secret and token for redaction on every Genie call, inside the turn's own scrub scope.
• Never hand Genie's raw reply text to the agent: a reply without structured fields becomes a fixed message, since that text is long reasoning that opens with instructions addressed to the agent.
• Tell the agent to use Genie's tools for a Genie connection, with no DS_* variables listed for it.
• Classify a Genie connection's fields from its own record, so the workspace host in Genie's links is no longer redacted.
• Give MCP sessions the SDK's HTTP timeouts (30 s, 300 s read) instead of httpx's 5 s default, for every MCP connector.
• Retry a dropped MCP call without crossing task scopes: each client lives in its own owner task, so a dropped call no longer leaves the turn waiting forever.
• Recover a session after a failed reconnect and a cancelled owner: the next call reconnects, and a close from another event loop returns normally.
• Close a client a reconnect opens after the session was closed, so nothing leaks at turn end.
• Hide a connection's secret bookkeeping fields from the start of every turn, so _client_secret is redacted before the first Genie call too.
• Log a Genie session's retry, reconnect and close errors by class name only; HubSpot and Notion keep their lines.
• Name a Genie connection's tools in the prompt the way registration names them, including two Genie connections and access mode "none".
• Label Genie's held secrets by connection, skip a Genie connection with tools off, and close the HTTP client when a transport fails to build.
• Tidy the Genie connector tests: literal URLs pinned, one source for shared test values, imports at the top.
• State why hidden secure fields are held at any length.
• Do not log a reconnect stopped by close as a failure.

Notes for the reviewer

• Stacked on Notion MCP connector: server URL, read-only tools, general MCP fixes #549. This builds on the MCP fixes in Notion MCP connector: server URL, read-only tools, general MCP fixes #549 and targets its branch; it retargets to staging when Notion MCP connector: server URL, read-only tools, general MCP fixes #549 merges.
• Shared MCP client changes reach HubSpot and Notion. Longer timeouts, one owner task per client, recovery after a failed reconnect. Their log lines and tool access are unchanged and pinned by tests.
• No token renewal inside a turn. The token lasts an hour; a lapse shows the existing reconnect text and the next turn mints a new one.
• Genie is refused in org mode (cowork-server side). Cloud MCP is not built, and the secret must never sit in the pod.
• Open: within one live turn the Genie One connection dropped and reconnected a few times; five spikes outside the app (idle sessions up to 240 s, one and three real questions in one session) did not reproduce it. Investigation continues; the turn still answered.

Ships with

• Notion MCP connector: server URL, read-only tools, general MCP fixes #549 first, then this PR, then an anton release.
• [ENG-3549] Connect Databricks Genie with a service principal, and check connection edits with their stored secrets cowork-server#744: the Genie connector and connection-edit PR. It works with an older anton (Genie reports this build can't connect it); a later commit raises its anton floor to the release.
• [ENG-3549] Databricks Genie credential form, and editing a saved connection cowork#1267: the Genie form and edit-form PR, with cowork-server's.

Checks

Check	Observed result
uv run pytest tests/core/mcp tests/test_scrubbing.py tests/test_datasources_prompt.py tests/test_publisher_state.py -q	602 passed
uv run pytest -q (full suite, at 34bda71)	7420 passed, 271 skipped
Live, desktop dev app against a Databricks trial workspace (this branch installed editable, at 31104b9 and e9178aa, before the review fixes)	5 test questions answered correctly through databricks__genie_ask; secret masked in the connection API and absent from the agent's environment. Not yet rerun on the final head

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T22:13:52.279549Z	34bda71	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34bda71f96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve Genie fields in the turn-key vault

Cloud turns pass a TurnKeyDataVault here, but its _fetch_uncached() still requires an access_token and only retains the existing OAuth allowlist plus _method; it drops workspace_url, client_id, _client_secret, and genie_space_id. Consequently this vault.load() can never provide the fields resolve_mcp_endpoint() requires, so a cloud Genie connection is either omitted before discovery or reported as incomplete and registers no tools. Extend the turn-key response handling and secure-key classification for the Genie service-principal fields.

Useful? React with 👍 / 👎.

[martyna-mindsdb]

Non-blocking: I think this doesn't apply for now: cowork-server refuses Genie in org mode, so cloud never has a Genie connection. Can you confirm and resolve?

[chatgpt-codex-connector]

Reopen clients whose owner loop has stopped

In the cloud path, discover_mcp_tools() opens sessions inside asyncio.run(), whose shutdown cancels the still-waiting _owner task and closes the SDK client, while _client remains non-None. The first tool call later runs on the turn's event loop and this fast path returns that closed client, guaranteeing a failed attempt and consuming the sole retry merely to establish a usable session; any transient failure on the retried call is then surfaced without its intended retry. Treat a done/cancelled owner as a missing client and reconnect before returning it.

Useful? React with 👍 / 👎.

[martyna-mindsdb]

Non-blocking: agree with this one. On the cloud path, the first tool call gets a dead client and uses up its only retry reconnecting. Could we treat a finished _owner task as "no client" in _acquire_client and reconnect right away? This affects HubSpot and Notion on cloud too.

[martyna-mindsdb]

Here are my comments:

1. Blocking: The live test ran before the review fixes. Can you rerun it on the final code? Please include a couple of Genie questions plus one HubSpot or Notion chat, since client.py changed for all MCP connectors.
2. Non-blocking: The PR mentions Genie One dropping and reconnecting several times in one turn. Can we open a ticket for it, so it's tracked after merge?
3. Non-blocking: The docstrings are quite long across the PR. Could you trim them to just the non-obvious parts?
4. Non-blocking: The PR description reads like a commit log. Could you shorten it to why, what changed and how it was tested? Also, please put the Linear link on a Linear: line.

Inline comments below.

[martyna-mindsdb]

Non-blocking: this caps the number of rows but not their size, so a wide table could still be very large. Worth adding a size limit too?

[martyna-mindsdb]

Non-blocking: if the turn is cancelled during this wait, Genie keeps working on the question. Should we call genie_cancel_response on cancel?

[martyna-mindsdb]

Non-blocking: this runs on every turn with a Genie connection, even when the user doesn't ask about Databricks. A failure can add about 40 s to the turn. Fine for now, but worth keeping in mind.

[martyna-mindsdb]

Non-blocking: can we drop the date from this comment? The reason is enough on its own.