[dongjoon-hyun]

What changes were proposed in this pull request?

This PR makes PipelinesHandler.blockUnsupportedSqlCommand, the guard for spark.sql(...) in Spark Declarative Pipelines (SDP), reject three kinds of statements. The parser does not produce a Command for them, but they run a command or a procedure once they are analyzed.

Statement	Top node of the parsed plan	What happens during analysis
EXECUTE IMMEDIATE '<sql>'	UnresolvedExecuteImmediate	ResolveExecuteImmediate parses and analyzes <sql>, and a command payload then runs as ExecuteImmediateCommand.
CALL cat.ns.proc(...)	Call (an ExecutableDuringAnalysis)	InvokeProcedures runs the procedure.
A statement named by an IDENTIFIER clause that the parser does not build yet, e.g. USE SCHEMA IDENTIFIER('a' || 'b'), CREATE TABLE IDENTIFIER('a' || 'b') (id INT), or CREATE TEMPORARY VIEW IDENTIFIER('v') AS SELECT 1	PlanWithUnresolvedIdentifier	ResolveIdentifierClause evaluates the clause and builds the actual command.

The guard now does the following.

• It blocks UnresolvedExecuteImmediate and ExecutableDuringAnalysis, the trait that Call implements.
• For a top-level PlanWithUnresolvedIdentifier, it builds the statement through the node's planBuilder with a placeholder name and checks that statement with the same allowlist and blocklist. As a result, USE SCHEMA IDENTIFIER('a' || 'b') is rejected as SetNamespaceCommand, like USE SCHEMA IDENTIFIER('ab'), while read-only forms such as SHOW COLUMNS FROM IDENTIFIER(...) and TABLE IDENTIFIER(...) are still allowed.

Because the change is in blockUnsupportedSqlCommand, it covers both a plain SQL relation and the root SQL of a SQL-with-references WITH_RELATIONS (SPARK-57189).

Like the existing checks, the new checks are applied to the top node of the parsed plan. Nested cases such as WITH ... INSERT ... are out of scope here. They are the top-node limitation discussed in #59425.

Why are the changes needed?

The guard checks only the class of the top node of the parsed plan. The statements above pass that check, and session.sql then analyzes them, which runs the command or the procedure. For example, the following line in a pipeline definition file creates a table while the pipeline is being registered, although spark.sql("CREATE TABLE ...") is rejected.

spark.sql("EXECUTE IMMEDIATE 'CREATE TABLE spark_catalog.default.t (id INT)'")

Similarly, spark.sql("USE SCHEMA IDENTIFIER('some_' || 'schema')") runs SetNamespaceCommand although USE SCHEMA some_schema is rejected, and CALL runs a procedure of a configured ProcedureCatalog. Inside a flow function, the guard lets these statements pass as well.

Does this PR introduce any user-facing change?

Yes. In an SDP pipeline definition, spark.sql(...) with EXECUTE IMMEDIATE, CALL, or a command named by such an IDENTIFIER clause now fails with errors like the following instead of running.

[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'UnresolvedExecuteImmediate' is not supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'Call' is not supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'SetNamespaceCommand' is not supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000

This includes EXECUTE IMMEDIATE of a query such as EXECUTE IMMEDIATE 'SELECT 1', because its SQL string is parsed only during analysis. This is a bug fix for the SDP SQL guard, which has existed since Apache Spark 4.1.0 (2025-12-11).

How was this patch tested?

Pass the CIs with the newly added test cases in PythonPipelineSuite.

• unsupportedSqlCommandList: EXECUTE IMMEDIATE, CALL, USE SCHEMA with a non-literal IDENTIFIER clause, and CREATE TEMPORARY VIEW with a literal IDENTIFIER clause. Each is checked outside and inside a query function. Without the fix, these 8 test cases fail because the guard does not reject the statements. For example, outside a query function, EXECUTE IMMEDIATE creates the table, and USE SCHEMA IDENTIFIER(...) runs SetNamespaceCommand and fails with SCHEMA_NOT_FOUND.
• supportedSqlCommandList: SHOW COLUMNS FROM IDENTIFIER(...) and TABLE IDENTIFIER(...) with a non-literal IDENTIFIER clause, to make sure that the new check does not block them.

$ build/sbt 'connect/testOnly org.apache.spark.sql.connect.pipelines.PythonPipelineSuite -- -z "SQL command"'

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

[spark-test]

Review summary

This review found no defects in the updated SDP SQL guard. The new PlanWithUnresolvedIdentifier branch builds the deferred statement with a placeholder name and sends it through the same allowlist and blocklist check. As a result, USE SCHEMA IDENTIFIER(...) and CREATE TEMPORARY VIEW IDENTIFIER(...) AS ... are rejected, while SHOW COLUMNS FROM IDENTIFIER(...) and TABLE IDENTIFIER(...) are still allowed. The new UnresolvedExecuteImmediate and ExecutableDuringAnalysis blocklist entries reject every root-level EXECUTE IMMEDIATE and CALL. The added grid-test entries cover each newly rejected statement both outside and inside a query function, and they cover the read-only IDENTIFIER forms that stay allowed.

Findings

0 total: 0 P0, 0 P1, 0 P2, 0 P3.

No findings.

PR description suggestions

• In the user-facing change section, say that every CALL is now rejected in pipeline spark.sql, including procedures that only read data. This is the same trade-off the description already accepts for EXECUTE IMMEDIATE 'SELECT 1'.

Generated by Omnigent on Databricks