[lyne7-sc]

Which issue does this PR close?

Closes #2690

Rationale for this change

Native hash joins and sort-merge joins can generate columns that a parent projection immediately discards. This change lets joins output only the requested columns, avoiding unnecessary output work.

What changes are included in this PR?

• Forward column-pruning requests to the existing native join implementations.
• Fix the full-projection check and preserve support for reordered and duplicate projections.

Are there any user-facing changes?

This is an internal execution optimization with no new configuration or public APIs.

How was this patch tested?

• Native column-pruning and join tests passed.
• Existing related suites: AuronBroadcastHashJoinConditionSuite and AuronSortMergeJoinConditionSuite
• Validated with Spark SQL on Parquet inputs:

SELECT /*+ BROADCAST(r) */ l.v
FROM prune_l l JOIN prune_r r ON l.k = r.k;

SELECT /*+ MERGE(l,r) */ l.v
FROM prune_l l JOIN prune_r r ON l.k = r.k;

Native logs recorded join output before the parent Project:

Before:

JOIN_OUTPUT node=HashJoin columns=3 rows=8 fields=["#0", "#1", "#2"]
JOIN_OUTPUT node=SortMergeJoin columns=3 rows=2 fields=["#0", "#1", "#2"]
JOIN_OUTPUT node=SortMergeJoin columns=3 rows=6 fields=["#0", "#1", "#2"]

After:

JOIN_OUTPUT node=HashJoin columns=1 rows=8 fields=["#1"]
JOIN_OUTPUT node=SortMergeJoin columns=1 rows=6 fields=["#1"]
JOIN_OUTPUT node=SortMergeJoin columns=1 rows=2 fields=["#1"]

Was this patch authored or co-authored using generative AI tooling?

• Yes
• No

If yes, include: Generated-by: OpenAI Codex (GPT-6.1)

ASF guidance: https://www.apache.org/legal/generative-tooling.html

[slfan1989]

SortMergeJoinExec caches JoinParams in a OnceCell, so the first projection wins. I verified locally that calling execute_projected with [1] and then [2] on the same instance returns column 1 both times.

A subsequent execute() also returns just one column.

Could we cache only the projection-independent parameters and add a regression test that reuses the same join instance?

[lyne7-sc]

Thanks for catching this! I changed SMJ to create JoinParams per execution and updated the existing test to cover different projections followed by execute() on the same instance.

While checking this path, I also fixed an existing hybrid join layout assumption exposed by column pruning: with a right-only projection, the row-number column used to preserve probe order was placed incorrectly in the output schema. The targeted regression test and Spark SQL reproducer now pass.