[jayzhan211]

Which issue does this PR close?

• Closes #.

Rationale for this change

x NOT IN (subquery) is planned as a null-aware LeftAnti hash join in
CollectLeft mode with several probe partitions. If any probe partition sees
a NULL key, the predicate is UNKNOWN for every build row and the join must
return nothing.

The final stage of HashJoinStream read the shared "probe side saw NULL"
flag before decrementing the probe-partition counter. That allowed this
interleaving:

1. partition B finishes its probe input and reads the flag as false;
2. partition A processes a batch with a NULL key, sets the flag, finishes and
   decrements the counter;
3. partition B decrements the counter, becomes the last partition, and emits
   the unmatched build rows.

The result then contains rows that NOT IN must suppress. The counter was
also decremented with Ordering::Relaxed, so nothing ordered the flag stores
of a finishing partition before the reads of the last one, even when the
reads happened after the decrement.

What changes are included in this PR?

• JoinLeftData::report_probe_completed now decrements with AcqRel and
  returns Option<ProbeSideSummary>: Some only for the last partition, with
  the shared flags read after the decrement. The two AtomicBool flags are
  private; the probe phase stores through record_probe_batch and the early
  exit reads a probe_side_has_null_hint that is documented as best-effort.
  The final stage therefore cannot read the flags before its own decrement,
  which guards the fix structurally rather than by a test that would have to
  force a few-instruction race window.
• process_unmatched_build_batch decrements first and hands the summary to
  the null-aware helpers. The NULL-probe rule of LeftAnti moved into
  null_aware_left_anti_final_indices, so all its final-stage rules are in
  one place.
• No change for joins that are not null-aware beyond the AcqRel ordering
  on the counter.

What is the testing strategy for this PR?

Two new tests in hash_join/exec.rs,
test_null_aware_anti_join_probe_null_in_other_partition and
test_null_aware_left_mark_probe_null_in_other_partition, run a CollectLeft
join with a two-partition probe side where only partition 0 has a NULL key,
draining the partitions in both orders. They cover the cross-partition flag
propagation that had no test. The race itself cannot be reproduced
deterministically without a test hook, which is why the fix makes the wrong
call order unrepresentable instead. Existing null-aware join tests and the
join unit test suite pass unchanged.

Are there any user-facing changes?

Wrong results from NOT IN under this interleaving are fixed. No API changes;
all touched items are pub(super).

[kumarUjjawal]

These partitions are drained sequentially, so the old implementation also passes both orders: the NULL flag is always written before the final partition examines it. The tests would also remain green if the counter ordering changed back to Relaxed.

Could we force the failing overlap with a synchronization hook where the non-NULL partition reads false, the sibling records NULL, and the non-NULL partition then becomes last or add a model-checked atomic test? This wrong-results race otherwise has no regression coverage.

[jayzhan211]

I add the loom test

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.52778% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.90%. Comparing base (46ec37d) to head (9834679).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...tafusion/physical-plan/src/joins/hash_join/exec.rs	95.12%	2 Missing and 2 partials ⚠️
...sical-plan/src/joins/hash_join/probe_completion.rs	97.87%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25076      +/-   ##
==========================================
+ Coverage   81.89%   81.90%   +0.01%     
==========================================
  Files        1131     1132       +1     
  Lines      419945   420155     +210     
  Branches   419945   420155     +210     
==========================================
+ Hits       343901   344121     +220     
+ Misses      55763    55744      -19     
- Partials    20281    20290       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@jayzhan211, thanks for working on this. The synchronization change looks good to me. Encapsulating the probe-side state behind the final summary makes the ordering requirement much clearer, and the new two-partition tests cover the race in both completion orders. I have one non-blocking suggestion for some additional dictionary-key coverage.

[kosiew]

Could we also add a two-partition LeftAnti case where the NULL-bearing probe partition uses a dictionary key containing only a logical NULL? The existing dictionary test is single-partition, while this new helper covers physical Int32 NULLs. I think that would be useful for exercising the logical_null_count() path through the new cross-partition summary. This is non-blocking.

[jayzhan211]

added them

[kumarUjjawal]

Thanks @jayzhan211

Looks good!

[kosiew]

🚀
@jayzhan211
Thank you for your contribution.

[jayzhan211]

One of my CI failure is nested_key_spill_keeps_groups_unique which seems added in #24889
Another is sql::unparser::test_tpch_unparser_roundtrip stack overflow — most likely collateral from the same memory pressure. 🤔