Skip to content

Null-aware anti join keeps rows whose dictionary build key points at a NULL value #26568

Closed

Description

@jayzhan211

Describe the bug

An uncorrelated null-aware LeftAnti hash join (x NOT IN (subquery)) emits a build row whose key is a dictionary entry pointing at a NULL value. NULL NOT IN (non-empty, NULL-free set) is UNKNOWN, so the row should be dropped, as it is when the key is a physical NULL.

The final step checks the key with the physical is_null (datafusion/physical-plan/src/joins/hash_join/stream.rs, null_aware_left_anti_final_indices). A dictionary key that points at a NULL value is a logical NULL but not a physical one. The null-aware LeftMark join already uses logical NULLs for the same decision (#25693).

To Reproduce

HashJoinExec, JoinType::LeftAnti, null_aware = true, CollectLeft (or a single-partition Partitioned):

  • build c1: Dictionary(Int32, Int32), values [1, NULL, 2], keys [0, 1, 2]
  • probe c2: [5, 6]

Result: [NULL, 1, 2]. Expected: [1, 2].

I could not reach it from SQL: nullif, CASE, regexp_replace, substr and arrow_cast over dictionary columns put NULLs in the keys, not the values, and give correct results. It needs a plan or data source whose dictionaries carry NULL values.

Expected behavior

A logical NULL build key is treated as NULL: dropped when the probe side is non-empty and has no NULL, kept when the probe side is empty.

Activity

  1. abp002 commented on Oct 21, 2026

    @abp002
    Contributor

    take

  2. added a commit that references this issue on Oct 23, 2026
    a79a694
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions