Support null aware hash mark-joins

[AdamGS]

This is still a draft, I'm putting it up because because might want to weigh in, and I find it useful to be able to see the diff clearly.

Which issue does this PR close?

• Closes Mark joins don't support null mark columns #21309.

Rationale for this change

This change is about correctness/sql completeness, but is also a step towards better subquery de-correlation.

Before this change, mark joins only produced boolean true / false results, so queries such as (id NOT IN (...)) IS NULL could
return incorrect results, especially for correlated scalar subqueries.

What changes are included in this PR?

1. Adds support for null-aware mark joins.
2. Make sure queries that joins that require null awareness go through a join implementation that supports that.
3. Extends decorrelation so scalar NOT IN mark joins use null-aware semantics when the predicate can be represented as hash join keys.

Are these changes tested?

1. Existing SLT tests that explicitly showed bad results.
2. New dedicated SLT tests.
3. New unit tests.

Are there any user-facing changes?

This PR changes planning behavior and introduces more public API around hash joins, I'll finalize this section as it gets closer to a reviewable state. It also introduces one minor public function - build_join_schema_with_null_aware.

AI Usage

AI was used in the process of developing this PR, mostly around testing and planning

[Dandandan]

@AdamGS are you planning to continue on this soon?

[AdamGS]

yes! I have some more changes locally that I haven't pushed. Codex also came up with a test case that I'm trying to figure out if it should be covered by this change or is a different correctness corner that DataFusion currently misses.
I'll try and clean up what I have over the weekend and make progress on this.

[Dandandan]

LMK when you are ready, I can allocate some time to review

[AdamGS]

pushed some stuff I have locally, mostly to show a regression test that I think is relevant.
I have more work locally, hoping to have tomorrow to mostly push on this.

[AdamGS]

@Dandandan got distracted by a lot of other stuff, but I think its getting pretty close. I'm going to spend more time this weekend trying to review it and think through the change with the paper.