Skip to content

fix: do not treat concat as preserving lexicographical ordering#23804

Open
buraksenn wants to merge 3 commits into
apache:mainfrom
buraksenn:dont-preserve-sort-ordering-concat
Open

fix: do not treat concat as preserving lexicographical ordering#23804
buraksenn wants to merge 3 commits into
apache:mainfrom
buraksenn:dont-preserve-sort-ordering-concat

Conversation

@buraksenn

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Please check the issue details but main idea is concat does not guarantee lexicographical ordering

What changes are included in this PR?

Main change is simply deleting preserves_lex_ordering method from concat and let it fallback to default false. Also added relevant tests and adjusted comments

Are these changes tested?

Yes adjusted existing and added new tests

Are there any user-facing changes?

No api changes but there is a behavior change I think on optimization part for concat

@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate functions Changes to functions implementation labels Jul 22, 2026

@neilconway neilconway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix; overall looks reasonable to me.

As far as I can tell, there are no longer any expressions for which preserves_lex_ordering will return true. Do we think this API is still useful? Can we come up with any use-cases where either

  1. we might add support for it in the future, or
  2. where extensions would want support for it?

If we can't, we might consider deprecating and removing it.

If we keep it, I'm a bit nervous doing so with no actual implementations / test coverage in-tree...

Comment thread datafusion/physical-expr/src/equivalence/properties/dependency.rs Outdated
Comment thread datafusion/physical-expr/src/equivalence/properties/dependency.rs Outdated
Comment thread datafusion/core/tests/physical_optimizer/enforce_sorting.rs Outdated
@buraksenn

Copy link
Copy Markdown
Contributor Author

Thanks for this fix; overall looks reasonable to me.

As far as I can tell, there are no longer any expressions for which preserves_lex_ordering will return true. Do we think this API is still useful? Can we come up with any use-cases where either

  1. we might add support for it in the future, or
  2. where extensions would want support for it?

If we can't, we might consider deprecating and removing it.

If we keep it, I'm a bit nervous doing so with no actual implementations / test coverage in-tree...

I thought maybe a downstream would depend on it so did not think of removing it but I understand it can be deprecated if its not usable. I'll check other comments thanks for the review

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) and removed core Core DataFusion crate labels Jul 22, 2026

@neilconway neilconway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks!

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.71%. Comparing base (c118002) to head (24c8360).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23804      +/-   ##
==========================================
- Coverage   80.72%   80.71%   -0.01%     
==========================================
  Files        1089     1089              
  Lines      368911   368869      -42     
  Branches   368911   368869      -42     
==========================================
- Hits       297785   297716      -69     
- Misses      53374    53398      +24     
- Partials    17752    17755       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

concat should NOT be reported as preserve lex ordering - wrong output

3 participants