perf(core): cut datafusion core compile time ~10x

[Dandandan]

Which issue does this PR close?

Adresses: #13814

Third and largest instance of the problem from #24325 (datafusion-catalog),
#24326 (datafusion-session) and #24330 (datafusion-catalog-listing).
Independent of all of them — different crates, so they can merge in any order.

Rationale for this change

datafusion core is the last unit of a cold cargo build -p datafusion and
compiles alone, so its cost lands directly on the build's wall clock. 76% of
its compile time was the trait solver: -Zself-profile reported 75.0s of
evaluate_obligation out of 98.6s total, essentially all of it proving
Send/Sync.

#[async_trait] gives each async fn a where 'life0: 'async_trait, ..
clause. rustc only serves auto-trait obligations from its global evaluation
cache when the ParamEnv is empty, so the Send/Sync proof for everything
the returned future captures is redone per method. In this crate the captured
sets include SessionState, &LogicalPlan and ListingTableConfig, each of
which reaches a large fraction of the logical-plan type graph.

Grouping the goals by the Self type in their ParamEnv shows how concentrated
this was — 14 impls, top 10 = 78% of the total:

impl	trait solving		impl	trait solving
ParquetReadOptions	7.73s		DynamicListTableFactory	5.18s
JsonReadOptions	7.54s		ListingTableFactory	4.56s
DefaultPhysicalPlanner	7.07s		TestTableFactory	4.47s
CsvReadOptions	6.59s		ListingTableConfig	4.30s
DataFrameTableProvider	5.68s		DefaultQueryPlanner	4.28s
(trait default bodies)	5.37s		DefaultTableFactory	4.15s
			SessionState	4.11s
			ArrowReadOptions	3.72s

For contrast, in the same compile 31,818 goals with an empty ParamEnv cost
0.19s in total — 6µs each, against ~1.3ms for the same kind of goal under
async_trait's bounds.

What changes are included in this PR?

First commit. Each of those methods becomes the hand-written desugaring of
async fn, which only forwards; the coroutine is built in a shim with no
where-clauses, so its auto-trait obligations are proved in an empty ParamEnv
and land in the global cache. Method bodies are moved verbatim into inherent fns.

The ReadOptions family (25.6s across four impls, plus the 5.37s default body)
collapses to a single proof: all five impls already delegated to the
_get_resolved_schema default body, which now hands the coroutine to a free
infer_schema_boxed. Because that helper is a plain function with no generics
and no where-clauses, its proof is cached once and shared by every impl.

Second commit, from re-profiling after the first. 11.15s of trait solving
remained, in exactly two places:

1. ListingTableConfigExt::infer was still an async fn capturing
   self: ListingTableConfig (4.37s). It now uses the same shim as
   infer_options beside it.
2. ReadOptions::_get_resolved_schema still carried Self: Sync, which
   #[async_trait] needed while its body was a coroutine capturing &self.
   After the first commit it is neither, so the bound is dead weight — and it
   forced every caller to prove its own type Sync structurally, through
   arrow's DataType/Schema, in a non-empty ParamEnv (2.1–2.4s each for
   Csv/Json/Parquet; ArrowReadOptions was already cheap, having fewer fields).

Two things worth noting for review:

• DefaultPhysicalPlanner::create_initial_plan already used exactly this shape
  (-> BoxFuture<'a, _> plus Box::pin(async move ..)) — there for recursion
  rather than for compile time. The idiom is not new to this codebase.
• The second commit relaxes a bound on a public trait method. Nothing in tree
  overrides _get_resolved_schema (all five impls only implement
  get_resolved_schema) and the underscore prefix marks it as internal, but an
  external override written with #[async_trait] would generate Self: Sync and
  no longer match. Happy to drop that commit if you would rather not touch it.

One body became eager: TestTableFactory::create_inner has no .await, so it is
a plain fn wrapped in ready(..). It builds a TestTableProvider and has no side
effects. Everything that awaits stays lazy — Box::pin(self.m_inner(..)) polls
nothing.

Are these changes tested?

• cargo test -p datafusion --lib — 442 passed
• cargo check -p datafusion --all-targets — clean (covers core's integration
  tests and benches, heavy users of these APIs)
• cargo clippy -p datafusion --lib — clean
• cargo doc with -D warnings — clean
• cargo fmt --check — clean

The compiler checks each rewritten signature against its trait declaration, and
every body is moved verbatim.

cargo rustc -p datafusion --lib with -Ztime-passes, alternated with the base
so machine drift cancels out:

	total	evaluate_obligation
base	88.9s	75.0s
after first commit	18.6s	11.15s
after second commit	8.2s	234ms

234ms over 34,724 goals is 6.7µs each — the same rate as goals that carry an
empty ParamEnv, i.e. the repeated proving is gone rather than merely reduced.
What remains in this crate is LLVM: 7.6s emitting objects and 4.2s in LLVM
passes.

An earlier interleaved wall-clock A/B of the first commit alone measured
74.4s/69.9s base against 16.6s/15.5s fixed.

Are there any user-facing changes?

No, other than the relaxed Self: Sync bound described above. No public
signature changes — after macro expansion these methods have the same signatures
as before.

🤖 Generated with Claude Code

[AdamGS]

LGTM 🥳

[codecov-commenter]

Codecov Report

❌ Patch coverage is 73.78277% with 70 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.13%. Comparing base (ab12f5e) to head (1afc9ec).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...afusion/core/src/datasource/file_format/options.rs	25.00%	42 Missing ⚠️
datafusion/core/src/dataframe/mod.rs	0.00%	25 Missing ⚠️
datafusion/core/src/datasource/listing/table.rs	93.93%	1 Missing and 1 partial ⚠️
datafusion/core/src/physical_planner.rs	97.05%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24329      +/-   ##
==========================================
- Coverage   81.14%   81.13%   -0.01%     
==========================================
  Files        1112     1112              
  Lines      386933   387419     +486     
  Branches   386933   387419     +486     
==========================================
+ Hits       313967   314326     +359     
- Misses      54476    54582     +106     
- Partials    18490    18511      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.