PERF: Optimize checked temporal fetch construction

[Jahnvi Thakkar (jahnvi480)]

Work Item / Issue Reference

GitHub Issue: #554

ADO Task: AB#48255

---

Summary

Reduce Python object-construction overhead when fetching DATE, TIME, and TIMESTAMP values. Previously, native temporal fields were converted into Python arguments and passed through a generic call to an already-cached constructor. Repeated imports were not the bottleneck.

The new helper uses checked CPython construction APIs when the cached constructor is the exact standard type; substituted constructors retain the original call. Six row-wise/batch conversion sites change. Field validation and final object allocation remain. NULLs, precision, timezone/fold, ownership, and exception behavior are preserved; DATETIMEOFFSET, UUID, Decimal, and text are untouched by this PR.

flowchart LR
    A["Native temporal fields after NULL checks"] --> B["Before: Python arguments and generic cached-constructor call"]
    A --> C{"After: exact standard type?"}
    C -->|"Yes"| D["Direct checked CPython construction"]
    C -->|"No: original fallback"| B
    B --> E["Validated Python object in result row"]
    D --> E

Loading

Fresh measurements

Temporal cases contain NULLs every seventh row. Pure cases have eight temporal columns; the row-wise case adds one harmless MAX column. Mixed has DATE/TIME/DATETIME2/DATETIMEOFFSET; narrow is an unchanged int/text/float control.

Workload / path	API / requested batch	Rows × columns	Before → after fetch time	Reduction
DATE / bounded	fetchmany(1000)	4,000 × 8	6.060 → 5.050 ms	16.67%
TIME(7) / bounded	fetchmany(1000)	4,000 × 8	6.815 → 5.479 ms	19.60%
DATETIME2(7) / bounded	fetchmany(1000)	4,000 × 8	8.077 → 5.359 ms	33.65%
DATETIME2(7) / MAX-forced row-wise	fetchall() / all remaining	4,000 × 9	13.906 → 9.562 ms	31.24%
Mixed temporal	Repeated fetchone() / 1	4,000 × 4	234.946 → 231.518 ms	1.46%; inconclusive
Unchanged narrow	fetchmany(1000)	10,000 × 3	6.493 → 6.634 ms	-2.16%

Method/build: September 21 Docker Linux x64; Python 3.13.15, pybind11 3.0.1, GCC 12.2 Release -O3 -DNDEBUG, profiling OFF, SQL Server 16.0.4225.2, ODBC 18.6.2.1. Main c963ee1e versus PR 5aaa6aae: 10 counterbalanced pairs × 5 samples × 14 cases, totaling 1,400 validated drains. Reductions are ratios of medians, excluding execute/validation; no outlier removal or retries. Fallback checks observed three callbacks per temporal type with 3/4/7 positional arguments—not optimized-path or allocation counts.

Limits: identical-build A/A calibration was noisy (speed-ratio interval 0.882×–1.345×). Mixed fetchone and bounded DATE fetchall remain inconclusive; unchanged narrow many/all had negative point estimates with intervals spanning zero. These are scoped bulk-temporal gains, not universal speedups or a no-regression guarantee.

Both builds passed six fresh-process compatibility modes covering boundaries, NULLs, types, substitutions, exceptions, recovery, and both fetch paths. No full-suite or all-OS success is claimed. Complete samples, intervals, provenance, and historical limitations remain in retained local evidence.

[Copilot]

🔵 Needs a closer look

Native extension changes span multiple fetch paths, with limited platform and validation coverage.

Pull request overview

Optimizes native SQL temporal fetch construction while preserving custom constructors and conversion behavior.

Changes:

• Adds checked datetime construction helpers.
• Integrates them into six temporal fetch paths.
• Adds regression tests and changelog documentation.

File summaries

File	Reviewed changes
tests/test_038_fetch_temporal.py	Temporal parity, constructor, exception, and fetch API coverage
mssql_python/pybind/fetch_temporal.hpp	Translation-unit-local checked construction helpers
mssql_python/pybind/ddbc_bindings.cpp	Integration into row-wise and batch temporal fetch paths
CHANGELOG.md	Documents the optimization and preserved behavior

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 92%

🎯 Overall Coverage 84%

📈 Total Lines Covered: 8815 out of 10455 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/pybind/ddbc_bindings.cpp (91.7%): Missing lines 4471
• mssql_python/pybind/fetch_temporal.hpp (92.1%): Missing lines 17-19

Summary

• Total: 50 lines
• Missing: 4 lines
• Coverage: 92%

mssql_python/pybind/ddbc_bindings.cpp

Lines 4467-4475

  4467                 case SQL_SS_TIME2: {
  4468                     const SQL_SS_TIME2_STRUCT& t2 = buffers.timeBuffers[col - 1][i];
  4469                     py::object timeObj =
  4470                         FetchTemporal::time(t2.hour, t2.minute, t2.second, t2.fraction / 1000);
! 4471                     PyList_SET_ITEM(row, col - 1, timeObj.release().ptr());
  4472                     break;
  4473                 }
  4474                 case SQL_SS_TIMESTAMPOFFSET: {
  4475                     SQLULEN rowIdx = i;

---

mssql_python/pybind/fetch_temporal.hpp

Lines 13-23

  13 
  14 // datetime.h keeps PyDateTimeAPI per translation unit, so these helpers must too.
  15 static inline void ensure_datetime_api() {
  16     if (PyDateTimeAPI == nullptr) {
! 17         PyDateTime_IMPORT;
! 18         if (PyDateTimeAPI == nullptr) throw py::error_already_set();
! 19     }
  20 }
  21 
  22 static inline py::object date(int year, int month, int day) {
  23     ensure_datetime_api();

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 64.1%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 78.4%
mssql_python.pybind.connection.connection.cpp: 82.5%
mssql_python.pybind.connection.connection_pool.cpp: 82.9%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.fetch_temporal.hpp: 92.1%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[github-actions]

PR Performance Report

✅ Performance improved

1 database task consistently improved across 1 measured environment. No consistent slowdowns were detected.

1 IMPROVEMENT 0 SLOWDOWNS 2/2 ENVIRONMENTS

Signal fingerprint

Database task	Unix / SQL Server 2025
Row fetching in batches of 10,000	21.0% faster

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Measured timings

Environment	Database task	Before	After	Change
Unix / SQL Server 2025	Row fetching in batches of 10,000	152.038 ms	119.004 ms	-21.0%

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

Unix / SQL Server 2025

Row fetching in batches of 10,000: py::fetchmany::cpp_call -31.017 ms; ddbc::FetchMany_wrap -30.945 ms; ddbc::FetchBatchData -18.479 ms.

All database tasks and timings

Unix / SQL Server 2022

Database task	Before	After	Paired change	Result
Connection opening	10.294 ms	10.133 ms	-1.6%	no signal
SELECT queries	1.054 ms	1.198 ms	+8.1%	no signal
Row insertion	33.963 ms	34.017 ms	-0.0%	no signal
Executemany inserts	153.475 ms	154.176 ms	+0.9%	no signal
Fetch-all queries	140.806 ms	120.117 ms	-15.4%	no signal
Row-by-row fetching	56.976 ms	55.149 ms	-3.2%	no signal
Batched row fetching	141.990 ms	123.918 ms	-13.5%	no signal
Transaction commit and rollback	113.336 ms	113.433 ms	-0.1%	no signal
Arrow row fetching	94.578 ms	95.271 ms	+1.0%	no signal
100,000-row insertion	442.567 ms	477.625 ms	+1.5%	no signal
Row fetching in batches of 100	193.217 ms	174.523 ms	-9.5%	no signal
Row fetching in batches of 10,000	153.197 ms	135.257 ms	-7.6%	no signal
Repeated positional queries	41.374 ms	41.298 ms	-0.2%	no signal
Repeated named-parameter queries	44.153 ms	43.943 ms	-0.9%	no signal
Legacy 100,000-row insertion	343.806 ms	344.655 ms	+1.4%	no signal
Insertion with explicit input sizes	481.229 ms	481.967 ms	-0.3%	no signal
Joined aggregation queries	176.117 ms	177.259 ms	+0.3%	no signal
Large joined-result fetching	190.935 ms	175.461 ms	-8.1%	no signal
1.2-million-row fetching	3439.841 ms	3461.804 ms	+0.8%	no signal
Common table expression queries	5.248 ms	5.324 ms	+2.8%	no signal

Unix / SQL Server 2025

Database task	Before	After	Paired change	Result
Connection opening	95.030 ms	95.305 ms	+0.2%	no signal
SELECT queries	1.157 ms	1.015 ms	-13.5%	no signal
Row insertion	32.630 ms	32.526 ms	+0.3%	no signal
Executemany inserts	146.585 ms	150.080 ms	+2.4%	no signal
Fetch-all queries	137.820 ms	118.032 ms	-13.9%	no signal
Row-by-row fetching	56.821 ms	54.893 ms	-3.4%	no signal
Batched row fetching	141.652 ms	121.223 ms	-14.4%	no signal
Transaction commit and rollback	110.450 ms	108.290 ms	-1.6%	no signal
Arrow row fetching	92.710 ms	91.827 ms	-1.3%	no signal
100,000-row insertion	418.185 ms	446.200 ms	+6.7%	no signal
Row fetching in batches of 100	192.194 ms	173.224 ms	-10.6%	no signal
Row fetching in batches of 10,000	152.038 ms	119.004 ms	-21.0%	consistent improvement
Repeated positional queries	39.909 ms	39.391 ms	-0.6%	no signal
Repeated named-parameter queries	42.292 ms	42.015 ms	-0.9%	no signal
Legacy 100,000-row insertion	329.731 ms	335.885 ms	+1.9%	no signal
Insertion with explicit input sizes	465.187 ms	468.077 ms	+2.3%	no signal
Joined aggregation queries	157.863 ms	157.454 ms	+0.3%	no signal
Large joined-result fetching	188.293 ms	173.568 ms	-7.9%	no signal
1.2-million-row fetching	3467.215 ms	3495.725 ms	+0.5%	no signal
Common table expression queries	5.082 ms	5.092 ms	-0.8%	no signal

Build and measurement details

ADO build 177131

PR head: a0bf434af1d6166d6a37d49d2186d0bf02df0111
Base: a5faa3289a4cec65495bc51bfdb1135c8e70d97f
Measured merge: 7088f708d9450ac1476793739977740c38037a88

• Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4295.3; 5 paired comparisons and 1 warmup.
• Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5005.3; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

[Copilot]

🔵 Needs a closer look

Native C++/CPython changes and stated cross-platform validation limitations warrant final human review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The isolated Linux wheel-validation job fails on a source-tree-relative module path assertion.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Native fetch-path and temporal-construction changes warrant final human review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Copilot review overview

🔵 Needs a closer look

Fix the installed-wheel Linux test path handling before approval.

Review effort: Lite
Findings: 1

Open (1)

• The Linux wheel-validation job copies tests/ into /test_isolated_${PYBIN} and runs the…

[Copilot]

Copilot review overview

🔵 Needs a closer look

Native fetch-path changes and scoped compatibility/performance validation warrant final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

• The Linux wheel-validation job copies tests/ into /test_isolated_${PYBIN} and runs the…

[Sumit Sarabhai (sumitmsft)]

PR #795: No actionable findings in the reviewed changes. The previous installed-wheel provenance concern is resolved through conditional source validation. Checked temporal construction preserves ranges, precision, substitutions, and exception propagation.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The changes affect multiple native fetch paths, with validation scoped rather than full-suite and all-OS coverage.

Review effort: Lite
Findings: None

[Gaurav Sharma (bewithgaurav)]

no regressions observed, lgtm

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical workflow and test findings, plus an incomplete-profiler verdict issue, remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• PR-controlled profiler code can exfiltrate the write-capable token · New
• New head.sha checkout conflicts with the existing test assertion · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

The workflow’s write-capable token can be used by checked-out PR code, and profiler fingerprint rendering lacks focused test coverage.

Review effort: Lite
Findings: None

Resolved since last review (2)

• New head.sha checkout conflicts with the existing test assertion
• PR-controlled profiler code can exfiltrate the write-capable token