Fix 64-bit integer truncation and REAL precision loss on fetch (#17) - #30
Merged
Conversation
GetColumnValue stringified every fetched column into a std::wstring before hydrating it back into the typed member. Two of these conversions were lossy: - SQLITE_INTEGER was read with the 32-bit sqlite3_column_int, silently truncating/wrapping any stored int64_t above INT32_MAX. - SQLITE_FLOAT was formatted with std::to_wstring(double), which uses %f with a fixed 6 decimal places, corrupting high-precision or large-magnitude doubles on read. Switch to sqlite3_column_int64 for integers, and format doubles with %.17g (full round-trip precision) instead of to_wstring. TEXT/BOOL/DATETIME handling is untouched. Add regression tests covering an int64 value beyond INT32_MAX and high-precision/large-magnitude doubles, verified to fail before this change and pass after. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK
StringUtilities::Join(list, char) has no call sites; only the std::string-separator overload is used. <sstream> in string_utilities.cc is likewise unused (conversions go through <codecvt>'s wstring_convert). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK
Merged
5 tasks
jkalias
added a commit
that referenced
this pull request
Jul 5, 2026
… wstring round-trip (#17) (#31) * Hydrate fetched rows directly from typed SQLite columns, removing the wstring round-trip (#17) FetchRecordsQuery::GetResults stringified every fetched column into a std::wstring and materialized the whole result set as a vector<vector<wstring>> (FetchQueryResults) before Hydrate parsed each string back into the typed member. This is the performance/memory half of #17 (the correctness half - int64 truncation and REAL precision loss - was already fixed in #30). Replace the two-pass string round-trip with row-by-row direct hydration: FetchRecordsQuery now exposes StepRow() (advance the prepared statement) and HydrateCurrentRow() (read the current row straight into the destination member via sqlite3_column_int64/_double/_text+_bytes). Database::FetchRecords<T> owns the construction loop, materializing only the final vector<T> - the string double-buffer and per-value wstring allocation are both gone. FetchQueryResults, GetResults, GetColumnValue and the string-based Hydrate are removed as they're no longer needed. NULL/empty-string skip semantics are preserved exactly: a NULL column of any storage class is left unassigned, and an empty TEXT column is likewise left unassigned, matching the prior text-path behavior bit for bit (the NULL vs empty-string conflation itself is issue #20 and remains out of scope). Benchmarked FetchAll<Company>() over 100k rows against the pre-refactor build: ~2.9x faster wall-clock (225ms -> 77ms) and ~6x fewer allocations (2.4M -> 400k). Added a 50k-row round-trip test and a dedicated NULL/empty-string semantics test; full suite (62 tests) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK * Remove now-unused StringUtilities::ToInt/ToDouble Direct hydration (previous commit) reads ints and doubles straight from the prepared statement via sqlite3_column_int64/_double, so these two string-parsing helpers no longer have any callers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK * Skip BLOB columns during hydration, matching the pre-refactor behavior (#17) The removed text-based path returned an empty string for any column whose runtime SQLite type wasn't INTEGER/FLOAT/TEXT, so both NULL and BLOB columns were skipped during Hydrate regardless of the member's declared storage class. Direct hydration only replicated the NULL half of that: a BLOB value (reachable via UnsafeSql or a foreign database file, even for a column declared as a different storage class) fell through to the typed accessors unconditionally, which could throw out of StringUtilities::FromUtf8 for invalid UTF-8 blob bytes on TEXT/DATETIME members, and silently assigned 0/0.0 instead of skipping for INT/BOOL/REAL members. Extend the skip check to SQLITE_NULL || SQLITE_BLOB, matching the old GetColumnValue's default case exactly. Adds a regression test inserting an invalid-UTF8 BLOB into a TEXT-affinity column and asserting Fetch neither throws nor corrupts the other members. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK * Bound hydration by the statement's actual column count (#17) Database::Initialize only ever runs CREATE TABLE IF NOT EXISTS, so a table created by an older version of a reflected struct (before a field was added) is never migrated - its actual column count can be smaller than record.member_metadata.size(). The removed text-based path bounded its loop by sqlite3_column_count(stmt_) (the statement's real column count); HydrateCurrentRow used record.member_metadata.size() instead, which can index sqlite3_column_* past the end of the result row - undefined behavior per SQLite's docs, even though the vendored SQLite build happens to guard against it defensively today. Bound the loop by min(sqlite3_column_count(stmt_), member_metadata.size()), matching the old behavior exactly. Adds a regression test that drops columns from an existing table (simulating schema drift) and fetches successfully. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the silent data-corruption defects described in #17 (which consolidates #15 and #16).
FetchRecordsQuery::GetColumnValue(src/queries.cc) stringifies every fetched column into astd::wstringbefore it's parsed back into the typed member. Two of these conversions were lossy on read:SQLITE_INTEGERwas read with the 32-bitsqlite3_column_int, silently truncating/wrapping any storedint64_taboveINT32_MAX(large explicit ids, epoch-millis timestamps, big quantities, andAUTOINCREMENTids once they grow).SQLITE_FLOATwas formatted withstd::to_wstring(double), which uses%fwith a fixed 6 decimal places — e.g.0.123456789round-tripped back as0.123457, and large-magnitude doubles were mangled.Writes were already correct (
sqlite3_bind_int64/sqlite3_bind_doublestore the true value); the loss was purely on the read path.Fix
This is approach A from the issue — minimal and text-path-preserving:
sqlite3_column_int64instead ofsqlite3_column_int.%.17g(full round-trip precision) viasnprintf, instead ofstd::to_wstring.TEXT, BOOL, and DATETIME handling is untouched. The parse side (
StringUtilities::ToInt→std::stoll,StringUtilities::ToDouble→std::stod) already round-trips these representations exactly, so no changes were needed there.Note on scope: this fixes both data-corruption bugs and satisfies the two round-trip acceptance criteria in #17, but it does not remove the intermediate
std::wstringround-trip — so it doesn't meet #17's "hydration reads typed columns directly" bullet (the direct-hydration/streaming redesign). I'd suggest keeping #17 open (or retitling it) to track that as a separate performance follow-up, since it's a larger, more invasive change touchingFetchQueryResultsand its consumers.Out of scope, left untouched per the issue boundaries: #19 (
ToInt/ToDoublereturning 0 on parse failure), #20 (NULL vs empty-string conflation), #14 (identifier quoting), relationships/nullable fields/predicates.Test plan
DatabaseTest.FetchPreservesInt64ValuesBeyondInt32Range— savesCompanyrecords withagevalues aboveINT32_MAX(up to nearINT64_MAX), fetches them back, asserts exact equality.DatabaseTest.FetchPreservesHighPrecisionDoubleValues— savesCompanyrecords with a high-precision fractionalsalaryand a large-magnitudesalary, fetches them back, asserts exact equality viaEXPECT_DOUBLE_EQ.0.123457instead of the full-precision double).cmake --build build && ./build/tests/unit_tests) — all 60 tests pass.Generated by Claude Code