Skip to content

Fix 64-bit integer truncation and REAL precision loss on fetch (#17) - #30

Merged
jkalias merged 2 commits into
mainfrom
claude/fix-fetch-data-corruption-x6852n
Jul 5, 2026
Merged

Fix 64-bit integer truncation and REAL precision loss on fetch (#17)#30
jkalias merged 2 commits into
mainfrom
claude/fix-fetch-data-corruption-x6852n

Conversation

@jkalias

@jkalias jkalias commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes the silent data-corruption defects described in #17 (which consolidates #15 and #16).

FetchRecordsQuery::GetColumnValue (src/queries.cc) stringifies every fetched column into a std::wstring before it's parsed back into the typed member. Two of these conversions were lossy on read:

  • SQLITE_INTEGER was read with the 32-bit sqlite3_column_int, silently truncating/wrapping any stored int64_t above INT32_MAX (large explicit ids, epoch-millis timestamps, big quantities, and AUTOINCREMENT ids once they grow).
  • SQLITE_FLOAT was formatted with std::to_wstring(double), which uses %f with a fixed 6 decimal places — e.g. 0.123456789 round-tripped back as 0.123457, and large-magnitude doubles were mangled.

Writes were already correct (sqlite3_bind_int64 / sqlite3_bind_double store the true value); the loss was purely on the read path.

Fix

This is approach A from the issue — minimal and text-path-preserving:

  • Integers: read with sqlite3_column_int64 instead of sqlite3_column_int.
  • Doubles: format with %.17g (full round-trip precision) via snprintf, instead of std::to_wstring.

TEXT, BOOL, and DATETIME handling is untouched. The parse side (StringUtilities::ToIntstd::stoll, StringUtilities::ToDoublestd::stod) already round-trips these representations exactly, so no changes were needed there.

Note on scope: this fixes both data-corruption bugs and satisfies the two round-trip acceptance criteria in #17, but it does not remove the intermediate std::wstring round-trip — so it doesn't meet #17's "hydration reads typed columns directly" bullet (the direct-hydration/streaming redesign). I'd suggest keeping #17 open (or retitling it) to track that as a separate performance follow-up, since it's a larger, more invasive change touching FetchQueryResults and its consumers.

Out of scope, left untouched per the issue boundaries: #19 (ToInt/ToDouble returning 0 on parse failure), #20 (NULL vs empty-string conflation), #14 (identifier quoting), relationships/nullable fields/predicates.

Test plan

  • Added DatabaseTest.FetchPreservesInt64ValuesBeyondInt32Range — saves Company records with age values above INT32_MAX (up to near INT64_MAX), fetches them back, asserts exact equality.
  • Added DatabaseTest.FetchPreservesHighPrecisionDoubleValues — saves Company records with a high-precision fractional salary and a large-magnitude salary, fetches them back, asserts exact equality via EXPECT_DOUBLE_EQ.
  • Confirmed both new tests fail against the pre-fix code (observed truncated/wrapped integers and 0.123457 instead of the full-precision double).
  • Confirmed both new tests pass after the fix.
  • Ran the full suite (cmake --build build && ./build/tests/unit_tests) — all 60 tests pass.

Generated by Claude Code

claude added 2 commits July 5, 2026 17:01
GetColumnValue stringified every fetched column into a std::wstring before
hydrating it back into the typed member. Two of these conversions were lossy:

- SQLITE_INTEGER was read with the 32-bit sqlite3_column_int, silently
  truncating/wrapping any stored int64_t above INT32_MAX.
- SQLITE_FLOAT was formatted with std::to_wstring(double), which uses %f with
  a fixed 6 decimal places, corrupting high-precision or large-magnitude
  doubles on read.

Switch to sqlite3_column_int64 for integers, and format doubles with %.17g
(full round-trip precision) instead of to_wstring. TEXT/BOOL/DATETIME
handling is untouched.

Add regression tests covering an int64 value beyond INT32_MAX and
high-precision/large-magnitude doubles, verified to fail before this change
and pass after.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK
StringUtilities::Join(list, char) has no call sites; only the
std::string-separator overload is used. <sstream> in string_utilities.cc is
likewise unused (conversions go through <codecvt>'s wstring_convert).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK
@jkalias
jkalias merged commit 6c4db76 into main Jul 5, 2026
15 checks passed
@jkalias
jkalias deleted the claude/fix-fetch-data-corruption-x6852n branch July 5, 2026 19:47
jkalias added a commit that referenced this pull request Jul 5, 2026
… wstring round-trip (#17) (#31)

* Hydrate fetched rows directly from typed SQLite columns, removing the wstring round-trip (#17)

FetchRecordsQuery::GetResults stringified every fetched column into a
std::wstring and materialized the whole result set as a
vector<vector<wstring>> (FetchQueryResults) before Hydrate parsed each string
back into the typed member. This is the performance/memory half of #17 (the
correctness half - int64 truncation and REAL precision loss - was already
fixed in #30).

Replace the two-pass string round-trip with row-by-row direct hydration:
FetchRecordsQuery now exposes StepRow() (advance the prepared statement) and
HydrateCurrentRow() (read the current row straight into the destination
member via sqlite3_column_int64/_double/_text+_bytes). Database::FetchRecords<T>
owns the construction loop, materializing only the final vector<T> - the
string double-buffer and per-value wstring allocation are both gone.
FetchQueryResults, GetResults, GetColumnValue and the string-based Hydrate
are removed as they're no longer needed.

NULL/empty-string skip semantics are preserved exactly: a NULL column of any
storage class is left unassigned, and an empty TEXT column is likewise left
unassigned, matching the prior text-path behavior bit for bit (the NULL vs
empty-string conflation itself is issue #20 and remains out of scope).

Benchmarked FetchAll<Company>() over 100k rows against the pre-refactor
build: ~2.9x faster wall-clock (225ms -> 77ms) and ~6x fewer allocations
(2.4M -> 400k). Added a 50k-row round-trip test and a dedicated
NULL/empty-string semantics test; full suite (62 tests) passes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK

* Remove now-unused StringUtilities::ToInt/ToDouble

Direct hydration (previous commit) reads ints and doubles straight from the
prepared statement via sqlite3_column_int64/_double, so these two
string-parsing helpers no longer have any callers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK

* Skip BLOB columns during hydration, matching the pre-refactor behavior (#17)

The removed text-based path returned an empty string for any column whose
runtime SQLite type wasn't INTEGER/FLOAT/TEXT, so both NULL and BLOB columns
were skipped during Hydrate regardless of the member's declared storage
class. Direct hydration only replicated the NULL half of that: a BLOB value
(reachable via UnsafeSql or a foreign database file, even for a column
declared as a different storage class) fell through to the typed accessors
unconditionally, which could throw out of StringUtilities::FromUtf8 for
invalid UTF-8 blob bytes on TEXT/DATETIME members, and silently assigned
0/0.0 instead of skipping for INT/BOOL/REAL members.

Extend the skip check to SQLITE_NULL || SQLITE_BLOB, matching the old
GetColumnValue's default case exactly. Adds a regression test inserting an
invalid-UTF8 BLOB into a TEXT-affinity column and asserting Fetch neither
throws nor corrupts the other members.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK

* Bound hydration by the statement's actual column count (#17)

Database::Initialize only ever runs CREATE TABLE IF NOT EXISTS, so a table
created by an older version of a reflected struct (before a field was added)
is never migrated - its actual column count can be smaller than
record.member_metadata.size(). The removed text-based path bounded its loop
by sqlite3_column_count(stmt_) (the statement's real column count);
HydrateCurrentRow used record.member_metadata.size() instead, which can
index sqlite3_column_* past the end of the result row - undefined behavior
per SQLite's docs, even though the vendored SQLite build happens to guard
against it defensively today.

Bound the loop by min(sqlite3_column_count(stmt_), member_metadata.size()),
matching the old behavior exactly. Adds a regression test that drops columns
from an existing table (simulating schema drift) and fetches successfully.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NUt3c1wdseRtRSSXfg3MCK

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants