Skip to content

feat(stdlib): Sqlite.affine codegen foundation — db-theory #1a (6 externs wired Deno-ESM end-to-end)#522

Merged
hyperpolymath merged 1 commit into
mainfrom
feat/db-1a-sqlite-foundation
Jun 1, 2026
Merged

feat(stdlib): Sqlite.affine codegen foundation — db-theory #1a (6 externs wired Deno-ESM end-to-end)#522
hyperpolymath merged 1 commit into
mainfrom
feat/db-1a-sqlite-foundation

Conversation

@hyperpolymath
Copy link
Copy Markdown
Owner

Summary

First step of the db-theory programme. SQLite's surface looked partial in the roadmap but was actually vapourware: 6 externs declared in stdlib/Sqlite.affine, zero codegen / zero adapter / zero tests / zero consumers. This PR lands the foundation so the extension PR (#1b — prepared statements, schema introspection, bulk I/O) has a real pathway.

Changes

  • lib/codegen_deno.ml deno_builtins — 6 new entries lowering each extern to its __as_db* JS helper.
  • lib/codegen_deno.ml prelude// ---- Sqlite ---- block with 6 const __as_db* helpers delegating to globalThis.__as_sqlite. Same shape as Motion / Pixi / PixiSound (only Tier-2 binding without it). Inline comment gives the Deno (jsr:@db/sqlite) and Node (better-sqlite3) adapter recipes.
  • tests/codegen-deno/sqlite_smoke.{affine,harness.mjs} — 3 smoke functions exercising full lifecycle, WHERE id = ? param-bound projection, and ordered multi-row retrieval. Harness installs an in-memory SQL mock (CREATE / INSERT / SELECT / WHERE / ORDER BY / COUNT(*) subset) explicitly marked as smoke-only.

Verification

  • dune build bin/main.exe: clean
  • dune runtest: 357 / 357 green
  • tools/run_codegen_deno_tests.sh: all 31 harnesses pass (was 30)

Next (db-theory #1b — separate PR)

  • Stmt opaque type + db_prepare / db_bind_* / db_step / db_column_* / db_finalize
  • db_schema_tables() + db_schema_columns(table) introspection
  • db_import_csv / db_export_csv bulk I/O
  • Typed DbError + Result<T, DbError> returns

Test plan

  • dune build clean
  • dune runtest 357 / 357 green (codegen ⊆ stdlib consistency picks up the 6 new builtins)
  • tools/run_codegen_deno_tests.sh 31 / 31 harnesses pass
  • sqlite_smoke mock validates all three smoke functions

🤖 Generated with Claude Code

…erns wired Deno-ESM end-to-end)

First step of the database-theory functionality programme (see survey
in 2026-06-01 session). The audit found the SQLite surface had a
deceptive "partial" status: `stdlib/Sqlite.affine` declared 6 externs
(`db_open` / `db_close` / `db_execute` / `db_query` / `db_query_one` /
`db_query_int`) but NO codegen lowering existed in any backend, NO JS
runtime adapter shipped in the prelude, NO interpreter builtins were
registered, and ZERO consumer code or tests exercised the surface.
Functionally `Sqlite.affine` was vapourware behind a stable-looking
extern signature.

This PR lands the foundation so future extension PRs (prepared
statements, schema introspection, bulk I/O — db-theory #1b) have a
working pathway to build on:

## Codegen lowerings (`lib/codegen_deno.ml`)

6 new entries in `deno_builtins`, mapping each AffineScript extern to
the matching `__as_db*` JS helper.

## JS runtime helpers (`lib/codegen_deno.ml` prelude)

A new `// ---- Sqlite ----` block adds six `const __as_db*` helpers
that delegate to `globalThis.__as_sqlite`, an adapter namespace the
host installs once at startup. The shape mirrors the Motion / Pixi /
PixiSound binding pattern (the canonical "host installs a globalThis
shim" idiom in this repo) — Sqlite was the only Tier-2 binding without
one. Two concrete adapter recipes are inlined in the runtime comment:

- Deno: `jsr:@db/sqlite`'s `Database` class adapted to the contract.
- Node: `better-sqlite3`'s `Database` adapted likewise.

Parameter marshalling stays JSON-string-encoded to match the existing
extern signatures (richer typed bindings ship in #1b alongside a
prepared-statement `Stmt` type).

## Smoke harness (`tests/codegen-deno/sqlite_smoke.{affine,harness.mjs}`)

Three smoke functions exercise the full lifecycle, single-row binding
+ projection, and ordered multi-row retrieval. The `.harness.mjs`
installs a deliberately minimal in-memory SQL mock satisfying the
`__as_sqlite` contract — just enough CREATE / INSERT / SELECT / WHERE-?
/ ORDER BY / COUNT(*) subset to drive the 6 externs. The mock is
explicitly NOT a real SQL engine and the harness commentary directs
production deployments to swap to `@db/sqlite` or `better-sqlite3`.

## Verification

- `dune build bin/main.exe`: clean
- `dune runtest`: 357 / 357 green (codegen ⊆ stdlib consistency test
  picks up the 6 new builtins and verifies the matching `pub extern fn`
  decls exist in `stdlib/Sqlite.affine` — they do, from before)
- `tools/run_codegen_deno_tests.sh`: all 31 harnesses pass (was 30)

## db-theory #1b scope (next PR, separate)

- `Stmt` opaque type + `db_prepare` / `db_bind_int` / `db_bind_text` /
  `db_bind_null` / `db_step` / `db_column_int` / `db_column_text` /
  `db_finalize` for prepared statements.
- `db_schema_tables() -> [String]` + `db_schema_columns(table)` for
  introspection.
- `db_import_csv` / `db_export_csv` for bulk I/O.
- Typed `DbError` extern + `Result<T, DbError>` returns where the JSON
  string form is too lossy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@hyperpolymath hyperpolymath enabled auto-merge (squash) June 1, 2026 12:38
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 1, 2026

🔍 Hypatia Security Scan

Findings: 89 issues detected

Severity Count
🔴 Critical 2
🟠 High 16
🟡 Medium 71

⚠️ Action Required: Critical security issues found!

View findings
[
  {
    "reason": "Action perpolymath/standards/.github/workflows/governance-reusable.yml@main\n needs attention",
    "type": "unpinned_action",
    "file": "governance.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Action ons/checkout@v6\n    needs attention",
    "type": "unpinned_action",
    "file": "publish-jsr.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Action land/setup-deno@v2\n    needs attention",
    "type": "unpinned_action",
    "file": "publish-jsr.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in affine-vscode-publish.yml",
    "type": "missing_timeout_minutes",
    "file": "affine-vscode-publish.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in casket-pages.yml",
    "type": "missing_timeout_minutes",
    "file": "casket-pages.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in casket-pages.yml",
    "type": "missing_timeout_minutes",
    "file": "casket-pages.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in casket-pages.yml",
    "type": "missing_timeout_minutes",
    "file": "casket-pages.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "missing_timeout_minutes",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "missing_timeout_minutes",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "missing_timeout_minutes",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  }
]

Powered by Hypatia Neurosymbolic CI/CD Intelligence

@hyperpolymath hyperpolymath merged commit fffae8b into main Jun 1, 2026
27 checks passed
@hyperpolymath hyperpolymath deleted the feat/db-1a-sqlite-foundation branch June 1, 2026 12:43
hyperpolymath added a commit that referenced this pull request Jun 1, 2026
…yped externs) (#524)

## Summary

Layers the prepared-statement surface on top of the #522 convenience
surface. Use this for anything carrying **user input**, anything
iterating over **more than a handful of rows**, and anything where
**typed value marshalling** matters (Int / Text / NULL).

## New stdlib surface (`stdlib/Sqlite.affine`)

| extern | purpose |
|---|---|
| `extern type Stmt` | opaque statement handle |
| `db_prepare(d, sql) -> Stmt` | compile SQL with `?` placeholders |
| `db_bind_int(s, idx, v) -> Int` | bind 1-indexed param (sqlite3
convention) |
| `db_bind_text(s, idx, v) -> Int` | bind text |
| `db_bind_null(s, idx) -> Int` | bind NULL |
| `db_step(s) -> Int` | `1` = `SQLITE_ROW`, `0` = `SQLITE_DONE` |
| `db_column_count(s) -> Int` | current-row width |
| `db_column_int(s, idx) -> Int` | 0-indexed read; NULL → 0 |
| `db_column_text(s, idx) -> String` | 0-indexed read; NULL → "" |
| `db_reset(s) -> Int` | re-step from row 0 without recompiling |
| `db_finalize(s) -> Int` | release |

Bind-index is 1-based, column-index is 0-based — matches both
`jsr:@db/sqlite` and `better-sqlite3` (the two reference `__as_sqlite`
adapters). NULL coercion happens at the JS-helper layer; callers needing
NULL/value-zero discrimination use the convenience `db_query_one` path.

## Smoke harness
(`tests/codegen-deno/sqlite_prepared.{affine,harness.mjs}`)

| smoke fn | exercises |
|---|---|
| `smoke_prepare_bind_int_step_finalize` | `INSERT(?, ?)` with two int
binds + finalize + query-back |
| `smoke_step_iteration` | `SELECT` N rows, loop `db_step` until 0,
accumulate via `db_column_int` |
| `smoke_text_bind_and_column` | bind_text → step → column_text
round-trip |
| `smoke_null_bind` | bind_null → column_int coerces to 0 |
| `smoke_reset_and_reuse` | prepare once + bind/step + reset +
re-bind/step → 2 rows |
| `smoke_column_count_basic` | `SELECT a, b, c FROM t` → column_count =
3 |

Mock extends #522's in-memory adapter with 10 statement-side methods;
production adapters (`jsr:@db/sqlite` / `better-sqlite3`) provide
near-1:1 wrappers.

## Verification

- `dune build bin/main.exe`: clean
- `dune runtest`: **363 / 363** green (codegen ⊆ stdlib consistency
picks up the 10 new builtins; all match `stdlib/Sqlite.affine`)
- `tools/run_codegen_deno_tests.sh`: **32 / 32** harnesses pass (was 31)

## Stacked on #522

#522 merged 12:43Z; this branch forks from the resulting main HEAD.
Clean diff.

## db-theory #1c (next PR, separate)

- Schema introspection (`db_schema_tables`, `db_schema_columns`)
- Bulk I/O (`db_import_csv`, `db_export_csv`)
- Typed `DbError` + `Result<T, DbError>` variants

## Test plan

- [x] dune build clean
- [x] dune runtest 363/363 green
- [x] tools/run_codegen_deno_tests.sh 32/32 pass
- [x] 6 prepared-statement smoke assertions pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hyperpolymath added a commit that referenced this pull request Jun 2, 2026
…on — db-theory #1c (6 externs) (#525)

## Summary

Closes out the SQLite stdlib triad (#522 foundation + #524 prepared
statements + this PR). Adds the three remaining practical surfaces:

| layer | externs |
|---|---|
| **Schema introspection** | `db_schema_tables(d)`,
`db_schema_columns(d, table)`, `db_table_exists(d, table)` |
| **Bulk CSV I/O** | `db_import_csv(d, table, path, has_header)`,
`db_export_csv(d, sql, params, path)` |
| **Error inspection** | `db_last_error(d)` (typed `DbError` carrier
lands in #1d) |

## Codegen + adapter

6 `b "name"` registrations + 6 `__as_db*` JS helpers, each delegating to
one new method on `globalThis.__as_sqlite` (`schemaTables`,
`schemaColumns`, `tableExists`, `importCsv`, `exportCsv`, `lastError`).
Real adapters back these with one-line wrappers (PRAGMA table_info,
prepare().iterate(), fs writes).

## Smoke harness
(`tests/codegen-deno/sqlite_introspect_bulk.{affine,harness.mjs}`)

8 smoke functions × 9 assertions — schema_tables ordering, column
descriptor shape (incl. PK flag), table_exists true/false, CSV import
skipping header, CSV export including header, last_error
empty/fault-injected. Mock extends the #1a/#1b adapter with the 6 new
methods plus a virtual filesystem map for hermetic CSV testing.

## Verification

- `dune build`: clean
- `dune runtest`: **363 / 363** green
- `tools/run_codegen_deno_tests.sh`: **33 / 33** harnesses pass (was 32)

## Stacked on #524

#524 (db-theory #1b — prepared statements) **MERGED 13:15Z**. This
branch forks from its head and rebases cleanly.

## db-theory #1d (next PR, separate)

- `extern type DbError` opaque carrier
- `Result<T, DbError>`-shaped variants of the convenience surface
- Design-blocked on the wider error-model question (do we add `Result`
returns to all extern fns or keep the JSON-string sentinel pattern?)

## Test plan

- [x] dune build clean
- [x] dune runtest 363/363 green
- [x] 33/33 codegen-deno harnesses pass
- [x] 9 introspection/bulk/error smoke assertions pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hyperpolymath added a commit that referenced this pull request Jun 2, 2026
## Summary

Refresh the authoritative status surfaces in this repo to reflect the
stdlib **db-theory triplet** that landed yesterday and today:

- PR #525 — `Sqlite.affine` schema introspection + bulk I/O + error
inspection (db-theory #1c, 6 externs)
- PR #526 — `Transaction.affine` (db-theory #2, affine-bounded write-set
isolation, 8 externs)
- PR #527 — `Aggregate.affine` (db-theory #3, SQL group-by + aggregation
primitives, 7 externs)
- PR #528 — CHANGELOG sync (already merged)

`CHANGELOG.md` is already up to date via #528, so this PR touches only
the four files that ledger that work as status, not as history.

## What changed

- `docs/CAPABILITY-MATRIX.adoc` — replace the stale "19/19 stdlib files"
sentence (the AOT gate now discovers files dynamically) and call out the
db-theory triplet as the current authoring frontier alongside Http /
Json / Dict-Map.
- `docs/TECH-DEBT.adoc` §C STDLIB — add a `STDLIB-05` row capturing the
full Sqlite arc (#522 / #524 / #525), Transaction (#526), and Aggregate
(#527), with pointers to the two academic proofs the PRs already shipped
(`docs/academic/proofs/db-theory-{2,3}-*.md`).
- `docs/stdlib-roadmap.adoc` — promote `Sqlite.affine` from `partial /
Coverage audit needed` to `usable`, and add `Aggregate.affine` +
`Transaction.affine` rows to the inventory snapshot.
- `.machine_readable/6a2/STATE.a2ml` — bump `last-updated` to
`2026-06-02` (this file mirrors, it does not lead — per its own
drift-flag).

## What was deliberately *not* touched

- `docs/ROADMAP.adoc` — language/compiler progress; the db-theory PRs
are stdlib-side and don't move the language roadmap.
- A `TEST-NEEDS` file — none exists at repo root; the three PRs each
shipped their own Deno-ESM smoke harness under `tests/codegen-deno/`
(`sqlite_introspect_bulk` / `transaction_smoke` / `aggregate_smoke`) and
the AOT gate auto-picked the new modules up.
- License / SPDX headers — per the estate's no-automated-licence-edits
rule.

## Test plan

- [x] `bash tools/check-doc-truthing.sh` passes locally — banner /
primacy / mirror invariants intact, over-claim ratchet unchanged (no new
baseline entry needed).
- [x] Commit GPG-signed; verified `gpg: Good signature`.

Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant