Skip to content

perf(ci): disable semantic search in default test fixtures#938

Merged
phernandez merged 3 commits into
mainfrom
perf/disable-semantic-in-default-tests
Jun 10, 2026
Merged

perf(ci): disable semantic search in default test fixtures#938
phernandez merged 3 commits into
mainfrom
perf/disable-semantic-in-default-tests

Conversation

@phernandez

@phernandez phernandez commented Jun 10, 2026

Copy link
Copy Markdown
Member

Summary

semantic_search_enabled silently defaults to enabled whenever fastembed + sqlite-vec are importable — which they are in dev and CI. Neither tests/conftest.py nor test-int/conftest.py overrode it, so every test that synced a note ran the real ONNX embedding stack (~5–7s per sync). The dedicated semantic suites (test-int/semantic/) configure the flag explicitly in their own conftest and are unaffected.

On top of that, the four search performance benchmarks in test-int/test_search_performance_benchmark.py force-enable semantic themselves and were running inside every CI integration job (~85s, worst single test 41s) — they aren't collected by just test-semantic and are meant to be run on demand.

Changes

  • Set semantic_search_enabled=False in both shared app_config fixtures (with constraint comments)
  • Deselect benchmark-marked tests from the CI integration recipes: -m "not semantic and not benchmark"
  • Fix six tests that implicitly relied on the enabled default:
    • two reindex_vectors wiring tests: stub delete_stale_vector_rows (the one call in that flow that requires the semantic stack)
    • three embedding-status tests: create the search_vector_embeddings stub table that the file's other tests already use
    • the vec0 regression test: patch the config_manager property for the status call, same pattern as its unit siblings

Measured (local, same machine)

Suite Before After
int SQLite 337s 110s (3.1x)
unit SQLite (2903 tests) 254s
tests/sync/test_watch_service.py 31s 2.9s (10.7x)
worst single int test 40.2s (benchmark) 5.0s

Test plan

  • uv run pytest tests -q — 2903 passed
  • uv run pytest -m "not semantic and not benchmark" test-int -q — 368 passed
  • Postgres int suite running locally; CI validates the full matrix here

🤖 Generated with Claude Code

Reviewed SHA: 8c0ce6d8a7e8134acdf005453a0a9f1a8e9295ab
Verdict: approve
Status: success - BM Bossbot approved this head SHA

Summary:
Reviewed the full provided diff against the engineering style guidance and the touched search/project status code paths. The default test fixtures now explicitly disable semantic search, the semantic and benchmark suites remain opt-in through their own targets, and the latest Postgres portability fix only changes a SQLite-specific monkeypatch to be non-failing on Postgres where that method is not called. No concrete merge-blocking issues found.

Blocking findings:

  • None

Non-blocking findings:

  • None

BM Bossbot image for PR #938

BM Bossbot image choices
  • Pull request: #938
  • Generated asset: /home/runner/work/basic-memory/basic-memory/docs/assets/infographics/pr-938.webp
  • Image model: gpt-image-2
  • Size: 1536x1024
  • Quality: high
  • Image mode: editorial-image
  • Theme source: auto

Theme / visual direction:

classic black-and-white photography: documentary field report, contact sheet, street photography, civic infrastructure, and darkroom contrast

phernandez and others added 2 commits June 9, 2026 20:38
Semantic search silently defaults to enabled whenever fastembed and
sqlite-vec are importable, so every test that synced a note paid the
ONNX embedding stack (~5-7s per sync). The dedicated semantic suites
configure the flag explicitly and are unaffected.

- set semantic_search_enabled=False in tests/ and test-int/ app_config
  fixtures
- deselect on-demand benchmark tests from the CI integration recipes
  (they force-enable semantic themselves and cost ~85s per job)
- fix six tests that implicitly relied on the enabled default: stub the
  vector purge in the reindex_vectors wiring tests, create the
  search_vector_embeddings stub in the embedding-status tests, and patch
  the ConfigManager property in the vec0 regression test

Measured locally: int SQLite 337s -> 206s (-> ~120s with benchmarks
deselected), watch-service unit file 31s -> 2.9s.

Signed-off-by: phernandez <paul@basicmachines.co>
delete_stale_vector_rows is SQLite-only; monkeypatch.setattr raised
AttributeError on PostgresSearchRepository. raising=False attaches an
unused attribute there — the Postgres purge path never calls it.
Verified on both backends.

Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez phernandez merged commit aa9594d into main Jun 10, 2026
29 checks passed
@phernandez phernandez deleted the perf/disable-semantic-in-default-tests branch June 10, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant