diff --git a/docs/06_vendor_docs/README.md b/docs/06_vendor_docs/README.md index 44a49773..dba649e9 100644 --- a/docs/06_vendor_docs/README.md +++ b/docs/06_vendor_docs/README.md @@ -9,6 +9,7 @@ Engine-, provider-, or vendor-specific references, adapter notes, version quirks | [`github-branch-protection.md`](github-branch-protection.md) | Two procedures (modern Rulesets + classic Branch Protection) for requiring CI status checks before merge to `main`; the three exact check names for the `relyloop` repo; verification + gotchas | `infra_foundation` plan §7.5 manual handoff #3; every operator who needs to update branch rules | | [`solr-10/`](solr-10/) | Apache Solr 10.0 ref-guide pages (asciidoc source, tag `releases/solr/10.0.0`) — matches the `solr:10.0` image. Module loading (`SOLR_MODULES`, no ``), configsets + UPLOAD API, LTR, auth | `infra_adapter_solr` (MVP2 Solr adapter + Compose infra) | | [`solr-9/`](solr-9/) | Apache Solr 9.9 ref-guide pages (asciidoc source, tag `releases/solr/9.9.0`) for cross-version comparison | `infra_adapter_solr` — confirm which behaviours differ between 9.x and 10 | +| [`relevance-tools/`](relevance-tools/) | Distilled capability snapshots (with upstream URLs + access dates) of the adjacent/competing relevance tools — OpenSearch SRW + Relevance Agent, Quepid, RRE, Chorus, Elasticsearch native, Splainer. Competitive-landscape references, not integration targets | Evidence base for [`docs/07_research/comparison.md`](../07_research/comparison.md); refresh when a tool ships a release that flips a comparison cell | ### Solr docs detail diff --git a/docs/06_vendor_docs/relevance-tools/README.md b/docs/06_vendor_docs/relevance-tools/README.md new file mode 100644 index 00000000..8ecab60f --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/README.md @@ -0,0 +1,32 @@ +# Adjacent relevance tools — distilled snapshots + +Distilled capability snapshots of the OSS/commercial relevance-tuning tools +that overlap RelyLoop's surface. These are the **evidence base** for the +competitive matrix in +[`docs/07_research/comparison.md`](../../07_research/comparison.md): each note +records what the tool does, the version/status, the license/tier, the upstream +URLs, and the **access date** the claims were verified against. + +These are *competitive-landscape* references, distinct from the +engine-integration references at the top level (`solr-9/`, `solr-10/`) — RelyLoop +does not integrate with these tools; it is compared against them. + +## Index + +| Note | Tool(s) | Backs comparison columns | +|---|---|---| +| [`opensearch-relevance.md`](opensearch-relevance.md) | OpenSearch Search Relevance Workbench (SRW) + OpenSearch Relevance Agent | "OpenSearch SRW", "OpenSearch Relevance Agent" | +| [`quepid.md`](quepid.md) | Quepid (OpenSource Connections / o19s) | "Quepid" | +| [`rated-ranking-evaluator.md`](rated-ranking-evaluator.md) | Rated Ranking Evaluator (Sease) | "RRE" | +| [`chorus.md`](chorus.md) | Chorus (Querqy / o19s) | "Chorus" | +| [`elasticsearch-native.md`](elasticsearch-native.md) | Elasticsearch native (`_rank_eval`, LTR, deprecated Behavioral Analytics / Search Applications) | "Elasticsearch (native)" | +| [`splainer.md`](splainer.md) | Splainer (o19s) | "Splainer" | + +## Maintenance + +Refresh a note when the tool ships a release that changes a comparison cell. +Always update the **Access date** line and re-verify the upstream URLs when you +touch a note, then propagate any cell change into `comparison.md` (and bump its +"Last updated" line). Corrections preferred over new claims. + +**Last audited:** 2026-05-31. diff --git a/docs/06_vendor_docs/relevance-tools/chorus.md b/docs/06_vendor_docs/relevance-tools/chorus.md new file mode 100644 index 00000000..6f26e1b2 --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/chorus.md @@ -0,0 +1,39 @@ +# Chorus + +**Vendor:** Querqy / OpenSource Connections community (Apache 2.0). +**Access date:** 2026-05-31. + +A **reference integration stack** for e-commerce search — a way to stand up a +realistic relevance stack (search engine + Querqy rules + SMUI + UBI + +Quepid/RRE) rather than a single tuning tool. + +**Capabilities (verified 2026-05-31):** + +- **Engine support — three editions:** Solr (the original/primary, + [`querqy/chorus`](https://github.com/querqy/chorus)), **Elasticsearch** + ([`querqy/chorus-elasticsearch-edition`](https://github.com/querqy/chorus-elasticsearch-edition)), + and **OpenSearch** + ([`o19s/chorus-opensearch-edition`](https://github.com/o19s/chorus-opensearch-edition)). + The ES/OpenSearch editions trail the Solr original in completeness (e.g. SMUI + was built for Solr; **Querqy was removed from the OpenSearch edition** until + updated for recent OpenSearch versions). → **This corrects the earlier + comparison claim of "Solr (primary) + OpenSearch (partial)," which omitted + the Elasticsearch edition.** +- **UBI** — Chorus (esp. the OpenSearch edition) is a reference **UBI showcase**; + UBI itself has plugins for OpenSearch, Elasticsearch, and Solr. +- **Evaluation / A/B** — via the bundled tools (Quepid/RRE), not a native Chorus + feature. +- **Optimization** — none of its own. No automated sweep, no Bayesian search, + no LLM-driven loop, no apply path. It's an assembly of components. + +**Why it matters vs RelyLoop:** Chorus shows how the OSS relevance pieces fit +together across engines, but the *optimization* is left to the human using the +bundled tools — there is no autonomous loop or Git-PR apply path. + +**Upstream URLs:** + +- Solr (primary) — https://github.com/querqy/chorus +- Elasticsearch edition — https://github.com/querqy/chorus-elasticsearch-edition +- OpenSearch edition — https://github.com/o19s/chorus-opensearch-edition +- Querqy umbrella — https://querqy.org/ +- UBI — https://www.ubisearch.dev/ diff --git a/docs/06_vendor_docs/relevance-tools/elasticsearch-native.md b/docs/06_vendor_docs/relevance-tools/elasticsearch-native.md new file mode 100644 index 00000000..f5b4c58e --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/elasticsearch-native.md @@ -0,0 +1,39 @@ +# Elasticsearch (native relevance tooling) + +**Vendor:** Elastic. **License:** Elastic License 2.0 + SSPL (not OSI open +source); feature gating by subscription tier. +**Access date:** 2026-05-31. **Latest major reviewed:** Elasticsearch 9.x. + +What Elasticsearch ships *natively* for relevance evaluation/tuning — distinct +from third-party tools that run against it. + +**Capabilities (verified 2026-05-31):** + +- **`_rank_eval` API** — an evaluation **primitive** (computes MRR, precision, + DCG/nDCG over a query set + rated docs). Available in the **Basic** tier. It + is an API, not a UI/workbench, and it is **not deprecated** in 9.x. +- **No native SRW/RelyLoop equivalent.** Elastic **deprecated** its + higher-level **Behavioral Analytics** and **Search Applications** in + **9.0.0** (Behavioral Analytics emits deprecation warnings; **App Search is + discontinued** with Stack 9.0 and Enterprise Search ships no new majors past + 8.x). Search Applications + Behavioral Analytics required a paid tier + (trial/Platinum/Enterprise) even before deprecation. +- **Learning to Rank (LTR)** — Elasticsearch supports **model inference** for + LTR (GBDT/LambdaMART; training happens outside ES). LTR + ML inference are + **paid-tier** features (Platinum or higher), not Basic. +- **Subscription tiers (2026):** Free/Basic, Gold (Elastic Cloud Hosted only), + Platinum, Enterprise. +- **Optimization / apply path** — none native. No automated sweep, no Bayesian + search, no Git-PR apply. + +**Why it matters vs RelyLoop:** ES gives you the evaluation primitive +(`_rank_eval`) but no optimizer, no workbench (the higher-level products are +deprecated), and its advanced ranking (LTR) is paywalled. RelyLoop runs +against ES (Apache-2.0) and adds the loop + apply path on top. + +**Upstream URLs:** + +- 9.0 release notes (deprecations) — https://www.elastic.co/guide/en/elastic-stack/9.0/release-notes-elasticsearch-9.0.0.html +- `_rank_eval` — https://www.elastic.co/docs/reference/elasticsearch/rest-apis/search-rank-eval +- LTR — https://www.elastic.co/docs/solutions/search/ranking/learning-to-rank-ltr +- Subscriptions / tiers — https://www.elastic.co/subscriptions diff --git a/docs/06_vendor_docs/relevance-tools/opensearch-relevance.md b/docs/06_vendor_docs/relevance-tools/opensearch-relevance.md new file mode 100644 index 00000000..fc2ba7e5 --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/opensearch-relevance.md @@ -0,0 +1,82 @@ +# OpenSearch relevance tooling (SRW + Relevance Agent) + +**Vendor:** OpenSearch project (Apache 2.0). +**Access date:** 2026-05-31. **Latest version reviewed:** OpenSearch 3.6. + +Two distinct, related products ship under the OpenSearch umbrella. RelyLoop is +compared against both. + +--- + +## OpenSearch Search Relevance Workbench (SRW) + +A Dashboards-based workbench for evaluating and comparing search +configurations. The closest competitor to RelyLoop. + +**Capabilities (verified 2026-05-31):** + +- **Query sets, judgments, experiments** — sampling test queries, managing + relevance ratings, and comparing configurations with metrics. New metrics + added through 3.5/3.6: Recall@K, MRR, DCG@K. +- **LLM-as-judge** — GA in 3.5; customizable prompts; requires a configured + LLM connector. +- **UBI-derived judgments** — COEC click model, GA. +- **Scheduled experiments** — nightly/weekly cadence, GA in 3.5. +- **Multi-datasource** — run experiments across multiple sources, added 3.6. +- **Hybrid Search Optimizer** — a **grid search**, not Bayesian. The "Global" + optimizer tests **66 combinations** = `{2 normalization × 3 combination × + 11 weight steps}`, over **hybrid lexical/neural weights only** — not field + boosts or broader query-time parameters. A separate "Dynamic" optimizer + (ML regression / random forest to predict per-query "neuralness") and the + grid optimizer both live in an **external o19s notebook repository**, not as + built-in OpenSearch features. Bayesian optimization is listed as + **"not yet implemented"** in that repo and is tracked in + [RFC #934](https://github.com/opensearch-project/neural-search/issues/934). +- **Apply path** — none. The SRW RFC scopes the product to evaluation and + analysis (query sets / judgments / experimentation); it does not describe + applying winning configs to production, promoting settings, or managing + rollouts. No PR/Git apply mechanism. + +**Why it matters vs RelyLoop:** SRW is the most feature-complete competitor, +but (1) its optimizer is a 66-cell grid over hybrid weights, versus RelyLoop's +Optuna/TPE Bayesian search over the full query-time space; (2) it has no +apply path; (3) it is OpenSearch-only. + +**Upstream URLs:** + +- Workbench docs — https://docs.opensearch.org/latest/search-plugins/search-relevance/using-search-relevance-workbench/ +- Hybrid search optimizer docs — https://docs.opensearch.org/latest/search-plugins/search-relevance/optimize-hybrid-search/ +- Optimization blog — https://opensearch.org/blog/hybrid-search-optimization/ +- External optimizer notebooks (o19s) — https://github.com/o19s/opensearch-hybrid-search-optimization +- Judgments (COEC) — https://docs.opensearch.org/latest/search-plugins/search-relevance/judgments/ +- SRW scope RFC #17735 — https://github.com/opensearch-project/OpenSearch/issues/17735 +- Hybrid optimizer RFC #934 — https://github.com/opensearch-project/neural-search/issues/934 +- Repo — https://github.com/opensearch-project/search-relevance +- 3.5 / 3.6 release notes — https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-3.5.0.md · https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-3.6.0.md + +--- + +## OpenSearch Relevance Agent + +An **experimental** (3.6) conversational, agent-driven tuning system in +OpenSearch Dashboards. A separate product from SRW. + +**Capabilities (verified 2026-05-31):** + +- A **multi-agent, hypothesis-driven** workflow: a User-Behavior-Analysis agent + identifies relevance gaps (from UBI data or query patterns), a + Hypothesis-Generator agent proposes tuning strategies, and an Evaluator agent + validates them via **offline evaluation on historical data**. +- Produces **query-DSL-level optimizations** — refining search fields, + adjusting weights, tuning boost functions. Human-in-the-loop ("you remain the + ultimate decision-maker"). +- **Does not** run large automated parameter sweeps or Bayesian optimization, + and **does not** apply changes to production (online/interleaving tests are + listed as future work). + +**Why it matters vs RelyLoop:** it's conversational like RelyLoop's chat agent, +but it recommends/validates DSL changes one hypothesis at a time rather than +running thousands of Bayesian trials, and it is OpenSearch-only with no +Git-PR apply path. + +**Upstream URL:** https://opensearch.org/blog/introducing-opensearch-relevance-agent-ai-powered-search-tuning/ diff --git a/docs/06_vendor_docs/relevance-tools/quepid.md b/docs/06_vendor_docs/relevance-tools/quepid.md new file mode 100644 index 00000000..1148d297 --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/quepid.md @@ -0,0 +1,39 @@ +# Quepid + +**Vendor:** OpenSource Connections / o19s (Apache 2.0). +**Access date:** 2026-05-31. + +The de-facto open-source **interactive relevance workbench**: create test +cases ("cases"/"books"), gather human judgments, compute metrics (nDCG, etc.), +and watch them move as you tweak the query configuration by hand. + +**Capabilities (verified 2026-05-31):** + +- **Engine support** — OpenSearch, Elasticsearch, Solr, plus Vectara, Algolia, + and custom Search APIs. (Diagnostics are powered by the shared `splainer-search` + library.) +- **LLM-as-judge — built into the OSS core.** Quepid ships an **"AI Judge"** + ("LLM as a Judge") feature, added in **v8.0.0 (2025-02-14)** in the + open-source repo (not a paid/hosted-only feature). You set an LLM key, supply + a customizable judgment prompt, and the AI Judge rates query/document pairs. + Later releases added Ollama support and encrypted key storage (8.2.0). → + **This corrects the earlier comparison claim that Quepid's LLM-judge was a + "community plugin, not in the OSS core."** +- **A/B comparison** — yes, but **manual/human-driven**: a person changes the + query config and compares cases. No automated parameter sweep. +- **Multi-cluster** — yes (multiple cases/configs). +- **Optimization** — none. Quepid does not run automated parameter sweeps or + Bayesian optimization, and does not write/apply search configs (it manages + judgments and measurement, not config deployment). + +**Why it matters vs RelyLoop:** Quepid is the strongest tool for human-rated +judgment management and manual A/B, and it now applies LLMs to *judging*. But +its LLM use is for judging, not for *driving an optimization loop* — there is +no autonomous sweep and no apply path. + +**Upstream URLs:** + +- Repo — https://github.com/o19s/quepid +- Changelog (AI Judge debuts v8.0.0) — https://github.com/o19s/quepid/blob/main/CHANGELOG.md +- Product site — https://www.quepidapp.com/ +- User manual — https://quepid-docs.dev.o19s.com/ diff --git a/docs/06_vendor_docs/relevance-tools/rated-ranking-evaluator.md b/docs/06_vendor_docs/relevance-tools/rated-ranking-evaluator.md new file mode 100644 index 00000000..d0a2b5bb --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/rated-ranking-evaluator.md @@ -0,0 +1,32 @@ +# Rated Ranking Evaluator (RRE) + +**Vendor:** Sease Ltd (Apache 2.0). +**Access date:** 2026-05-31. + +An **offline search-quality evaluation** library/toolkit for Apache Solr and +Elasticsearch. Aimed at the search engineer's CI workflow. + +**Capabilities (verified 2026-05-31):** + +- **Engine support** — Apache Solr + Elasticsearch. +- **How you run it** — primarily a **Java library invoked via a Maven plugin** + (set up a Maven project, import RRE, run the evaluation); there is also an + RRE Server (web console). A standalone **CLI is in development / not part of + the current release** — so "CLI-driven" overstates it; "Maven-plugin / CI-driven" + is accurate. +- **A/B comparison / evaluation** — yes, offline; compares rankings across + configurations/versions and computes IR metrics. +- **Scheduled / unattended** — via CI/cron driving the Maven build (not a + built-in scheduler). +- **Optimization** — none. RRE evaluates and reports; it does not run automated + parameter sweeps, is not LLM-driven, and has no apply path. + +**Why it matters vs RelyLoop:** RRE is a mature offline evaluation harness for +the Solr/ES world, but it's a measurement tool, not an optimizer — no Bayesian +search, no LLM, no Git-PR apply. + +**Upstream URLs:** + +- Repo — https://github.com/SeaseLtd/rated-ranking-evaluator +- Wiki — https://github.com/SeaseLtd/rated-ranking-evaluator/wiki +- Overview — https://sease.io/2021/01/offline-search-quality-evaluation-rated-ranking-evaluator-rre.html diff --git a/docs/06_vendor_docs/relevance-tools/splainer.md b/docs/06_vendor_docs/relevance-tools/splainer.md new file mode 100644 index 00000000..2b047a10 --- /dev/null +++ b/docs/06_vendor_docs/relevance-tools/splainer.md @@ -0,0 +1,29 @@ +# Splainer + +**Vendor:** OpenSource Connections / o19s (Apache 2.0). +**Access date:** 2026-05-31. + +A **relevance diagnostics sandbox** — paste a search URL with all its query +params and Splainer shows the results alongside parsed, summarized `explain` +information so you can understand *why* documents rank where they do. A +debugging tool, not an optimizer. + +**Capabilities (verified 2026-05-31):** + +- **Engine support** — Solr, Elasticsearch, and **OpenSearch** (with + experimental support for others, e.g. Vectara). → **This corrects the earlier + comparison claim of "Solr + ES," which omitted OpenSearch.** The underlying + `splainer-search` library also powers Quepid's diagnostics. +- **What it does** — interactive explain/score breakdown and query tweaking in a + sandbox. +- **What it doesn't** — no judgments/metrics workflow, no automated A/B at + scale, no parameter sweep, no LLM, no scheduling, no apply path. + +**Why it matters vs RelyLoop:** Splainer is the "why did this rank?" microscope; +it's complementary to RelyLoop, not a competitor on optimization or apply. + +**Upstream URLs:** + +- Sandbox app — https://splainer.io/ +- `splainer-search` library — https://github.com/o19s/splainer-search +- Splainer app repo — https://github.com/o19s/splainer diff --git a/docs/07_research/comparison.md b/docs/07_research/comparison.md index eb9f3d99..ecbf2356 100644 --- a/docs/07_research/comparison.md +++ b/docs/07_research/comparison.md @@ -1,25 +1,25 @@ # Comparison with adjacent tools **Status:** Factual reference. Updated when a referenced tool ships a release that changes a row. -**Last updated:** 2026-05-27. +**Last updated:** 2026-05-31. **Last full accuracy audit:** 2026-05-31 — every cell re-verified against live upstream docs; distilled per-tool snapshots (with access dates and upstream URLs) live in [`docs/06_vendor_docs/relevance-tools/`](../06_vendor_docs/relevance-tools/). **Scope:** OSS and commercial relevance-tuning tools that overlap RelyLoop's surface. Excludes general-purpose ML observability (Phoenix, LangSmith, Helicone) — those are complementary, not competitive. This page is a factual matrix, not a sales sheet. Each row links to the tool's own docs so readers can verify claims independently. Where a capability is partial, the cell describes the partial state rather than claiming "no." ## Snapshot matrix -| Capability | RelyLoop (v0.1.0) | OpenSearch SRW (3.6) | OpenSearch Relevance Agent (3.6) | Quepid | RRE | Chorus | Elasticsearch (native) | Splainer | +| Capability | RelyLoop (v0.1.3) | OpenSearch SRW (3.6) | OpenSearch Relevance Agent (3.6) | Quepid | RRE | Chorus | Elasticsearch (native) | Splainer | |---|---|---|---|---|---|---|---|---| | **Bayesian / TPE optimization over full query-time search space** | yes (Optuna TPE, thousands of trials) | no — Hybrid Search Optimizer is a 66-cell grid search over `{2 norms × 3 combiners × 11 weight steps}` for hybrid weights only ([docs](https://docs.opensearch.org/latest/search-plugins/search-relevance/optimize-hybrid-search/)); Bayesian is in [RFC #934](https://github.com/opensearch-project/neural-search/issues/934) with no shipped code | no | no | no | no | no | no | -| **LLM-as-judge with customizable prompts** | yes | yes — GA in 3.5 ([release notes](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-3.5.0.md)) | suggests judgments conversationally | community plugin, not in the OSS core | no | no | no — operators DIY against `_rank_eval` | no | -| **UBI-derived judgments (click streams → ratings)** | planned MVP2 (`feat_ubi_judgments`); hybrid UBI+LLM is the differentiated mode | yes — COEC click model GA ([docs](https://docs.opensearch.org/latest/search-plugins/search-relevance/judgments/)) | no | no | no | yes — reference UBI showcase ([repo](https://github.com/o19s/chorus-opensearch-edition)) | no native UI; UBI plugin available as community fork ([repo](https://github.com/o19s/user-behavior-insights-elasticsearch)) | no | -| **Engine support** | Elasticsearch 8.11+/9.x + OpenSearch 2.x/3.x + **Apache Solr 9.x/10.x (MVP2 shipped)**; one adapter, one workflow | OpenSearch only | OpenSearch only | Solr (since day one) + ES + OpenSearch | Solr + ES | Solr (primary) + OpenSearch (partial) | ES only | Solr + ES | -| **Git-PR apply path (winning configs land as PRs, named approvers merge)** | yes (GitHub today; GitLab + Bitbucket in backlog) | **no — explicitly out of scope by RFC** ([RFC #17735](https://github.com/opensearch-project/OpenSearch/issues/17735): "focuses on evaluation and analysis, not production deployment mechanisms") | no | no — Quepid writes judgments, not configs | no | no | no | no | -| **Search-configuration A/B comparison runner** | indirect (compare studies) | yes — GA in 3.1 ([docs](https://docs.opensearch.org/latest/search-plugins/search-relevance/using-search-relevance-workbench/)) | suggests configs, doesn't compare them at scale | yes (manual) | yes (CLI) | yes (via Quepid) | no — `_rank_eval` is an API primitive, no UI | yes (drill-down) | -| **Scheduled / unattended experiment runs** | yes (Optuna study runs overnight; `feat_auto_followup_studies` chains them) | yes — GA in 3.5 (nightly/weekly/monthly cadence) | no | no | yes (cron-driven CLI) | no | no | no | +| **LLM-as-judge with customizable prompts** | yes | yes — GA in 3.5 ([release notes](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-3.5.0.md)) | suggests judgments conversationally | yes — built-in **AI Judge** ("LLM as a Judge"), customizable prompts, in the OSS core since **v8.0.0 (2025-02-14)**; supports OpenAI + Ollama ([changelog](https://github.com/o19s/quepid/blob/main/CHANGELOG.md)) | no | no | no — operators DIY against `_rank_eval` | no | +| **UBI-derived judgments (click streams → ratings)** | yes — shipped MVP2 (`feat_ubi_judgments`); hybrid UBI+LLM is the differentiated mode | yes — COEC click model GA ([docs](https://docs.opensearch.org/latest/search-plugins/search-relevance/judgments/)) | no | no | no | yes — reference UBI showcase ([repo](https://github.com/o19s/chorus-opensearch-edition)) | no native UI; UBI plugin available as community fork ([repo](https://github.com/o19s/user-behavior-insights-elasticsearch)) | no | +| **Engine support** | Elasticsearch 8.11+/9.x + OpenSearch 2.x/3.x + **Apache Solr 9.x/10.x (MVP2 shipped)**; one adapter, one workflow | OpenSearch only | OpenSearch only | Solr + ES + OpenSearch (+ Vectara, Algolia, custom APIs) | Solr + ES | Solr (primary) + ES + OpenSearch editions (ES/OS editions trail Solr; Querqy removed from the OS edition) | ES only | Solr + ES + OpenSearch | +| **Git-PR apply path (winning configs land as PRs, named approvers merge)** | yes (GitHub today; GitLab + Bitbucket in backlog) | **no** — the SRW scope ([RFC #17735](https://github.com/opensearch-project/OpenSearch/issues/17735)) is bounded to query sets, judgments, and experimentation; it describes no mechanism for applying winning configs to production | no | no — Quepid writes judgments, not configs | no | no | no | no | +| **Search-configuration A/B comparison runner** | indirect (compare studies) | yes — GA in 3.1 ([docs](https://docs.opensearch.org/latest/search-plugins/search-relevance/using-search-relevance-workbench/)) | suggests configs, doesn't compare them at scale | yes (manual) | yes (Maven plugin / CI) | yes (via Quepid) | no — `_rank_eval` is an API primitive, no UI | yes (drill-down) | +| **Scheduled / unattended experiment runs** | yes (Optuna study runs overnight; `feat_auto_followup_studies` chains them) | yes — GA in 3.5 (nightly/weekly/monthly cadence) | no | no | yes (cron/CI-driven Maven build) | no | no | no | | **Multi-cluster support** | yes (one tool, many `clusters` rows) | yes — added in 3.6 ([release notes](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-3.6.0.md)) | yes (3.6) | yes | yes | yes | no | no | | **Conversational agent that runs the loop** | yes (chat orchestrator dispatches `start_study`, `generate_judgments_*`, `open_proposal` tools) | no (SRW UI is form-driven; the Relevance Agent is a separate experimental product) | yes — DSL recommender, **but does not run multi-thousand-trial sweeps** ([blog](https://opensearch.org/blog/introducing-opensearch-relevance-agent-ai-powered-search-tuning/)) | no | no | no | no | no | -| **Local-first LLM observability (self-hosted Langfuse/SigNoz)** | planned MVP2 | n/a (no LLM-as-judge observability surface) | n/a | n/a | n/a | n/a | n/a | n/a | +| **Local-first LLM observability (self-hosted Langfuse/SigNoz)** | planned MVP3 | n/a (no LLM-as-judge observability surface) | n/a | n/a | n/a | n/a | n/a | n/a | | **Apache 2.0 license** | yes | yes ([repo](https://github.com/opensearch-project/search-relevance)) | yes (OpenSearch project) | yes ([repo](https://github.com/o19s/quepid)) | yes ([repo](https://github.com/SeaseLtd/rated-ranking-evaluator)) | yes ([repo](https://github.com/querqy/chorus)) | Elastic License 2.0 + SSPL (not OSI-OSS); `_rank_eval` API is Basic-tier | yes ([repo](https://github.com/o19s/splainer-search)) | | **License tier for relevance-tuning features** | Apache 2.0, all tiers | Apache 2.0, all tiers | Apache 2.0, all tiers | Apache 2.0, all tiers | Apache 2.0, all tiers | Apache 2.0, all tiers | `_rank_eval` Basic; native LTR + ML inference Platinum or higher ([subscriptions](https://www.elastic.co/subscriptions)) | Apache 2.0, all tiers | @@ -27,8 +27,8 @@ This page is a factual matrix, not a sales sheet. Each row links to the tool's o Each individual capability above has at least one OSS comparable. The combination — *Bayesian/TPE optimization across the full search space, on every major open-source engine, with a Git-PR apply path* — does not. Concretely: -- OpenSearch SRW is the closest competitor and ships GA query sets, judgment lists, A/B comparison, LLM-as-judge, scheduled experiments, and UBI judgments — but its optimizer is a 66-cell grid restricted to hybrid weights, and it has no apply path by explicit RFC decision. -- Quepid is the closest *workbench* (manual A/B with judgments) and is the strongest tool for human-rated judgment management; it does not run automated sweeps and is not LLM-driven. +- OpenSearch SRW is the closest competitor and ships GA query sets, judgment lists, A/B comparison, LLM-as-judge, scheduled experiments, and UBI judgments — but its optimizer is a 66-cell grid restricted to hybrid weights (a Bayesian approach is still RFC-only / "not yet implemented" in the external optimizer notebooks), and its RFC scopes it to evaluation with no production-apply path. +- Quepid is the closest *workbench* (manual A/B with judgments) and is the strongest tool for human-rated judgment management. It now ships a built-in LLM-as-judge ("AI Judge") in its OSS core, but it applies LLMs to *judging*, not to *optimization*: it does not run automated parameter sweeps, and it manages judgments/measurement rather than writing or applying search configs. - Elasticsearch ships `_rank_eval` (an API primitive) and deprecated its higher-level Behavioral Analytics and Search Applications products in 9.0 ([release notes](https://www.elastic.co/guide/en/elastic-stack/9.0/release-notes-elasticsearch-9.0.0.html)). There is no native ES equivalent to SRW or RelyLoop. - Solr's ecosystem (Quepid + Chorus + RRE) is mature for manual evaluation but has no auto-optimizer. The UBI schema (`ubi_queries` / `ubi_events`) is standardized across engines, so RelyLoop reads behavioral signals on Solr the same way it reads them on ES/OpenSearch ([UBI tools](https://www.ubisearch.dev/tools/)). (A live `solr.UBIComponent` capture plugin is not yet part of the stock Solr distribution; RelyLoop's read-path is identical regardless of how the events were captured.) @@ -71,8 +71,12 @@ Pull requests welcomed from operators using any of the listed tools — correcti - [Apache Solr LTR reference guide](https://solr.apache.org/guide/solr/latest/query-guide/learning-to-rank.html) - [Sease: Solr 10 — Vector Search and LTR (March 2026)](https://sease.io/2026/03/apache-solr-10-what-is-new-for-vector-search-and-ltr.html) - [Quepid repository](https://github.com/o19s/quepid) +- [Quepid CHANGELOG (AI Judge / "LLM as a Judge" debuts v8.0.0, 2025-02-14)](https://github.com/o19s/quepid/blob/main/CHANGELOG.md) +- [OpenSearch hybrid-search-optimization notebooks (o19s — grid impl; Bayesian "not yet implemented")](https://github.com/o19s/opensearch-hybrid-search-optimization) - [Chorus repository (Solr-centric reference stack)](https://github.com/querqy/chorus) +- [Chorus Elasticsearch edition](https://github.com/querqy/chorus-elasticsearch-edition) - [Chorus OpenSearch edition](https://github.com/o19s/chorus-opensearch-edition) +- [Distilled per-tool snapshots (this audit's evidence base)](../06_vendor_docs/relevance-tools/) - [Rated Ranking Evaluator (Sease)](https://github.com/SeaseLtd/rated-ranking-evaluator) - [Splainer](https://github.com/o19s/splainer-search) - [Elasticsearch 9.0 release notes (Behavioral Analytics + Search Applications deprecation)](https://www.elastic.co/guide/en/elastic-stack/9.0/release-notes-elasticsearch-9.0.0.html)