[pull] develop from baserow:develop - #439
Merged
Merged
Conversation
* feat: let a button field start an automation workflow * fix: start only a workflow from the button's own workspace * fix: drop an imported workflow the button's workspace does not hold * test: cover what a click on a start workflow action does * feat: offer the start workflow action in the button field editor * test: cover the start workflow form inside the field editor * docs: say what the start workflow action does and does not do * docs: point to the start workflow action from phase 4's summary * fix: keep a snapshot's start workflow action pointing at its workflow * test: cover the create and type swap routes into the workspace guard * test: pin the start workflow icon, label and immediate dispatch error * docs: say what a click reaches through a started workflow * fix: keep an imported workflow only when the reference is this instance's Ids are one global sequence, so a file written elsewhere can name a workflow number the destination workspace happens to own. Presence in the workspace was read as identity, and the button started work nobody chose. An imported action now keeps its workflow only when this import remapped it, or when the data never left the instance, and in both cases when the workspace still matches. Field duplication and a field type change say so the way every other copy does, with an `is_duplicate` config. The same rule now also refuses a workflow whose trigger cannot be dispatched immediately, which a save with the same id already refused, and a copy asked for by a person drops a workflow that person may not read. * fix: say in the editor that the started workflow cannot be found Trashing the automation leaves the id on the action. The click fails, and the editor said nothing: the shared service type reports a workflow that is missing from the list the same way it reports one still being fetched. The database action type now says it, once the applications have actually been fetched. Every automation of the workspace carries its workflows in that same payload, so before it lands an empty store is what a load still running looks like, and nothing is said. The applications are filtered by what the reader may see, so absence answers whether this editor can see the workflow, not whether it exists. The copy says that, rather than sending someone to replace an action that works for everybody else. * fix: treat a template install as the file import it is A template is imported with `is_duplicate=True`, so the rule that keeps an imported workflow read it as a copy that never left the instance. It is the one exception to that: the file was written on another installation, where the same number meant a different workflow, which is the collision the rule exists to stop. `ImportExportConfig` now says which of the two an import is, and the start workflow action drops a workflow a template names by a number this workspace happens to own. Nothing else reads the new flag, so a template install keeps behaving as it did. Also drops the id mapping `_check_workflow` no longer receives: the import path stopped calling it, and leaving the parameter suggested the save path handles the snapshot case, which only the import path does. * test: cover a duplicated button field keeping its workflow Field duplication skips the serialization import path and builds its own import config, so nothing covered the one call site that config was added for: remove it and every duplicated button silently loses its workflow while the suite stays green. Also says in a comment that a field type change reaching the same code is a restore of what the conversion backed up, not a copy, and that the read check applies to it too. * refactor: read the workspace from the caller, not the store The action list already knows which workspace the button field is in and passes it in the context every other check reads. Taking it from the selected workspace instead said something the type had no business knowing. * fix: resolve an imported workflow reference instead of following it An export of a database without its automation names a workflow id this installation does not have. The foreign key is deferred, so the service row is written and following the reference is what fails, ending the whole import job over a reference that simply has to go. Look the workflow up and blank it when it is not here. Read the id off the service defensively too: the service type builds whatever type the file named, so a hand-edited or version skewed export can leave a start workflow action holding a service that has no workflow at all. * fix: ask the id mapping what it actually remapped `MirrorDict` answers `in` and `get` for every key, and an import installs one under this very key when a workflow is duplicated. Asked that way the guard is told this import remapped an id it never touched, and the collision check below it never runs, so the copy could keep a workflow nobody picked. The key view answers only for what was written. * test: cover a restored button keeping its action target Converting away from a button deletes the row its actions cascade off, so only the backup brings them back, and nothing covered what a restored action ends up pointing at. * refactor: trim comments on the start workflow action * fix: decide a start workflow import before the row is written The import callbacks run after the application's transaction has committed, so a workflow id this installation lacks failed on insert. Permission is now checked before the trigger, through the service. * fix: restore the serializers import develop moved out of the module
* feat: add Phoenix LLM tracing service to the dev stack (ai profile)
* chore(deps): add openinference pydantic-ai instrumentation (dev)
* feat: export assistant traces to Phoenix when BASEROW_ASSISTANT_PHOENIX_URL is set
* fix: harden Phoenix tracing setup after review
* feat: Phoenix postgres storage, ai-evals profile and auth support
* docs: ADR 007 and assistant tracing/evals guides
* feat: assistant eval platform core (types, registries, scenarios)
* feat: eval harness with shared assistant factory
* feat: phoenix dataset sync for assistant evals
* feat: assistant eval experiments via phoenix client + CLI
* feat: migrate core, database and docs evals to the platform
Ports the 42 kuma-core/kuma-database/kuma-docs pytest evals into
declarative EvalCase/EvalScenario registrations, plus structure-level
tests asserting dataset counts, ids, and scenario wiring.
* feat: migrate builder and automation evals to the platform
Ports the 12 builder + 2 proactive + 2 user-source cases into
evals/datasets/builder.py (kuma-builder, 16 cases) and the 7 automation
cases into evals/datasets/automation.py (kuma-automation), following the
Task 5 pattern established in core.py/database.py. Extends
eval_platform/test_datasets.py with counts, id lists, a mode-per-case
assertion, and pre_state snapshot checks for the two cases that need one.
* fix: link traces and final counts for single-case eval runs
* fix: root spans for single-case eval runs use the assistant tracer provider
* feat: assistant eval runner service with run page
Adds the assistant_eval_runner management command (migrate -> instrumentation
-> dataset sync -> single worker thread -> wsgiref server) and its
runner.py module: a stdlib WSGI app serving a plain server-rendered page
(GET /, POST /run, GET /healthz) backed by a bounded in-memory run history
and one background worker draining a queue.Queue via run.run_experiment_for.
Wires the assistant-eval-runner compose service (ai-evals profile, own
baserow_evals DB), renames phoenix-db-init to ai-evals-db-init and extends
it to also create baserow_evals, and adds BASEROW_EVAL_RUNNER_PORT to both
env example files and the dev docs.
* fix: eval runner integration fixes
assistant-eval-runner needs its own SECRET_KEY/DATABASE_*/REDIS_* env vars:
dev-overlay compose services don't inherit docker-compose.yml's
x-backend-variables anchor (that's file-scoped YAML), so the container was
starting without a Django SECRET_KEY and without a Redis password, crashing
on auth. Also links to finished experiments pointed at the container-internal
BASEROW_ASSISTANT_PHOENIX_URL (http://phoenix:6006), unreachable from the
developer's browser.
Adds BASEROW_ASSISTANT_PHOENIX_PUBLIC_URL (defaulting to
http://localhost:${BASEROW_PHOENIX_PORT:-6060}) and has runner.py build the
Phoenix link from it, deep-linking straight to the finished run's experiment
compare view when its dataset_id/experiment_id are known, falling back to
the datasets list otherwise.
* refactor: retire the pytest eval harness
All 65 eval cases now live in baserow_enterprise.assistant.evals.datasets;
delete the legacy pytest-based suite and its `eval` markers.
* docs: rewrite assistant evals guide for the Phoenix platform
* feat: docs judge, prompt sync, git stamping and UI-contributed eval examples
- answer_quality LLM-as-judge evaluator for kuma-docs (BASEROW_EVAL_JUDGE_MODEL)
- Kuma prompts versioned into Phoenix on sync; experiments stamped with
prompt hashes, git branch/commit and judge model
- sync preserves UI-added dataset examples with prompt-match adoption;
eval-export emits paste-ready docs.py snippets
- final-review fixes: skip-safe scoring, runner CSRF/bind hardening,
doc corrections
* feat: eval runner UI round — tabs, fan-out, reference answers, 64-case docs dataset
- Runner page: dataset tabs with selection badges, per-dataset and global
all/none, cross-dataset fan-out (one experiment per dataset), 5s
auto-refresh of runs, persistent run history across restarts, per-dataset
Phoenix links, in-page Help tab rendering the repo docs via /docs.json.
- Reference answers for kuma-docs: EvalCase.reference_answer, judge grades
against it, sync preserves UI-curated ones, eval-export round-trips them.
- kuma-docs grown to 64 cases: reference answers for all, PostHog-mined
frequent questions, and cannot-do guardrail cases.
- Gemini (google:gemini-3.6-flash) in the eval model list.
* feat: runnable UI examples, prompt overrides, eval baseline and results view
- UI-added Phoenix examples now execute: docs questions get the standard
docs checks + judge; tool cases declare scenario/expected_tools/
answer_contains in metadata (empty-workspace default scenario). They are
tickable per dataset on the runner page and run in whole-dataset
experiments too.
- Prompt overrides: run any experiment with a prompt's latest Phoenix
version instead of the code constant (runner checkbox panel with soft
per-tab ordering, or --override-prompt). Agent singletons keep their
dynamic instructions; effective hashes + override list stamped on the
experiment.
- Committed baseline snapshot (evals/baseline.json, 111 cases) with
just b eval-baseline capture/import; the runner imports it on startup so
every Phoenix instance gets a baseline experiment per dataset,
idempotent by content hash and run count.
- Results tab: all datasets in one view for a named experiment, with
baseline deltas and a case-weighted overall row, via Phoenix experiment
annotation summaries.
- New evaluation guide (docs/testing/ai-assistant-eval-analysis.md) served
in the Help tab; layout uses horizontal space better (single tab row,
fluid case columns).
* feat: time and cost in eval results, frozen into the baseline snapshot
- Results tab gains time (sum of run latencies) and cost/token columns per
dataset and summed overall, with lower-is-better baseline deltas.
- eval-baseline capture freezes each dataset's time/cost/token totals into
baseline.json; import stamps them as metadata so imported baselines
(which carry no traces to price) still show time and cost.
- Importing a new baseline supersedes the previously imported one instead
of accumulating duplicates.
- Changelog entry for the eval platform.
* fix: OpenInference attributes on subset-run task root spans
Bare wrapper spans rendered as kind 'unknown' with Unset status and empty
input/output in Phoenix. The Task root span now carries the CHAIN kind, the
case prompt as input, the run result as JSON output, and an explicit OK
status.
* polish; minor fixes
* fix: address eval platform review feedback
Apply production model settings and retries, isolate automation checks, and fix runner configuration, links, and result status.
Defer GPT-5.6 compatibility and baseline comparisons to #6034.
* fix: preserve model profiles in rebased eval harness
Use an explicit model profile for evals and share concrete models with tools. Manage model clients through the existing lifecycle helper and preserve regression coverage from the retired harness.
* fix: harden eval persistence and Phoenix access
Co-authored-by: Przemyslaw Kukulski <przemyslaw+gh@baserow.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )