Add support for Vector Search Endpoint (direct only) by janniklasrose · Pull Request #4887 · databricks/cli

janniklasrose · 2026-04-02T11:38:26Z

Changes

Adds vector_search_endpoints as a first-class resource type, using the direct deployment engine (only, no TF support).

New configuration surface

resources:
  vector_search_endpoints:
    my_endpoint:
      name: my-endpoint
      endpoint_type: STANDARD
      min_qps: 1
      budget_policy_id: my-policy
      permissions:
        - level: CAN_USE
          group_name: data-team

Required fields: name, endpoint_type. Optional: min_qps, budget_policy_id, permissions.

Key points to note

State ID = endpoint name. The CRUD API identifies endpoints by name; the UUID
(endpoint_uuid) is stored separately in the refresh output for use by the permissions API.

endpoint_type is immutable. Changing it triggers delete + recreate (resources.yml).

Two separate update APIs. DoUpdate dispatches to:

UpdateEndpointBudgetPolicy when budget_policy_id changes
PatchEndpoint when min_qps changes

These can fire in the same deploy if both fields change.

budget_policy_id drift is suppressed. The API returns effective_budget_policy_id
(which includes inherited workspace policies), not the user-set value. Until the SDK
exposes budget_policy_id separately, remote changes to this field are ignored
(reason: effective_vs_requested in resources.yml). See TODO in
bundle/direct/dresources/vector_search_endpoint.go:53.

Permissions use UUID, not name. The PreparePermissionsInputConfig function uses
${...endpoint_uuid} as the object ID when constructing the permissions API path for
vector search endpoints.

Direct-only validation. ValidateDirectOnlyResources (bundle/config/mutator/) emits
an error at plan/deploy time if vector_search_endpoints are present in a non-direct bundle.
Vector Search Endpoints have no Terraform provider.

No dev-mode name prefix. Like UC resources, vector search endpoint names are NOT
prefixed with the dev user name in development mode.

Tests

Acceptance & Unit tests.
Tested e2e with CLI build.

…date Co-authored-by: Jan Rose

…s#5046) ## Changes Address review nits on databricks#4887: - **`update/min_qps/script`**: drop redundant `--keep` + manual `rm` pair in `print_requests()`. `print_requests.py` already deletes `out.requests.txt` when `--keep` is omitted, so the pair was a no-op. ([thread](databricks#4887 (comment))) - **`drift/min_qps/script`**: record `bundle plan --output json` alongside the existing `contains.py` summary check, so the test pins down that `min_qps` is the *only* field detected as changed (old=1, new=1, remote=5), not just the overall count. ([thread](databricks#4887 (comment))) Not in this PR: the [`recreated_same_name` badness thread](databricks#4887 (comment)) — that requires real behavior change (storing `endpoint_uuid` in state and comparing it via `OverrideChangeDesc`, similar to `dashboards.go`'s etag pattern), so it'll get its own follow-up PR. ## Tests - `go test ./acceptance -run TestAccept/bundle/resources/vector_search_endpoints/update/min_qps` - `go test ./acceptance -run TestAccept/bundle/resources/vector_search_endpoints/drift/min_qps`

## Release v0.298.0 ### CLI * Added `--limit` flag to all paginated list commands for client-side result capping ([#4984](#4984)). On `jobs list` and `jobs list-runs` the former API page-size flag was renamed to `--page-size` (hidden) to avoid collision. * Accept `yes` in addition to `y` for confirmation prompts, and show `[y/N]` to indicate that no is the default. * Cache `/.well-known/databricks-config` lookups under `~/.cache/databricks/<version>/host-metadata/` so repeat CLI invocations against the same host skip the ~700ms discovery round trip. * Deprecated `auth env`. The command is hidden from help listings and prints a deprecation warning to stderr; it will be removed in a future release. ### Bundles * Remove `experimental-jobs-as-code` template, superseded by `pydabs` ([#4999](#4999)). * Prompt before destroying or recreating Lakebase resources (database instances, synced database tables, postgres projects and branches) ([#5052](#5052)). * Treat deleted resources as not running in the `fail-on-active-runs` check ([#5044](#5044)). * engine/direct: Added support for Vector Search Endpoints ([#4887](#4887)). * engine/direct: Exclude deploy-only fields (e.g. `lifecycle`) from the Apps update mask so requests that change both `description` and `lifecycle.started` in the same deploy no longer fail with `INVALID_PARAMETER_VALUE` ([#5042](#5042), [#5051](#5051)). * engine/direct: Fix phantom diffs from `depends_on` reordering in job tasks ([#4990](#4990)). ### Dependency updates * Bump `github.com/databricks/databricks-sdk-go` from v0.126.0 to v0.128.0 ([#4984](#4984), [#5031](#5031)). * Bump Go toolchain to 1.25.9 ([#5004](#5004)).

…ting principal (#5151) ## Summary The invariant test config used \`user_name: viewer@example.com\`, which doesn't exist in the cloud workspaces. The Permissions Set API silently drops the unknown user, so a Read after deploy returns an ACL without that entry — the no_drift invariant then sees a phantom update and the test fails on aws-prod-ucws. Pre-existing bug from #4887, not caught earlier because deploy itself was failing on the 50-char endpoint name limit (#5108) before reaching the no_drift check. ### Failure shape (before this fix) \`\`\` "resources.vector_search_endpoints.bar.permissions": { "action": "update", "new_state": { "value": { "__embed__": [ { "level": "CAN_USE", "user_name": "viewer@example.com" }, { "level": "CAN_MANAGE", "service_principal_name": "[USERNAME]" } ] } }, "remote_state": { "__embed__": [ { "level": "CAN_MANAGE", "service_principal_name": "[USERNAME]" } ] }, ... } \`\`\` ### Change Use \`group_name: users\` (always present in every workspace) to match the pattern used by the other \`*_with_permissions\` invariant configs (\`job_with_permissions\`, \`model_with_permissions\`, \`secret_scope_with_permissions\`). ## Test plan - [x] Local: \`go test ./acceptance -run 'TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=vector_search_endpoint'\` passes - [x] Cloud: same target passes on aws-prod-ucws This pull request was AI-assisted by Isaac.

## Changes Adds `vector_search_endpoints` as a first-class resource type, using the direct deployment engine (only, no TF support). ### New configuration surface ```yaml resources: vector_search_endpoints: my_endpoint: name: my-endpoint endpoint_type: STANDARD min_qps: 1 budget_policy_id: my-policy permissions: - level: CAN_USE group_name: data-team ``` Required fields: `name`, `endpoint_type`. Optional: `min_qps`, `budget_policy_id`, `permissions`. ## Key points to note **State ID = endpoint name.** The CRUD API identifies endpoints by name; the UUID (`endpoint_uuid`) is stored separately in the refresh output for use by the permissions API. **`endpoint_type` is immutable.** Changing it triggers delete + recreate (`resources.yml`). **Two separate update APIs.** `DoUpdate` dispatches to: - `UpdateEndpointBudgetPolicy` when `budget_policy_id` changes - `PatchEndpoint` when `min_qps` changes These can fire in the same deploy if both fields change. **`budget_policy_id` drift is suppressed.** The API returns `effective_budget_policy_id` (which includes inherited workspace policies), not the user-set value. Until the SDK exposes `budget_policy_id` separately, remote changes to this field are ignored (`reason: effective_vs_requested` in `resources.yml`). See TODO in `bundle/direct/dresources/vector_search_endpoint.go:53`. **Permissions use UUID, not name.** The `PreparePermissionsInputConfig` function uses `${...endpoint_uuid}` as the object ID when constructing the permissions API path for vector search endpoints. **Direct-only validation.** `ValidateDirectOnlyResources` (`bundle/config/mutator/`) emits an error at plan/deploy time if vector_search_endpoints are present in a non-direct bundle. Vector Search Endpoints have no Terraform provider. **No dev-mode name prefix.** Like UC resources, vector search endpoint names are NOT prefixed with the dev user name in development mode. ## Tests - Acceptance & Unit tests. - Tested e2e with CLI build. --------- Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

## Changes Address review nits on #4887: - **`update/min_qps/script`**: drop redundant `--keep` + manual `rm` pair in `print_requests()`. `print_requests.py` already deletes `out.requests.txt` when `--keep` is omitted, so the pair was a no-op. ([thread](#4887 (comment))) - **`drift/min_qps/script`**: record `bundle plan --output json` alongside the existing `contains.py` summary check, so the test pins down that `min_qps` is the *only* field detected as changed (old=1, new=1, remote=5), not just the overall count. ([thread](#4887 (comment))) Not in this PR: the [`recreated_same_name` badness thread](#4887 (comment)) — that requires real behavior change (storing `endpoint_uuid` in state and comparing it via `OverrideChangeDesc`, similar to `dashboards.go`'s etag pattern), so it'll get its own follow-up PR. ## Tests - `go test ./acceptance -run TestAccept/bundle/resources/vector_search_endpoints/update/min_qps` - `go test ./acceptance -run TestAccept/bundle/resources/vector_search_endpoints/drift/min_qps`

## Release v0.298.0 ### CLI * Added `--limit` flag to all paginated list commands for client-side result capping ([#4984](#4984)). On `jobs list` and `jobs list-runs` the former API page-size flag was renamed to `--page-size` (hidden) to avoid collision. * Accept `yes` in addition to `y` for confirmation prompts, and show `[y/N]` to indicate that no is the default. * Cache `/.well-known/databricks-config` lookups under `~/.cache/databricks/<version>/host-metadata/` so repeat CLI invocations against the same host skip the ~700ms discovery round trip. * Deprecated `auth env`. The command is hidden from help listings and prints a deprecation warning to stderr; it will be removed in a future release. ### Bundles * Remove `experimental-jobs-as-code` template, superseded by `pydabs` ([#4999](#4999)). * Prompt before destroying or recreating Lakebase resources (database instances, synced database tables, postgres projects and branches) ([#5052](#5052)). * Treat deleted resources as not running in the `fail-on-active-runs` check ([#5044](#5044)). * engine/direct: Added support for Vector Search Endpoints ([#4887](#4887)). * engine/direct: Exclude deploy-only fields (e.g. `lifecycle`) from the Apps update mask so requests that change both `description` and `lifecycle.started` in the same deploy no longer fail with `INVALID_PARAMETER_VALUE` ([#5042](#5042), [#5051](#5051)). * engine/direct: Fix phantom diffs from `depends_on` reordering in job tasks ([#4990](#4990)). ### Dependency updates * Bump `github.com/databricks/databricks-sdk-go` from v0.126.0 to v0.128.0 ([#4984](#4984), [#5031](#5031)). * Bump Go toolchain to 1.25.9 ([#5004](#5004)).

…ting principal (#5151) ## Summary The invariant test config used \`user_name: viewer@example.com\`, which doesn't exist in the cloud workspaces. The Permissions Set API silently drops the unknown user, so a Read after deploy returns an ACL without that entry — the no_drift invariant then sees a phantom update and the test fails on aws-prod-ucws. Pre-existing bug from #4887, not caught earlier because deploy itself was failing on the 50-char endpoint name limit (#5108) before reaching the no_drift check. ### Failure shape (before this fix) \`\`\` "resources.vector_search_endpoints.bar.permissions": { "action": "update", "new_state": { "value": { "__embed__": [ { "level": "CAN_USE", "user_name": "viewer@example.com" }, { "level": "CAN_MANAGE", "service_principal_name": "[USERNAME]" } ] } }, "remote_state": { "__embed__": [ { "level": "CAN_MANAGE", "service_principal_name": "[USERNAME]" } ] }, ... } \`\`\` ### Change Use \`group_name: users\` (always present in every workspace) to match the pattern used by the other \`*_with_permissions\` invariant configs (\`job_with_permissions\`, \`model_with_permissions\`, \`secret_scope_with_permissions\`). ## Test plan - [x] Local: \`go test ./acceptance -run 'TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=vector_search_endpoint'\` passes - [x] Cloud: same target passes on aws-prod-ucws This pull request was AI-assisted by Isaac.

…ting principal (databricks#5151) ## Summary The invariant test config used \`user_name: viewer@example.com\`, which doesn't exist in the cloud workspaces. The Permissions Set API silently drops the unknown user, so a Read after deploy returns an ACL without that entry — the no_drift invariant then sees a phantom update and the test fails on aws-prod-ucws. Pre-existing bug from databricks#4887, not caught earlier because deploy itself was failing on the 50-char endpoint name limit (databricks#5108) before reaching the no_drift check. ### Failure shape (before this fix) \`\`\` "resources.vector_search_endpoints.bar.permissions": { "action": "update", "new_state": { "value": { "__embed__": [ { "level": "CAN_USE", "user_name": "viewer@example.com" }, { "level": "CAN_MANAGE", "service_principal_name": "[USERNAME]" } ] } }, "remote_state": { "__embed__": [ { "level": "CAN_MANAGE", "service_principal_name": "[USERNAME]" } ] }, ... } \`\`\` ### Change Use \`group_name: users\` (always present in every workspace) to match the pattern used by the other \`*_with_permissions\` invariant configs (\`job_with_permissions\`, \`model_with_permissions\`, \`secret_scope_with_permissions\`). ## Test plan - [x] Local: \`go test ./acceptance -run 'TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=vector_search_endpoint'\` passes - [x] Cloud: same target passes on aws-prod-ucws This pull request was AI-assisted by Isaac.

janniklasrose added 20 commits March 30, 2026 14:04

Add testserver

21f14c2

Add bundle/config/resources

0d8d10a

[autogen] make schema

7709fa9

[autogen] make schema-for-docs

1373144

[autogen] make generate-validation

114911e

Fix endpoint URL

f252c6d

Add bundle/direct/dresources

86a7ffe

Add generate-refschema target

be3589d

[autogen] make generate-refschema

c2692ec

Add TODO for budgetPolicyId

f31ca1e

[autogen] make generate-direct-resources

f10545a

Add resources.yml override

3603137

Ignore remote changes for budget_policy_id

0533a01

Add acceptance tests

ca87557

Add invariant tests

e700521

Exclude from migrate invariant test

885309a

Fix tests

8d8f392

Appease linters

7a95921

Add direct-only validation

63532ee

Handle 404 in Exists

56c99d9

janniklasrose had a problem deploying to test-trigger-is April 2, 2026 11:39 — with GitHub Actions Failure

janniklasrose added 9 commits April 10, 2026 15:54

[WIP] Add permissions, some gen'ed files

c95e932

Comments

0b615d5

Use wrapper for uuid for permissions

08c6565

Fix apply-bundle-permissions

e5cbab8

Add permissions acceptance test

08b590e

Fix tests & schemas

097a2bf

Missing out.test.toml for permission test

bf43e56

Capture badness regarding permissions update if endpoint is updated

bbe139f

Add minQps to test server

5bfac54

janniklasrose changed the title ~~Vector Search endpoint support (direct only)~~ Add support for Vector Search Endpoint (direct only) Apr 15, 2026

Update changelog for Vector Search Endpoints support

81394d1

denik reviewed Apr 16, 2026

View reviewed changes

janniklasrose and others added 11 commits April 17, 2026 10:47

Merge branch 'main' into janniklasrose/vs-endpoint

0046217

Use e.Name guard instead of e.ID in InitializeURL

908371d

Revert Makefile target

3b8a1e5

Update acceptance tests

42cf5d6

Regenerate files after SDK bump

7343c22

Address comments

31bf314

Rename budget_policy_ignored test

54b6038

Add new UsagePolicyId field to RemapState, but dont force-send-field

94fe508

Fix knownMissingInRemoteType for vector_search_endpoints after SDK up…

2138780

…date Co-authored-by: Jan Rose

Fix tests

8877ec4

Capture badness regarding name vs id

ba4f7e8

denik approved these changes Apr 20, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/vector_search_endpoints/update/min_qps/script

Comment thread acceptance/bundle/resources/vector_search_endpoints/drift/recreated_same_name/test.toml

Comment thread acceptance/bundle/resources/vector_search_endpoints/drift/min_qps/script

janniklasrose added this pull request to the merge queue Apr 20, 2026

Merged via the queue into main with commit 5202ec2 Apr 20, 2026
24 checks passed

janniklasrose deleted the janniklasrose/vs-endpoint branch April 20, 2026 09:50

janniklasrose mentioned this pull request Apr 21, 2026

acceptance: clean up vector_search_endpoints min_qps tests #5046

Merged

janniklasrose mentioned this pull request Apr 29, 2026

Add vector_search_indexes resource (direct engine) #5123

Merged

shreyas-goenka mentioned this pull request May 1, 2026

acceptance: fix vector_search_endpoint permissions config to use existing principal #5151

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Vector Search Endpoint (direct only)#4887

Add support for Vector Search Endpoint (direct only)#4887
janniklasrose merged 49 commits into
mainfrom
janniklasrose/vs-endpoint

janniklasrose commented Apr 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

janniklasrose commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

New configuration surface

Key points to note

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

janniklasrose commented Apr 2, 2026 •

edited

Loading