bundle: Clear local state on destroy to prevent lineage mismatch by janniklasrose · Pull Request #6210 · databricks/cli

janniklasrose · 2026-08-10T09:45:48Z

Follow-up to #6207 (which adds the reproducing acceptance test).

bundle destroy now deletes local state (resources.json in direct, terraform.tfstate in TF).

eng-dev-ecosystem-bot · 2026-08-10T10:18:01Z

Integration test report

Commit: 5d6ee06

Run: 34987819995

	Env	💚RECOVERED	✅pass	🙈skip	Time
💚	aws linux	1	275	18	6:53
💚	aws windows	1	277	16	5:31
💚	azure linux	1	274	18	7:49
💚	azure windows	1	276	16	6:28
💚	gcp linux	1	275	18	9:59
💚	gcp windows	1	277	16	8:09

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R

Top 8 slowest tests (at least 2 minutes):

duration	env	testname
4:32	azure windows	TestAccept
4:29	gcp windows	TestAccept
3:48	gcp linux	TestFilerWorkspaceFilesExtensionsStat
3:09	aws windows	TestAccept
2:33	gcp windows	TestFilerWorkspaceFilesExtensionsStat
2:32	gcp windows	TestSyncIncrementalFileSync
2:09	azure linux	TestFilerRecursiveDelete/workspace_files_extensions
2:01	gcp linux	TestImportDirDoesNotOverwrite

Destroy only deleted the remote state, leaving the local state file (and its lineage) behind. When the same bundle was later redeployed from a different machine that had no local state, the fresh deploy minted a new lineage that no longer matched the lingering local one, so every subsequent command failed with "lineage mismatch in state files". Remove the local state files on a successful destroy so no stale lineage is left behind; the next command then adopts the remote state cleanly. Both engines keep a lineage-bearing state file (direct: resources.json, terraform: terraform.tfstate) and both are read by validateStates, so remove both regardless of which engine ran the destroy. Reworks bundle/state/state_present, which relied on the local direct state surviving a destroy to assert serial continuity. Its engine- selection intent is unchanged; the mid-test destroy is dropped. The reproducing test now runs on both engines. Co-authored-by: Isaac

…ineage-on-destroy # Conflicts: # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/out.test.toml # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/output.txt # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/script # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/test.toml

Now that destroy removes the local state file, read_state.py called after a destroy (e.g. bundle/resources/pipelines/update) hit FileNotFoundError on its unconditional open(). Treat a missing state file the same as a missing entry within the file and print "State not found", matching print_state.py. Co-authored-by: Isaac

janniklasrose · 2026-08-14T21:14:03Z

+	// Remove the local state files now that the deployment is gone. Destroy only
+	// deletes the remote state; leaving a local state file behind keeps its
+	// lineage around, so a later fresh deploy of the same bundle (e.g. from
+	// another machine that has no local state) mints a new lineage that no longer
+	// matches this lingering one, and every subsequent command fails with a
+	// lineage mismatch. Both engines keep a lineage-bearing state file, so remove
+	// both regardless of which engine ran the destroy.
+	_, localDirectPath := b.StateFilenameDirect(ctx)
+	_, localTerraformPath := b.StateFilenameTerraform(ctx)


destroy can only run on one engine at a time. we know the engine, so we should only delete the appropriate one here and not touch other files

Done in 6e9baa7 — now selects the running engine's state file (direct: resources.json, terraform: terraform.tfstate) and removes only that one.

Destroy runs on a single engine, which we already know here, so remove only that engine's local state file instead of both. Addresses review feedback. Co-authored-by: Isaac

…ineage-on-destroy # Conflicts: # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/script # acceptance/bundle/state/state_present/output.txt # bundle/phases/destroy.go

…ineage-on-destroy # Conflicts: # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/out.test.toml

read_state.py now names the state file it consulted in the "State not found" message; normalize that engine-specific path in pipelines/update so the terraform and direct matrix variants stay identical. Rework the lineage-mismatch test to keep the reproduction (a lingering local state still mismatches after an out-of-band redeploy) alongside a new assertion that a clean destroy removes the local state file. Run it on both engines, recording the engine-specific mismatch error in out.mismatch.<engine>.txt. Co-authored-by: Isaac <no-reply@databricks.com>

…ineage-on-destroy Co-authored-by: Isaac <no-reply@databricks.com>

Keeping the terraform deploy/destroy pair, the destroy now runs on the higher-serial direct state and removes it (the local-state cleanup this PR adds). The following direct deploy then migrates from the leftover terraform state instead of staying on direct, so assert that behavior and ignore the terraform provider artifacts the deploy leaves behind. Co-authored-by: Isaac <no-reply@databricks.com>

…ineage-on-destroy

… files The mismatch error names the engine-specific state file (resources.json vs terraform.tfstate). Normalize it to [STATE_FILE]/[ENGINE] via a scoped Repl so both matrix variants share one output.txt, dropping the out.mismatch.<engine>.txt split. Co-authored-by: Isaac <no-reply@databricks.com>

A fresh-create plan no longer carries a lineage (it is minted at deploy, not plan), matching main. Regenerate the stale golden. Co-authored-by: Isaac <no-reply@databricks.com>

janniklasrose · 2026-09-15T12:48:46Z

 {
  "plan_version": [PLAN_VERSION],
  "cli_version": "[CLI_VERSION]",
-  "lineage": "[UUID]",


this got removed because the plan is captured after the destroy

shreyas-goenka

minor comments, thanks!

shreyas-goenka · 2026-09-15T13:17:17Z

 Destroy: 1 deleted

 >>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy
+Warn: Direct engine selected via DATABRICKS_BUNDLE_ENGINE environment variable but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.


just curious, why is this warning showing up now but was not present before? Unexpected diff?

There should not be a TF state at this point?

before we cleaned up local state on destroy:

remote direct state gone, local direct state remained, DATABRICKS_BUNDLE_ENGINE=direct deploy is deployed as direct because the higher .serial wins.

after removing local state:

destroy runs on direct (higher serial) and deletes remote and local direct states, so only terraform/terraform.tfstate remains. per existing logic, DATABRICKS_BUNDLE_ENGINE=direct deploy deploys as TF.

We just ran bundle destroy before running deploy. That should have removed terraform/terraform.tfstate right (given we are removing local state).

The recorded (DMS) path reads deployment history, not a file, so it cannot print a filename; keeping it on the file-based paths made the pipelines/update DMS matrix variants diverge. Drop the filename for now (revisit denik's suggestion later) and remove the now-dead [STATE_FILE] Repl. Co-authored-by: Isaac <no-reply@databricks.com>

directories around it (.internal/, sync-snapshots/) are left behind empty. Prune every directory that ends up empty under the bundle+target local state directory. Symlinks count as content and are not traversed, so terraform's provider mirror survives, and the walk stays within the target's state directory so a sibling engine's state is untouched. Follow-up to #6210 (comment) Co-authored-by: Isaac <no-reply@databricks.com>

- Replace the `find -type d -empty` / hand-rolled snapshot checks with a full `find` of the state tree in both destroy acceptance tests, so a stray file and an empty scaffolding dir both show up. Prune the terraform engine dir: its contents differ by engine and embed the provider version and OS/arch, which would make the golden fragile; the bundle scaffolding under test lives directly under the target dir. - Fold the empty-dir cleanup into the clear-lineage-on-destroy changelog fragment (spanning #6210 and #6685) and drop the separate fragment. Co-authored-by: Isaac <no-reply@databricks.com>

databricks#6210 makes `bundle destroy` remove the local state file, but the directories around it (`.internal/`, `sync-snapshots/`) are left behind empty. This PR has `destroy` prune every directory that ends up empty under the bundle+target local state directory. Symlinks count as content and are not traversed, so terraform's provider mirror survives, and the walk stays within the target's state directory so a sibling engine's state is untouched. Follow-up to databricks#6210 (comment) --------- Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.17.0 ### Notable Changes * Bump the direct deployment state version to 3. Clients older than v1.8.0 will reject bundles deployed with this release. ([#6713](#6713)) ### CLI * Add an `INVALID_REFRESH_TOKEN` error code to `databricks auth token --output json` failures. ([#6684](#6684)) * Add experimental `databricks auth docker configure` to configure Docker credential helper access for Databricks Artifact Registry. ([#6700](#6700)) * Add experimental `databricks auth docker token` to generate Docker credentials for Databricks Artifact Registry. ([#6699](#6699)) * `databricks environments setup-local` now reports the `E_PROVISION_CONFLICT` error code instead of the generic `E_PROVISION` when `uv sync` fails to resolve a dependency conflict. ([#6666](#6666)) * Preserve SSH sessions across temporary tunnel disconnects, with bounded replay and backpressure for large transfers. ([#6650](#6650)) * Allow OAuth U2M logins to override the CLI client ID with `--client-id`, profile `client_id`, or `DATABRICKS_CLIENT_ID`. ([#6594](#6594)) ### Bundles * direct: Store a dashboard's `serialized_dashboard` in state as a content hash instead of its full contents. ([#6105](#6105)) * direct: Fix pipelines recreation when the whole `ingestion_definition` block is added or removed. ([#6589](#6589)) * `bundle plan`, `deploy`, and `destroy` no longer report removing `permissions`, `grants`, or secret scope ACLs from a bundle as a deletion, since it leaves the resource untouched. ([#6647](#6647)) * `bundle plan` and `deploy` no longer list or count a resource that was already deleted remotely as a deletion, matching `bundle destroy`; applying still cleans up its stale state entry. ([#6675](#6675)) * Fix `bundle run` failing with `expected an int, found a string` when an unrelated resource references another resource that is not deployed. `bundle run` now resolves `${resources.*}` references only within the resource being run. ([#6690](#6690)) * Add grants support for the AI Gateway `model_service`, `mcp_service`, and `model_provider_service` resources (direct engine). ([#6635](#6635)) * Add bundle support for the AI Gateway `mcp_service` resource (direct engine). ([#6633](#6633)) * Add bundle support for the AI Gateway `model_provider_service` resource (direct engine). ([#6634](#6634)) * Add bundle support for the AI Gateway `model_service` resource (direct engine). ([#6525](#6525)) * Prevent resource drift on catalogs if `storage_root` contained a trailing slash in the URL. ([#6622](#6622)) * Fixed a "lineage mismatch in state files" error that could occur after destroying a bundle and redeploying it from another machine. `bundle destroy` now removes the local state file so no stale lineage is left behind, and prunes the state directories it leaves empty (such as `.internal/` and `sync-snapshots/`). ([#6210](#6210), [#6685](#6685)) * direct: `bundle plan` no longer reports a permanent update on a cluster that uses a cluster policy: when the cluster spec sets `policy_id`, a field present in the remote but absent from the bundle config is not treated as drift. ([#6531](#6531)) * `bundle deploy` on the direct engine now reports each resource as soon as it is deployed, instead of listing them all after the deployment finishes. A deploy that fails part way through now reports the resources it did apply. ([#6361](#6361)) * Direct-engine bundles no longer flag phantom drift on server-populated nested fields under reused config types (e.g. `external_locations` file-event-queue resource IDs, `database_instances` parent-instance refs, `apps` git credential ID). ([#6618](#6618)) * `databricks bundle generate app` now reproduces a git-backed app's `git_repository` and `git_source` configuration instead of emitting a workspace `source_code_path`, so generating from a Git-deployed app no longer silently converts it to workspace source. ([#6656](#6656)) * Improved configuration load time for bundles with many included files. ([#6195](#6195)) * `bundle destroy` no longer deletes triggered job runs, leaving them untouched on the backend. ([#6672](#6672)) * direct: resources.job\_runs: new lifecycle.triggers.on\_file\_change setting to restart the run when monitored files change. Can be set to a series of paths or globs. ([#6309](#6309)) * Bundle summary now shows a name for Postgres branches, endpoints, databases, and roles instead of a blank Name field. ([#6663](#6663)) * Added PyDABs (Python) support for cluster policies, dashboards, and Genie spaces. ([#6585](#6585)) * CLI commands no longer imply that a resource whose type has no workspace URL is merely not deployed yet. ([#6583](#6583)) * Capture the implicit dependency a vector search index has on a catalog or schema defined in the same bundle, so the catalog and schema are deployed first. ([#6655](#6655)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#6695](#6695)) * Bump `github.com/databricks/databricks-sdk-go` from v0.177.0 to v0.178.0. ([#6673](#6673)) * Bump Terraform provider from v1.131.0 to v1.132.0. ([#6671](#6671))

Make the destroy summary line ("Destroy: N deleted") print even when the destroy encounters errors partway through. This is a follow-up to PR #6210. Before this change, early returns on errors (during the resource Apply, DMS finalization, or file deletion) skipped the summary entirely. Extract the summary into logDestroySummary and invoke it via defer in destroyCore so it prints on every return path; partial deletions may have already succeeded. The count is the planned deletions, not the actually-succeeded ones. Making the count reflect only successful deletions is a separate follow-up. Add an acceptance test that injects a failed resource delete and asserts the summary still prints after the destroy error. The dms/failed-delete golden gains the same summary line for the same reason. Co-authored-by: Isaac <no-reply@databricks.com>

acc: Reproduce linear mismatch error after destroy and oob redeploy

39f1814

Base automatically changed from janniklasrose/lineage-mismatch-message to main August 10, 2026 10:08

janniklasrose force-pushed the janniklasrose/clear-lineage-on-destroy branch from f28302d to 5445a04 Compare August 14, 2026 08:18

janniklasrose added 2 commits August 14, 2026 13:08

janniklasrose commented Aug 14, 2026

View reviewed changes

bundle: Remove only the running engine's state file on destroy

6e9baa7

Destroy runs on a single engine, which we already know here, so remove only that engine's local state file instead of both. Addresses review feedback. Co-authored-by: Isaac

janniklasrose marked this pull request as ready for review August 14, 2026 21:22

Merge remote-tracking branch 'origin/main' into janniklasrose/clear-l…

965a086

…ineage-on-destroy # Conflicts: # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/script # acceptance/bundle/state/state_present/output.txt # bundle/phases/destroy.go

janniklasrose requested review from andrewnester, denik, pietern and shreyas-goenka August 18, 2026 07:56

denik reviewed Aug 18, 2026

View reviewed changes

Comment thread acceptance/bin/read_state.py

Comment thread acceptance/bundle/destroy/lineage-mismatch-after-redeploy/output.txt

Comment thread acceptance/bundle/state/state_present/script

denik reviewed Sep 9, 2026

View reviewed changes

Comment thread bundle/phases/destroy.go

janniklasrose and others added 7 commits September 9, 2026 12:48

Merge remote-tracking branch 'origin/main' into janniklasrose/clear-l…

bbf25f9

…ineage-on-destroy # Conflicts: # acceptance/bundle/destroy/lineage-mismatch-after-redeploy/out.test.toml

restore script and generate matching output

54a48ce

Merge remote-tracking branch 'origin/main' into janniklasrose/clear-l…

fe3924b

…ineage-on-destroy Co-authored-by: Isaac <no-reply@databricks.com>

fix changelog

796d0ec

Merge remote-tracking branch 'origin/main' into janniklasrose/clear-l…

c937442

…ineage-on-destroy

janniklasrose commented Sep 15, 2026

View reviewed changes

Comment thread acceptance/bundle/destroy/lineage-mismatch-after-redeploy/output.txt

janniklasrose and others added 2 commits September 15, 2026 12:41

acc: regenerate whl-not-uploaded plan golden (drop stale lineage)

908c22b

A fresh-create plan no longer carries a lineage (it is minted at deploy, not plan), matching main. Regenerate the stale golden. Co-authored-by: Isaac <no-reply@databricks.com>

janniklasrose commented Sep 15, 2026

View reviewed changes

janniklasrose requested a review from denik September 15, 2026 12:53

janniklasrose mentioned this pull request Sep 15, 2026

Clean up empty local state directories on destroy #6685

Merged

janniklasrose commented Sep 15, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/pipelines/update/output.txt Outdated

janniklasrose added this pull request to stack #6687 September 15, 2026 13:08

shreyas-goenka reviewed Sep 15, 2026

View reviewed changes

denik reviewed Sep 15, 2026

View reviewed changes

Comment thread bundle/phases/destroy.go

denik reviewed Sep 15, 2026

View reviewed changes

Comment thread bundle/phases/destroy.go

janniklasrose and others added 4 commits September 15, 2026 14:15

Move error handling to the bottom in bundle/phases/destroy.go

90850bc

add back .serial assertion

576287e

Remove unnecessary testtoml override

5d6ee06

denik approved these changes Sep 16, 2026

View reviewed changes

shreyas-goenka approved these changes Sep 16, 2026

View reviewed changes

janniklasrose added this pull request to the merge queue Sep 16, 2026

Merged via the queue into main with commit 6bbe0d4 Sep 16, 2026
39 checks passed

janniklasrose deleted the janniklasrose/clear-lineage-on-destroy branch September 16, 2026 12:46

rugpanov mentioned this pull request Sep 17, 2026

Update Databricks CLI to v1.17.0 databricks/databricks-vscode#2200

Open

Uh oh!

Conversation

janniklasrose commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

janniklasrose Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

janniklasrose Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janniklasrose Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shreyas-goenka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shreyas-goenka Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

janniklasrose Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

janniklasrose commented Aug 10, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Aug 10, 2026 •

edited

Loading