Fix whole-block plan misclassification via bidirectional rule matching - #6589
Merged
Merged
Conversation
Keep structdiff emitting whole-block changes: when a nested struct is nil on one side it is one change at the block level, which is the right granularity for a whole add/remove. Fix the two bundle-plan consumers that need to relate a rule to such a change: - findMatchingRule is now bidirectional (matchesFieldRule): a rule on foo.bar matches a change at foo, so adding/removing a whole block that contains an immutable field recreates instead of updating. - prepareChanges keeps one level per subtree: when one diff descended to a leaf while the other stayed at the block, the block-level entry wins and inner-field entries are dropped. update_mask is left alone, so the postgres default_endpoint_settings message-mask case stays a documented bug. structdiff and configsync are unchanged, so there is no empty-parent regression. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: cde9ff2
Top 3 slowest tests (at least 2 minutes):
|
Applying bidirectional matching through the shared findMatchingRule and deduping
in prepareChanges both leaked into config-remote-sync (which consumes the plan's
ChangeDesc): ignore_remote rules wrongly skipped whole blocks (dropping a
job_clusters rename), and the dedup dropped fine-grained ops the split logic
needs (leaving a num_workers remnant).
So bidirectional matching is now recreate-only (findRecreateRule); ignore/skip
rules stay descendant-only, and prepareChanges no longer dedups. The PR now only
fixes the recreate misclassification (pipelines/recreate-{add,remove}-whole-block).
The plan "one level" overlap and update_mask remain documented follow-ups.
Co-authored-by: Isaac <no-reply@databricks.com>
It is a generic bidirectional matcher, not recreate-specific; recreate is just its only current caller. Co-authored-by: Isaac <no-reply@databricks.com>
denik
marked this pull request as ready for review
September 11, 2026 07:51
Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Sep 11, 2026
denik
enabled auto-merge
September 11, 2026 08:13
denik
added a commit
that referenced
this pull request
Sep 11, 2026
…lan fix #6589 made the direct engine match recreate rules bidirectionally, so a change to a nested field inside a block covered by a recreate rule is now caught. Seven ingestion_definition.* transitions on pipelines move OK -> OK_RECREATE as a result -- connector_type, the data_staging_options catalog/schema, and several table_configuration leaves. Both are passing verdicts; this records the improved classification. Co-authored-by: Isaac
denik
added a commit
that referenced
this pull request
Sep 11, 2026
…lan fix #6589 made the direct engine match recreate rules bidirectionally, so a change to a nested field inside a block covered by a recreate rule is now caught. Seven ingestion_definition.* transitions on pipelines move OK -> OK_RECREATE as a result -- connector_type, the data_staging_options catalog/schema, and several table_configuration leaves. Both are passing verdicts; this records the improved classification. Co-authored-by: Isaac
denik
added a commit
that referenced
this pull request
Sep 15, 2026
…lan fix #6589 made the direct engine match recreate rules bidirectionally, so a change to a nested field inside a block covered by a recreate rule is now caught. Seven ingestion_definition.* transitions on pipelines move OK -> OK_RECREATE as a result -- connector_type, the data_staging_options catalog/schema, and several table_configuration leaves. Both are passing verdicts; this records the improved classification. Co-authored-by: Isaac
janniklasrose
pushed a commit
that referenced
this pull request
Sep 15, 2026
#6589) Follow-up to #6566 (the reproducer tests, now merged). Fixes the whole-block **recreate misclassification** without touching structdiff, so configsync and the rest of the plan pipeline are unaffected. structdiff keeps emitting one change at the block level when a nested struct is nil on one side — the right granularity for a whole add/remove. The fix is a single, tightly-scoped change: - **Bidirectional recreate matching** (`findRecreateRule` / `matchesFieldRuleBidirectional`): a recreate rule on `foo.bar` now also matches a change recorded at `foo`, so adding/removing a whole block that contains an immutable field (`ingestion_definition.connection_name`) is classified `recreate` instead of `update`. This is applied **only** to `recreate_on_changes`; `findMatchingRule` (ignore_remote/ignore_local, backend_default, normalize) stays descendant-only, because a whole block that merely *contains* an ignored/defaulted leaf is still a real change and must not be skipped. Test flips: `pipelines/recreate-{add,remove}-whole-block` → `recreate` on both engines. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Sep 16, 2026
## Release v1.17.0 ### Notable Changes * Bump the direct deployment state version to 3. Clients older than v1.8.0 will reject bundles deployed with this release. ([#6713](#6713)) ### CLI * Add an `INVALID_REFRESH_TOKEN` error code to `databricks auth token --output json` failures. ([#6684](#6684)) * Add experimental `databricks auth docker configure` to configure Docker credential helper access for Databricks Artifact Registry. ([#6700](#6700)) * Add experimental `databricks auth docker token` to generate Docker credentials for Databricks Artifact Registry. ([#6699](#6699)) * `databricks environments setup-local` now reports the `E_PROVISION_CONFLICT` error code instead of the generic `E_PROVISION` when `uv sync` fails to resolve a dependency conflict. ([#6666](#6666)) * Preserve SSH sessions across temporary tunnel disconnects, with bounded replay and backpressure for large transfers. ([#6650](#6650)) * Allow OAuth U2M logins to override the CLI client ID with `--client-id`, profile `client_id`, or `DATABRICKS_CLIENT_ID`. ([#6594](#6594)) ### Bundles * direct: Store a dashboard's `serialized_dashboard` in state as a content hash instead of its full contents. ([#6105](#6105)) * direct: Fix pipelines recreation when the whole `ingestion_definition` block is added or removed. ([#6589](#6589)) * `bundle plan`, `deploy`, and `destroy` no longer report removing `permissions`, `grants`, or secret scope ACLs from a bundle as a deletion, since it leaves the resource untouched. ([#6647](#6647)) * `bundle plan` and `deploy` no longer list or count a resource that was already deleted remotely as a deletion, matching `bundle destroy`; applying still cleans up its stale state entry. ([#6675](#6675)) * Fix `bundle run` failing with `expected an int, found a string` when an unrelated resource references another resource that is not deployed. `bundle run` now resolves `${resources.*}` references only within the resource being run. ([#6690](#6690)) * Add grants support for the AI Gateway `model_service`, `mcp_service`, and `model_provider_service` resources (direct engine). ([#6635](#6635)) * Add bundle support for the AI Gateway `mcp_service` resource (direct engine). ([#6633](#6633)) * Add bundle support for the AI Gateway `model_provider_service` resource (direct engine). ([#6634](#6634)) * Add bundle support for the AI Gateway `model_service` resource (direct engine). ([#6525](#6525)) * Prevent resource drift on catalogs if `storage_root` contained a trailing slash in the URL. ([#6622](#6622)) * Fixed a "lineage mismatch in state files" error that could occur after destroying a bundle and redeploying it from another machine. `bundle destroy` now removes the local state file so no stale lineage is left behind, and prunes the state directories it leaves empty (such as `.internal/` and `sync-snapshots/`). ([#6210](#6210), [#6685](#6685)) * direct: `bundle plan` no longer reports a permanent update on a cluster that uses a cluster policy: when the cluster spec sets `policy_id`, a field present in the remote but absent from the bundle config is not treated as drift. ([#6531](#6531)) * `bundle deploy` on the direct engine now reports each resource as soon as it is deployed, instead of listing them all after the deployment finishes. A deploy that fails part way through now reports the resources it did apply. ([#6361](#6361)) * Direct-engine bundles no longer flag phantom drift on server-populated nested fields under reused config types (e.g. `external_locations` file-event-queue resource IDs, `database_instances` parent-instance refs, `apps` git credential ID). ([#6618](#6618)) * `databricks bundle generate app` now reproduces a git-backed app's `git_repository` and `git_source` configuration instead of emitting a workspace `source_code_path`, so generating from a Git-deployed app no longer silently converts it to workspace source. ([#6656](#6656)) * Improved configuration load time for bundles with many included files. ([#6195](#6195)) * `bundle destroy` no longer deletes triggered job runs, leaving them untouched on the backend. ([#6672](#6672)) * direct: resources.job\_runs: new lifecycle.triggers.on\_file\_change setting to restart the run when monitored files change. Can be set to a series of paths or globs. ([#6309](#6309)) * Bundle summary now shows a name for Postgres branches, endpoints, databases, and roles instead of a blank Name field. ([#6663](#6663)) * Added PyDABs (Python) support for cluster policies, dashboards, and Genie spaces. ([#6585](#6585)) * CLI commands no longer imply that a resource whose type has no workspace URL is merely not deployed yet. ([#6583](#6583)) * Capture the implicit dependency a vector search index has on a catalog or schema defined in the same bundle, so the catalog and schema are deployed first. ([#6655](#6655)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#6695](#6695)) * Bump `github.com/databricks/databricks-sdk-go` from v0.177.0 to v0.178.0. ([#6673](#6673)) * Bump Terraform provider from v1.131.0 to v1.132.0. ([#6671](#6671))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #6566 (the reproducer tests, now merged).
Fixes the whole-block recreate misclassification without touching structdiff, so configsync and the rest of the plan pipeline are unaffected.
structdiff keeps emitting one change at the block level when a nested struct is nil on one side — the right granularity for a whole add/remove. The fix is a single, tightly-scoped change:
findRecreateRule/matchesFieldRuleBidirectional): a recreate rule onfoo.barnow also matches a change recorded atfoo, so adding/removing a whole block that contains an immutable field (ingestion_definition.connection_name) is classifiedrecreateinstead ofupdate. This is applied only torecreate_on_changes;findMatchingRule(ignore_remote/ignore_local, backend_default, normalize) stays descendant-only, because a whole block that merely contains an ignored/defaulted leaf is still a real change and must not be skipped.Test flips:
pipelines/recreate-{add,remove}-whole-block→recreateon both engines.This pull request and its description were written by Isaac.